# WO-02 — Typed semantic control, coordinates and focus

Status: in progress. Maps to source phases 1–2 and visual transforms in phase 3.

Implement the managed-browser path first using its existing owning Playwright
session. Preserve exact frame/page scope, opaque node identity, generation,
revision, supported operations, live eligibility and observed predicates. A
policy must only select a runtime-owned candidate. A model cannot supply code or
replace candidate arguments after selection. Use current tool/task authorization.

Expose the controller through normal tool dispatch, not only an unused library.
Include an observe/abstain path and bounded no-progress outcome. File predicates
consume actual saved-byte receipts. Retain cancellation and context ownership.

Test at least 25 distinct fixture workflows covering form fields, checks/selects,
focus, delayed modal, frame/shadow scope, navigation, stale/replaced/reordered
nodes, disabled/ambiguous controls, download custody, cancellation, rejected
actions, and deliberately pixel-only controls. Report real local-browser tests
separately from mocked adapter tests.

For native input, introduce a single explicit capture-to-input transform covering
normalization, resizing, letterbox, crop, display scale and window origin. Reject
out-of-bounds/padding/stale geometry; do not extrapolate the old window fixture
(640×480 geometry with a 1920×1080 capture). Serialize focus-dependent operations
per desktop session and stop before another input when interrupted or focus drifts.

Remaining phase gates must be recorded individually: Firefox BiDi and existing
user-browser extension permissions; Linux AT-SPI2 and consented Wayland input;
Windows UIA/action patterns/elevation denial; optional macOS accessibility; and
browser/native window identity merging. Chromium coverage does not establish
parity for these providers.

## Delivered semantic adapter slice (2026-10-12)

- `browser.ts` uses the tool's owned Playwright page and frame sessions, private
  per-document WeakMap identities and pinned element handles. Reordering cannot
  redirect an existing action. Navigation, replacement, focus and scroll state
  invalidate observations. Verified download bytes survive browser close/Stop.
- `playwright_browser` exposes `ui_snapshot`/`ui_step`. Host constructor options
  select Chromium or Firefox explicitly; baseline Firefox uses Playwright's
  Juggler transport and is not labeled BiDi. An optional host visual adapter
  uses the same finite controller and exact authority/effect checks.
- `native_ui` exposes bounded AT-SPI2/UIA observations and native patterns.
  Linux tests use their own D-Bus session, Xvfb display and GTK applications.
  A real GtkFileChooserDialog required recognizing the `file chooser` role;
  the filename, Documents destination and eligible Save action are now observed.
- Native workers ship as embedded, versioned JSON IPC implementations. They
  validate application/window identity, foreground focus, generations, supported
  patterns and secret fields. The Windows worker explicitly denies higher
  integrity targets; no elevation or global input fallback is implemented.
- Registration covers ordinary TUI and Telegram admin DM. Desktop discovery
  promotes the semantic tool with the existing visual tools. Public/admin group
  policies keep host desktop access denied; existing narration delivery remains.

Validation: 82 real browser workflows (41 per engine), 39 existing browser
regressions, 35 native checks (20 unit/protocol, 14 real GTK, one actual PowerShell
parse/C# compile), 112 selected CLI checks and 105 orchestrator checks pass.
Windows UIA interaction itself is unverified on this Linux host. Wayland semantic
AT-SPI is supported where exposed; RemoteDesktop portal input/capture, existing
browser extension permissions and browser/native identity merging remain open.

## Explicit Firefox BiDi transport

The optional Firefox BiDi mode uses an installed exact executable and verifies
the new browser's active BiDi session over its reserved loopback endpoint before
labeling observations `webdriver-bidi`. Ordinary TUI/admin construction can select
the engine/protocol/executable through validated `OMNIUS_PLAYWRIGHT_*` host
settings. Missing support fails explicitly; no browser installation or protocol
fallback occurs.

An owned local fixture established a real upstream limitation: Firefox 151 emits
Blob download begin/end events with `navigation: null`, and Playwright 1.61 drops
them. This mode advertises direct HTTP(S) download custody separately and rejects
known button/Blob targets before clicking. A changed script on an eligible link
can still leave the action unresolved; it must not cause another click or a saved
file claim. Standard Playwright Blob downloads remain supported. Exact final
regression counts are recorded in WO-05.

## Owned browser visual execution

`browser-visual.ts` binds screenshot bytes, dimensions, viewport transform,
current page/session/scope and semantic revision through grounding and input.
Configured point services receive copies; they cannot replace the page or mutate
the retained capture. Unknown or ambiguous points, changed scope/revision and
cancellation cannot dispatch a click. The normal host grant check occurs before
visual inference, and the click still requires independent effect verification.

The new helper's focused checks and real browser fixtures pass. The live GUI
integration then exercised three semantic actions and one actual Moondream
canvas target under four Kev selections. An X11 visible-window receipt and
inspected screenshot bind the result to the owned headed browser. Stop/review
checks remain offline regression evidence; the live run completed normally.
Measured results and exact cleanup are recorded in WO-04.
