# Rapid computer use and verified recovery

Status: implementation in progress. The isolated Qwen, Laya and Kev baselines have measured
finite-choice smoke results; Moondream passed twenty synthetic grounding cases.
A configured Qwen planner completed the two-action local browser fixture.
The visible three-model run now also completed four actions through the real
runner, with actual Kev selections, one Moondream point, progress, verified
effects and a terminal receipt.
The remaining model and platform gates stay open.
Requested: 2026-10-11. Baseline: `4ba37c16` on `origin/main`.

The user supplied [the architecture and test scheme](architecture-source.md) and
reported a live Save As failure. This work continues the delivered
[Telegram progress/browser repair](../telegram-progress-browser-20261011/README.md).
Its scope is the supplied architecture, with the observed save/recovery defect
first in delivery order. It does not replace the earlier narration requirements.

## Current failure, anchored in the running task

Read-only inspection of the `telegram_test/.omnius` activity archive
`0d4052ce3e9cba8e` found the filename and **Documents** breadcrumb before Save
(entry 333, turn 36). Save was clicked on turn 37. The exact file exists in
Documents, with 466314 bytes, PNG signature, and a timestamp matching the action;
the matching Downloads path is absent. This proves local bytes, not media
provenance or Telegram delivery.

The next 34 shell calls included 22 identical negative Downloads searches. The
correct destination remained in the model's context through turn 71. The vision
tool promoted a changed screenshot to `proven`, without checking a saved file.
The observation ledger omitted empty output. The focus supervisor's forced replan
was deliberately advisory for medium/large native models and silent mode. These
are separate defects; hiding diagnostics alone cannot fix them.

## Work orders

| ID | Deliverable | Status | Gate |
| --- | --- | --- | --- |
| [WO-01](WO-01-native-save-recovery.md) | Exact native-save receipt and bounded negative-search recovery | Verified | Documents fixture, unchanged old file, timeout, cancellation, dispatch replay |
| [WO-02](WO-02-semantic-control.md) | Typed UI state, legal actions, browser adapters and native geometry/focus | In progress | Guarded local-browser fixtures; native platform coverage stated separately |
| [WO-03](WO-03-policy-workflows.md) | Finite policy providers, verified workflows and replay | In progress | Unknown choices cannot execute; service failure, drift and stop remain bounded |
| [WO-04](WO-04-live-model-gates.md) | Pinned source/model manifest and local model evaluation | Qwen/Laya/Kev choices and Moondream 20 captures measured; further comparisons open | Actual runner placement, smoke tests, 20 captures, measured latency/VRAM |
| [WO-05](WO-05-verification-delivery.md) | Integrated tests, measured results, scoped commits and delivery | In progress | Build, focused regressions, clean task commits, tracked-branch push |
| [WO-06](WO-06-medium-horizon-runtime.md) | Medium-horizon browser cases, sourced extraction, speed and ordinary runtime availability | Source and controlled evaluation verified; live/deployment gates open | Four visible cases passed; TUI/admin-DM integration verified separately from model quality and the unchanged installed daemon |
| [WO-07](WO-07-tool-routing-prompts.md) | Shared tier, compact-context and permitted surface awareness of browser/native routing | Source verified; daemon unchanged | All three tiers and real compaction tested; model-directed recovery distinguished from automatic runtime retry |
| [WO-08](WO-08-cross-application-desktop.md) | Browser API docs, file-manager navigation and gedit append/save | Controlled integration verified; model/deployment gates open | Real Xvfb and visible Xephyr completions, exact saved bytes/source spans, Stop and cleanup |

## Invariants and remaining architecture phases

- UI state is a working projection with opaque identity, revision and generation;
  selectors and OCR text are observations, not authority.
- Every meaningful mutation has a typed expected effect, or an explicit unresolved
  outcome. A click, screenshot change, elapsed delay or filename mention cannot
  establish saved-file success.
- Runtime-owned finite actions and existing tool/task authority control execution;
  model confidence and page instructions cannot grant permission.
- Semantic operations precede vision. Vision coordinates retain their exact
  capture transform and are revalidated before input.
- Repeated unchanged negative searches cause bounded recovery within that task;
  a new task, relevant changed evidence or distinct scope can be inspected.
- Progress stays separate from final answers. Existing Telegram topic/run,
  cancellation, artifact custody, review and delivery contracts remain in force.
- Preserve the active Telegram session and existing user-browser windows. Use an
  isolated checkout and owned browser contexts. The user's 2026-10-12 steering
  explicitly requests a visible GUI browser; its integrated run may use a
  separate owned window on the current desktop. Routine regressions stay isolated.

The opt-in local host command now connects the planner and finite policy to
runner-owned semantic execution, terminal verification and a durable completed
ledger. Its build and 477 initial regression checks pass, including real isolated
browser completion and cancellation. Subsequent planner repairs passed 419
affected checks plus 18 independent schema cases. One live Qwen retry then
completed the local form; the earlier rejected plan remains in the evidence.

The source's remaining model bring-up, existing-browser extension support, live
Windows UIA, Wayland input, further model comparisons and release profiling
remain individually gated. The controlled live visual-fallback fixture passed;
general application coverage, automatic deployment into the active bot and
app-disjoint model comparisons remain open. A completed integration slice does
not close those phases.
Each work order records implementation evidence and outstanding work explicitly.

## Visible three-model execution: controlled fixture verified

The user's requested GUI integration ran on X11 `:0` in a separate owned Chromium
window. One pinned Qwen plan drove four actual Kev finite decisions. Title,
Ready and Preview used semantic controls; Moondream was invoked once for the
pixel-only Apply control. Every selected candidate matches its dispatched action
ID. Its capture, transform and click receipt agree, all four effects were verified,
and the committed terminal receipt agrees with the ledger and objective frontier.

The task completed in 25.05 seconds, emitted eleven progress explanations, then
kept the completed page visible for 90 seconds. The [retained screenshot](../../../computer-use/evidence/visible-host-attempt2-20261012/headed-smoke-01/visible-result.png)
shows Report, checked Ready and Applied Report. The [69-artifact manifest](../../../computer-use/evidence/visible-host-attempt2-manifest-20261012.json)
binds inputs, physical model receipts, valid trace, terminal authority, actual
placement and independently verified cleanup. All three models used a single
scoped A100 lease; the voice workload was preserved. The earlier startup failure
and launch repair remain in WO-04. This is one controlled live task, with no
fallback or replan; broader quality and performance claims require further runs.

The opt-in entry point and explicit live harness are documented in
[HOST_SESSION.md](../../../computer-use/HOST_SESSION.md). Continue with the
remaining platform/model gates and active-bot integration as separate work orders.

The user explicitly delegated automatic endpoint/model/GPU selection. The host's
GPU policy still requires exact selected tags/devices, actual runner placement and
broker-scoped CUDA deployments. Hardware inventory alone does not satisfy that
gate. See WO-04 for live preflight and test status.

## Resume after host restart (2026-10-12)

The host restarted and cleared the temporary checkout. Delivered commits
`1a490b44` and `291f8373` remain on `origin/main`; uncommitted browser/controller/
native/model draft work did not survive. Continued work is in the persistent
checkout `/home/roko/Documents/Projects/Adjacent/omnius/omnius-rapid-computer-use-20261012`.
Its replacement implementation and test results are recorded separately from the
earlier temporary draft. The original dirty main checkout remains untouched.

Fresh broker discovery reports `admission_blocked: true`, missing GPU
`GPU-fc9d264d-8a20-bf8c-eb72-72a6dcbdaf2a`, and NVIDIA recovery action `Reboot`.
The later recorded preflight reports an NVML driver/library mismatch and all
three selected A100s unverifiable; WO-04 links the timestamped result.
No model was loaded while those failures were present. At 04:48 UTC the host
reported healthy hardware again. Automatic selection remains authorized; WO-04
records the subsequent isolated, broker-leased zero-token load and exact runner
verification. The existing voice service and the user's active bot remain intact.
