# Reconcile legacy desktop tools with the rapid computer-use controller

Status: reconciliation implemented; isolated build, focused regressions and real
read-only desktop observation/capture verified. Live Kev/Moondream inference
and visual input acceptance remain incomplete: the model services and operator
profile have not been provisioned. The earlier host GPU-health fault is historical;
current readiness and remaining deployment work are recorded below. No claim that every application/control is supported.

## Observed production disconnect

The current terminal run in `telegram_test`,
`tui-3144597-mv5p6bvu-1791939658797-1`, was asked to open Firefox and search
archive.org. At 2026-10-14T01:04:57Z it invoked `vision_action_loop` with
`action=observe`. The result captured the desktop using `gnome-screenshot`, ran
Tesseract OCR, reported `Vision analysis unavailable`, and performed no input
action. The next recorded tool was a todo update. This is not evidence that
Kev, Moondream, or the new rapid decision/executor loop ran.

## Accepted work

1. Identify the canonical new controller and trace the legacy tool, registration,
   desktop discovery, prompt guidance and configured model routing into it.
2. Reconcile existing desktop entry points with the new controller. Ordinary
   authorized desktop tasks must discover the intended semantic observation,
   finite-action policy, visual grounding and verified executor rather than an
   independent OCR-only observe/planner loop.
3. Preserve OS focus, exact current target identity, task authority, Stop and
   public/private tool boundaries. An observation or tool success cannot claim
   an action, model invocation or verified goal progress.
4. Report actual policy/vision availability and invoked model identities. Use
   the configured Kev/alternative decision and Moondream adapters when required;
   semantic fast paths may correctly avoid model calls. Do not silently load a
   different checkpoint or allocate an unintended GPU.
5. Cover the actual runtime registration/discovery and legacy-to-controller
   transition. Perform live model/desktop acceptance only after the repository's
   exact endpoint, model and accelerator preflight; distinguish offline fixtures
   from actual GUI actions and inference.

## Scope and delivery

Continue the already authorized capacity, source-processing and Telegram
delivery repairs. Keep this migration separately reviewable and committed after
its own checks; preserve unrelated work and the operator's publication staging.
The operator publishes and updates the active runtime.

Architecture anchor:
`/home/roko/Documents/open_weight_rapid_computer_use_architecture_2026-10-11.md`.

## Reconciled runtime

- TUI, its subagents and Telegram admin construct one configured native tool per
  scope and pass the same instance to the legacy wrapper. Browser and native
  factories share operator-controlled pinned policy/vision clients, preflight
  gates, request quarantine and explicit capability/model receipts.
- `vision_action_loop` now returns canonical native observations and actionable
  handoff to `native_ui`. Its old coordinates/keyboard/allow-actions parameters
  cannot dispatch input. The previous manual utility is unregistered and absent
  from default discovery. Existing camera/image tools retain their own purposes.
- The remaining `desktop_click` and `desktop_describe` registrations have also
  been removed from the TUI, its children and Telegram admin. Exact searches for
  either retired name resolve to the same configured `native_ui` schema and
  session through aliases. Their package exports now provide observation/handoff
  compatibility only. Old screenshot/OCR/raw-coordinate implementations are
  explicitly named manual utilities, excluded from the package index and runtime
  registries; their arguments cannot select them through compatibility tools.
- Desktop discovery and every shared model-tier route select `native_ui`;
  browser UI selects `playwright_browser`. Telegram and visual-sensor guidance
  no longer recommends the retired OCR loop or global keyboard injection.
- Unique semantic controls skip inference. Ambiguous candidates can use pinned
  Kev; missing semantic targets can use pinned Moondream. Selection/grounding
  cannot dispatch: a separate host-admitted step checks the exact task effect,
  current window/control, Stop and observed outcome.
- Native visual capture matches AT-SPI's active PID/bounds against actual X11
  XID/PID/geometry before and after capture. Input rechecks that binding, moves
  to the finite point, verifies actual pointer arrival and rechecks ownership
  before a targeted click. Accessibility IDs are never cast to XIDs and titles
  never establish window identity. Capture failure retains real semantic state.
- This visual input adapter targets Linux X11. Existing native semantic adapters
  remain available; unavailable visual backends are explicit. There is no new
  global raw-key fallback or claimed universal visual control.

## Verification

The CU-only isolated recursive build and artifact audit pass. Its focused checks
pass 261 cases: execution 164, CLI runtime/actual Telegram discovery/guidance 60,
and shared routing/discovery 37. The model transport tests use fixtures and are
not counted as live inference.

The residual legacy-registration repair has its own isolated recursive build
and 105 focused checks passing: execution 65, real Telegram registry/native
runtime/guidance 39, and the changed TUI media-registration check. These tests
verify exact legacy-name discovery, shared native handles, separate action
admission, cancellation, no legacy perception/input and public-scope denial.
They make no live model requests or desktop input claims.

The final isolated bundle SHA-256 is
`359772f23c1dbc48c3cac29253ba57115b7f024650180b8268997d2729a830da`.
Its tarball audit passes with 588 files and zero warnings. A fresh global npm
install also passes, including the required SQLite prebuild without a native
compiler. The isolated package was not published; the operator owns publication.

Actual read-only OS validation observed 84 AT-SPI nodes and captured the active
786x557 window with verified X11/accessibility geometry. It made zero model
requests, input actions or service changes. Both model capabilities correctly
reported `not_configured`. Receipt:
`evidence/native-reconciliation-readonly-live-20261014.json`; repeatable probe:
`scripts/live-native-reconciliation-harness.mjs`.

## Historical runtime preflight — 2026-10-14T01:15Z

Read-only checks at approximately 2026-10-14T01:15Z found no configured
`~/.omnius/computer-use-runtime.json` and no resident Kev or Moondream process.
GPU0 is an A100 80 GB with 6 MiB allocated; GPU1 hosts the existing voice
workloads. `docker gpu discover` reports `gpu_health.admission_blocked=true`
because selected GPU2 is missing and reports recovery required. The scoped
lease policy therefore does not currently permit starting fresh model workloads.
No GPU lease, model load, inference request, reboot or service change was made.
This limits live model acceptance; it does not prevent implementing and testing
the shared runtime, canonical discovery and guarded execution contracts.


## Current model readiness — 2026-10-14

The operator authorized release of the controlled post-boot inference hold. The
current official broker discovery reports `inference_hold.active=false`,
`gpu_health.admission_blocked=false`, all three A100 80 GB devices selected,
no recovery actions or missing devices, no active external leases, and no warnings.
The previous missing-GPU/recovery condition above is not a current deployment
blocker. The earlier controlled local Qwen acceptance was preflighted on GPU0 at
that test's timestamp; it does not authorize assumptions about later placement.

At this read-only readiness observation, `nvidia-smi` instead reports managed
Ollama runner PID 624996 on GPU UUID
`GPU-fc9d264d-8a20-bf8c-eb72-72a6dcbdaf2a` (GPU2), with 26,944 MiB allocated.
The public `/api/ps` lists
`omnius-robit-qwen38-27b-e03-obliterated-omni:latest`, digest
`927216926bdf3c84747a51a9f0da49e8615cb46a9832931f3a1b9eaffa0621cf`.
These are separate current process/catalog observations, not a new inference
acceptance test or a permanent model-to-GPU assignment.

The rapid-loop model prerequisites remain incomplete:

- `~/.omnius/computer-use-runtime.json` is absent. No running Kev or Moondream
  point service or listener was found at the historical test ports 36155/59613.
  `mapsource-photon` is a place/address search service; ports 8091/8092/8093 serve
  map tiles. They are not compatible Moondream endpoints.
- Pinned artifacts remain installed under
  `/home/roko/Documents/Projects/Adjacent/omnius/omnius-rapid-computer-use-20261012/.omnius/rapid-model-eval`.
  These include the Kev Python environment and manifest, the pinned Kev adapter
  and Qwen3.5-4B-Base shards, and the Photon Python overlay, Moondream checkpoint,
  tokenizer, manifest and isolated CUDA 12.9 runtime. File existence and sizes
  were checked; this observation did not rehash every model byte or execute a
  model/import CUDA. The recorded immutable manifests must be verified again
  when provisioning a current runtime.
- Old selection/runtime/approval files are historical. Their PIDs, boot identity,
  lease scopes and expired authorization cannot satisfy the current model gate.
  Creating a profile that points at those files would not make a healthy service.
- `createRuntimeUiModelClients` reads the private profile when a tool instance is
  constructed. It does not watch for profile creation/change. Existing registered
  tools therefore need controlled recreation/reload after provisioning; a new
  profile alone cannot update their captured model clients.
- The resident gate authorizations last 120 seconds by default and at most 300
  seconds. Each model call verifies fresh process/GPU/lease evidence and the
  current authorization. A persistent deployment needs a host-owned renewal path
  under the operator's standing deployment authorization, with current placement
  checks; extending or removing this gate is not the renewal mechanism.
- `integrated_models.py` is an existing bounded acceptance supervisor. It requires
  Qwen, Kev and Photon together on one exact leased GPU, supports a 60–1800 second
  lifetime, and deliberately does not issue load/engine/inference approvals. It
  is useful for acceptance but is not a persistent two-model deployment service.
- The npm deployment assets currently include the model gate, GPU-lease helper,
  Laya service, Photon service, point helper and model lock. They omit
  `kev_service.py` and `integrated_models.py`. An externally provisioned compatible
  Kev endpoint can still be consumed by the client; self-provisioning from the
  installed package requires packaging the selected service/supervisor pieces.

Read-only inventory: [current model readiness](evidence/computer-use-runtime-readiness-20261014.json).
This observation made no model requests, desktop input, service changes, lease
acquisitions or model loads.

## Minimal supported runtime profile

The main agent remains the planner; this profile supplies only finite policy and
visual grounding to both the browser and native factories. The architecture's
Qwen3.8-Flash-Next checkpoint is a separate uninstalled/unselected planner target;
the working 27B local model is not evidence that this larger target was deployed.

The following is a shape example, not an installed or authorized profile. Ports
36155/59613 are historical examples and must be bound and verified for the new
services. Every referenced state file must be generated for that current runtime.
The actual profile must be an owner-controlled regular file with mode 0600;
state directories must also be private.

```json
{
  "schema": "omnius.browser-runtime.v1",
  "policy": {
    "kind": "system-one",
    "family": "kev",
    "endpoint": "http://127.0.0.1:36155/v1/systemone",
    "model": "jaredpalmer/kev-4b@6cfce5c2fa4b4bd64026336ab649c5ca78857d52",
    "revision": "6cfce5c2fa4b4bd64026336ab649c5ca78857d52",
    "preflight": {
      "selectionFile": "/home/roko/.omnius/computer-use/kev/current/selection.json",
      "receiptFile": "/home/roko/.omnius/computer-use/kev/current/resident-runtime.json",
      "approvalFile": "/home/roko/.omnius/computer-use/kev/current/inference-approval.json"
    }
  },
  "visual": {
    "kind": "moondream-point",
    "endpoint": "http://127.0.0.1:59613/v1/point",
    "model": "moondream3.1-9B-A2B",
    "revision": "1da27e6c2d2f9118de14560df456881dc62d91f3",
    "preflight": {
      "selectionFile": "/home/roko/.omnius/computer-use/photon/current/selection.json",
      "receiptFile": "/home/roko/.omnius/computer-use/photon/current/resident-runtime.json",
      "approvalFile": "/home/roko/.omnius/computer-use/photon/current/inference-approval.json"
    }
  }
}
```

## Remaining work before live rapid-loop completion

1. Provide a supervised production lifecycle for the selected Kev and Moondream
   roles: scoped broker leases, heartbeats, prepare/ready transitions, bounded
   shutdown, current-runtime authorization renewal and recovery after restart.
   Reuse the existing pinned services and gate; package the required launcher
   assets if Omnius owns deployment from its npm installation.
2. Revalidate the installed artifact/dependency manifests, select available A100
   scopes from current broker discovery, start the services with exact reserved
   CUDA visibility, and perform the staged weights/engine placement preflight.
   Moondream's engine warmup needs its separate current approval. Do not reuse
   old endpoint/PID receipts or bypass the current lease/health gate.
3. Verify the live `/v1/systemone` and `/v1/point` model identities, generate the
   current selection/resident/authorization bundles, then install the private
   profile. Add or exercise controlled tool recreation so ordinary active TUI
   and Telegram-admin tools receive the new clients.
4. Run actual TUI and admin-DM acceptance with the registered canonical tools:
   demonstrate a unique semantic fast path, an ambiguous finite choice invoking
   Kev, and a missing semantic target invoking Moondream before guarded input.
   Include browser navigation/forms and native file/folder/editor operations,
   current task goal/state in decisions, observed outcomes, and Stop during a
   pending model call. Record model and action receipts plus end-to-end latency.

Until those steps pass, the completed claim is canonical controller registration,
legacy handoff, semantic execution and verified visual capture. Live, persistent
Kev+Moondream visual task completion remains open, independently of the current
inference/publication repair.
