# Open-Weight Rapid Computer Use: Decision Models + Moondream + Qwen3.8

**Single-file implementation and research package**  
**Reference date:** 2026-10-11  
**Target:** Linux/Windows desktop and Firefox/Chromium browsers; local inference; 3 × NVIDIA A100 80 GB PCIe  
**Intended reader:** engineering / autonomous implementation agent

> **Status and evidence discipline.** This is a sourced engineering design and implementation work order, **not** a claim that the assembled system has been tested. Model quality and speed figures below are author-reported unless specifically labeled independent. A TypeSafe-compatible API does not imply identical model weights, calibration, coverage, or performance. Pin revisions and recheck licenses at deployment. All recommended inference models have downloadable weights; some additional *toolkits* provide one-pass decisions using third-party open weights.

## Contents

1. [Mission, design principles, and boundaries](#1-mission-design-principles-and-boundaries)
2. [The rapid decision-model landscape](#2-the-rapid-decision-model-landscape)
3. [Vision, GUI specialists, and reasoner sources](#3-vision-gui-specialists-and-reasoner-sources)
4. [Recommended combined architecture](#4-recommended-combined-architecture)
5. [Unified interface state and action contracts](#5-unified-interface-state-and-action-contracts)
6. [Browser observation and control](#6-browser-observation-and-control)
7. [Desktop observation and control](#7-desktop-observation-and-control)
8. [Decision orchestration and execution policy](#8-decision-orchestration-and-execution-policy)
9. [Visual grounding and coordinate handling](#9-visual-grounding-and-coordinate-handling)
10. [Workflow compilation, learning, and data](#10-workflow-compilation-learning-and-data)
11. [A100 deployment and verified launch entry points](#11-a100-deployment-and-verified-launch-entry-points)
12. [Security, authority, and recovery](#12-security-authority-and-recovery)
13. [Benchmark and model-selection protocol](#13-benchmark-and-model-selection-protocol)
14. [Agent implementation work order](#14-agent-implementation-work-order)
15. [Primary references](#15-primary-references)

---

## 1. Mission, design principles, and boundaries

### Objective

Build a **low-latency computer-use controller** that works on both browser content and desktop applications and runs locally with open-weight models:

- **Qwen3.8-Flash-Next:** higher-order goal decomposition, uncertainty resolution, recovery, and planning.
- **Moondream 3.1:** image understanding, pointing and object detection when semantic UI APIs cannot resolve a target.
- **Jev-class rapid decision model:** score or select among a **runtime-bounded set of valid actions**, preferably with probability outputs and no text-generation loop.
- **Native observer/executor:** DOM/accessibility/UI Automation structures, first-class element operations, guarded input, and observation of effects.
- **Persistent interface graph:** store reusable handles, relationships, state revisions, observed outcomes, and successful workflow fragments.

A plan should normally run without calling Qwen for every click. A visual pass should normally be skipped if native semantics already identify the target. An action should complete on **observed state**, never on an arbitrary sleep alone.

### Non-negotiable boundaries

- Open weights for **inference**; TypeSafe Jev itself is **closed/hosted and reference-only**, not part of the local stack [D01].
- No inference that a JEV-like text decision model can natively locate screen coordinates: it cannot unless expressly multimodal or paired with Moondream/another GUI VLM.
- The policy can choose **only actions validated and advertised by the current runtime**, never arbitrary free-form executable code.
- The observer must revalidate stale handles and visual coordinates after navigation, focus changes, screen transforms, layout changes, scrolling, or virtualization of lists.
- No silently executing sensitive side effects. Define a permission/approval and audit model.
- Benchmark end-to-end outcomes on the actual machine, not only a model's millisecond forward-pass figure.

---

## 2. The rapid decision-model landscape

### 2.1 What "Jev-like" actually means

TypeSafe's System One / Jev takes a shared state and typed questions such as **Choice** (finite options), **Noul** (probability of a binary proposition) and **Score** (ordinal rubric); the result is a bounded answer/probability rather than prose [D01]. Its hosted weights are not an open-weight dependency. Independent research evaluates its decision behavior, calibration, and task limitations [D02].

There are three relevant *implementation families*:

1. **Purpose-built heads / discriminative encoders:** one representation of state plus direct, typed decision heads, often very small (Laya, Von, CUA-S1-FORMS).
2. **Language-model option-probability readout:** a model processes context and its answer-token logits are compared, without completing JSON/prose (Kev, decider, JevK5, SemIf, and generic readout toolkits). Some fine-tune the base; others reuse it.
3. **Contrastive state↔action retrieval:** embed the state and reusable candidate actions independently; cache action embeddings and score by similarity (CLM-8B). This is particularly appealing for dense interactive loops [D08].

**The Jev-class model is a policy selector, not a complete computer-use agent.** GUI understanding and reliable actuation remain separate responsibilities.

### 2.2 Primary open-weight decision/action models

| Family | Released size(s) | How decisions are produced | Best computer-use role | Code and weight sources | Caveat |
|---|---|---|---|---|---|
| **Kev 1.0** | 0.8B / 4B / 9B / 27B | LoRA+head on Qwen bases for typed Choice/Noul/Score; locally served System One-style API | General action arbitration with rich state | [Code D03](https://github.com/jaredpalmer/kev); [models](https://huggingface.co/jaredpalmer/kev-4b); [release](https://github.com/jaredpalmer/kev/releases) | Kev's held-out decision metrics are **not** GUI task-success rates. |
| **Laya** | 421M English / 322M multilingual / 421M typed-decisions | Non-autoregressive encoder and typed heads; RLCD-based calibration | Lowest-footprint fast policy, yes/no guards | [Code D04](https://github.com/NandhaKishorM/laya); [weights](https://huggingface.co/convaiinnovations/laya) | Short-state/domain testing essential. |
| **Von** | 395M | ModernBERT encoder; direct Choice/Noul/Score in one pass | CPU-resident fallback; frequent classification | [Code D05](https://github.com/wfzyx/von); [weights link in README](https://github.com/wfzyx/von#readme) | Vendor-reported sub-15 ms is workload/hardware-specific. |
| **decider** | 0.8B / 2B / 4B / 12B / 31B / 35B-A3B; some GGUF | Trained one-pass readout and calibrated probabilities; vision variant exists | General small local policy, 2B/4B GGUF evaluation | [Code D06](https://github.com/Mapika/decider); [4B weights](https://huggingface.co/Mapika/decider-4b); [2B GGUF](https://huggingface.co/Mapika/decider-2b-GGUF); [2B vision](https://huggingface.co/Mapika/decider-2b-vision) | Different variants have materially different base/training/runtime. |
| **JevK5** | 4B / 9B; 2B and GGUF derivatives | Qwen3.5 fine-tune; option-logit readout; CUDA-graph runtime | Fast GPU policy with typed endpoint | [Code D07](https://github.com/allebee/jevk5); [4B weights](https://huggingface.co/alibiserikbay/JevK5); [9B weights](https://huggingface.co/alibiserikbay/JevK5-9B) | Benchmarks and numerical revisions must be pinned; some learned from hosted teachers, but weights are public. |
| **CLM-8B** | Qwen3-8B encoder + learned projection head | **Contrastive state/action embeddings**; cached candidate embeddings; dot-product ranking | Repeated actions, variable legal-action sets, high-throughput candidate ranking | [Code D08](https://github.com/Contrastive-LM/CLM); [model/data collection](https://huggingface.co/Contrastive-LM) | Requires embedding-serving infrastructure; logits/probabilities not directly equivalent to Kev. |
| **OneJev** | 0.8B / 4B / 9B / 27B, 27B FP8 | **Multimodal** typed decisions over screenshots/video/text in one pass | Direct visual action arbitration; cross-check Moondream | [Code D09](https://github.com/OmniJev/OneJev); [0.8B](https://huggingface.co/OmniJev/OneJev-0.8B); [4B](https://huggingface.co/OmniJev/OneJev-4B) | Promising but new; verify on actual UI controls and coordinate task. |
| **PlayJev** | 0.8B | Visual frame + constrained moves in one pass; trained on browser games | Research baseline for pixel→finite-action policy | [Code D10](https://github.com/OmniJev/PlayJev); [weights](https://huggingface.co/OmniJev/PlayJev-0.8B) | Browser-game specialist, not a general-purpose desktop agent. |
| **CUA-S1-FORMS** | 706,048 parameters | Per-form-field FILL/CHECK/CLICK/SKIP scorer | Specialized forms sub-policy, extremely small | [Weights D11](https://huggingface.co/cua-ai/cua-s1-forms); [research repo](https://github.com/cua-ai/Cua-S1) | Very narrow domain; model-card evaluation does not generalize to other UI. |
| **NanoJev** | 0.6B | Open small parallel typed-decision model and training pipeline | Minimal research/training baseline | [Code D12](https://github.com/TianyuCodings/NanoJev) | Research-grade, validate checkpoint and outputs. |
| **Luce** | Qwen3-4B-base training recipe | Synthetic-data generation, LoRA+head, temperature calibration | Fine-tune a custom policy on your own UI traces | [Code D13](https://github.com/scienthoon/luce) | A **recipe** rather than proof of general UI competence. |
| **OpenJev / OpenJev weights** | Depends on specific project | Generic local typed decisions from LM logits | Interoperability / experiments | [Open weights D14](https://huggingface.co/openjev/openjev); [local runtime D15](https://github.com/lookski/openjev) | Multiple unrelated projects use this name; do **not** conflate them. |

**Recommended first evaluation group:** Laya (lightweight policy), Kev-4B, decider-4B, JevK5-4B, CLM-8B (cached option ranking), OneJev-0.8B (visual decisions). Include a rules-only baseline. Use an identical dataset and action vocabulary for every candidate; report unsupported question/option shapes rather than silently translating away differences.

### 2.3 One-pass frameworks that reuse open weights, not new checkpoint families

| Framework | Source | What it contributes | Important distinction |
|---|---|---|---|
| **SemIf (formerly OpenJev)** | [GitHub D16](https://github.com/TheoLeeCJ/SemIf-OpenJev) | Direct option-logit readout, CUDA/MLX/llama.cpp paths, WebGPU experiment and shared-state evaluation | Runtime over third-party frozen model; does **not** mean its own new trained checkpoint. |
| **Open Alternative to Jev** | [GitHub D17](https://github.com/ikermoel/open-alternative-jev) | In-process option scoring using Transformers/vLLM; packed shared-state questions | **Not** itself a wire-compatible `/v1/systemone` server. |
| **Together Tev1 experimental** | [GitHub D18](https://github.com/togethercomputer/tev1) | Reproducible data/finetune and constrained-label examples | Source states training data and model weights are **not bundled in the repo**; treat local availability/terms as **unverified**, not a default open-weight deployment. |

### 2.4 Independent context and latency evidence

- [JevBench](https://github.com/fstandhartinger/jevbench) provides a comparable protocol for finite typed decisions, with a published [v1.4.2 report](https://github.com/fstandhartinger/jevbench/blob/main/docs/RELEASE-v1.4.2.md) listing **decider-4B v2** first overall and **Jev 1.13.0** second by its composite scoring. The metric mixes intelligence, calibration, latency and cost; it is **not** a direct GUI-control leaderboard [E01].
- [Kev 1.0 release](https://github.com/jaredpalmer/kev/releases) describes a frozen checkpoint suite and honest development/locked-test splits. Quoted accuracy requires reading the exact column and release tag [D03].
- [CLM](https://github.com/Contrastive-LM/CLM) reports up to **9×** lower latency than Jev for selected computer-use/gaming/tool-ranking tasks with cached candidates, not a universal 9× guarantee [D08].
- [Laya](https://github.com/NandhaKishorM/laya) reports **33 ms** one-question and **7.2 ms/question** batched on a **T4** for its own workload [D04].
- [JevK5](https://github.com/allebee/jevk5) reports about **13 ms p50** on an **H100** for short 4B decisions in its CUDA-graph path. Longer documents are slower [D07].
- [OneJev](https://github.com/OmniJev/OneJev) reports for a 1280×720 screenshot on an **H200**: **0.8B** ~31 ms for one question, ~51 ms for ten, measured in its harness [D09].
- Published hardware and test suites are different. **Do not** transpose H100/H200/T4 inference figures onto A100 end-to-end click latency; measure observe + queue + forward + execute + verify.

### 2.5 Practical model-selection matrix

| Need | Start with | Compare with | Reason |
|---|---|---|---|
| Lowest possible decision footprint | Laya 421M or Von 395M | CUA-S1-FORMS for forms | Small discriminative model; can stay hot on CPU or GPU. |
| Fast general typed action choices | Kev-4B | decider-4B, JevK5-4B | Trainable and supports dynamically supplied choices. |
| Thousands of recurring action descriptions | CLM-8B | Kev-4B | Candidate embeddings can be computed once and cached. |
| Screenshot→typed action without a separate VLM | OneJev-0.8B/4B | PlayJev for benchmark demonstrations | Native visual decision models, but must validate grounding. |
| Extremely narrow form routines | CUA-S1-FORMS | deterministic form engine | Smallest specialized policy, but domain limited. |
| Custom task-specific calibration/finetuning | Kev, Laya, Luce recipe | decider, CLM | Fine-tune on real state/action traces, held-out applications. |
| Existing open LLM with no extra trained policy | SemIf or Open Alternative to Jev | rules + typed constrained decoder | Fastest integration path, but calibration is not automatic. |

---

## 3. Vision, GUI specialists, and reasoner sources

These are **not interchangeable** with decision-only models. Their role is to interpret pixels, reason about task context or propose actions when UI semantics are unavailable.

| Model / layer | Role | Open-weight / license observations | Primary source |
|---|---|---|---|
| **Moondream 3.1 9B-A2B** | `point`, `detect`, `query`, `caption` on local screenshots; 9B total / 2B active claimed | Downloadable; **Moondream Model License**, not Apache. Local Photon documented. | [Model card V01](https://huggingface.co/moondream/moondream3.1-9B-A2B) |
| **Qwen3.8-Flash-Next** | Reasoner/planner; multimodal fallback, long-horizon recovery | Downloadable; **Qwen Community License 1.0**; review commercial conditions | [Official model V02](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) |
| **UI-Venus-1.5-2B** | Compact screenshot GUI agent | Model card lists **Apache-2.0** | [Weights V03](https://huggingface.co/inclusionAI/UI-Venus-1.5-2B) |
| **UI-Venus-2-9B** | Larger screenshot GUI agent | Weight availability confirmed; **license not explicit on the checked model card**: verify upstream terms | [Weights V04](https://huggingface.co/inclusionAI/UI-Venus-2-9B) |
| **UI-TARS-1.5-7B** | Screenshot-driven GUI grounding and action generation | Model card lists **Apache-2.0** | [Weights V05](https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B) |
| **OS-Atlas-Pro-4B** | Lightweight grounding / GUI action modeling | Confirm model-specific terms before deployment | [Weights V06](https://huggingface.co/OS-Copilot/OS-Atlas-Pro-4B) |
| **OpenCUA-7B** | Desktop GUI trajectories and computer-use | Model card lists **MIT** | [Weights V07](https://huggingface.co/xlangai/OpenCUA-7B) |
| **Fara1.5-27B** | Screenshot-centric browser action model | Model card lists **MIT** | [Weights V08](https://huggingface.co/microsoft/Fara1.5-27B) |
| **OneJev-0.8B/4B** | Direct finite visual decisions | Apache-2.0 in repo/model cards; also catalogued above | [OneJev V09](https://github.com/OmniJev/OneJev) |

**Default:** Moondream 3.1 is the first visual grounding tool. Optionally benchmark OneJev and UI-Venus against it for tasks that fail with pixel pointing alone. Qwen3.8 is the planner, **not** the cursor driver. The reasoner should receive compact symbolic state, relevant screenshots and verification failures, not a continuous firehose of complete screenshots.

**Correction to earlier sample code:** the Moondream 3.1 card currently documents `md.vl(local=True, model="moondream3.1-9B-A2B")` rather than assuming `md.photon(...)` as the supported entrypoint [V01].

---

## 4. Recommended combined architecture

```text
                         USER GOAL / APPROVAL POLICY
                                     |
                        +------------v------------+
                        | Qwen3.8 task planner     |
                        | plan, goals, pre/post    |
                        | invariants, recovery    |
                        +------------+------------+
                                     |
             +-----------------------v------------------------+
             | UNIFIED GRAPH / CONTEXT ENGINE                 |
             | windows, tabs, elements, handles, relations,   |
             | geometry, revisions, action capabilities       |
             +--------------+------------------+--------------+
                            ^                  ^
          semantic events   |                  |    visual detections
              +-------------+--+          +----+-------------+
              | DOM / CDP/BiDi |          | Moondream        |
              | AX / UIA /     |          | pointing /       |
              | AT-SPI / AXAPI |          | detection         |
              +----------------+          +------------------+
                                     |
                           VALID ACTION GENERATOR
                          (typed finite candidates)
                                     |
                        +------------v------------+
                        | RAPID DECISION POLICY    |
                        | Laya/Kev/CLM/decider/    |
                        | JevK5/OneJev            |
                        +------------+------------+
                                     |
                           ACTION / CAPABILITY GUARD
                                     |
                          NATIVE EXECUTOR / INPUT
                                     |
                          EFFECT OBSERVER + VERIFIER
                                     |
                     changed? satisfied? retry? replan?
                                     +----------> GRAPH / QWEN
```

### Paths ordered by speed and robustness

- **Path A / deterministic:** known workflow + valid target + sufficient state → native API action → verify; **no model call**.
- **Path B / policy:** multiple legal actions → single local typed decision over compact state → native API action → verify; **no Qwen or Moondream call**.
- **Path C / visual:** semantic handle unavailable → capture relevant screen region → Moondream point/detect → geometry validation → policy/executor → verify.
- **Path D / replan:** task ambiguous, repeated verification failures, unexpected modal, lost permissions, conflicting evidence → Qwen and optional additional vision → new preconditioned plan.

Do not make a large model an obligatory per-action dependency. Maintain event-driven observations. A periodic watchdog may detect stalled channels, but never advance a task merely because a fixed delay elapsed.

---

## 5. Unified interface state and action contracts

A minimal TypeScript schema for the control plane (use Zod/JSON Schema at every service boundary):

```ts
type Source = 'dom' | 'uia' | 'atspi' | 'ax' | 'vision';
type CoordinateSpace = 'css-viewport' | 'device-pixel' | 'screen-logical';
type Capability =
  | 'invoke' | 'set_value' | 'set_checked' | 'select'
  | 'focus' | 'scroll' | 'drag' | 'type_text' | 'key_chord';

type Rect = {
  x: number; y: number; width: number; height: number;
  space: CoordinateSpace;
  frameId?: string; displayId?: string;
  scale?: number;
};

type Element = {
  id: string;                 // scoped, opaque ID, not a recycled array index
  generation: number;         // invalidate on navigation/frame teardown/rebinding
  source: Source;
  appId: string;
  windowId: string;
  frameId?: string;
  parentId?: string;
  role: string;
  name: string;
  rect?: Rect;
  state: {
    visible?: boolean; enabled?: boolean; focused?: boolean;
    checked?: boolean; expanded?: boolean; selected?: boolean;
    value?: string;
  };
  capabilities: Capability[];
  confidence?: number;      // source-specific; never assume cross-model calibration
  selectorHint?: string;    // non-authoritative re-find hint
  observedAt: string;
};

type UIState = {
  revision: number;
  activeWindowId: string;
  focusedElementId?: string;
  elements: Element[];
  observations: Array<{ kind: string; source: Source; revision: number }>;
};

type Action = {
  idempotencyKey: string;
  stateRevision: number;
  target: { id: string; generation: number };
  op: Capability;
  args: Record<string, unknown>;
  preconditions: Array<{ predicate: string; args: unknown }>;
  expectedEffects: Array<{ predicate: string; args: unknown }>;
  risk: 'read' | 'reversible-write' | 'external-side-effect' | 'destructive';
};
```

### Identity, geometry, and consistency

1. Key native handles by `browserContext+target+frame+backendNodeId` or `process+window+accessibility-runtime-id` as appropriate. IDs are opaque and may be invalidated at any time.
2. Distinguish **viewport CSS px**, **captured-image px**, **logical desktop px**, and **physical device px** explicitly.
3. Maintain a monotonic **graph revision** and per-element **generation**. An action with a stale revision requires re-resolve or predicate revalidation.
4. Favor selectors by role, accessible name, test ID, or stable application-specific identity over coordinates.
5. Attach `source`, `confidence`, and `observedAt` to visual detections. Do not treat VLM point accuracy as known merely because a point was returned.
6. Only expose operations the adapter can actually perform. `set_checked` should map to a supported native toggle/checkbox pattern when possible; otherwise perform a guarded physical click.
7. The graph stores a **working projection** of app state, not secret content or a permanent screenshot archive by default.

### Legal candidate envelope for the decision model

```json
{
  "state_revision": 417,
  "goal": "Enable dark mode in the active application",
  "focused_window": "settings",
  "context": {
    "visible_dialog": false,
    "last_action": "open_preferences",
    "last_effect": "preferences_visible"
  },
  "actions": [
    {
      "key": "a1", "target": "e42", "op": "set_checked",
      "description": "Enable the Dark mode switch",
      "preconditions": ["e42.enabled", "e42.checked == false"]
    },
    {
      "key": "a2", "target": "e18", "op": "invoke",
      "description": "Open Appearance section"
    },
    {
      "key": "wait", "target": null, "op": "none",
      "description": "No safe action; request new observation"
    }
  ]
}
```

The policy returns `a1 | a2 | wait` with its option distribution if supported. **Application code** resolves the chosen ID, checks preconditions, authorizes side effects, and invokes the executor. Never run free-form code returned by the model.

---

## 6. Browser observation and control

### 6.1 Dedicated automation browser

For a managed Chromium instance:

- **CDP**: DOM snapshot, accessibility tree, frame tree, layout boxes/quads, input and lifecycle events. Official protocol domain index: [B01](https://chromedevtools.github.io/devtools-protocol/).
- **Playwright**: high-level locator operations and checks. `getByRole`, `getByLabel`, `getByTestId`, auto-wait/visibility/enabled/stability checks. See [locators B02](https://playwright.dev/docs/locators) and [actionability B03](https://playwright.dev/docs/actionability).
- **WebDriver BiDi**: the browser-neutral evented protocol, relevant for Firefox. It remains a W3C Working Draft as of October 8, 2026: [B04](https://www.w3.org/TR/webdriver-bidi/). Implement feature detection, not blind CDP parity.

Suggested browser observer pipeline:

```text
BrowserContext/Target discovery
  -> frame and tab lifecycle tracking
  -> semantic tree / DOM/AX capture for new target
  -> annotate actionable elements and unique IDs
  -> MutationObserver + relevant accessibility/lifecycle events
  -> bounded incremental patch to persistent graph
  -> availability of snapshots only when invalidated / needed
```

**Avoid:** taking a full screenshot and asking a VLM to find every button on every loop; serializing megabytes of irrelevant DOM to Qwen; using a raw JS string passed back from a model as unrestricted `Runtime.evaluate` code.

### 6.2 Existing user browser

Use a **WebExtension** with approved host permissions and an optional native messaging host if interaction outside the browser is needed. A page opened at an ordinary URL cannot access arbitrary tabs or the desktop by itself. For Firefox, prioritize extension/content-script observation with WebDriver BiDi for managed automation. Never assume the user's regular browser exposes remote debugging or other tabs to an unprivileged webpage.

### 6.3 Browser edge conditions

Handle cross-origin iframes, shadow DOM, portals, canvas/WebGL, editor contenteditable regions, offscreen virtualized lists, native dialogs, file pickers, multiple tabs, pop-ups, navigation races, focus-stealing pages, and permission prompts. If DOM and accessibility routes cannot meaningfully resolve a target, escalate to visual grounding. For browser-bound work in a browser VM, keep the capture and input coordinate reference on the same rendering surface.

### 6.4 Browser executor choices

1. Native semantic action, if uniquely resolved and eligible: `locator.click`, `fill`, `selectOption`, `check`, `press`.
2. ARIA/action API or accessibility control pattern, if available and safer.
3. Physical pointer/keyboard injection only when necessary, after a visible-element and geometry check.
4. Re-read the relevant element or task predicate after every meaningful state change.

---

## 7. Desktop observation and control

| OS/session | Semantic surface | Native execution/capture | Engineering caution |
|---|---|---|---|
| Windows | Microsoft UI Automation: `AutomationElement`, control patterns | Native UIA patterns; guarded `SendInput`; OS screen/window capture | UIPI integrity boundaries, elevated windows, secure desktop. [T01](https://learn.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32) |
| Linux X11 | AT-SPI2 exposed through D-Bus | AT-SPI actions; XTest/window focus/capture | App/toolkit accessibility gaps, display scaling. [T02](https://www.freedesktop.org/wiki/Accessibility/AT-SPI2/) |
| Linux Wayland | AT-SPI2 for accessibility | XDG RemoteDesktop portal, libei/EIS, PipeWire for consented input/capture | Compositor/security/session support varies; no global unrestricted injection by default. [T03](https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.RemoteDesktop.html) |
| macOS | Accessibility API / `AXUIElement` | Accessibility action patterns; Quartz / ScreenCaptureKit | Explicit accessibility/screen-recording permissions. [T04](https://developer.apple.com/documentation/applicationservices/accessibility) |

Treat browser windows as desktop windows until a browser adapter positively identifies them; then attach DOM/AX subgraphs under the corresponding native window. Maintain exact pixel-transform mapping between desktop capture and the browser's CSS viewport. Focus is a **stateful exclusive resource**: serialize focus-dependent native keyboard/mouse actions per OS session, while independent browser contexts can often operate concurrently.

---

## 8. Decision orchestration and execution policy

### 8.1 Finite-action control state machine

```text
NEW_GOAL
  -> PLAN (Qwen)
  -> OBSERVE
  -> ENUMERATE_CANDIDATES
  -> DETERMINISTIC_IF_UNIQUE_AND_PROVEN
      else DECIDE_FAST
      else GROUND_VISUALLY
  -> GUARD / AUTHORIZE
  -> EXECUTE
  -> AWAIT_RELEVANT_STATE_PREDICATE
  -> VERIFY
       success -> NEXT_STEP / COMPLETE
       stale   -> REFRESH_AND_RESOLVE
       blocked -> ASK_PERMISSION / ABSTAIN
       failed  -> RECOVER_OR_REPLAN (Qwen)
```

**Fast policy request design:** keep the state representation compact and bounded; candidate labels should describe **effects** and not merely bare verbs. For example, "Click Save, committing draft settings" is superior to "Click button #13". Include current progress, last action, last observed effect, possible blockers, active app, and a strict list of valid options.

**Sampling:** default to deterministic top-choice or calibrated decision rule. Keep a legal `ABSTAIN/OBSERVE/ESCALATE` alternative; without it, a low-confidence model is forced to pick a potentially unsafe action.

**Risk handling:** model confidence is evidence, **not permission**. Require an independent risk-class authorization even for 99.9% model confidence. High-impact effects must use explicit task-scoped approval.

### 8.2 Essential executor pseudocode

```python
async def run(goal, observer, reasoner, policy, vision, executor, guard):
    plan = await reasoner.make_plan(goal, await observer.snapshot())

    while not plan.satisfied:
        state = await observer.state_delta_or_snapshot()
        step = plan.current_step(state)
        candidates = await observer.legal_actions(step, state)

        if step.precondition_failed(state):
            plan = await reasoner.replan(plan, state, reason="precondition")
            continue

        if not candidates:
            crop = await observer.capture_relevant_region(step)
            detection = await vision.ground(crop, step.target_description)
            candidates = await observer.resolve_visual_candidate(detection, crop)

        if not candidates:
            plan = await reasoner.replan(plan, state, reason="target_unresolved")
            continue

        action = (
            candidates[0] if observer.is_deterministic(step, candidates, state)
            else await policy.choose(state, candidates)
        )
        if action is None or action.key == "abstain":
            plan = await reasoner.replan(plan, state, reason="policy_abstained")
            continue

        await observer.revalidate_action(action)
        await guard.authorize(action, goal)
        await executor.perform(action)  # idempotency-aware, serialized where needed

        outcome = await observer.await_predicates(
            step.expected_effects,
            cancellation=goal.cancellation,
            watchdog=step.watchdog,  # safety bound; not an action delay
        )
        if outcome.satisfied:
            plan.advance()
        elif outcome.stale:
            await observer.invalidate(action.target)
        else:
            plan = await reasoner.replan(plan, await observer.snapshot(),
                                         reason=outcome.reason)

    return await observer.verify_goal(plan)
```

This is **architectural pseudocode**, not a runnable package. In production include retry bounds, terminal failure, human approval states, runaway-loop protection, authenticated channels, and structured log correlation IDs. Never keep a `while` loop alive indefinitely on an unrecoverable external failure.

### 8.3 Should CLM or Kev be the main policy?

- **CLM advantage:** many action descriptions recur across windows/tasks; embedding cache and action retrieval avoid rerunning expensive candidate encoders. Particularly attractive for tool catalogs, repeated desktop commands, and large action menus [D08].
- **Kev/decider advantage:** direct typed probabilities over custom runtime questions; well suited when the state has subtle textual constraints and per-step candidates change [D03][D06].
- **Laya/Von advantage:** low resource footprint; guard models can run hot without monopolizing main reasoner VRAM [D04][D05].
- **OneJev advantage:** direct multimodal decisions from screenshot + typed options; measure whether it reduces a separate Moondream→policy cascade [D09].

A pragmatic winning configuration may be **rules first → Laya or CLM → Kev on uncertain cases → Moondream → Qwen3.8**. Benchmark **cost of escalations** and total task success, not merely standalone policy accuracy.

---

## 9. Visual grounding and coordinate handling

### 9.1 Moondream local API

The model card's supported local Photon interface is:

```bash
python -m pip install moondream pillow
```

```python
import moondream as md
from PIL import Image

vision = md.vl(local=True, model="moondream3.1-9B-A2B")
screenshot = Image.open("screenshot.png").convert("RGB")
result = vision.point(screenshot, "the Save button inside Preferences")
print(result["points"])
```

Source: [Moondream 3.1 primary model card](https://huggingface.co/moondream/moondream3.1-9B-A2B). First load downloads weights; validate installed Photon, GPU compatibility and its exact point coordinate representation for the pinned version.

### 9.2 Coordinate transforms

A **grounded point** is only meaningful relative to the exact image supplied to the model. Record the crop origin, dimensions, resizing/letterboxing, device-pixel ratio, display ID, window position, client-area insets and any scroll offset. Implement one tested transform:

```text
model output in model/image coordinates
  -> undo normalization / letterbox / image resizing
  -> add crop offset in full capture
  -> map capture physical px to display logical coordinates
  -> map into target input API coordinate system
  -> hit-test against current target's live rectangle
```

If a model returns normalized coordinates, for an **uncropped, unresized** screenshot `x_px = x_norm * image.width` and likewise for `y`; do **not** apply this simplification after a crop or letterbox without undoing transforms. Wrong-coordinate clicks must be a first-class failure in the evaluator.

### 9.3 Where vision should be triggered

- Canvas/WebGL/custom icon or unlabeled hit target.
- Native/embedded widget whose accessibility tree is empty or incorrect.
- Occluded, partially rendered or ambiguous semantic control.
- UI graph mismatch with screenshot after a material layout change.
- Complex drag-and-drop, spatial ordering or visually represented state.

Do not invoke vision for stable inputs with a correct `set_value`, `invoke`, `toggle`, or `select` pattern.

---

## 10. Workflow compilation, learning, and data

### 10.1 Reusable workflow representation

```yaml
id: preferences.set_dark_mode
version: 1
scope: app:example-settings
preconditions:
  - active_app == "example-settings"
steps:
  - resolve: {role: button, name: Preferences}
    action: invoke
    effect: {predicate: section_visible, value: Appearance}
  - resolve: {role: switch, name: Dark mode}
    action: set_checked
    args: {checked: true}
    effect: {predicate: checked, value: true}
postconditions:
  - {predicate: switch_checked, name: Dark mode, value: true}
escalation:
  missing_target: moondream
  unexpected_state: qwen_replan
```

Compile only after successful **verification**, not simply after action submission. Plans must remain **conditional**: if a precondition fails, re-observe instead of replaying blindly. Cache semantic locators, not absolute pointer coordinates. Include task-scoped permissions with every replay.

### 10.2 Data schema for policy distillation

```json
{
  "trace_id": "uuid",
  "application_family": "browser_settings",
  "state_digest": "hash",
  "state_features": {},
  "goal_step": "toggle_dark_mode",
  "legal_actions": [{"key": "a1", "description": "..."}],
  "chosen_action": "a1",
  "result": "verified_success",
  "cost": {"latency_ms": null, "vision_calls": 0, "qwen_calls": 0},
  "risk": "reversible-write",
  "manual_override": false,
  "provenance": {"model_revision": "...", "ui_revision": 417}
}
```

Never use **unverified success** as a positive label. Add hard negatives: plausible buttons with incorrect role, stale handles, visually similar but destructive icons, masked/disabled controls, occluded targets, permissions dialogs, changed labels, wrong focus. Split train/test by application, website, workflow and UI version to detect leakage. Calibrate thresholds on a validation split and freeze test sets.

### 10.3 Distillation order

1. Instrument robust baseline with rules + Qwen + Moondream; capture allowed action sets and actual outcomes.
2. Replay to evaluate Laya/Kev/CLM/decider/OneJev **without executing their proposed action**.
3. Select promising model(s) and fine-tune on verified successful outcomes and carefully constructed negatives.
4. Calibrate per action-risk tier, app category, and candidate-set size; measure ECE, Brier and selective risk.
5. Enable live policy behind action guard for low-risk tasks; compare against shadow baseline.
6. Promote successful action fragments into conditional workflows, with regression replay on every change.

---

## 11. A100 deployment and verified launch entry points

### 11.1 Layout (starting hypothesis)

| Resource | Assignment | Why |
|---|---|---|
| **A100 #0 + #1 (80 GB each)** | Qwen3.8-Flash-Next, suitable quantization/tensor-parallel topology | Reasoner; high total parameter count despite low activated-parameter count. Validate memory and runtime support first. |
| **A100 #2 (80 GB)** | Moondream 3.1 (Photon), plus small decision policy (Kev-4B or Laya/Von or OneJev) | Keep visual grounding and decision loop hot and isolated. |
| **CPU/RAM** | Browser sessions, UI graph, desktop broker, event bus, observation, action guards, verification | Must remain responsive independently of model queue. |

Alternative: put Laya/Von on CPU, CLM pooling on GPU #2 if its throughput benefit outweighs Moondream contention; or use 27B Kev only for exception tiers. **Do not promise that 125B total-parameter Qwen3.8 fits any arbitrary 2×A100 quantization.** Verify actual selected checkpoint's weight shards, model architecture support, quantization format, KV cache, tensor-parallel compatibility, throughput and headroom before choosing placement [V02]. A100 is Ampere, not Hopper/Blackwell; do not assume FP8/NVFP4 fast paths supported by newer GPUs will work unmodified.

### 11.2 Verified repository launch examples

**Kev local server** (official entrypoint; installation may download checkpoints):

```bash
git clone https://github.com/jaredpalmer/kev.git
cd kev
uv sync --extra serve
uv run --extra serve python -m kev.serve --run jaredpalmer/kev-4b@v1.0 --port 8009
```

See [Kev instructions](https://github.com/jaredpalmer/kev/blob/main/README.md), and its [v1.0 release](https://github.com/jaredpalmer/kev/releases) for frozen revisions. Confirm `--run` tag compatibility with installed repo checkout.

**Laya** (official Python API; model loads on first predict):

```bash
python -m pip install laya
```

```python
from laya import Router
router = Router()
result = router.predict(
    "Active panel: Preferences. Dark mode switch is unchecked.",
    {"next": {"type": "choice", "instructions": "Best legal next step?",
              "criteria": {"toggle": "Enable the existing Dark mode switch",
                           "inspect": "Observe the settings panel again"}}},
)
print(result)
```

See [Laya README](https://github.com/NandhaKishorM/laya). The listed alternative of **Von** uses its own SDK; don't pretend the exact Python signatures are shared by every provider.

**CLM** (repo's documented architecture; requires an embedding server):

```bash
python -m pip install contrastive-lm
vllm serve Qwen/Qwen3-8B --served-model-name qwen3-8b --runner pooling --max-model-len 2048 --port 8090
# in a separate process
clm-serve
```

See [CLM README](https://github.com/Contrastive-LM/CLM). The 75 MB reference projection head is downloaded by CLM as documented. Allocate compatible GPUs and keep the pooling server separate from the Qwen3.8 planner.

**OneJev direct screenshot decisions**:

```bash
pip install "qev[torch] @ git+https://github.com/OmniJev/OneJev.git"
qev serve --model OmniJev/OneJev-0.8B
```

See [OneJev model card](https://huggingface.co/OmniJev/OneJev-0.8B); its server extends the System One-shaped request with a media field. Do not send images to a text-only TypeSafe-compatible policy and expect it to understand them.

**Moondream**: use `md.vl(local=True, model="moondream3.1-9B-A2B")` per §9 and [official card](https://huggingface.co/moondream/moondream3.1-9B-A2B).

**Qwen3.8**: use the official [model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) and select **a server version explicitly supporting its architecture and selected quantization**; do not hardcode unverified `vllm serve` flags or assume arbitrary GGUF ports. If a matching GGUF is used in Ollama/llama.cpp, document the quantizer, model revision, image modality support and tokenizer behavior in the run manifest.

### 11.3 Deployment contracts

- Distinct localhost ports; authenticated connections even when exposing through a remote reverse proxy.
- Versioned `/health`, `/ready`, `/metrics`; fail closed when model unavailable.
- Time-bounded and cancellation-aware model calls; bounded queues and priority for interactive action decisions.
- Graph and executor available even when planner/vision service is reloading; no frozen desktop UI loop.
- Startup self-test: text decision, visual point, target-selector invocation, state predicate, cancellation, model revision and GPU memory report.
- Collect p50/p95/p99 model calls *and* user-visible operation and task latency.

---

## 12. Security, authority, and recovery

**Threat model:** the model observes untrusted web page content, email, application labels, document text, and images that can contain instructions addressed to an AI. These are **data**, not an authorized change of task. The true user's intent and capability policy outrank page content.

Implement:

- **Explicit approval** before sending messages, running untrusted commands, installing packages, purchasing goods, publishing or deleting data, changing permissions, exposing secrets or sending network traffic to new destinations outside the task's authority.
- **Allowlisted capability routing**: the model's finite action key must map to a code-owned operation and target; code refuses unknown verbs, stale IDs and out-of-scope apps.
- **Safe secrets handling**: never transmit credentials/screenshots of sensitive areas to cloud endpoints; by default redact sensitive fields from policy input and traces, and disable raw logging of passwords.
- **No unreviewed page code execution** through arbitrary `Runtime.evaluate` strings. DOM extraction scripts are maintained, versioned and code-owned.
- **Reversible checkpoints** for low-risk state changes; idempotency tokens for actions that can be repeated; unambiguous terminal outcomes for side effects.
- **Abstention and rollback** where supported; bound retries; circuit breaker for repeated invalid targets, broken accessibility endpoints and focus thrashing.
- **Operator interruption** (stop/pause/cancel), full local audit trace, application/window-level permissions, least privilege, and consent for Wayland/macOS capture/control.
- **No fabricated verification:** success requires observed postconditions (or explicit uncertainty if external completion cannot be observed).

---

## 13. Benchmark and model-selection protocol

### 13.1 Benchmarks to run

| Test family | Examples | Metric |
|---|---|---|
| Static action decision | 2–50 legal controls, role/name/state/risk conflicts | top-1 action, expected effect, ECE, Brier, abstain coverage |
| Browser semantic | Form fill, tab switch, dropdown, modal, nested iframe, virtual list | success rate, steps, stale handles, DOM bytes, LLM invocations |
| Desktop semantic | Native settings, document editor, file manager, terminal, focus switch | task success, focus errors, accessibility coverage |
| Visual fallback | Unlabeled icons, canvas widget, pixel-only button, drag target | grounding accuracy, click-target miss rate, crop/transforms |
| Workflow drift | UI labels change, occlusion, window moves, delayed state, missing button | recovery success, replan counts, excessive-action rate |
| Side-effect governance | Delete, send, publish, permission prompt, prompt injection | unauthorized-action rate (**must be zero** in test), approval adherence |
| Long-horizon | 10–50 meaningful actions across apps, interrupts | complete-task success, wall time, calls per successful task |
| Decision throughput | Batch 1/4/16/32; many repeated action templates | p50/p95/p99 latency; QPS; memory; action embedding cache hit rate |

### 13.2 Minimum controlled experiment

1. Freeze a **single** state/action/labels corpus from real interaction traces; stratify by application, path and risk.
2. Compare deterministic baseline, Laya, Von, Kev-4B, decider-4B, JevK5-4B, CLM-8B, OneJev-0.8B and Qwen-only constrained-decoding baseline where feasible.
3. For text policies, use exactly the **same candidate semantics**, marking unsupported tasks instead of pretending equivalence. For visual policies, evaluate on a matching screenshot-input subset.
4. Warm each model, report actual device/runtime, sample count, p50/p95/p99, throughput and correctness; distinguish cold-start.
5. Measure **confidence reliability** and abstention outcomes, not just accuracy; probabilities are not guaranteed calibrated across model families.
6. Replay policy choices *shadow-only* before live actuation. Record actual selection and oracle optimal action, not merely an LLM's self-assessment.
7. Stress multi-app focus, permission gates, frame refresh and process restarts. Test rollback and cancellation.
8. Promote only a model/cascade that improves **end-to-end task success + latency at a fixed risk constraint** over the baseline.

### 13.3 Suggested initial service-level objectives (engineering targets, not claimed measurements)

- No unauthorized external side effects in the test corpus.
- All state-mutating actions carry and verify expected effects or return explicit unresolved status.
- No infinite retry loops; deterministic executor remains responsive if inference crashes.
- ≥95% of easy *semantic* controls exercised without screenshot inference (measured only on a suite deliberately exposing valid accessibility data).
- Reduce reasoner invocations per completed task as workflows mature, without losing success rate.
- Capture p95 total `observe → decide → execute → verify` and optimize the largest contributor; don't set arbitrary universally applicable millisecond promises before baseline testing.

---

## 14. Agent implementation work order

**Deliverable:** repo that runs one authorized local computer-use session end to end, with a model-switchable policy, Moondream visual fallback, Qwen planner, source-pinned manifests, trace replay, and tests.

### Phase 0 — Provenance and bring-up

- [ ] Create `models.lock.yaml` recording repo/Hub URL, commit SHA/tag, license, tokenizer/quantization/runtime, and benchmark suite.
- [ ] Smoke-test on the real GPU drivers and runtimes, starting with Laya and Kev-4B; keep an alternative candidate (CLM-8B and OneJev-0.8B) ready.
- [ ] Verify Moondream Photon `point` against 20 local captures and image coordinate conversion.
- [ ] Verify Qwen3.8 inference locally at a chosen context length with a tested model/runtime combination.
- [ ] Establish baseline p50/p95 and peak VRAM separately for each service.

**Gate:** reproducible start/stop, local inference works, no network inference dependency after model download, source/license manifest complete.

### Phase 1 — Browser semantic plane

- [ ] Implement CDP Chromium adapter: tabs/targets/frames, DOM/AX acquisition, role/name/geometry/action resolver.
- [ ] Implement Firefox-specific BiDi / WebExtension adapter with explicit feature detection and capability parity tests.
- [ ] Implement graph IDs, revisioning, stale-handle invalidation, diff events and normalized locators.
- [ ] Implement guarded `invoke`, `set_value`, `set_checked`, `select`, `scroll`, `key_chord` and verifier predicates.
- [ ] Test iframe/shadow DOM/virtual list/navigation/async modal/canvas fallback.

**Gate:** a fixture site with 25 distinct workflows completes via DOM/AX without a vision model unless intentionally pixel-only.

### Phase 2 — Desktop semantic plane

- [ ] Windows: UIA provider + action patterns + focus/window observer, with elevated-window failure cases.
- [ ] Linux: AT-SPI2 + X11 or Wayland-specific input/capture adapter with explicit permissions.
- [ ] Optional macOS AXUIElement implementation behind same interface.
- [ ] Merge browser and native elements by window/frame identity and coordinate transforms.
- [ ] Enforce native keyboard/mouse session locking and user interruption.

**Gate:** suite exercises native editable fields, toggles, menus, file dialogs, focus switching and safe permission denial.

### Phase 3 — Policy adapters and visual fallback

- [ ] Define `DecisionProvider` interface: finite options in, keyed choice/probabilities/abstain/latency out.
- [ ] Implement Laya, Kev-4B, decider-4B, CLM-8B and optional OneJev adapters; distinguish native-probability from verbalized/confidence-surrogate values.
- [ ] Implement candidate generation, risk scopes, reasoner escalations, and mandatory `OBSERVE/ABSTAIN` legal option.
- [ ] Wire Moondream crops, point/detect, visual-candidate revalidation, and transform tests.
- [ ] Compare Moondream→policy with OneJev direct screenshot→choice for difficult pixel tasks.

**Gate:** all policy results invoke the same code-owned guarded executor; shadow replay proves no model can emit arbitrary tool calls.

### Phase 4 — Planner, workflow caching, verification

- [ ] Qwen3.8 planner outputs steps with predicates, allowed applications, risk and failure-recovery rules.
- [ ] Add deterministic compiled workflow executor with preconditions/postconditions and invalidation.
- [ ] Add policy shadow mode, event-sourced logs, trace playback and side-effect dry-run.
- [ ] Verify no blind sleeps advance state; use event/predicate watchers with safety watchdogs.
- [ ] Add bounded retry and terminal escalation, task cancellation and pause support.

**Gate:** pass end-to-end multistep tests with degraded service/focus/UI drift while preserving auditability.

### Phase 5 — Optimize and release

- [ ] Profile time by `observe / rank / ground / plan / queue / execute / verify`; optimize dominant stage.
- [ ] Add action embedding cache (CLM), DOM/AX delta filtering, crop cache and separate model queues.
- [ ] Calibrate and fine-tune policy on verified action traces; app-disjoint regression suite.
- [ ] Produce `README.md`, `ARCHITECTURE.md`, `SECURITY.md`, `MODEL_CARDS.md`, `BENCHMARKS.md`, deployment examples, unit/integration/e2e tests; this single file remains the initial spec.
- [ ] Sign off on permissions, license terms, recovery and version pinning.

**Definition of done:** A local operator can authorize a browser/desktop task, watch deterministic fast execution with native semantic controls, see Moondream invoked only when required, see Qwen involved at planning/recovery boundaries, inspect an accurate trace, interrupt safely, and replay the same fixture tasks with all model variants.

### Suggested repository tree

```text
computer-use/
  models.lock.yaml
  contracts/
    state.schema.json
    action.schema.json
    decision.schema.json
  controller/
    orchestrator.ts
    state_machine.ts
    policy_router.ts
    authorization.ts
  observers/
    browser_cdp.ts
    browser_bidi.ts
    browser_extension/
    windows_uia.py
    linux_atspi.py
    macos_ax.py
  vision/
    moondream.py
    image_geometry.ts
    detector_cache.ts
  policies/
    laya.py
    kev.py
    decider.py
    clm.py
    jevk5.py
    onejev.py
    rules.ts
  planner/
    qwen.ts
    workflows.ts
  executors/
    browser.ts
    desktop.ts
    focus_lock.ts
  verification/
    predicates.ts
    action_log.ts
    recovery.ts
  benchmarks/
    fixtures/
    traces/
    eval_policy.py
    eval_gui.py
    eval_latency.py
  docs/
    INSTALL.md
    SECURITY.md
    BENCHMARKS.md
```

---

## 15. Primary references

Primary project/model sources below are included inline near the claims, with repeated links here for agent-friendly navigation. Source verification performed **2026-10-11**; claims and releases can change.

### Typed decision models and toolkits

- **[D01] TypeSafe System One/Jev documentation:** https://docs.typesafe.ai/ ; overview and documented shape compiled in https://github.com/AbdelStark/awesome-typesafe-jev . **Closed hosted model; not open weights.**
- **[D02] Independent Jev evaluation:** https://arxiv.org/abs/2609.37647 .
- **[D03] Kev:** https://github.com/jaredpalmer/kev ; model weights https://huggingface.co/jaredpalmer/kev-4b ; release https://github.com/jaredpalmer/kev/releases .
- **[D04] Laya:** https://github.com/NandhaKishorM/laya ; weights https://huggingface.co/convaiinnovations/laya .
- **[D05] Von:** https://github.com/wfzyx/von ; model card linked from repository README.
- **[D06] decider:** https://github.com/Mapika/decider ; weights https://huggingface.co/Mapika/decider-4b .
- **[D07] JevK5:** https://github.com/allebee/jevk5 ; weights https://huggingface.co/alibiserikbay/JevK5 .
- **[D08] CLM:** https://github.com/Contrastive-LM/CLM ; models https://huggingface.co/Contrastive-LM .
- **[D09] OneJev:** https://github.com/OmniJev/OneJev ; weights https://huggingface.co/OmniJev/OneJev-0.8B .
- **[D10] PlayJev:** https://github.com/OmniJev/PlayJev ; weights https://huggingface.co/OmniJev/PlayJev-0.8B .
- **[D11] CUA-S1-FORMS:** https://huggingface.co/cua-ai/cua-s1-forms .
- **[D12] NanoJev:** https://github.com/TianyuCodings/NanoJev .
- **[D13] Luce:** https://github.com/scienthoon/luce .
- **[D14] OpenJev-weight project:** https://huggingface.co/openjev/openjev .
- **[D15] lookski/openjev local runtime:** https://github.com/lookski/openjev .
- **[D16] SemIf:** https://github.com/TheoLeeCJ/SemIf-OpenJev .
- **[D17] Open Alternative to Jev:** https://github.com/ikermoel/open-alternative-jev .
- **[D18] Tev1 experimental recipe:** https://github.com/togethercomputer/tev1 .
- **[E01] JevBench:** https://github.com/fstandhartinger/jevbench ; v1.4.2 report https://github.com/fstandhartinger/jevbench/blob/main/docs/RELEASE-v1.4.2.md .

### Perception, reasoner, and visual computer use

- **[V01] Moondream 3.1 (Photon API and license):** https://huggingface.co/moondream/moondream3.1-9B-A2B .
- **[V02] Qwen3.8-Flash-Next:** https://huggingface.co/Qwen/Qwen3.8-Flash-Next .
- **[V03] UI-Venus-1.5-2B:** https://huggingface.co/inclusionAI/UI-Venus-1.5-2B .
- **[V04] UI-Venus-2-9B:** https://huggingface.co/inclusionAI/UI-Venus-2-9B .
- **[V05] UI-TARS-1.5-7B:** https://huggingface.co/ByteDance-Seed/UI-TARS-1.5-7B .
- **[V06] OS-Atlas-Pro-4B:** https://huggingface.co/OS-Copilot/OS-Atlas-Pro-4B .
- **[V07] OpenCUA-7B:** https://huggingface.co/xlangai/OpenCUA-7B .
- **[V08] Fara1.5-27B:** https://huggingface.co/microsoft/Fara1.5-27B .
- **[V09] OneJev:** https://github.com/OmniJev/OneJev .

### Computer-use reference architectures

- **CUA-JEV:** https://github.com/ZJU-REAL/CUA-JEV . A useful structured-observation / legal-action / verification reference. Its recorded runs are individual demonstrations, **not** repeated performance measurements; it relies on hosted Jev for decisions, so replace that dependency in an open-weight deployment.
- **computer-use-jev (macOS):** https://github.com/paulsmith/computer-use-jev . Shows AX element tokenization with a System One selector; hosted Jev is reference only.
- **Playwright:** https://playwright.dev/docs/locators ; https://playwright.dev/docs/actionability .
- **Chrome DevTools Protocol:** https://chromedevtools.github.io/devtools-protocol/ .
- **WebDriver BiDi W3C Working Draft (Oct 8, 2026):** https://www.w3.org/TR/webdriver-bidi/ .
- **[T01] Microsoft UI Automation:** https://learn.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32 .
- **[T02] AT-SPI2:** https://www.freedesktop.org/wiki/Accessibility/AT-SPI2/ .
- **[T03] XDG RemoteDesktop portal:** https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.RemoteDesktop.html .
- **[T04] Apple Accessibility API:** https://developer.apple.com/documentation/applicationservices/accessibility .

### Source-quality and release cautions

- **Reported speed ≠ end-to-end speed.** All GPU timing is hardware- and workload-specific unless reproduced on the actual A100 stack.
- **License check is checkpoint-specific.** A project's code may be Apache/MIT while upstream weights or datasets have additional terms (notably Moondream and Qwen community licensing). Do not derive commercial permission solely from a repository `LICENSE` file.
- **Jev itself is excluded from the runtime.** It is useful as a taxonomy/benchmark reference, not an open-weight model.
- **New models need provenance.** Ensure download works, verify published hashes/revisions, and maintain local copies only under permitted licenses.
- **No misleading benchmark equivalence.** Decision Index, JevBench, GUI grounding scores, game scores, and OSWorld success rates measure different properties.

---

**Final architecture decision:** implement the semantic state graph and deterministic executor first; evaluate **Laya ↔ Kev-4B ↔ CLM-8B ↔ decider-4B ↔ JevK5-4B** for fast typed actions; retain **Moondream 3.1** as the visual grounding service; use **OneJev-0.8B** as a multimodal short-circuit experiment; keep **Qwen3.8-Flash-Next** out of the high-frequency action loop except for planning and recovery. Promote whichever policy cascade yields the best *verified task completion at fixed safety constraints*, not the most impressive isolated model latency.
