# OMNI Conductor

This package is the local, model-agnostic control plane for the OMNI voice UI. It
does not start or download models. All defaults bind or connect to loopback only.

## Fixed local topology

| Service | Port | Contract |
| --- | ---: | --- |
| Conductor | 8100 | FastAPI, event replay, SSE/WebSocket |
| Streaming ASR | 8101 | WebSocket; JSON control + binary PCM frames |
| Optional router | 8102 | OpenAI-compatible `/v1/chat/completions` |
| Qwen | 8103 | Owned by Hermes; OpenAI-compatible serving |
| Hermes Agent | 8642 | `/v1/runs`, run-event SSE, `/stop` |
| Model manager | 8110 | On-demand leases |
| Media worker | 8111 | Jobs, job-event SSE, cancellation |

Start only the conductor (after its dependencies are deliberately configured):

```powershell
python -m orchestrator.main
```

The optional learned router is off by default. Set `OMNI_ROUTER_ENABLED=1` only
when a local router is already serving on port 8102. `OMNI_ALLOW_REMOTE_ENDPOINTS`
must be explicitly enabled before any non-loopback dependency URL is accepted.
The conductor itself binds to loopback by default. A container bind such as
`0.0.0.0` requires both `OMNI_ALLOW_REMOTE_ENDPOINTS=1` and a non-empty
`OMNI_CONDUCTOR_TOKEN`. When the token is configured, every HTTP request and both
WebSocket upgrades require `Authorization: Bearer <token>`; credentials in query
strings are deliberately unsupported. Run the module as `python -m
orchestrator.main` (not `python -m orchestrator.app`).

The Hermes adapter follows the official Runs API: create uses `input` and
`session_id` (plus bounded `instructions` only for structured OMNI delegation),
then consumes `/v1/runs/{id}/events`. Set `HERMES_API_KEY`, `API_SERVER_KEY`, or
`OMNI_HERMES_API_KEY` to send Bearer authentication. Tool events and
`message.delta` are progress; only `run.completed`, `run.failed`, and
`run.cancelled` terminate the operation.

When Hermes emits `approval.request`, that session enters hold. A final spoken
unambiguous German/English yes submits `{"choice":"once"}`; an unambiguous no
submits `{"choice":"deny"}` to `/v1/runs/{id}/approval`. Voice can never grant
`session` or `always`. Any other response asks for an explicit yes/no and causes
no action. Approval responses are idempotent and appear as `approval.responded`.

## Event protocol

Every event has `protocol_version`, `type`, `event_id`, monotonic `seq`, UTC
`timestamp`, `session_id`, `turn_id`, `operation_id`, `trace_id`, optional
`causation_id`, `source`, and a JSON `payload`. `GET /v1/events?after_seq=N`
replays persisted JSONL events. `/v1/events/stream` and `/v1/events/ws` replay the
same range before continuing live.

Only `input.transcript.final` is routed to a side-effecting service. Partial
transcripts are persisted for UI feedback only. Reusing an idempotency key with
the same request returns the original operation; reusing it with different input
returns HTTP 409. Each operation can append exactly one terminal event.

Media completion payloads contain `artifact_id`, an absolute URL/path and
`mime_type`. Embedded Base64/data URLs are rejected.

## Browser audio bridge

Connect to `ws://127.0.0.1:8100/v1/asr/ws`. The first frame is JSON:

```json
{"type":"input.start","protocol_version":"1.0","session_id":"s1","turn_id":"t1","trace_id":"trace1","sample_rate_hz":16000,"channels":1,"encoding":"pcm_s16le"}
```

Send subsequent audio as raw binary PCM frames. Send `{"type":"input.commit"}`
to finalize or `{"type":"input.cancel"}` to discard. The bridge returns JSON
`input.transcript.partial`, `input.transcript.final`, or `error` frames and feeds
the same transcript into the versioned conductor event stream.

Text takes the identical final-transcript route through `POST /v1/input/text`:

```json
{"session_id":"s1","turn_id":"t2","text":"Öffne den Browser","needs_vision":false}
```

`POST /v1/sessions/{session_id}/agent-focus` with
`{"turn_id":"control-1","armed":true,"cancel_if_active":true,"needs_vision":false}` cancels
active work; if none is active it arms a one-shot forced Hermes route for the next
final voice or text turn. `POST /v1/sessions/{session_id}/operations/active/cancel`
always cancels active work. `POST /v1/sessions/{session_id}/end` with a `turn_id`
cancels work, clears the one-shot route and emits `session.ended`.

Set `armed:false` to clear an unused one-shot route. `target_node` may be a paired
node ID or its exact display name. `required_capabilities` and
`requested_permissions` are sets; the provider pool will only select a node that
advertises and was explicitly granted every requested item.

## Paired remote Qwen / Hermes nodes

Remote providers are genuinely off unless `OMNI_REMOTE_NODES_ENABLED=1`.
While it is `0`, persisted trusted nodes are not loaded into provider selection
and pairing is rejected. mDNS additionally requires
`OMNI_MDNS_DISCOVERY_ENABLED=1`; enabling discovery without remote nodes is a
configuration error.

There is no address-range scan and no automatic trust. `GET /v1/nodes/discovered`
performs only a bounded `_omni-node._tcp.local.` mDNS lookup (and returns an empty
list if the optional `zeroconf` library is absent). Discovery never pairs.

Pair a selected HTTPS node explicitly with `POST /v1/nodes/pair`:

```json
{
  "endpoint": "https://studio-pc.local:9443",
  "one_time_code": "123456",
  "expected_fingerprint_sha256": "<64 hex characters>",
  "ca_cert_path": "C:/.../ca.pem",
  "client_cert_path": "C:/.../client.pem",
  "client_key_path": "C:/.../client.key",
  "requested_permissions": ["desktop.control", "files.read"]
}
```

The one-time code is sent with `Cache-Control: no-store` and is never written to
the trust store or event log. The response fingerprint and HTTPS origin must
match. Every later health, run, event and cancellation call uses the configured
mTLS identity. `GET /v1/nodes` reports the allowlisted capabilities, granted
permissions, health and measured latency. `DELETE /v1/nodes/{node_id}` removes
trust. `POST /v1/nodes/kill-switch` with `{"engaged":true}` cancels known remote
runs, removes remote providers from selection and persists the switch; send false
to deliberately enable paired nodes again.

Selection is: exact requested node (when present), otherwise existing session
affinity, then lowest measured latency among healthy eligible providers. A failed
run submission is retried against the next provider with the same idempotency key.
Voice ASR and video/image media stay local; only the Hermes job prompt, explicitly
attached context/artifact references and its event stream cross the paired link.

The direct local Hermes provider advertises only `hermes.run`, code, structured
tools and internet search, with `tool.execute`/`network.search` permissions.
Desktop, files, webcam and visual context always require a run-bound ticket from
a paired mTLS sidecar provider. The main computer follows the same path; there is
no privileged local fallback.

Paired-node parity is carried inside `/v1/runs`, not exposed as an unauthenticated
desktop-control API. The formal action types are `desktop.screenshot`,
`desktop.control`, `files.read`, `files.write`, `tool.execute`, `webcam.start`,
`webcam.frame`, and `webcam.stop`. Each action declares its matching permission.
Node SSE can return `node.activity`, `desktop.*`, `files.*`, `webcam.*`, and
`permission.*`; the conductor copies these into the versioned UI event stream.
Screenshots/webcam/media are artifact references, never Base64 blobs.

## Unified Nelloz agent layer

The conductor includes a small model-agnostic agent core in
`orchestrator/agent_system.py`. Nelloz owns the persistent session and task
state; models are providers selected by logical roles: `voice`, `fast`, `agent`,
`vision`, `max`, `image`, and `video`.

The layer provides a central `ModelRegistry`, versioned `TaskState` with stale
action rejection, an in-process `EventBus`, deterministic intent detection, a
bounded `ContextBuilder`, and a permission-aware `ToolBroker`. Existing Hermes,
Media, ASR, and EventStore adapters remain the downstream transport layer.

Role bindings can be changed without modifying the agent core:

```text
NELLOZ_MODEL_AGENT=Qwen3.8-27B
NELLOZ_PROVIDER_AGENT=qwen
NELLOZ_LOCAL_AGENT=1
NELLOZ_MODEL_MAX=frontier
NELLOZ_PROVIDER_MAX=cloud-frontier
```

`GET /v1/agent/models` reports the active role bindings. Tool and operation
payloads can carry `task_id` and `task_version`; actions from an older task
version are rejected before downstream execution.