# Voice, Audio, Vision, And Voicechat

## Endpoint Summary

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/v1/voice/state` | Voice runtime status |
| `GET` | `/v1/voice/models` | List TTS voice models |
| `POST` | `/v1/voice/models/switch` | Switch active TTS model |
| `POST` | `/v1/voice/models/{modelId}/pull` | Install runtime prerequisites and pull one managed TTS model |
| `POST` | `/v1/voice/models/{modelId}/deploy` | Pull and deploy one managed CUDA TTS model |
| `GET`/`POST` | `/v1/voice/supertonic-settings` | Read or update voice tuning settings |
| `GET` | `/v1/asr/engines` | List ASR systems, models, capabilities, and readiness |
| `GET` | `/v1/asr/status` · `/v1/asr/selection` | Read selection and runtime status |
| `GET` | `/v1/asr/downloads` | List persistent ASR download and deployment transitions |
| `POST` | `/v1/asr/downloads` | Start or resume an ASR weight download |
| `GET` | `/v1/asr/downloads/{engineId}/{modelId}` | Poll one ASR download and deployment transition |
| `POST` | `/v1/asr/downloads/{engineId}/{modelId}` | Start or resume one model download |
| `POST` | `/v1/asr/downloads/{engineId}/{modelId}/retry` | Retry or resume a failed or interrupted download |
| `PATCH` | `/v1/asr/selection` | Persist and activate an exact ASR engine/model |
| `POST` | `/v1/asr/activate` | Activate and persist an exact ASR engine/model |
| `POST` | `/v1/asr/engines/{engineId}/setup` | Install a managed ASR runtime and pinned weights |
| `POST` | `/v1/asr/engines/{engineId}/models/{modelId}/pull` | Pull one exact model and validate its managed runtime |
| `POST` | `/v1/asr/engines/{engineId}/models/{modelId}/deploy` | Pull, select, and activate one exact model |
| `POST` | `/v1/asr/transcriptions` · `/v1/asr/test` | Transcribe/test with the selected backend |
| `GET`/`POST` | `/v1/voice/asr-models` · `/v1/voice/asr-models/switch` | Compatibility registry/activation aliases |
| `POST` | `/v1/voice/tts` | Synthesize text |
| `POST` | `/v1/audio/speech` | OpenAI-compatible TTS alias |
| `GET` | `/v1/audio/classify/health` | Jetson CUDA/TensorRT YAMNet readiness |
| `POST` | `/v1/audio/classify/setup` | Provision and warm pinned JetPack TensorRT YAMNet |
| `POST` | `/v1/audio/classify` | Classify caller-supplied WAV via direct-tool compatible contract |
| `GET` | `/v1/audio/embed/health` | Role-typed embedding readiness (`?kind=acoustic|speaker|semantic`) |
| `POST` | `/v1/audio/embed/setup` | Provision/warm one role runtime (admin; `?kind=...`) |
| `POST` | `/v1/audio/embed` | Managed role-typed embedding via `audio_analyze` (`?kind=...`) |
| `GET` | `/v1/audio/diarization/live/readiness` | Non-mutating Sortformer worker readiness |
| `POST` | `/v1/audio/diarization/live/setup` | Verify and warm a local Sortformer runtime (admin) |
| `POST` | `/v1/audio/diarization/live` | Managed live/session-local speaker-turn diarization |
| `POST` | `/v1/audio/diarization/live/cancel` | Cancel live worker work and clear its queue |
| `GET` | `/v1/audio/diarization/reconcile/readiness` | Non-mutating offline Community-1 worker readiness |
| `POST` | `/v1/audio/diarization/reconcile/setup` | Verify and warm a local Community-1 runtime (admin) |
| `POST` | `/v1/audio/diarization/reconcile` | Managed offline/dream reconciliation proposals |
| `POST` | `/v1/audio/diarization/reconcile/cancel` | Cancel reconciliation work and clear its queue |
| `POST` | `/v1/voice/transcribe` | Transcribe audio |
| `POST` | `/v1/audio/transcriptions` | OpenAI-compatible transcription alias |
| `POST` | `/v1/voice/transcribe/stream` | Isolated final transcription over SSE |
| `GET` | `/v1/voice/clone-refs` | List clone references |
| `POST` | `/v1/voice/clone-refs/upload` | Upload clone reference |
| `POST` | `/v1/voice/clone-refs/from-url` | Fetch clone reference server-side |
| `POST` | `/v1/voice/clone-refs/{filename}/activate` | Activate clone reference |
| `POST` | `/v1/voice/clone-refs/{filename}/rename` | Rename clone reference |
| `DELETE` | `/v1/voice/clone-refs/{filename}` | Delete clone reference |
| `POST` | `/v1/voice/speak` | Synthesize and broadcast to voicechat clients |
| `WS` | `/v1/voicechat/ws` | Full-duplex voicechat WebSocket |
| `POST` | `/v1/vision/describe` | Vision describe placeholder/deferred endpoint |
| `GET` | `/v1/vision/embed/readiness` | Non-mutating isolated OpenCLIP readiness |
| `POST` | `/v1/vision/embed/setup` | Explicit OpenCLIP setup (admin scope) |
| `POST` | `/v1/vision/embed` | Offline-only verified OpenCLIP image embedding |
| `GET` | `/v1/ocr/readiness` | Probe exact OCR package/engine/backend readiness without mutation |
| `POST` | `/v1/ocr/setup` | Create and verify the isolated OCR venv (admin scope) |
| `POST` | `/v1/ocr/advanced` | Agent-equivalent managed advanced OCR (alias of `/v1/tools/ocr_image_advanced/call`) |
| `GET` | `/v1/media/av/status` | Inspect configured Omni adapter availability without loading weights |
| `GET` | `/v1/media/omni/setup` | Inspect pinned Omni setup profiles, prerequisites and active job |
| `POST` | `/v1/media/omni/setup/jobs` | Start an explicit profile/accelerator setup job (admin) |
| `GET` | `/v1/media/omni/setup/jobs/{id}` | Poll durable setup progress and verified receipts |
| `POST` | `/v1/media/omni/setup/jobs/{id}/cancel` | Request cancellation; poll until owned work settles (admin) |
| `POST` | `/v1/media/omni/setup/attach` | Save an explicit existing-adapter selection without restarting it (admin) |
| `POST` | `/v1/media/av/analyze` | Analyze a supplied media file with separate transcript/audio/visual evidence and current-input provenance |

The [Omni comprehension setup guide](../../guides/omni-stack-setup.md) covers the
shared model modules, pinned core-stack deployment and existing-adapter attach
flow. Setup reads are passive; installation, attachment and cancellation require
`admin` scope. Adapter availability alone does not attest loaded weights or GPU
residency, and file analysis does not open a live microphone or camera.

## Jetson Microphone Routing

The live `/listen` microphone path does not trust PulseAudio's mutable default
source. When the documented Seeed ReSpeaker USB 4-Mic Array source
`alsa_input.usb-SEEED_ReSpeaker_4_Mic_Array__UAC1.0_-00.multichannel-input`
is present as a six-channel Pulse source, Omnius selects it ahead of a running
`nx_remapped_out` desktop/TTS remap and FFmpeg extracts DSP/beamformed channel
0 to mono PCM16/16 kHz. `GET /v1/voice/state` reports the actual source,
source kind, channel count, and selected channel.

`OMNIUS_MIC_DEVICE` remains an explicit operator override (`pulse:<source>` or
`hw:N,M`). A saved live-input selection of `pulse:nx_remapped_out` is upgraded
only when that exact six-channel ReSpeaker source is concurrently available;
other explicit selections are left unchanged.

This affects only Omnius-owned live microphone capture. Egg owns its physical
capture and sends conditioned mono WAV files to audio-comprehension endpoints;
those file-analysis endpoints do not open a microphone or reselect a Pulse
source.

## Vision Embeddings

`POST /v1/vision/embed` uses an isolated, CUDA-enabled OpenCLIP
`ViT-B-32` / `laion2b_s34b_b79k` backend and returns a 512-dimensional,
L2-normalized image vector plus the verified artifact digest. It does not make
descriptive claims or invoke an LLM.

First poll `GET /v1/vision/embed/readiness`. It is non-mutating and returns
HTTP 503 until the isolated runtime and checksum-manifested weights are ready.
Use the admin-scoped `POST /v1/vision/embed/setup` to create the
private runtime under `~/.omnius/runtimes/vision/open-clip` with
`include-system-site-packages = false`, link only the already-validated vendor
Torch provider, install the pinned non-Torch closure (including `wcwidth`), and
fetch/verify the checkpoint from immutable revision
`1a25a446712ba5ee05982a381eed697ef9b435cf`. Setup is single-flight and
daemon-bootstrapped by default on JetPack; set `OMNIUS_VISION_AUTO_SETUP=0`
to disable it. Inference never installs packages or downloads artifacts, and
its Python worker is network-denied.

JetPack Torch is inherited and never replaced. `torchvision` must either be
present in the compatible vendor stack or be supplied explicitly with the local
`OMNIUS_VISION_TORCHVISION_WHEEL` and immutable
`OMNIUS_VISION_TORCHVISION_WHEEL_SHA256`; generic PyPI Torch/torchvision is
blocked. Bootstrap discovery may inspect the selected vendor interpreter, but
the managed runtime installs its own pinned support closure and proves Torch,
torchvision, and OpenCLIP with user-site packages disabled. OpenCLIP reports
its own `component: openclip` memory-admission result, governed by
`OMNIUS_VISION_MIN_AVAILABLE_MB`, rather than reusing CLAP telemetry. On
memory-constrained Jetson systems model loading and embedding fail with a typed 503 rather than
evicting a resident ASR/Ollama model. Dependency and
weight provisioning itself is allowed to finish under transient pressure;
readiness separately reports `installed`, `weightsReady`,
`inferenceAvailable`, and the current admission result. The model is loaded
only after at least 8 GiB is available.

## Advanced OCR

First poll `GET /v1/ocr/readiness?language=eng`. On Jetson, a privileged or
interactive npm installation/bootstrap provisions `tesseract-ocr`, `python3-venv`,
an import-verified system OpenCV provider (including NVIDIA
`libopencv-python`), `python3-numpy`, `python3-pil`, and `python3-reportlab`.
Daemon startup then creates (or repairs)
`~/.omnius/runtimes/vision/ocr-advanced` with `--system-site-packages` and
installs only `pytesseract==0.3.13 --no-deps`. `POST /v1/ocr/setup` is the
admin-scoped observable retry/repair path; set `OMNIUS_OCR_AUTO_SETUP=0` to
disable startup setup. Set `OMNIUS_OCR_PYTHON` to use an operator-managed,
already-ready interpreter instead.

`POST /v1/ocr/advanced` is a compatibility alias for the exact
agent-facing `ocr_image_advanced` tool. Use the same direct-tool envelope;
the response retains display text in `result.output` and returns the parsed
pipeline payload in `result.data`.

```bash
curl -sS -X POST http://127.0.0.1:11435/v1/ocr/advanced \
  -H 'content-type: application/json' \
  -d '{"args":{"image":"/data/invoice.png","language":"eng","psm":6},"timeout_ms":90000}'
```

Inference never invokes sudo, apt, pip, or venv creation. If readiness is
false, the tool returns `result.data.code=ocr_runtime_not_ready` with setup and
readiness URLs. Jetson never installs generic CUDA, Torch, OpenCV, or NumPy
packages. A valid `OMNIUS_OCR_PYTHON` is accepted from its isolated import and
version results without requiring particular dpkg package names. The caller
needs `run` scope because optional
`output_dir`, batch, and debug modes write OCR artifacts.

The managed pipeline is bounded: it starts with high-yield preprocessing and
expands variants only for low-evidence or larger images. Small crops avoid the
former all-variant/all-PSM explosion and use a six-second cap per Tesseract
attempt. If the first small-crop pass proves high-volume very-low-confidence or
distinctly symbol-heavy garbage, recovery stops immediately; blanks and
plausibly recoverable text still receive the second high-yield attempt. The
REST default is 90 seconds (maximum 180 seconds), while the worker has an
80-second internal deadline so it can return diagnostics. A timeout or cancellation returns
`result.data.schema=omnius.ocr-diagnostic.v1` with code `ocr_timeout` or
`ocr_cancelled`; cancellation terminates the Python/Tesseract process group
with TERM followed by KILL.

OCR text is evidence-gated before it is returned. Strong text with adequate
confidence is accepted. An empty or tiny non-substantive result is a successful
`omnius.ocr-evidence.v1` `low_information` observation with diagnostic code
`ocr_low_information`, not a fabricated transcript. High-volume low-confidence
or symbol-heavy output is suppressed and returned as
`ocr_evidence_rejected`; its raw text is not exposed as alternate OCR evidence.

## TTS

`POST /v1/voice/tts` returns audio bytes. `format` can be `wav` or `pcm`. `X-Sample-Rate` reports the sample rate.

`POST /v1/voice/models/{modelId}/pull` installs runtime prerequisites and
weights without starting inference. `POST /v1/voice/models/{modelId}/deploy`
also verifies the selected CUDA device and starts the persistent managed
runtime. A ready model returns `200`. A model without the matching managed
adapter returns `404`. Installation, download, CUDA preflight, and startup
failures return `500`.

LuxTTS uses a two-part health contract on NVIDIA Jetson ARM64. The dependency
probe hides CUDA, verifies all import-time modules, requires
`onnxruntime==1.23.2` with `CPUExecutionProvider`, and reports missing `k2` as
an optional performance degradation. LuxTTS imports its ONNX CPU path even
when later synthesis uses PyTorch CUDA. The separate operational probe then
requires a CUDA-capable Torch build matched to the host L4T, JetPack, CUDA,
and Python ABI. It never falls back to generic PyPI Torch on AGX Orin.

Repair preserves the existing venv, model cache, and clone references. A
health-probe timeout does not start package repair. ARM dependencies use
prebuilt wheels where available. Host build prerequisites are not installed
from daemon or media-tool calls. An administrator can set
`OMNIUS_VOICE_AUTO_SYSTEM_DEPS=1` for a deliberate setup run. Set
`JETSON_TORCH_WHEEL` only to an exact wheel that matches the current Jetson
runtime when automatic provider discovery cannot resolve one.

### AGX Orin LuxTTS smoke procedure

Run this procedure on the target AGX Orin after the administrator provisions
the exact JetPack-compatible Torch wheel. Do not use it to select a GPU for an
Ollama inference test. The separate live-inference hardware gate still applies
to model requests.

1. Record `uname -m`, `/etc/nv_tegra_release`, the Python version, and the
   configured `JETSON_TORCH_WHEEL` source.
2. Run the LuxTTS dependency probe with CUDA hidden. Confirm that it imports
   `zipvoice.luxvoice`, reports `onnxruntime==1.23.2`, lists
   `CPUExecutionProvider`, and treats a missing `k2` module as degraded only.
3. Run the separate Torch operational probe. Confirm that `torch.cuda` is
   available and that CUDA device `0` is the integrated AGX Orin GPU. Stop if
   the driver, CUDA runtime, Torch build, or Python ABI does not match.
4. Start Omnius through the normal operator path. Request one short WAV from
   `POST /v1/voice/tts` with the exact LuxTTS model and clone reference.
5. Verify HTTP success, a non-empty WAV header, `X-Voice-Backend: luxtts`, the
   exact `X-Voice-Model`, and the expected sample rate. Play or inspect the
   artifact before accepting the smoke result.
6. Repeat the readiness check after restart. Confirm that the existing venv,
   model cache, and clone references were reused and that no automatic sudo or
   generic PyPI Torch installation occurred.

Record each observation as pass, fail, or not run. A mocked installer test does
not replace this device smoke procedure.

OpenAI-compatible alias:

```text
POST /v1/audio/speech
```

## ASR

Managed ASR downloads use persistent, pollable transitions. Start a collection
download with `POST /v1/asr/downloads` and body
`{"engineId":"...","modelId":"...","device":"..."}`. The device is
optional. The response is `200` when weights are ready or `202` with
`statusUrl`, `retryUrl`, and `pollAfterMs` while work is pending. Duplicate
requests coalesce and interrupted snapshots can resume.

Poll `GET /v1/asr/downloads/{engineId}/{modelId}`. A missing job returns `404`.
The POST form starts or resumes that exact model. The `/retry` POST is
idempotent for active work. Both return `200` when ready or `202` when accepted.
Use `/v1/asr/engines/{engineId}/models/{modelId}/pull` to download and validate
the managed runtime and exact CUDA placement. Use the matching `/deploy` route
to persist the selection after readiness. Pull and deploy return a pollable
`202` transition when work remains and `500` on runtime, download, deployment,
or CUDA validation failure.

`POST /v1/asr/transcriptions` transcribes uploaded audio with the selected
engine. `/v1/asr/test` runs the same real path for readiness checks. The
OpenAI-compatible alias is:

```text
POST /v1/audio/transcriptions
```

VibeVoice is exposed as `vibevoice-transformers/vibevoice-asr-7b`. It is a
completed-file backend, not an incremental PCM stream: the response preserves
speaker IDs, segment timestamps, raw structured text, and warnings. Pass
`?context=` for customized hotwords/background context. Setup installs the
exact pinned `microsoft/VibeVoice-ASR` snapshot into the unified ASR cache;
activation requires an explicit GPU and verified placement, and never falls
back to CPU or another device. Discrete Linux uses `nvidia-smi`; Jetson/L4T,
where NVIDIA does not ship `nvidia-smi`, uses `tegrastats` plus CUDA Torch
device properties.

## Jetson Audio Classification

`audio_analyze` classification on Jetson is a persistent, CUDA-enabled FP16
TensorRT YAMNet worker, not a per-call TensorFlow/TF-Hub process. Normal
inference never installs packages or downloads weights. Operators may invoke
`POST /v1/audio/classify/setup` with admin scope, or explicitly opt into managed
startup provisioning with `OMNIUS_AUDIO_AUTO_SETUP=1`. Setup
downloads the pinned ONNX model and class map once, verifies their SHA-256
digests, builds an engine for the local JetPack/TensorRT stack, checks the
approved integrated CUDA device (`OMNIUS_AUDIO_CUDA_VISIBLE_DEVICES=0`), and
warms the worker. `OMNIUS_AUDIO_PYTHON` explicitly selects the operator's
JetPack-compatible Torch 2.2/CUDA 12.2 interpreter. When unset, Omnius probes
conventional project venvs (including `~/Documents/<project>/.venv`) and reuses
the first interpreter that proves CUDA Torch placement; it never modifies that
source environment. Omnius pins `cuda-python==12.2.0 --no-deps`
only inside its isolated audio venv, verifies Orin compute capability 8.7, and
handles L4T R36.3 `tegrastats` builds which do not support `--count`.

Poll `GET /v1/audio/classify/health` before sending classification work. A
general `/health` success is not sufficient. The response reports readiness,
backend (`tensorrt-fp16`), device, model digest, warmup timing, queue depth, and
errors. If the worker exits, readiness becomes false and the next health poll
schedules its local restart without restarting Egg or Omnius.

The compatible request shape is:

```json
{
  "args": {
    "action": "classify",
    "file": "data/cognitive-memory/media/audio/2026/08/13/<utterance-id>.wav",
    "top_k": 5
  },
  "timeout_ms": 90000
}
```

Send it to either `POST /v1/tools/audio_analyze/call` or the
`POST /v1/audio/classify` alias. Inputs must be caller-supplied mono PCM16,
16 kHz WAV files; classification never captures from `default`, normalizes the
Egg signal again, installs packages, or fetches weights. The response is
machine-readable at `result.data.classifications` with model/backend/digest,
audio facts, a `low_information` quiet-input signal, and
queue/decode/inference/total timings. One job runs and one may wait; additional
requests receive a typed `audio_classifier_busy` failure in
`result.data.code` while retaining the direct-tool HTTP 200 envelope.

## Role-Typed Audio Embeddings

`POST /v1/audio/embed` is a stable adapter to the exact agent-facing
`audio_analyze` direct tool. It requires an explicit role kind; Omnius does
not claim that an acoustic vector is a speaker-identity or semantic-audio
vector. Check the non-provisioning role control plane first:

```text
GET /v1/audio/embed/health?kind=acoustic
```

Each readiness response names the exact requested vector space and worker:

- `acoustic`: YAMNet TensorRT, 1024d,
  `yamnet-onnx/ac2ca3bd45d12ec1f19f1144205ea529b4e9dedf/output_1:mean-l2`.
- `speaker`: CPU-only WeSpeaker CAM++, 512d,
  `speaker-identity-wespeaker-voxceleb-campplus-acf623ad8ca746e50baa432255cf8fc57c669c45-fbank80-cmn-fullclip-l2`.
- `semantic`: JetPack CUDA CLAP, 512d,
  `clap-htsat-unfused/84bcbbd1d619e407a8216371ddef36e458d95d93/audio:segment-mean-l2`.

An unprovisioned or unready role returns its own typed HTTP 503 readiness body;
callers must never fall back to another role's space.

The invoke shape remains the direct-tool envelope, with `action` forced to
`embed` by the route:

```json
{
  "args": { "file": "data/cognitive-memory/media/audio/2026/08/13/<utterance-id>.wav" },
  "timeout_ms": 90000
}
```

Use the canonical query form: `POST /v1/audio/embed?kind=acoustic` (replace
`acoustic` with `speaker` or `semantic` as needed). `kind` in the JSON body or
`args` is supported as a compatibility alias. Existing clients may send
top-level `path` or `file` and Omnius normalizes it into `args`; binary
`bytesBase64` input is rejected because each managed worker is intentionally a
supplied-retained-WAV-only boundary.

`POST /v1/audio/embed/setup?kind=<role>` is the admin-only provisioning path.
It is the only endpoint that installs dependencies or retrieves pinned weights:
acoustic reuses the pinned JetPack YAMNet/TensorRT setup, speaker provisions
the isolated CPU-only WeSpeaker runtime, and semantic provisions the isolated
JetPack CUDA CLAP runtime. It activates only the selected role. Normal startup
does not provision these heavy roles. Set `OMNIUS_AUDIO_AUTO_SETUP=1` to opt
into acoustic and speaker setup. Set `OMNIUS_SEMANTIC_AUDIO_AUTO_SETUP=1` as
well to opt into CLAP setup. The explicit admin setup endpoint remains
available when automatic setup is off.

CLAP provisioning installs a checksum-locked CPython 3.10/aarch64 wheel
closure—including Pillow and the complete probed Transformers CLAP import
surface—inside a private venv with `include-system-site-packages = false`.
Only the validated vendor CUDA Torch site is linked explicitly. The immutable
model revision is provisioned without loading the model. It is
allowed to finish while unified memory is busy. Worker activation and
inference retain the 8 GiB admission gate and idle eviction, so setup cannot
silently evict ASR, Ollama, or another CUDA workload.

On JetPack, speaker setup deliberately leaves `OMNIUS_AUDIO_PYTHON` (including
an Egg project `.venv`) unchanged. It creates a private, non-system-site
`~/.omnius/runtimes/audio/speaker/venv`, then installs only a locked,
checksum-verified CPython 3.10/aarch64 CPU dependency set: NumPy 1.26.4,
ONNX Runtime 1.17.3, and the exact CPU support wheels required by that ONNX
Runtime release. Each wheel is materialized and SHA-256 verified before the
managed venv installs it with `--no-deps --no-index`.

The worker implements the WeSpeaker CAM++ 80-bin Kaldi configuration in pure
NumPy: 25 ms / 10 ms Hamming frames, dither disabled, Kaldi pre-emphasis and
mel bank behavior, then full-clip CMN without CVN. Readiness invokes a
versioned CPU preprocessing probe. It checks the fixed Kaldi configuration,
feature shape, CMN, energy range, and expected mel-band profile with explicit
cross-platform numeric tolerances before it can report ready. The speaker path neither imports
nor links Torch or Torchaudio, so the generic `torchaudio-2.2.0` ABI mismatch
cannot bind against, replace, or otherwise affect JetPack's CUDA-enabled Egg
Torch. Inference remains install-free and network-free; failed package imports
or calibration leave the role unready with the exact probe error.

The dedicated route returns typed HTTP 503 whenever its requested runtime is
unprovisioned or unavailable: `audio_classifier_not_ready` (acoustic),
`speaker_embedding_not_ready` (speaker), or
`audio_semantic_embedding_not_ready` (semantic). Callers should run the
matching admin setup route and poll its health URL. Once active, speaker
saturation and deadline errors stay role-specific
(`speaker_embedding_busy`, `speaker_embedding_timeout`). The generic
`POST /v1/tools/audio_analyze/call` interface intentionally retains its normal
HTTP 200 direct-tool envelope for these structured errors. Likewise, the
dedicated semantic adapter surfaces local CLAP admission pressure as HTTP 503
`audio_semantic_memory_pressure`, without substituting another role.

Direct-tool auth, profile, origin, output-cap, timeout, and cancellation
policy all remain in effect. Inference never creates environments, installs
packages, downloads weights, or falls back to the retired generic Python
`embed-audio.py` provider.

## Speaker Diarization And Reconciliation

Omnius provides two managed, role-specific persistent workers. They never
fall back to generic transcription, create a durable speaker identity, or
provision themselves during inference.

- `POST /v1/audio/diarization/live` uses NVIDIA
  `nvidia/diar_streaming_sortformer_4spk-v2` for retained, mono PCM16/16 kHz
  WAV windows of at most six seconds. It returns typed turn spans with explicit
  overlap and labels scoped only to the supplied `session_id`.
- `POST /v1/audio/diarization/reconcile` uses
  `pyannote/speaker-diarization-community-1` over retained uncompressed PCM16
  WAV audio of any length. It requires persisted observation IDs and returns
  review-required cluster proposals, never an identity assertion.

`GET /v1/audio/diarization/{live|reconcile}/readiness` is non-mutating: it
does not download weights, install packages, create an environment, or load a
model. It returns HTTP 200 only when the matching worker has already proved it
loaded the verified local model; otherwise it returns typed HTTP 503 with the
exact remediation. Inference is likewise non-provisioning and returns typed
`speaker_diarization_runtime_unavailable` until admin setup succeeds.

Admin setup is asynchronous and single-flight. It returns HTTP 202 while it
provisions, and readiness exposes `setup.stage`, `setup.failed_stage`, the
complete captured setup `stderr`, and the terminal error. Those diagnostics
are persisted under the managed runtime so they survive daemon restart.
Existing ready workers return HTTP 200. Inference never performs these setup
actions.

On JetPack, live setup with an empty object downloads the immutable,
checksum-pinned Q8 Sortformer artifact and builds NVIDIA NeMo-Speech.cpp at a
pinned source revision under `~/.omnius`. Build tooling is isolated and the
JetPack Torch installation is not imported or modified:

```json
POST /v1/audio/diarization/live/setup
{}
```

The native build requires the JetPack CUDA compiler plus `git` and a C++17
compiler. Omnius installs pinned CMake/Ninja only in its private build-tools
venv and never invokes `sudo`; missing native prerequisites are reported by
readiness for operator installation outside inference. CUDA compiler discovery
checks `OMNIUS_DIAR_NVCC`, `/usr/local/cuda-12.2/bin/nvcc`, then
`/usr/local/cuda/bin/nvcc`, and reports those exact locations when none exists. Set
`OMNIUS_NEMO_SPEECH_BIN` to reuse a prebuilt CUDA-enabled binary.

Managed Community-1 setup creates a separate CPU-only CPython environment, so
its PyTorch/Torchaudio packages cannot replace or conflict with JetPack CUDA
Torch. TorchCodec is deliberately omitted because it has no CPython
3.10/aarch64 wheel; the worker decodes retained PCM16 WAV itself and passes
pyannote its supported in-memory waveform mapping. After accepting the gated
model conditions, expose `OMNIUS_HF_TOKEN` to the daemon and submit:

```json
POST /v1/audio/diarization/reconcile/setup
{ "pyannote_terms_accepted": true }
```

The token is used only for the pinned setup download, is rejected in HTTP
bodies, is never written to disk, and is removed from the offline worker
environment. `OMNIUS_PYANNOTE_TERMS_ACCEPTED=1` may persist only the terms
attestation, not the credential.

Every manifest is `omnius.audio-diarization-model-snapshot.v1` and pins the
exact role model, a full immutable 40- or 64-hex revision, a relative artifact, and its
`sha256:<digest>`. Community-1 is a directory pipeline: its manifest also
needs a complete `files` checksum object for every regular model/config file.
Setup streams and verifies all declared bytes; later readiness and inference
reject changed, missing, or added snapshot files by metadata before a worker
can load them. Advanced operators may still provide `snapshot_path`,
`manifest_path`, and a role-specific `python_path`; those paths retain the
same immutable manifest validation.

Both roles allow one active request plus one queued request. A third request is
typed busy. `POST /v1/audio/diarization/{live|reconcile}/cancel` terminates the
worker process group before clearing both active and queued work. Inference and
readiness may warm only a previously verified binding and never download or
install anything. The separate JetPack daemon bootstrap may provision the
public live model by default; gated Community-1 is provisioned only when both
the terms attestation and setup-only token are configured.

Live responses are `omnius.speaker-diarization.v1` `SpeakerTurnSpan[]` with
`start_ms`, `end_ms`, `channel`, explicit overlap, and session-local
`speaker_*` labels. Labels may populate
`AudioObservationInput.diarizationClusterId`; they are never durable person
IDs, voice enrollments, or biometric matches. Reconciliation emits only
review-required proposals with `durable_identity: "forbidden"`.

## Voicechat WebSocket

`/v1/voicechat/ws` supports full-duplex realtime voice.

Client to server:

- Binary PCM Int16 mono 16 kHz mic audio.
- JSON text frame `{ "type": "start" }`.
- JSON text frame `{ "type": "stop" }`.
- JSON text frame `{ "type": "speak", "text": "..." }`.
- JSON text frame `{ "type": "ping" }`.

Server to client:

- `hello`
- `loaded`
- `transcript`
- `agent_text`
- `tts_header`
- binary TTS PCM
- `tts_start`
- `tts_end`
- `state`
- `session_state`
- `error`
- `keepalive`
- `pong`
- `session_started`
- `session_stopped`

Each binary TTS batch is preceded by a `tts_header` frame that announces the sample rate.

## Realtime REST Plus Voice

For ASR/TTS clients that do not use `/v1/voicechat/ws`, call `/v1/chat` with `realtime: true` and send the text transcript as `message`.
