# Voice, Audio, Vision, And Voicechat

## Endpoint Summary

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/v1/voice/state` | Voice runtime status |
| `GET` | `/v1/voice/models` | List TTS voice models |
| `POST` | `/v1/voice/models/switch` | Switch active TTS model |
| `GET`/`POST` | `/v1/voice/supertonic-settings` | Read or update voice tuning settings |
| `GET` | `/v1/asr/engines` | List ASR systems, models, capabilities, and readiness |
| `GET` | `/v1/asr/status` · `/v1/asr/selection` | Read selection and runtime status |
| `PATCH` | `/v1/asr/selection` | Persist and activate an exact ASR engine/model |
| `POST` | `/v1/asr/activate` | Activate and persist an exact ASR engine/model |
| `POST` | `/v1/asr/engines/{engineId}/setup` | Install a managed ASR runtime and pinned weights |
| `POST` | `/v1/asr/transcriptions` · `/v1/asr/test` | Transcribe/test with the selected backend |
| `GET`/`POST` | `/v1/voice/asr-models` · `/v1/voice/asr-models/switch` | Compatibility registry/activation aliases |
| `POST` | `/v1/voice/tts` | Synthesize text |
| `POST` | `/v1/audio/speech` | OpenAI-compatible TTS alias |
| `GET` | `/v1/audio/classify/health` | Jetson CUDA/TensorRT YAMNet readiness |
| `POST` | `/v1/audio/classify/setup` | Provision and warm pinned JetPack TensorRT YAMNet |
| `POST` | `/v1/audio/classify` | Classify caller-supplied WAV via direct-tool compatible contract |
| `GET` | `/v1/audio/embed/health` | Role-typed embedding readiness (`?kind=acoustic|speaker|semantic`) |
| `POST` | `/v1/audio/embed/setup` | Provision/warm one role runtime (admin; `?kind=...`) |
| `POST` | `/v1/audio/embed` | Managed role-typed embedding via `audio_analyze` (`?kind=...`) |
| `GET` | `/v1/audio/diarization/live/readiness` | Non-mutating Sortformer worker readiness |
| `POST` | `/v1/audio/diarization/live/setup` | Verify and warm a local Sortformer runtime (admin) |
| `POST` | `/v1/audio/diarization/live` | Managed live/session-local speaker-turn diarization |
| `POST` | `/v1/audio/diarization/live/cancel` | Cancel live worker work and clear its queue |
| `GET` | `/v1/audio/diarization/reconcile/readiness` | Non-mutating offline Community-1 worker readiness |
| `POST` | `/v1/audio/diarization/reconcile/setup` | Verify and warm a local Community-1 runtime (admin) |
| `POST` | `/v1/audio/diarization/reconcile` | Managed offline/dream reconciliation proposals |
| `POST` | `/v1/audio/diarization/reconcile/cancel` | Cancel reconciliation work and clear its queue |
| `POST` | `/v1/voice/transcribe` | Transcribe audio |
| `POST` | `/v1/audio/transcriptions` | OpenAI-compatible transcription alias |
| `POST` | `/v1/voice/transcribe/stream` | Isolated final transcription over SSE |
| `GET` | `/v1/voice/clone-refs` | List clone references |
| `POST` | `/v1/voice/clone-refs/upload` | Upload clone reference |
| `POST` | `/v1/voice/clone-refs/from-url` | Fetch clone reference server-side |
| `POST` | `/v1/voice/clone-refs/{filename}/activate` | Activate clone reference |
| `POST` | `/v1/voice/clone-refs/{filename}/rename` | Rename clone reference |
| `DELETE` | `/v1/voice/clone-refs/{filename}` | Delete clone reference |
| `POST` | `/v1/voice/speak` | Synthesize and broadcast to voicechat clients |
| `WS` | `/v1/voicechat/ws` | Full-duplex voicechat WebSocket |
| `POST` | `/v1/vision/describe` | Vision describe placeholder/deferred endpoint |
| `GET` | `/v1/vision/embed/readiness` | Non-mutating isolated OpenCLIP readiness |
| `POST` | `/v1/vision/embed/setup` | Explicit OpenCLIP setup (admin scope) |
| `POST` | `/v1/vision/embed` | Offline-only verified OpenCLIP image embedding |
| `GET` | `/v1/ocr/readiness` | Probe exact OCR package/engine/backend readiness without mutation |
| `POST` | `/v1/ocr/setup` | Create and verify the isolated OCR venv (admin scope) |
| `POST` | `/v1/ocr/advanced` | Agent-equivalent managed advanced OCR (alias of `/v1/tools/ocr_image_advanced/call`) |
| `POST` | `/v1/media/av/analyze` | Grounded AV/audio comprehension of a media file into entities/events |

## Jetson Microphone Routing

The live `/listen` microphone path does not trust PulseAudio's mutable default
source. When the documented Seeed ReSpeaker USB 4-Mic Array source
`alsa_input.usb-SEEED_ReSpeaker_4_Mic_Array__UAC1.0_-00.multichannel-input`
is present as a six-channel Pulse source, Omnius selects it ahead of a running
`nx_remapped_out` desktop/TTS remap and FFmpeg extracts DSP/beamformed channel
0 to mono PCM16/16 kHz. `GET /v1/voice/state` reports the actual source,
source kind, channel count, and selected channel.

`OMNIUS_MIC_DEVICE` remains an explicit operator override (`pulse:<source>` or
`hw:N,M`). A saved live-input selection of `pulse:nx_remapped_out` is upgraded
only when that exact six-channel ReSpeaker source is concurrently available;
other explicit selections are left unchanged.

This affects only Omnius-owned live microphone capture. Egg owns its physical
capture and sends conditioned mono WAV files to audio-comprehension endpoints;
those file-analysis endpoints do not open a microphone or reselect a Pulse
source.

## Vision Embeddings

`POST /v1/vision/embed` uses an isolated, CUDA-enabled OpenCLIP
`ViT-B-32` / `laion2b_s34b_b79k` backend and returns a 512-dimensional,
L2-normalized image vector plus the verified artifact digest. It does not make
descriptive claims or invoke an LLM.

First poll `GET /v1/vision/embed/readiness`. It is non-mutating and returns
HTTP 503 until the isolated runtime and checksum-manifested weights are ready.
Use the admin-scoped `POST /v1/vision/embed/setup` to create the
`--system-site-packages` runtime under
`~/.omnius/runtimes/vision/open-clip`, install pinned non-Torch dependencies,
and fetch/verify the checkpoint from immutable revision
`1a25a446712ba5ee05982a381eed697ef9b435cf`. Setup is single-flight and
daemon-bootstrapped by default on JetPack; set `OMNIUS_VISION_AUTO_SETUP=0`
to disable it. Inference never installs packages or downloads artifacts, and
its Python worker is network-denied.

JetPack Torch is inherited and never replaced. `torchvision` must either be
present in the compatible vendor stack or be supplied explicitly with the local
`OMNIUS_VISION_TORCHVISION_WHEEL` and immutable
`OMNIUS_VISION_TORCHVISION_WHEEL_SHA256`; generic PyPI Torch/torchvision is
blocked. Bootstrap discovery may inspect the selected vendor interpreter, but
the managed runtime installs its own pinned support closure and proves Torch,
torchvision, and OpenCLIP with user-site packages disabled. On memory-constrained
Jetson systems model loading and embedding fail with a typed 503 rather than
evicting a resident ASR/Ollama model. Dependency and
weight provisioning itself is allowed to finish under transient pressure;
readiness separately reports `installed`, `weightsReady`,
`inferenceAvailable`, and the current admission result. The model is loaded
only after at least 8 GiB is available.

## Advanced OCR

First poll `GET /v1/ocr/readiness?language=eng`. On Jetson, a privileged or
interactive npm installation/bootstrap provisions `tesseract-ocr`, `python3-venv`,
an import-verified system OpenCV provider (including NVIDIA
`libopencv-python`), `python3-numpy`, `python3-pil`, and `python3-reportlab`.
Daemon startup then creates (or repairs)
`~/.omnius/runtimes/vision/ocr-advanced` with `--system-site-packages` and
installs only `pytesseract==0.3.13 --no-deps`. `POST /v1/ocr/setup` is the
admin-scoped observable retry/repair path; set `OMNIUS_OCR_AUTO_SETUP=0` to
disable startup setup. Set `OMNIUS_OCR_PYTHON` to use an operator-managed,
already-ready interpreter instead.

`POST /v1/ocr/advanced` is a compatibility alias for the exact
agent-facing `ocr_image_advanced` tool. Use the same direct-tool envelope;
the response retains display text in `result.output` and returns the parsed
pipeline payload in `result.data`.

```bash
curl -sS -X POST http://127.0.0.1:11435/v1/ocr/advanced \
  -H 'content-type: application/json' \
  -d '{"args":{"image":"/data/invoice.png","language":"eng","psm":6},"timeout_ms":90000}'
```

Inference never invokes sudo, apt, pip, or venv creation. If readiness is
false, the tool returns `result.data.code=ocr_runtime_not_ready` with setup and
readiness URLs. Jetson never installs generic CUDA, Torch, OpenCV, or NumPy
packages. A valid `OMNIUS_OCR_PYTHON` is accepted from its isolated import and
version results without requiring particular dpkg package names. The caller
needs `run` scope because optional
`output_dir`, batch, and debug modes write OCR artifacts.

The managed pipeline is bounded: it starts with high-yield preprocessing and
expands variants only for low-evidence or larger images. Small crops avoid the
former all-variant/all-PSM explosion and use a six-second cap per Tesseract
attempt. If the first small-crop pass proves high-volume very-low-confidence or
distinctly symbol-heavy garbage, recovery stops immediately; blanks and
plausibly recoverable text still receive the second high-yield attempt. The
REST default is 90 seconds (maximum 180 seconds), while the worker has an
80-second internal deadline so it can return diagnostics. A timeout or cancellation returns
`result.data.schema=omnius.ocr-diagnostic.v1` with code `ocr_timeout` or
`ocr_cancelled`; cancellation terminates the Python/Tesseract process group
with TERM followed by KILL.

OCR text is evidence-gated before it is returned. Strong text with adequate
confidence is accepted. An empty or tiny non-substantive result is a successful
`omnius.ocr-evidence.v1` `low_information` observation with diagnostic code
`ocr_low_information`, not a fabricated transcript. High-volume low-confidence
or symbol-heavy output is suppressed and returned as
`ocr_evidence_rejected`; its raw text is not exposed as alternate OCR evidence.

## TTS

`POST /v1/voice/tts` returns audio bytes. `format` can be `wav` or `pcm`. `X-Sample-Rate` reports the sample rate.

OpenAI-compatible alias:

```text
POST /v1/audio/speech
```

## ASR

`POST /v1/asr/transcriptions` transcribes uploaded audio with the selected
engine. `/v1/asr/test` runs the same real path for readiness checks. The
OpenAI-compatible alias is:

```text
POST /v1/audio/transcriptions
```

VibeVoice is exposed as `vibevoice-transformers/vibevoice-asr-7b`. It is a
completed-file backend, not an incremental PCM stream: the response preserves
speaker IDs, segment timestamps, raw structured text, and warnings. Pass
`?context=` for customized hotwords/background context. Setup installs the
exact pinned `microsoft/VibeVoice-ASR` snapshot into the unified ASR cache;
activation requires an explicit GPU and verified placement, and never falls
back to CPU or another device. Discrete Linux uses `nvidia-smi`; Jetson/L4T,
where NVIDIA does not ship `nvidia-smi`, uses `tegrastats` plus CUDA Torch
device properties.

## Jetson Audio Classification

`audio_analyze` classification on Jetson is a persistent, CUDA-enabled FP16
TensorRT YAMNet worker—not a per-call TensorFlow/TF-Hub process. The daemon
auto-provisions it on a Jetson unless `OMNIUS_AUDIO_AUTO_SETUP=0`; operators may
also explicitly invoke `POST /v1/audio/classify/setup` with admin scope. Setup
downloads the pinned ONNX model and class map once, verifies their SHA-256
digests, builds an engine for the local JetPack/TensorRT stack, checks the
approved integrated CUDA device (`OMNIUS_AUDIO_CUDA_VISIBLE_DEVICES=0`), and
warms the worker. `OMNIUS_AUDIO_PYTHON` explicitly selects the operator's
JetPack-compatible Torch 2.2/CUDA 12.2 interpreter. When unset, Omnius probes
conventional project venvs (including `~/Documents/<project>/.venv`) and reuses
the first interpreter that proves CUDA Torch placement; it never modifies that
source environment. Omnius pins `cuda-python==12.2.0 --no-deps`
only inside its isolated audio venv, verifies Orin compute capability 8.7, and
handles L4T R36.3 `tegrastats` builds which do not support `--count`.

Poll `GET /v1/audio/classify/health` before sending classification work. A
general `/health` success is not sufficient. The response reports readiness,
backend (`tensorrt-fp16`), device, model digest, warmup timing, queue depth, and
errors. If the worker exits, readiness becomes false and the next health poll
schedules its local restart without restarting Egg or Omnius.

The compatible request shape is:

```json
{
  "args": {
    "action": "classify",
    "file": "data/cognitive-memory/media/audio/2026/08/13/<utterance-id>.wav",
    "top_k": 5
  },
  "timeout_ms": 90000
}
```

Send it to either `POST /v1/tools/audio_analyze/call` or the
`POST /v1/audio/classify` alias. Inputs must be caller-supplied mono PCM16,
16 kHz WAV files; classification never captures from `default`, normalizes the
Egg signal again, installs packages, or fetches weights. The response is
machine-readable at `result.data.classifications` with model/backend/digest,
audio facts, a `low_information` quiet-input signal, and
queue/decode/inference/total timings. One job runs and one may wait; additional
requests receive a typed `audio_classifier_busy` failure in
`result.data.code` while retaining the direct-tool HTTP 200 envelope.

## Role-Typed Audio Embeddings

`POST /v1/audio/embed` is a stable adapter to the exact agent-facing
`audio_analyze` direct tool. It requires an explicit role kind; Omnius does
not claim that an acoustic vector is a speaker-identity or semantic-audio
vector. Check the non-provisioning role control plane first:

```text
GET /v1/audio/embed/health?kind=acoustic
```

Each readiness response names the exact requested vector space and worker:

- `acoustic`: YAMNet TensorRT, 1024d,
  `yamnet-onnx/ac2ca3bd45d12ec1f19f1144205ea529b4e9dedf/output_1:mean-l2`.
- `speaker`: CPU-only WeSpeaker CAM++, 512d,
  `speaker-identity-wespeaker-voxceleb-campplus-acf623ad8ca746e50baa432255cf8fc57c669c45-fbank80-cmn-fullclip-l2`.
- `semantic`: JetPack CUDA CLAP, 512d,
  `clap-htsat-unfused/84bcbbd1d619e407a8216371ddef36e458d95d93/audio:segment-mean-l2`.

An unprovisioned or unready role returns its own typed HTTP 503 readiness body;
callers must never fall back to another role's space.

The invoke shape remains the direct-tool envelope, with `action` forced to
`embed` by the route:

```json
{
  "args": { "file": "data/cognitive-memory/media/audio/2026/08/13/<utterance-id>.wav" },
  "timeout_ms": 90000
}
```

Use the canonical query form: `POST /v1/audio/embed?kind=acoustic` (replace
`acoustic` with `speaker` or `semantic` as needed). `kind` in the JSON body or
`args` is supported as a compatibility alias. Existing clients may send
top-level `path` or `file` and Omnius normalizes it into `args`; binary
`bytesBase64` input is rejected because each managed worker is intentionally a
supplied-retained-WAV-only boundary.

`POST /v1/audio/embed/setup?kind=<role>` is the admin-only provisioning path.
It is the only endpoint that installs dependencies or retrieves pinned weights:
acoustic reuses the pinned JetPack YAMNet/TensorRT setup, speaker provisions
the isolated CPU-only WeSpeaker runtime, and semantic provisions the isolated
JetPack CUDA CLAP runtime. It activates only the selected role. Normal Jetson
startup provisions acoustic, speaker, and semantic roles by default; set
`OMNIUS_AUDIO_AUTO_SETUP=0` to disable all audio setup or
`OMNIUS_SEMANTIC_AUDIO_AUTO_SETUP=0` to disable CLAP specifically.

CLAP provisioning installs a checksum-locked CPython 3.10/aarch64 wheel
closure and the immutable model revision without loading the model. It is
allowed to finish while unified memory is busy. Worker activation and
inference retain the 8 GiB admission gate and idle eviction, so setup cannot
silently evict ASR, Ollama, or another CUDA workload.

On JetPack, speaker setup deliberately leaves `OMNIUS_AUDIO_PYTHON` (including
an Egg project `.venv`) unchanged. It creates a private, non-system-site
`~/.omnius/runtimes/audio/speaker/venv`, then installs only a locked,
checksum-verified CPython 3.10/aarch64 CPU dependency set: NumPy 1.26.4,
ONNX Runtime 1.17.3, and the exact CPU support wheels required by that ONNX
Runtime release. Each wheel is materialized and SHA-256 verified before the
managed venv installs it with `--no-deps --no-index`.

The worker implements the WeSpeaker CAM++ 80-bin Kaldi configuration in pure
NumPy: 25 ms / 10 ms Hamming frames, dither disabled, Kaldi pre-emphasis and
mel bank behavior, then full-clip CMN without CVN. Readiness invokes a
versioned CPU preprocessing probe. It checks the fixed Kaldi configuration,
feature shape, CMN, energy range, and expected mel-band profile with explicit
cross-platform numeric tolerances before it can report ready. The speaker path neither imports
nor links Torch or Torchaudio, so the generic `torchaudio-2.2.0` ABI mismatch
cannot bind against, replace, or otherwise affect JetPack's CUDA-enabled Egg
Torch. Inference remains install-free and network-free; failed package imports
or calibration leave the role unready with the exact probe error.

The dedicated route returns typed HTTP 503 whenever its requested runtime is
unprovisioned or unavailable: `audio_classifier_not_ready` (acoustic),
`speaker_embedding_not_ready` (speaker), or
`audio_semantic_embedding_not_ready` (semantic). Callers should run the
matching admin setup route and poll its health URL. Once active, speaker
saturation and deadline errors stay role-specific
(`speaker_embedding_busy`, `speaker_embedding_timeout`). The generic
`POST /v1/tools/audio_analyze/call` interface intentionally retains its normal
HTTP 200 direct-tool envelope for these structured errors. Likewise, the
dedicated semantic adapter surfaces local CLAP admission pressure as HTTP 503
`audio_semantic_memory_pressure`, without substituting another role.

Direct-tool auth, profile, origin, output-cap, timeout, and cancellation
policy all remain in effect. Inference never creates environments, installs
packages, downloads weights, or falls back to the retired generic Python
`embed-audio.py` provider.

## Speaker Diarization And Reconciliation

Omnius provides two managed, role-specific persistent workers. They never
fall back to generic transcription, create a durable speaker identity, or
provision themselves during inference.

- `POST /v1/audio/diarization/live` uses NVIDIA
  `nvidia/diar_streaming_sortformer_4spk-v2` for retained, mono PCM16/16 kHz
  WAV windows of at most six seconds. It returns typed turn spans with explicit
  overlap and labels scoped only to the supplied `session_id`.
- `POST /v1/audio/diarization/reconcile` uses
  `pyannote/speaker-diarization-community-1` over retained uncompressed PCM16
  WAV audio of any length. It requires persisted observation IDs and returns
  review-required cluster proposals, never an identity assertion.

`GET /v1/audio/diarization/{live|reconcile}/readiness` is non-mutating: it
does not download weights, install packages, create an environment, or load a
model. It returns HTTP 200 only when the matching worker has already proved it
loaded the verified local model; otherwise it returns typed HTTP 503 with the
exact remediation. Inference is likewise non-provisioning and returns typed
`speaker_diarization_runtime_unavailable` until admin setup succeeds.

Admin setup is asynchronous and single-flight. It returns HTTP 202 while it
provisions, and readiness exposes the exact phase or terminal error. Existing
ready workers return HTTP 200. Inference never performs these setup actions.

On JetPack, live setup with an empty object downloads the immutable,
checksum-pinned Q8 Sortformer artifact and builds NVIDIA NeMo-Speech.cpp at a
pinned source revision under `~/.omnius`. Build tooling is isolated and the
JetPack Torch installation is not imported or modified:

```json
POST /v1/audio/diarization/live/setup
{}
```

The native build requires the JetPack CUDA compiler plus `git` and a C++17
compiler. Omnius installs pinned CMake/Ninja only in its private build-tools
venv and never invokes `sudo`; missing native prerequisites are reported by
readiness for operator installation outside inference. Set
`OMNIUS_NEMO_SPEECH_BIN` to reuse a prebuilt CUDA-enabled binary.

Managed Community-1 setup creates a separate CPU-only CPython environment, so
its PyTorch/Torchaudio packages cannot replace or conflict with JetPack CUDA
Torch. TorchCodec is deliberately omitted because it has no CPython
3.10/aarch64 wheel; the worker decodes retained PCM16 WAV itself and passes
pyannote its supported in-memory waveform mapping. After accepting the gated
model conditions, expose `OMNIUS_HF_TOKEN` to the daemon and submit:

```json
POST /v1/audio/diarization/reconcile/setup
{ "pyannote_terms_accepted": true }
```

The token is used only for the pinned setup download, is rejected in HTTP
bodies, is never written to disk, and is removed from the offline worker
environment. `OMNIUS_PYANNOTE_TERMS_ACCEPTED=1` may persist only the terms
attestation, not the credential.

Every manifest is `omnius.audio-diarization-model-snapshot.v1` and pins the
exact role model, a full immutable 40- or 64-hex revision, a relative artifact, and its
`sha256:<digest>`. Community-1 is a directory pipeline: its manifest also
needs a complete `files` checksum object for every regular model/config file.
Setup streams and verifies all declared bytes; later readiness and inference
reject changed, missing, or added snapshot files by metadata before a worker
can load them. Advanced operators may still provide `snapshot_path`,
`manifest_path`, and a role-specific `python_path`; those paths retain the
same immutable manifest validation.

Both roles allow one active request plus one queued request. A third request is
typed busy. `POST /v1/audio/diarization/{live|reconcile}/cancel` terminates the
worker process group before clearing both active and queued work. Inference and
readiness may warm only a previously verified binding and never download or
install anything. The separate JetPack daemon bootstrap may provision the
public live model by default; gated Community-1 is provisioned only when both
the terms attestation and setup-only token are configured.

Live responses are `omnius.speaker-diarization.v1` `SpeakerTurnSpan[]` with
`start_ms`, `end_ms`, `channel`, explicit overlap, and session-local
`speaker_*` labels. Labels may populate
`AudioObservationInput.diarizationClusterId`; they are never durable person
IDs, voice enrollments, or biometric matches. Reconciliation emits only
review-required proposals with `durable_identity: "forbidden"`.

## Voicechat WebSocket

`/v1/voicechat/ws` supports full-duplex realtime voice.

Client to server:

- Binary PCM Int16 mono 16 kHz mic audio.
- JSON text frame `{ "type": "start" }`.
- JSON text frame `{ "type": "stop" }`.
- JSON text frame `{ "type": "speak", "text": "..." }`.
- JSON text frame `{ "type": "ping" }`.

Server to client:

- `hello`
- `loaded`
- `transcript`
- `agent_text`
- `tts_header`
- binary TTS PCM
- `tts_start`
- `tts_end`
- `state`
- `session_state`
- `error`
- `keepalive`
- `pong`
- `session_started`
- `session_stopped`

Each binary TTS batch is preceded by a `tts_header` frame that announces the sample rate.

## Realtime REST Plus Voice

For ASR/TTS clients that do not use `/v1/voicechat/ws`, call `/v1/chat` with `realtime: true` and send the text transcript as `message`.