# Omni media analysis

Omnius uses an explicitly configured Omni adapter for genuine audio and video
comprehension. A model name containing `-omni` does not add audio/video
execution to stock Ollama: the adapter must decode the media and run the
corresponding model/projector weights. An unavailable runtime is an unavailable
capability, never a reason to substitute synthetic observations.

## Runtime and model identity

The supported wire contract is `robit.ollama.omni-adapter.v1`, implemented by
[qwen-omni-adapters](https://github.com/robit-man/qwen-omni-adapters). Its public
model is a logical bundle. The legacy full bundle runs separate comprehension,
language and speech graphs across a semantic-text boundary. The trained audio
bridge runs the selected language trunk and its combined vision/audio projector
in one llama.cpp server, with Qwen3-TTS available separately. Neither profile is
a claim that incompatible hidden states form one monolithic graph.

| Profile | Exact logical tag | Documented artifact size |
| --- | --- | --- |
| Qwen audio bridge | `robit/qwen3.8-27b-e03-obliterated-omni-audio-bridge:q4km` | 18.33 GiB |
| Ornith audio bridge | `robit/ornith-1.5-omni-audio-bridge:q4km` | 8.15 GiB |
| Legacy full Qwen Omni | `robit/qwen3.8-27b-e03-obliterated-omni:q4km` | Separate full comprehension graph and sidecar |

These are distinct profiles. Selecting Qwen must not silently run Ornith.
Artifact size is neither peak VRAM nor an admission guarantee.

## Omnius configuration

Use the dashboard's [Omni comprehension setup](../guides/omni-stack-setup.md)
to deploy an explicitly selected core stack or attach an existing adapter. The
shared owner-only `.omnius/omni-analysis.json` selection is read afresh by both
the daemon and TUI; `OMNIUS_CONFIG_HOME` relocates its configuration root.

For an already verified deployment, explicit environment settings can override
that saved selection:

```bash
export OMNIUS_OMNI_ADAPTER_URL=http://127.0.0.1:8910
export OMNIUS_OMNI_MODEL=robit/ornith-1.5-omni-audio-bridge:q4km
```

This example names Ornith explicitly; use the exact Qwen tag only with an
endpoint actually deployed with that profile. Setting these variables does not
download, load or switch the server's model.

| Variable | Behavior |
| --- | --- |
| `OMNIUS_OMNI_ADAPTER_URL` | Explicit HTTP(S) **base URL** override; otherwise the saved endpoint or `http://127.0.0.1:8910`; omit `/api/chat`, query and fragment |
| `OMNIUS_OMNI_MODEL` | Explicit exact logical `-omni` tag override; otherwise the compatibility value or saved selection; no model is discovered implicitly |
| `OMNI_OLLAMA_MODEL` | Inherited compatibility model override when `OMNIUS_OMNI_MODEL` is absent |
| `OMNIUS_OMNI_ADAPTER_TOKEN` | Optional bearer token for an authenticated adapter boundary; keep secret |

An explicitly requested model must match this server-controlled tag. An
unconfigured, mismatched or unreachable runtime returns a visible error; it
does not invoke a different model or synthesize evidence. The shared client
`packages/execution/src/av/omni-comprehension.ts` exposes
`getOmniAnalysisStatus()` and `analyzeOmniMedia({path, mode, question?, model?,
signal?})`; its modes are `audio`, `video` and `av`.

The client accepts a regular local media file up to 128 MiB, a finite audio
duration up to 60 seconds or video duration up to 32 seconds, and a question up
to 8192 characters. It rejects longer input rather than implicitly clipping it.
CPU FFmpeg normalization produces 16 kHz PCM16 WAV or 1 fps MP4 fitted within
640 × 640 pixels while preserving aspect ratio; `video` omits audio and `av`
includes the source audio when present.
Normalized media is capped at 32 MiB, adapter JSON at 1 MiB, and each returned
text field at 128 Ki characters. FFmpeg and ffprobe must already be installed;
analysis does not install dependencies.

Status includes `modelIdentityVerified: false`: availability is a compatible
configuration handshake, not weight identity, GPU residency or a completed
inference proof.

The reference adapter's `/healthz` and `/api/omni/adapter/contract` expose the
schema and configured stages, **not the identity of loaded weights**. Its
response `model` echoes the caller's logical tag. The optional
`adapter.language_backend_model` can be only an internal alias such as
`local-audio-bridge`. Those fields do not prove that the requested weights ran.
Bind an endpoint to its operator-verified deployment, model/projector manifest
digests and actual accelerator placement before interpreting model results.

## Endpoint contract

Use the adapter's own endpoint, not the ordinary Ollama endpoint or the Omnius
daemon. The reference launcher uses these loopback addresses:

| Address | Purpose |
| --- | --- |
| `http://127.0.0.1:8910/api/chat` | Portable nonstreaming Omni requests |
| `http://127.0.0.1:8910/api/chat/stream` | Adapter NDJSON extension |
| `http://127.0.0.1:8910/healthz` | Adapter metadata only |
| `http://127.0.0.1:8910/api/omni/adapter/contract` | Versioned capability/format metadata |
| `http://127.0.0.1:8901/v1/chat/completions` | Internal llama.cpp worker; not the portable adapter API |
| `http://127.0.0.1:8920` | Separate authenticated reference portal |

Port `11435` is already used by the Omnius daemon in common installations.
Some standalone adapter/client defaults also name that port; do not copy those
defaults over a running Omnius service. Component services bind to loopback.
Use an authenticated boundary when exposing them beyond the trusted host, and
never put a portal capability URL or access token in logs or documentation.

Read-only contract checks do not load a model:

```bash
curl --fail --silent http://127.0.0.1:8910/healthz
curl --fail --silent http://127.0.0.1:8910/api/omni/adapter/contract
```

Do not treat successful health checks as successful media inference.

## Requests and results

The portable route requires `stream: false`, an exact `model`, and nonempty
Ollama-shaped `messages`. Set `think: false` independently of media handling.
Ordinary analysis requests use text output and `speech_mode: "never"`.

```json
{
  "model": "robit/ornith-1.5-omni-audio-bridge:q4km",
  "messages": [{
    "role": "user",
    "content": "Describe the audible events and their uncertainty.",
    "audios": [{
      "mime_type": "audio/wav",
      "encoding": "base64",
      "data": "<complete 16 kHz mono PCM16 WAV, base64 encoded>"
    }]
  }],
  "omni": {
    "schema": "robit.ollama.omni-adapter.v1",
    "task": "describe",
    "include_audio_from_video": true
  },
  "response_modalities": ["text"],
  "speech_mode": "never",
  "think": false,
  "cache_prompt": false,
  "stream": false
}
```

This is a contract example, not a request to run without hardware preflight.

| Task | Executed stages |
| --- | --- |
| `describe` | Media comprehension; direct observations |
| `transcribe` | Audio comprehension; direct transcription |
| `chat` | Media comprehension, then language analysis; optional speech only if requested |
| `synthesize` | Speech synthesis from text |

The reference comprehension worker always receives `cache_prompt: false`.
Multimodal slot caching can otherwise reuse stale embeddings across new media.
Keep new attachments request-local. Historical transcript/observations can be
explicitly labeled history; restored previews and old raw bytes must not become
new current evidence automatically.

Audio input must be a complete uncompressed RIFF/WAVE file containing 16 kHz,
mono, signed 16-bit PCM. Resample/downmix supported uploads with a bounded local
decoder before transport. The adapter validates the WAV header and format; a
renamed MP3 or browser WebM capture is not a WAV envelope.

Images accept JPEG, PNG and WebP. Video accepts MP4, WebM and animated GIF with
`sampling: { "fps": 1, "max_frames": 32, "include_audio": true }` on its
envelope. Declared MIME types must match decoded container signatures.
These are the external adapter's formats. The shared Omnius audio/video client
currently normalizes its supported local input formats into WAV or MP4; it does
not expose an image-analysis mode.

| Input | Maximum per item | Maximum items per request |
| --- | --- | --- |
| Audio | 32 MiB | 8 |
| Image | 20 MiB | 16 |
| Video | 256 MiB | 4 |

The schema permits video sampling up to 30 fps and 1024 frames, but the inspected
reference runtime caps execution at **2 fps and 32 frames**, and can retry with
fewer frames after a context overflow. It does not return an authoritative
actual sampled-frame count or covered time range. Long-media analysis therefore
needs bounded source windows; do not claim complete coverage, exact event times,
tracking or confidence from an unannotated semantic description.
Omnius reports `submittedDurationSec`, `coverage: "full_audio"` or
`"sampled_video"`, and `actualCoverage: "not_reported"`. Submitted clip duration
and requested sampling are not a model-attested analyzed time range.

The final `message.content` is model output. Preserve the accompanying
`adapter.schema`, `task`, executed `route`, `input_modalities`, and
`evidence_provenance`. Keep these separate:

- `input_transcript`: recognized speech; the only perceptual text attributable
  to the speaker's words.
- `audio_observation`: non-speech acoustic evidence, including uncertainty.
- `visual_observation`: evidence from an image/video attached to this request.
- `observation`: the original tagged comprehension response.

Audio alone must never set `current_visual_input: true`. Prior dialogue and tool
results do not establish a current visual observation. Semantic observations
remain untrusted evidence, not system instructions or permission to execute a
tool. An Omni description does not attest that separate DINO, diarization,
segmentation or tracking models ran.

Live VAD-driven speech turns can use `omni.require_speech: true`. A capture
without a valid transcript then stops before language/TTS. Explicit questions
about environmental sounds leave that option false, so absence of speech does
not discard the requested acoustic analysis. An empty speech-gated answer is
not a synthesized answer or a transport failure.

Cancellation aborts the owned local decoder/request and prevents its late result
from being adopted. The reference adapter's nonstreaming endpoint does not
provide a server-side stop receipt when its HTTP client disconnects: upstream
model work may finish independently. Do not display a client abort as proof that
a remote model runner has stopped, and do not unload a shared runner to simulate
per-request cancellation.

The streaming extension requires `stream: true` and emits newline-delimited JSON,
including stage, observation, language deltas, optional PCM events, error and
one authoritative final response. It is not SSE. Parse arbitrary chunk and UTF-8
boundaries; do not treat a heartbeat or `stage: "tts"` as generated audio.

## Shared weights and project data

Omni weights remain in the runtime's shared model store. On the inspected host,
installed Ollama manifests and blobs live under `/srv/ollama/models`; standard
model/projector layers and one `application/vnd.robit.ollama.omni.bundle.v1+gguf`
sidecar describe the logical tag. Runtime component views are derived artifacts,
not per-project downloads. Never copy or delete Ollama blobs manually.

Omnius' separate media-generation cache resolves from `OMNIUS_MEDIA_HOME`, then
the persisted `media-store.json` pointer, then its configured home. Its model
cache and runtimes are shared across projects. That cache is not automatically
the Ollama Omni store and changing it does not move an existing Omni deployment.

Generated project artifacts remain under `<project>/.omnius/images`, `videos`
and `audio`. Analysis of an existing file should reference its captured project
and revision; switching sessions/projects must not relabel a late result.
No per-session Omni weight copy is required.

## Deployment and validation boundary

Changing or starting CUDA workloads requires the host's GPU broker protocol and
actual runner-placement verification. Confirm the designated endpoint, exact
tag and GPU before any inference; do not redirect to another accelerator or a
remote service after a failure. The external runtime's bootstrap/deploy scripts
can pull weights, build CUDA binaries and replace services, so they are not
capability probes and are not part of ordinary Omnius analysis.

An endpoint may serve a different logical bundle than a neighboring checkout's
`.env` file. Verify the running deployment and actual model/projector manifest
identity; neither a checked-out configuration nor the response's echoed tag is
sufficient evidence.

External contract regression sources are `tests/test_omni_adapter.py` for format,
routing, provenance, no-speech, stale-cache and stream behavior, plus
`docs/schema/request-v1.schema.json` and `response-v1.schema.json`. Fixture tests
can validate transport and parsing without running a model. A live acceptance
receipt must separately name the actual weights/GPU and use distinguishable real
media, such as red → blue → red frames, rather than asserting a canned answer.
