# Bring Your Own Inference

Omnius routes local and hosted models through an explicit provider descriptor.
The descriptor determines the wire protocol, URL paths, authentication
headers, and supported operations. It is the source of truth used by setup,
model discovery, TUI runs, REST chat, summaries, voice, Telegram, sponsors,
and endpoint probes.

## Discover The Provider Contract

```bash
omnius discover "bring your own inference"
omnius show provider.anthropic
omnius show provider.gemini
omnius discover "provider protocol" --kind provider
```

From the daemon:

```bash
curl -s "http://127.0.0.1:11435/v1/discovery?q=provider&kind=provider"
curl -s "http://127.0.0.1:11435/v1/discovery/provider.anthropic"
```

## Protocol Matrix

| Provider | Protocol | Canonical base | Authentication |
| --- | --- | --- | --- |
| Ollama | `ollama` | `http://127.0.0.1:11434` | none by default |
| LM Studio, vLLM | `openai-chat` | local configured URL | optional Bearer |
| OpenAI | `openai-chat` | `https://api.openai.com/v1` | Bearer |
| Anthropic / Claude | `anthropic-messages` | `https://api.anthropic.com/v1` | `x-api-key` plus `anthropic-version` |
| Google Gemini | `openai-chat` | `https://generativelanguage.googleapis.com/v1beta/openai` | Bearer plus Omnius client header |
| OpenRouter, Groq, Chutes, DeepInfra, Together, Fireworks, Mistral, Cerebras, SambaNova, NVIDIA NIM, Hyperbolic | `openai-chat` | provider preset | Bearer |
| Sponsor and COHERE peers | negotiated Omnius endpoint | discovered peer | peer/session policy |
| Custom | explicitly selected | user supplied | configured for that endpoint |

Known aliases include `claude` for Anthropic and `google` / `google-ai` for
Gemini.

## Anthropic Messages Translation

Anthropic is not treated as an OpenAI-shaped URL. Omnius sends
`POST /v1/messages` with `x-api-key`, `anthropic-version: 2023-06-01`, and
JSON content type. The shared adapter translates:

- the OpenAI-style system message into Anthropic's top-level `system`;
- ordinary user and assistant messages into Messages content blocks;
- tool definitions into Anthropic tool declarations;
- assistant tool calls into `tool_use` blocks;
- tool results into `tool_result` blocks;
- Anthropic text, tool calls, streaming events, stop reasons, and usage back
  into Omnius' common completion shape.

This behavior follows Anthropic's
[Messages API](https://platform.claude.com/docs/en/api/messages/create) and
[authentication contract](https://platform.claude.com/docs/en/manage-claude/authentication).

## Gemini OpenAI Compatibility

Gemini uses Google's official OpenAI-compatible prefix:

```text
https://generativelanguage.googleapis.com/v1beta/openai
```

Omnius appends compatible resources such as `/models`,
`/chat/completions`, and `/embeddings`, authenticates with
`Authorization: Bearer <key>`, and identifies the client with
`x-goog-api-client: omnius/<version>`. See Google's
[OpenAI compatibility guide](https://ai.google.dev/gemini-api/docs/openai).

The compatibility API is not a promise that every Gemini-native feature is
available. Discover Omnius' advertised provider capabilities before depending
on a provider-specific extension.

## Key Resolution

Omnius resolves upstream provider credentials in this exact order:

```text
OMNIUS_PROVIDER_API_KEY
  → OMNIUS_MODEL_API_KEY
  → OMNIUS_UPSTREAM_API_KEY
  → OMNIUS_API_KEY
  → VLLM_API_KEY
  → persisted endpoint configuration
```

The first defined credential wins. REST bearer keys are a separate boundary:
use `OMNIUS_REST_API_KEY` or `OMNIUS_REST_API_KEYS` for callers of the daemon.
Never put either provider or REST credentials in URLs, logs, discovery data,
or agent guidance files.

## Automatic Remote Fallback

For normal requests to a loopback local inference endpoint, Omnius tries the
local endpoint first. Connection failures, local broker admission failures,
server failures, and an expired initial local attempt can trigger a bounded,
sequential remote cascade. Configured remote endpoints continue using their
selected provider. Requests explicitly pinned to the exact configured endpoint
also retain that endpoint and do not use this cascade.

Remote fallback is enabled by default. Configured fallback providers run first,
followed by compatible anonymous routes from the same
[free inference catalogue](https://robit-man.github.io/inference/) used during
onboarding. Hermes, LLM7, and OVHcloud seed routes remain available when the
catalogue cannot be read. Anonymous routes receive no provider credential;
LLM7's public `Bearer unused` value is an authentication sentinel. Tool-bearing
requests retain their tools and require a compatible route. Image requests
currently have no verified fallback route and fail closed.

`/fallback auto` is the default: a local outage automatically uses a compatible
remote provider. `/fallback manual` asks the requester before sending their
conversation and tool context to remote inference. Telegram shows buttons in
the original chat; only the original requester can answer. TUI shows a picker.
Declining or stopping sends no remote request. `/fallback status` (or `/fallback`)
shows the current mode. The choice is saved globally for subsequent requests,
including backends that are already running; models cannot issue this command.
Telegram approval covers the same owned input and its final answer review, and
cannot authorize another input or survive Stop. Manual mode without an
interactive approval handler stops with an approval-required diagnosis.

The remote request includes the retained conversation and tool context needed
for that inference. Host-known credentials are redacted from that wire data.
Once streaming output begins, an interrupted generation is not replayed on a
different provider. Stop cancels admission, discovery, and generation.

Configure the cascade in `~/.omnius/config.json`:

```json
{
  "inferenceFallback": {
    "enabled": true,
    "mode": "auto",
    "keyless": false,
    "providers": [
      {
        "providerId": "groq",
        "model": "YOUR_GROQ_MODEL_ID",
        "apiKeyEnv": "GROQ_API_KEY"
      },
      {
        "providerId": "openrouter",
        "model": "YOUR_OPENROUTER_MODEL_ID",
        "apiKeyEnv": "OPENROUTER_API_KEY"
      }
    ],
    "localTimeoutMs": 20000,
    "requestTimeoutMs": 20000,
    "totalTimeoutMs": 60000
  }
}
```

Use exact model IDs available to your account. Make the named key variables
available to the Omnius process through its service environment or a local
secret manager. `keyless: false` limits fallback to the configured providers;
omit it or set it to `true` to include the anonymous routes. Profiles accept a
canonical provider ID, model ID, and environment variable name, with at most
six profiles. They do not accept raw keys, custom URLs, or the generic primary
credential variables such as `OMNIUS_PROVIDER_API_KEY`. Missing keys skip their
profiles; malformed explicit policy disables remote fallback.

`localTimeoutMs` bounds the initial local attempt. `requestTimeoutMs` bounds
each remote attempt, and `totalTimeoutMs` bounds the remote cascade. These
defaults are 20,000 ms, 20,000 ms, and 60,000 ms respectively. Each per-attempt
timeout must fit within the total timeout. Streaming clears the initial local
timeout after its first frame. Failed local routes cool down for 15 seconds;
later real requests retry them without synthetic generation probes.

Disable fallback for a process with:

```bash
export OMNIUS_INFERENCE_FALLBACK=off
```

Or replace the configured provider list using environment-only references:

```bash
export OMNIUS_INFERENCE_FALLBACK_PROVIDERS='[{"providerId":"groq","model":"YOUR_GROQ_MODEL_ID","apiKeyEnv":"GROQ_API_KEY"}]'
```

Each key resolves on the host and goes only to its own registered provider's
pinned HTTPS authentication header. Authentication necessarily sends that
provider its own key. Keys are excluded from prompts, URLs, discovery traffic,
status text, and unrelated providers, and authenticated inference refuses HTTP
redirects.

After building the workspace, a synthetic check through the production backend
can verify remote text or native tool calls without contacting local inference:

```bash
node scripts/inference-fallback-smoke.mjs
node scripts/inference-fallback-smoke.mjs --tools --stream
```

These checks send only a synthetic readiness prompt and execute no returned tool.

Before publication, verify the actual bundled CLI against the broker's NVIDIA
recovery HTTP 503 and the manual approval paths, with all HTTP and DNS mocked:

```bash
node scripts/build-publish.mjs
node scripts/verify-inference-fallback-bundle.mjs publish/dist/index.js
```

This check uses the backend binding inside the distributable CLI bundle,
requires useful remote output, and verifies that manual denial or missing
approval makes no remote dispatch. Building and the pre-pack audit reject
bundles missing the fallback and manual approval implementation. A package
version or process restart alone does not prove these features are present.
The verifier requires the package's declared runtime dependencies. It also
accepts the path to `dist/index.js` in an isolated installed package; it does
not install dependencies or start services.

When a fallback model has a smaller context window, Omnius retains the complete
input and reduces only the maximum output allowance to fit that model. It also
honors an advertised output limit. A provider is skipped when the complete
input leaves insufficient room for output; fallback does not delete context
to make a request fit.

## Fail-Closed Custom Endpoints

Omnius may infer a protocol only for a registered provider or an unambiguous
legacy configuration:

- a known host selects its registered protocol;
- a legacy `ollama` backend selects `ollama`;
- a legacy `vllm` backend selects `openai-chat`;
- a custom or ambiguous endpoint must declare its protocol explicitly.

When the protocol is ambiguous, Omnius performs no model listing, health
probe, embeddings request, chat request, or generated-token request. It keeps
the saved endpoint disabled and returns an actionable configuration error.
It never falls back to Ollama, CPU, a different provider, or a guessed
OpenAI-compatible path.

## Hardware Gate

The repository's `AGENTS.md` requires a live hardware preflight before local
inference. Confirm the designated endpoint, exact model tag, actual GPU
placement with `nvidia-smi`, and absence of the test runner on an unintended
device before generating tokens. If the endpoint-to-GPU mapping cannot be
verified, stop instead of falling back.

## Verify Without Generating Tokens

Use discovery and configuration inspection first. Endpoint/model probes may
contact a provider and therefore still require the correct protocol and
credential. A local model-load probe must obey the hardware gate.

For a service integration, also require the minimum Omnius version described
in [Service Version Compatibility](../operations/version-compatibility.md).
