# @odla-ai/ai

> ⚠️ **Early access — pre-1.0.** Agents work from bounded runbooks; humans
> approve credentials, production changes, releases, and merges. APIs and exact
> package availability can change. Review the documented guarantees and
> limitations; this software is MIT-licensed and provided without warranty.

One general-purpose interface for AI inference across **Anthropic (Claude)**,
**OpenAI (GPT)**, and **Google (Gemini)** — text, image, and audio — plus a
tool-use **agent engine** and an **eval harness**. It's the place every odla app
reaches for when it needs AI.

- **Library surface.** Import it in-process and bring your own keys; this
  package exports no generic hosted gateway. Separately, odla's System AI
  broker uses the same facade server-side for bounded o11y/security purposes
  and returns purpose grants rather than provider credentials.
- **One canonical shape.** Anthropic-flavored content blocks, forced tool calls
  with object inputs, a normalized streaming event vocabulary, cache-aware usage,
  and a closed error taxonomy — the same regardless of which provider answers.
- **Capability-checked.** Every request is validated against the model before
  dispatch, so "audio to Claude" fails fast with a clear error instead of an
  opaque provider 400.
- **Server runtimes.** Node 20+ and Cloudflare Workers. Provider adapters are
  runtime-lazy, but all three provider SDKs are package dependencies. Do not put
  long-lived provider keys in browser code; use your own authenticated proxy.

```bash
npm install @odla-ai/ai
```

> **Ask the runbooks first.** odla's operational procedures live in a database,
> not in this file: `npx @odla-ai/cli runbook ask "<question>"` returns the current
> steps, and unlike anything written here it cannot be out of date. Use it before
> searching the web or working from memory. This README and the JSDoc in the
> shipped `.d.ts` are the version-matched **API** reference; a runbook is the
> **procedure**. Most tasks need an answer from both.

## Quick start

```ts
import { init } from "@odla-ai/ai";

const ai = init({
  keys: {
    anthropic: process.env.ANTHROPIC_API_KEY,
    openai: process.env.OPENAI_API_KEY,
    google: process.env.GOOGLE_API_KEY,
  },
  defaultModel: "claude-opus-4-8",
});

// One call, any provider — pick the model, the provider is resolved for you.
const res = await ai.chat({
  model: "gpt-5",
  messages: [{ role: "user", content: "Say hello in one word." }],
  maxTokens: 64,
});
console.log(res.content); // [{ type: "text", text: "Hello" }]
```

### Automatic observability

Every provider request made by `chat`, `stream`, `extract`, or `search` creates
one OpenTelemetry CLIENT span when the host has configured an OTel tracer
provider. The span includes provider/model/operation, all four token classes,
catalog-priced cost (or an explicit `unpriced` marker), provider latency, and
streaming time to first token. With no tracer provider, the OTel API is a no-op.

Use low-cardinality metadata to attribute a call to an app or feature:

```ts
await ai.chat({
  model: "gpt-5",
  messages,
  maxTokens: 512,
  telemetry: {
    operation: "support.answer",
    attributes: { "odla.app": "support", "deployment.environment": "prod" },
  },
});
```

Static defaults belong on `init({ telemetry: ... })`. Set `telemetry: false`
on `init` or on one request to suppress automatic spans. Telemetry metadata is
consumed by the facade and is never sent in a provider-native request.

### Capacity control

`init()` can share a FIFO capacity scheduler across callers. Pools are keyed by
the SHA-256 fingerprint of the active provider key plus the provider-native
model id, so aliases share capacity while credentials remain isolated. The
default 20% headroom makes configured TPM/RPM/TPD limits 80% effective;
estimated tokens advance the next-start clock, and only positive actual-token
excess advances it again.

```ts
const ai = init({
  keys: { openai: process.env.OPENAI_API_KEY },
  capacity: {
    limits: ({ model }) => model === "gpt-5-nano"
      ? { tokensPerMinute: 180_000_000, requestsPerMinute: 30_000,
          tokensPerDay: 15_000_000_000, maxParallelRequests: 64 }
      : undefined,
  },
});
```

Requests beyond the pacing or parallel ceiling wait FIFO. `deadline` includes
queue time and `signal` cancels a queued request.

### Streaming

```ts
for await (const ev of ai.stream({ model: "gemini-2.5-flash", messages: [...], maxTokens: 512 })) {
  if (ev.type === "content_block_delta" && ev.delta.type === "text_delta") {
    process.stdout.write(ev.delta.text);
  }
}
```

The provider event union (`message_start → content_block_start →
content_block_delta → content_block_stop → message_delta → message_stop`, plus
`error`) is identical across providers. Platform-hosted SSE adds a final
`receipt` event containing usage, cost, queue wait, TTFT, and total latency.

### Structured extraction (forced tool call)

```ts
const { value } = await ai.extract<{ sentiment: "positive" | "negative" | "neutral" }>({
  model: "claude-haiku-5-5",
  user: "Classify: 'I love this!'",
  tool: {
    name: "classify",
    parameters: {
      type: "object",
      properties: { sentiment: { type: "string", enum: ["positive", "negative", "neutral"] } },
      required: ["sentiment"],
    },
  },
});
// value.sentiment === "positive" (parsed and validated before it is returned)
```

`extract` and local agent tools validate provider arguments before returning or
executing them. The dependency-free validator supports the common tool-schema
subset: local `$ref`/`$defs`, `type`, `enum`, `const`, composition, object
properties/required/additional properties, arrays, and string/number constraints.
Unknown or unsupported schema keywords outside that documented subset fail
closed. `extract()` throws `ToolInputError`; local agent tools return an error
`tool_result` and their handlers do not execute.

### Web search

```ts
const { text, citations } = await ai.search({ model: "claude-sonnet-5", user: "Who won the 2026 World Cup?" });
```

`citations` is the normalized source list, parsed from Anthropic's
`web_search_tool_result` blocks and inline text citations, OpenAI's
`url_citation` annotations, and Gemini's `groundingMetadata`. A citation
carrying `citedText` or a `startIndex`/`endIndex` span was attributed to a
specific claim; one carrying only a `url` was merely consulted, which is much
weaker evidence that the answer came from there. It is `[]` when the provider
reported nothing — a grounded-*looking* answer is not the same as a sourced one.

Two caveats worth knowing before you compare providers: Gemini reports
`vertexaisearch.cloud.google.com/grounding-api-redirect/…` links rather than
publisher URLs, so hosts only line up after you resolve those redirects; and
streamed responses carry no citations, because the event set has no citation
event yet.

## Multimodal: text, image, audio

Content blocks carry the modality. Image is broadly supported; **audio is
accepted by OpenAI and Google, but not Anthropic** — sending an `audio` block to
a Claude model throws a `CapabilityError` before any network call.

```ts
// Image (works on Claude, GPT, Gemini)
await ai.chat({
  model: "gpt-4o",
  messages: [{ role: "user", content: [
    { type: "image", source: { type: "base64", mediaType: "image/png", data: pngB64 } },
    { type: "text", text: "What's in this image?" },
  ] }],
  maxTokens: 128,
});

// Audio (OpenAI: gpt-audio; Google: gemini-2.5-*)
await ai.chat({
  model: "gpt-audio",
  messages: [{ role: "user", content: [
    { type: "audio", source: { type: "base64", mediaType: "audio/wav", data: wavB64 } },
    { type: "text", text: "Transcribe this." },
  ] }],
  maxTokens: 256,
});
```

## Agents: build, run, and test

A **persona** is a role as config-as-data: a model, a system prompt, its tools,
and reasoning knobs. `runAgent` drives the tool-use loop (model turn → run tool
handlers → feed results back → repeat) until the model stops or a step limit hits.

```ts
import { init, runAgent, type Persona, type ToolDef } from "@odla-ai/ai";

const ai = init({ keys: { anthropic: process.env.ANTHROPIC_API_KEY } });

const add: ToolDef = {
  name: "add",
  description: "Add two integers.",
  inputSchema: { type: "object", properties: { a: { type: "number" }, b: { type: "number" } }, required: ["a", "b"] },
  handler: (input) => ({ content: String((input.a as number) + (input.b as number)) }),
};

const calculator: Persona = {
  name: "calculator",
  model: "claude-opus-4-8",
  system: "Use the add tool for arithmetic, then state the result.",
  tools: [add],
  maxSteps: 4,
};

const run = await runAgent(ai, calculator, { input: "What is 21 + 21?" });
console.log(run.finalText);   // "…42"
console.log(run.toolCalls);   // [{ toolUse, output }]
console.log(run.usage);       // accumulated across turns
```

Provider calls and tool handlers receive a combined cancellation signal. Runs
can also stop before additional effects when a run-wide budget is exhausted:

```ts
const run = await runAgent(ai, calculator, {
  input: "Reconcile these records",
  signal: request.signal,
  deadline: Date.now() + 30_000,
  budget: { maxInputTokens: 20_000, maxOutputTokens: 4_000, maxToolCalls: 8 },
});
```

Token limits are post-turn ceilings because provider input usage is known only
after a response. `maxInputTokens` and `maxTotalTokens` therefore cannot
tokenizer-bound the first prompt; they stop later turns and tool effects once
reported usage reaches the ceiling. `maxOutputTokens` and the remaining total
budget cap each request's `maxTokens` before dispatch. Limits must be integers
(`maxToolCalls` may be zero; token limits must be positive). When a limit blocks
tools, the loop records a matching error `tool_result` for every requested tool
before persisting memory, so retries never inherit an unmatched effect request.
`stoppedReason` is `"budget_exhausted"` when a limit prevents further work.

Tools can declare taint constraints (`outputTaint` / `acceptsTaint`) for a
lightweight CaMeL-style prompt-injection gate, and personas can carry a pluggable
`MemoryScope`.

### Skills

A **skill** is a named bundle of tools + instructions. Attach skills to a persona
and their tools are merged into the model's tool list and their instructions into
the system prompt — the primitive for "add a skill and call it as tools."

```ts
import { entityCrudSkill, runAgent } from "@odla-ai/ai";
import { init as odlaInit } from "@odla-ai/db";

const db = odlaInit({ appId, adminToken: appKey, endpoint: ODLA_URL });

// Turn an odla-db entity into list/create/update/delete tools that change data.
const todos = entityCrudSkill({
  db,
  entity: "todos",
  fields: { text: { type: "string" }, done: { type: "boolean" } },
  required: ["text"],
});

const assistant = { name: "todo-bot", model: "claude-opus-4-8", skills: [todos] };
const run = await runAgent(ai, assistant, { input: "Add 'buy milk' and show my open todos." });
// the agent calls create_todo then list_todos, writing/reading via odla-db
```

`entityCrudSkill` generates `list_<plural>`, `create_<singular>`, `update_<singular>`
(partial — pass only the fields that change, e.g. toggle `done`), and
`delete_<singular>` tools whose handlers call `db.transact`/`db.query`. Or hand-roll
a `Skill` = `{ name, instructions?, tools: ToolDef[] }` for anything else.

### Evals

```ts
import { evaluate, exactMatch, llmJudge } from "@odla-ai/ai";

const report = await evaluate({
  inference: ai,
  model: "claude-opus-4-8",
  system: "Answer with only the number.",
  grader: exactMatch(),
  cases: [
    { input: "What is 2+2?", expected: "4" },
    { input: "What is 10-3?", expected: "7" },
  ],
});
console.log(`${report.passed}/${report.total} passed`);
```

Graders: `exactMatch`, `includes`, `structuredMatch`, and `llmJudge` (which
dogfoods the inference core to grade open-ended output against a rubric). Pass a
`persona` instead of a `model` to evaluate a full agent.

### Harness: configuration as data, and proof it was sent

A `Persona` carries live tool handlers, so it can be written in TypeScript but
not in YAML, a database row, or a run record. A `HarnessSpec` is the same
configuration as plain JSON; a `ToolsetRegistry` holds the code, and `toPersona`
joins them.

```ts
import { toPersona, resolveHarness, verifyEcho } from "@odla-ai/ai";

const harness = { id: "fast", model: "gpt-5.5", effort: "low", toolset: "search" };

// Can this model run this configuration, exactly as written?
const res = resolveHarness(harness);
if (!res.ok) return skip(res.code); // "effort_unsupported", "temperature_ignored", …

const run = await runAgent(ai, toPersona(harness, { search: [searchTool] }), { input: q });

// Did the adapter actually send what was asked for?
const drift = verifyEcho(harness, res.spec, run.response.echo!);
if (drift.length) throw new Error(drift[0].detail);
```

`resolveHarness` answers rather than throws, so a caller enumerating many
model × setting combinations can mark the illegal ones skipped instead of
wrapping every one in `try`/`catch`. It has no "resolved with adjustments"
result: a harness runs as specified or it does not run.

`response.echo` reports what the adapter *actually built* — native model id,
the reasoning knob really sent, whether thinking was on, whether temperature
survived. Comparing it to the request (`verifyEcho`) is what catches a silent
substitution introduced below your code, which upfront validation cannot see.
One it still surfaces today: reasoning-tier models on Chat Completions drop
`temperature`. That is correct behavior, and it still makes a run
non-comparable to one that ran on the model's default.

### Per-step trace

`AgentRun.usage` says what a run cost in total; `AgentRun.trace` says where it
went — one `StepRecord` per model turn, carrying that turn's usage, `costUsd`
(absent when the model's pricing is unknown, never zero), `durationMs`,
`stopReason`, the tool calls it triggered, and its `echo`.

```ts
const run = await runAgent(ai, persona, { input: q });

for (const step of run.trace) {
  console.log(step.index, step.durationMs, step.costUsd, step.toolCalls.length);
}

// A mid-run substitution a final-turn echo cannot see:
const lostReasoning = run.trace.filter((s) => s.echo?.thinkingActive === false);
```

Pass `now` on the run input to inject a clock; it defaults to `Date.now`.

### Which OpenAI endpoint a model uses

OpenAI models are called on Chat Completions by default and on `/v1/responses`
when the model needs it. That routing exists for one reason: Chat Completions
refuses function tools whenever reasoning is in play on the gpt-5.6 generation,
so the adapter used to disable reasoning to keep tools working. `/v1/responses`
carries both, so those models go there instead and keep their reasoning. It also
accepts `temperature` alongside an effort level, and its reasoning vocabulary
covers the full `low`→`max` ladder rather than collapsing the top rungs.

Set `api: "chat" | "responses"` on a `ModelSpec` to pin the endpoint; omit it to
let the adapter choose. Every built-in catalog model stays on Chat Completions —
models move by explicit declaration, not by a wholesale switch.

## Keys & roles

Keys are **BYO** — a static per-provider map and/or a dynamic `resolveKey`
(multi-tenant / per-call). A provider with no resolvable key is simply
unavailable: requesting one of its models throws a `ConfigError`.

```ts
const ai = init({
  resolveKey: (provider) => lookupTenantKey(currentTenant, provider),
});
```

"Roles" are personas (above) — config-as-data, not RBAC.

## Storage, memory & BYO keys via odla-db

odla-ai follows odla-db's model: it's a self-contained npm package whose README
and exported TypeScript declarations/JSDoc describe the installed API, and it
integrates with `@odla-ai/db` for durable storage — **without taking a runtime
dependency on it**. You install `@odla-ai/db`, build a client, and inject it;
odla-ai talks to it through a structural interface. The rendered public
reference is at <https://odla.ai/docs/packages/ai>.

The agent handshake (an agent never takes a platform secret):

```ts
import { requestToken, init as odlaInit } from "@odla-ai/db";
import { provisionAgentApp, odlaDbKeyResolver, OdlaDbMemory, persistRun, runAgent, init } from "@odla-ai/ai";

const ODLA_PLATFORM = "https://odla.ai";
const ODLA_DB_URL = "https://db.odla.ai";

// 1. Device authorization — the matching account signs in, reviews, and
//    approves the exact code; the agent receives a revocable platform token.
const { token } = await requestToken({
  endpoint: ODLA_PLATFORM,
  email: process.env.ODLA_USER_EMAIL!, // identity hint, not a password or token
  onCode: ({ userCode, verificationUriComplete }) => console.log(verificationUriComplete ?? userCode),
});

// 2. Provision: create app, mint an app key, push the agent schema, and store the
//    user's BYO provider keys as odla-db secrets (encrypted at rest).
const { appId, appKey } = await provisionAgentApp({
  endpoint: ODLA_PLATFORM,
  token,                                   // operator/dev token — needed to WRITE secrets
  providerKeys: { openai: OPENAI_KEY, anthropic: ANTHROPIC_KEY },
});

// 3. Runtime: read keys from secrets with the app key, persist memory + run traces.
const db = odlaInit({ appId, adminToken: appKey, endpoint: ODLA_DB_URL });
const ai = init({ resolveKey: odlaDbKeyResolver(db) });   // keys come from odla-db secrets

const persona = { name: "assistant", model: "gpt-5", memory: new OdlaDbMemory({ db, sessionId: "user-42" }) };
const run = await runAgent(ai, persona, { input: "Pick up where we left off." });
await persistRun(run, { db, sessionId: "user-42" });
```

**BYO keys as secrets** — the write/read split mirrors odla-db: storing a secret
is a *privileged* action (operator or `odla_dev_` token, via `provisionAgentApp` /
`putSecret`); reading it at request time uses the ordinary **read-only** app key
(`odlaDbKeyResolver` → `db.secrets.get`). Keys are AES-GCM-encrypted at rest and
never returned to browser clients. `OdlaDbMemory` persists conversation turns and
`persistRun` writes run traces, both as natural-key upserts (idempotent on retry).

### Platform apps (odla.ai)

For apps registered on odla, **hosted AI is the normal path**. Admins choose a
priced model allowlist and guardrails; each app chooses one of those models.
The app uses its existing backend key, while provider credentials stay in the
platform vault. Legacy and explicitly selected BYOK configs remain supported.

```ts
import { initFromPlatform } from "@odla-ai/ai";
import { init as odlaInit } from "@odla-ai/db";

const db = odlaInit({ appId: TENANT, adminToken: env.ODLA_API_KEY, endpoint: "https://db.odla.ai" });
const { ai, provider, model, revision } = await initFromPlatform({
  platform: "https://odla.ai",
  appId: APP_ID,          // the registry app id, not the tenant
  env: "prod",
  appKey: env.ODLA_API_KEY,
  db,                     // needed only if this app is configured for BYOK
});
```

The anonymous `public-config` projection carries the mode, attested config and
policy revisions, default model, and only the approved model catalog. It is
cached for about 60 seconds, so model changes need no redeploy. Hosted calls go
to the Registry broker, which reserves quota before provider I/O, applies
server-side output/deadline bounds, prices normalized token usage, and records a
90-day metadata ledger by app/environment/model/status. Prompts are not stored
in that ledger. Non-streaming idempotent retries use a separate response receipt
bounded to 512 KiB; streaming responses are not replayed.

Hosted inference sends the request's prompts, messages, tool definitions/results,
and attachments to the selected third-party model provider. Do not send data
that the app is not authorized to disclose to that provider. `providerExtras`
is rejected in hosted mode so app code cannot route around platform policy.

BYOK still reads the configured provider key from `db.secrets` and filters the
facade to that provider. **Never put provider keys in a Worker var.**

### 0.3 migration notes

- Tool/extraction JSON Schemas now fail closed. `extract()` throws
  `ToolInputError`; `runAgent()` records an error `tool_result` and does not run
  a local handler when its arguments do not validate.
- Platform calls cannot cross the provider selected in Studio.
- `CancelledError` (`cancelled`) and `DeadlineExceededError`
  (`deadline_exceeded`) distinguish caller limits from provider failures.
- Invalid run budgets now throw `ConfigError`; budget stops persist paired tool
  results and may therefore add one final user/tool-result turn to memory.

## Models & capabilities

The catalog maps a canonical model id → provider, native id, and capabilities.
The built-in `DEFAULT_CATALOG` covers current Anthropic, OpenAI, and Google
models; extend or override it via `buildCatalog` / `init({ catalog })`.

| | image in | audio in | tools | thinking | effort | web search | structured |
|---|---|---|---|---|---|---|---|
| Claude (opus/sonnet/fable) | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Claude Haiku | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ |
| GPT-5 line | ✅ | ❌ | ✅ | — | ✅ | — | ✅ |
| GPT-4o | ✅ | ❌ | ✅ | — | ❌ | — | ✅ |
| gpt-audio | ❌ | ✅ | ✅ | — | ❌ | — | ✅ |
| Gemini 2.5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

> **Model ids drift.** The catalog is data — verify ids against each provider's
> current docs (the `claude-api` skill for Anthropic; `/v1/models` for OpenAI;
> Google GenAI docs for Gemini) rather than trusting them from memory.

## Pricing

Every catalog entry carries USD-per-MTok `pricing`, so a call's cost is a
lookup plus arithmetic:

```ts
import { priceCall, modelPricing } from "@odla-ai/ai/pricing";

const usd = priceCall("claude-opus-4-8", res.usage); // number | undefined
```

`priceCall` returns `undefined` — never `0` — for a model with no known
pricing, so missing catalog data can't masquerade as a free call. Cache-read
and cache-write tokens are billed at their own rates when the model declares
them, falling back to the input rate otherwise.

Some models charge a second, higher rate card once the prompt passes a length:
Claude Haiku 5.5 over 100K tokens, the GPT-5.4-and-later frontier models over
272K, Gemini Pro over 200K. Those entries carry it as `pricing.longContext`
(`aboveTokens` plus the same four rates), and `priceCall` / `costUsd` apply it
for you. Two things follow from how providers bill it:

- The higher rates apply to the **whole request**, not the tokens past the
  threshold, and the prompt length that selects them counts cache reads and
  cache writes as well as uncached input.
- So price **one request at a time** and add the dollars. Pricing a sum of
  several requests' usage would bill short prompts as one long one.

```ts
import { promptTokens, ratesForPrompt } from "@odla-ai/ai/pricing";

const pricing = modelPricing("claude-haiku-5-5")!;
ratesForPrompt(pricing, promptTokens(res.usage)); // the rate card this call paid
```

Import from **`@odla-ai/ai/pricing`** rather than the package root when you only
need to price calls: the subpath pulls in the catalog and nothing else — no
client, no adapters, and none of the three provider SDKs. That keeps cost
accounting (e.g. `@odla-ai/o11y`) small enough to sit in a Worker bundle, and
means there is one price list rather than a second one that drifts.

## Errors

Every error is an `OdlaAIError` subclass carrying a stable `code` — branch on
`err.code`, not message text: `ConfigError`, `AuthError`, `RateLimitError`,
`CapabilityError`, `ContextWindowError`, `InvalidRequestError`, `ToolInputError`,
`CancelledError`, `DeadlineExceededError`, `ProviderError`.

`OdlaAIError.usage` is present only when a provider response was already
received and a later forced-tool or schema check failed. It is a frozen,
sanitized token-usage record so accounting can charge a failed structured
response accurately; it may be absent for transport, authentication, timeout,
or preflight failures. `toShape()` and streaming `OracleErrorShape` deliberately
omit it.

## Develop

```bash
npm run typecheck   # tsc --noEmit (strict)
npm test            # vitest — offline, mock-provider based
npm run build       # tsup → dual ESM + CJS + d.ts
npm run smoke       # live check against real providers (needs .dev.vars)
```

For the live smoke, copy `.dev.vars.example` → `.dev.vars` and fill in whatever
keys you have; providers without a key are skipped.