--- name: warlock-js-ai-live description: 'Video generation and realtime voice sessions for Warlock AI. Importing this package mounts `ai.video` and `ai.realtime`; exports `video`, `realtime`, `VideoModelContract`, `RealtimeTransport`, `MockVideoModel` and `MockRealtimeTransport`. Use for "generate a video", "duplex voice", audio events or per-second costs. Not this package → @warlock.js/ai for chat, still images, speech and one-shot transcription.' --- # Live rich-media — `ai.video` + `ai.realtime` (`@warlock.js/ai-live`) `@warlock.js/ai` ships the synchronous modality verbs (`ai.agent`, `ai.image`, `ai.speech`, `ai.transcribe`). `ai-live` adds the two that need heavyweight, demand-gated machinery — a **long-running job** (video) and a **persistent network session** (realtime voice) — kept in their own package so the core stays dependency-light. Both mount onto the same `ai.*` facade by a **side-effect import**: ```ts import "@warlock.js/ai-live"; // lights up ai.video + ai.realtime on the shared Ai facade import { ai } from "@warlock.js/ai"; ``` The mount mirrors how `@warlock.js/ai-tools` mounts `ai.mcp` and `@warlock.js/ai-workspace` mounts `ai.workspace`: a `declare module "@warlock.js/ai"` interface-merge plus `ai.video = video; ai.realtime = realtime` at load. Forget the side-effect import and `ai.video` is a compile-time `undefined` — not a silent runtime miss. ## `ai.video` — text-to-video, uniform envelope Prompt in / one video out. The adapter hides the provider's async **submit→poll** lifecycle and resolves only when the clip is ready, so the verb returns the framework's uniform **never-throws** `{ data, error, usage, report }` — the exact shape `ai.image` returns, so video spend and traces fold into the same dashboards. ```ts import { ai } from "@warlock.js/ai"; const { data, error, usage, report } = await ai.video({ model: sora.video({ name: "sora-2", pricing: { perSecond: 0.1 } }), // VideoModelContract prompt: "a timelapse of a city skyline at dusk, cinematic", durationSeconds: 8, aspectRatio: "16:9", }); if (error) { console.warn(error.code); // typed AIError — NEVER thrown } else { const clip = data.video; // GeneratedVideo (discriminated) if (clip.type === "url") download(clip.url); else save(Buffer.from(clip.base64, "base64"), clip.mediaType); } ``` ### Shape ```ts type VideoResult = ExecuteResult & { type: "video"; report: VideoReport }; type VideoData = { video: GeneratedVideo }; type GeneratedVideo = | { type: "url"; url: string; mediaType?: string } | { type: "base64"; base64: string; mediaType: string }; // VideoParams (provider-neutral) — the model is the only required field beyond prompt: await ai.video({ model, // VideoModelContract from an adapter's video({ name }) prompt: "...", durationSeconds: 8, // requested clip length aspectRatio: "9:16", resolution: "1080p", // hint negativePrompt: "blurry, watermark", signal, // AbortSignal → status "cancelled" observe: collector, // route the report to an Observer (panoptic), like agents sessionId: "campaign-42", name: "hero-clip", // report node name (defaults to "video") options: { seed: 7 }, // provider-specific escape hatch, forwarded verbatim }); ``` `VideoModelContract` is the moving-image sibling of `ImageModelContract` — `{ name, provider, pricing?, generate(prompt, options) }` — produced by an adapter's `video()` factory. An adapter with no video API simply doesn't define `video()`. ## Video cost-truth — per-second first `VideoModelPricing` carries `{ perSecond?, input?, output? }`. Per-second wins when set (video is metered by clip length): `usage.cost = { input: 0, output: durationSeconds × perSecond }`, attributed to `cost.output`. A token-metered model falls back to the standard `computeCost` against reported tokens. No usable pricing → `usage.cost` stays `undefined` (honest "cost unknown", never a false zero). A pre-priced adapter response is honored, not overwritten. The final `durationSeconds` from the provider (not just the requested one) drives the math and lands on `report.durationSeconds`. ## `ai.realtime` — a duplex voice SESSION (not a one-shot) Unlike every other `ai.*` verb, `ai.realtime()` returns a **stateful, long-lived session** — its closest sibling is `ai.orchestrator`. You open it over a pluggable **`RealtimeTransport`** (the low-level connection to the provider's realtime endpoint), push microphone audio / text turns in, consume an async **event stream** out, and `close()` to end it and receive a `RealtimeReport` for the cost/observability surfaces. ```ts import { ai } from "@warlock.js/ai"; const session = await ai.realtime({ transport: openAiRealtime({ apiKey }), // RealtimeTransport (own adapter) model: "gpt-realtime", voice: "alloy", instructions: "You are a friendly phone receptionist.", sessionId: "call-8891", }); session.sendAudio(micChunkBase64, "audio/pcm"); // push mic audio upstream session.sendText("Please hold for one moment."); // or a text turn for await (const event of session.events()) { // duplex output switch (event.type) { case "audio": speaker.write(event.base64); break; case "transcript": if (event.final) log(event.role, event.text); break; case "tool-call": await handleTool(event.name, event.input); break; case "error": console.warn(event.error.code); break; case "done": break; } } const report = await session.close(); // RealtimeReport — idempotent ``` ### Session contract & event union ```ts interface RealtimeSession { sendAudio(base64: string, mediaType: string): void; sendText(text: string): void; events(): AsyncIterable; close(): Promise; // idempotent — second close() returns the first report } type RealtimeEvent = | { type: "audio"; base64: string; mediaType: string } | { type: "transcript"; role: "user" | "assistant"; text: string; final: boolean } | { type: "tool-call"; id: string; name: string; input: unknown } | { type: "error"; error: AIError } // typed, non-fatal | { type: "done" }; // server ended the session type RealtimeReport = { runId: string; rootRunId: string; type: "realtime"; name: string; status: "completed" | "failed" | "cancelled"; startedAt: string; endedAt: string; duration: number; sessionId?: string; }; ``` The transport seam is what lets `ai-live` ship the session surface without hard-wiring `ws` (it's an optional peer). A concrete transport implements just two interfaces: `RealtimeTransport.connect(config) → RealtimeConnection` (with `sendAudio` / `sendText` / `events()` / `close()`). ## Pattern — a live phone receptionist over a channel ```ts import "@warlock.js/ai-live"; import { ai } from "@warlock.js/ai"; async function handleCall(caller: PhoneChannel, transport: RealtimeTransport) { const session = await ai.realtime({ transport, model: "gpt-realtime", voice: "alloy", instructions: "Greet the caller and route them to the right department.", sessionId: caller.id, }); // Pump caller audio → session (fire-and-forget). caller.onAudio((chunk) => session.sendAudio(chunk, "audio/pcm")); // Pump session output → caller, until the model signals done. for await (const event of session.events()) { if (event.type === "audio") caller.playAudio(event.base64); if (event.type === "done") break; } const report = await session.close(); metrics.record({ runId: report.runId, seconds: report.duration / 1000 }); } ``` ## Testing — mocks, no HTTP, no sockets `@warlock.js/ai-live` exports deterministic doubles so both verbs test offline. ```ts import { ai } from "@warlock.js/ai"; import { MockVideoModel, MockRealtimeTransport } from "@warlock.js/ai-live"; // Video — script the returned clip / usage / duration; assert cost math + recorded calls. const model = new MockVideoModel("mock-video", [{ durationSeconds: 8 }], { perSecond: 0.1 }); const { data, usage } = await ai.video({ model, prompt: "x", durationSeconds: 8 }); // data.video → { type: "url", url: "https://mock/video.mp4", ... } // usage.cost.output === 0.8 (8 × 0.1) // model.calls[0] → { prompt: "x", options: { durationSeconds: 8, ... } } // Realtime — script the outbound event stream; assert what the session sent. const transport = new MockRealtimeTransport([ { type: "transcript", role: "assistant", text: "Hi!", final: true }, { type: "done" }, ]); const session = await ai.realtime({ transport, model: "mock-realtime" }); session.sendText("hello"); for await (const e of session.events()) { /* ... */ } const report = await session.close(); // transport.lastConnection.sentText → ["hello"] // transport.connectConfigs[0] → { model: "mock-realtime", ... } // report.status === "completed" ``` Feed `MockVideoModel` an `{ error }` response and `ai.video` lands it on `result.error` (never throws), setting `report.status` to `"failed"` (or `"cancelled"` when the signal aborted). ## Status — contracts now, transports next `4.6.0` introduces the package: the `VideoModelContract` / `RealtimeSession` contracts, the `ai.video()` verb (uniform envelope + per-second cost-truth, tested against `MockVideoModel`), and the `ai.realtime()` session primitive over a pluggable transport (tested against `MockRealtimeTransport`). The first **concrete provider transports** — an OpenAI Realtime **WebSocket** for `ai.realtime`, and **Sora / Veo** video adapters for `ai.video` — are the next implementation step; the seams are defined so they drop in without changing the verb surface. ## See also - [[generate-images]] — still-image OUTPUT (`ai.image`), the synchronous sibling `ai.video` mirrors. - [[generate-speech]] — one-shot text-to-speech / transcription (`ai.speech` / `ai.transcribe`); use `ai.realtime` instead when you need a live duplex conversation, not a single audio render. - The `observe-ai-flows` topic of the `warlock-js-ai` skill — the `observe` seam both verbs route their reports through. - The `run-orchestrator` topic of the `warlock-js-ai` skill — the other session-shaped primitive `ai.realtime` resembles.