# Voice Agents Build real-time voice agents with speech-to-text, text-to-speech, and conversation persistence. Audio streams over WebSocket — no SFU or meeting infrastructure required. ## Overview `@cloudflare/voice` provides two server-side mixins and matching React hooks: | Export | Import | Purpose | | ---------------- | -------------------------- | -------------------------------------------- | | `withVoice` | `@cloudflare/voice` | Full voice agent: STT, LLM, TTS, persistence | | `withVoiceInput` | `@cloudflare/voice` | STT-only: transcription without response | | `useVoiceAgent` | `@cloudflare/voice/react` | React hook for `withVoice` agents | | `useVoiceInput` | `@cloudflare/voice/react` | React hook for `withVoiceInput` agents | | `VoiceClient` | `@cloudflare/voice/client` | Framework-agnostic client | Built on Cloudflare Durable Objects, you get: - **Real-time audio** — mic audio streams as binary WebSocket frames, TTS audio streams back - **Automatic conversation persistence** — messages stored in SQLite, survive restarts - **Streaming TTS** — LLM tokens are sentence-chunked and synthesized concurrently - **Interruption handling** — user speech during playback cancels the current response - **Continuous STT** — per-call transcriber session, model handles turn detection - **Pipeline hooks** — intercept and transform text at every stage > **Experimental.** This API is under active development and will break between releases. Pin your version. The Voice mixins extend the Agents SDK. Its documentation is available at `agents/docs/index.md`. ## Quick Start ### Install ```sh npm install @cloudflare/voice agents ``` ### Server ```typescript import { Agent } from "agents"; import { withVoice, WorkersAIFluxSTT, WorkersAITTS, type VoiceTurnContext } from "@cloudflare/voice"; const VoiceAgent = withVoice(Agent); export class MyAgent extends VoiceAgent { transcriber = new WorkersAIFluxSTT(this.env.AI); tts = new WorkersAITTS(this.env.AI); async onTurn(transcript: string, context: VoiceTurnContext) { return "Hello! I heard you say: " + transcript; } } ``` ### Client (React) ```tsx import { useVoiceAgent } from "@cloudflare/voice/react"; function VoiceUI() { const { status, transcript, interimTranscript, audioLevel, isMuted, startCall, endCall, toggleMute } = useVoiceAgent({ agent: "MyAgent" }); return (

Status: {status}

{interimTranscript && (

{interimTranscript}

)} {transcript.map((msg, i) => (

{msg.role}: {msg.text}

))}
); } ``` ### Wrangler Config ```jsonc // wrangler.jsonc { "ai": { "binding": "AI" }, "durable_objects": { "bindings": [{ "name": "MyAgent", "class_name": "MyAgent" }] }, "migrations": [{ "tag": "v1", "new_sqlite_classes": ["MyAgent"] }] } ``` ## How It Works ``` Browser Durable Object (withVoice) ┌──────────┐ ┌──────────────────────────┐ │ Mic │ binary PCM (16kHz) │ Transcriber session │ │ │ ──────────────────────► │ (per-call, continuous) │ │ │ │ ↓ model detects turn │ │ │ JSON: transcript │ onTurn() → your LLM code │ │ │ ◄────────────────────── │ ↓ (sentence chunking) │ │ │ binary: audio │ TTS │ │ Speaker │ ◄────────────────────── │ │ └──────────┘ └──────────────────────────┘ ``` 1. The client captures mic audio and sends it as binary WebSocket frames (16kHz mono 16-bit PCM) 2. Audio streams continuously to the transcriber session (created at `start_call`, lives for the entire call) 3. The STT model detects when the user finishes an utterance and fires `onUtterance` 4. Your `onTurn()` method runs — typically an LLM call 5. The response is sentence-chunked and synthesized via TTS 6. Audio streams back to the client for playback ## Server API: `withVoice` `withVoice(Agent)` adds the full voice pipeline to an Agent class. ### Providers Set providers as class properties. Class field initializers run after `super()`, so `this.env` is available. | Property | Type | Required | Description | | ------------- | ------------- | -------- | -------------------------------- | | `transcriber` | `Transcriber` | Yes | Continuous per-call STT provider | | `tts` | `TTSProvider` | Yes | Text-to-speech | ```typescript import { withVoice, WorkersAIFluxSTT, WorkersAITTS } from "@cloudflare/voice"; const VoiceAgent = withVoice(Agent); export class MyAgent extends VoiceAgent { transcriber = new WorkersAIFluxSTT(this.env.AI); tts = new WorkersAITTS(this.env.AI); } ``` For runtime model switching (e.g. Flux vs Nova 3 dropdown), override `createTranscriber`: ```typescript export class MyAgent extends VoiceAgent { tts = new WorkersAITTS(this.env.AI); createTranscriber(connection: Connection): Transcriber { return new WorkersAIFluxSTT(this.env.AI); } } ``` Custom transcriber sessions can implement `waitUntilReady(): Promise` when they open an upstream streaming connection asynchronously. `withVoice()` waits for this optional method before sending `listening` or running `onCallStart()`. Resolve it when the session can accept audio and emit transcripts; reject it when startup failed and the call should return to `idle`. If `close()` abandons startup while readiness is pending, settle the readiness promise so server startup work does not wait forever. Providers that are ready synchronously can omit the method. ### `onTurn(transcript, context)` **Required.** Called when the user finishes speaking and the transcript is ready. Return a `string`, `AsyncIterable`, or `ReadableStream` for streaming responses: **Simple response:** ```typescript async onTurn(transcript: string, context: VoiceTurnContext) { return "You said: " + transcript; } ``` **Streaming response (recommended for LLM):** ```typescript import { streamText, convertToModelMessages } from "ai"; import { createWorkersAI } from "workers-ai-provider"; async onTurn(transcript: string, context: VoiceTurnContext) { const workersai = createWorkersAI({ binding: this.env.AI }); const result = streamText({ model: workersai("@cf/moonshotai/kimi-k2.7-code"), system: "You are a helpful voice assistant. Keep responses concise.", messages: [ ...context.messages.map(m => ({ role: m.role as "user" | "assistant", content: m.content })), { role: "user", content: transcript } ], abortSignal: context.signal }); return result.fullStream; } ``` The `context` object provides: | Field | Type | Description | | ------------ | ------------------------------------------ | ---------------------------------- | | `connection` | `Connection` | The WebSocket connection | | `messages` | `Array<{ role: string; content: string }>` | Conversation history from SQLite | | `signal` | `AbortSignal` | Aborted on interrupt or disconnect | ### Lifecycle Hooks | Method | Description | | ----------------------------- | ------------------------------------------- | | `beforeCallStart(connection)` | Return `false` to reject the call | | `onCallStart(connection)` | Called after a call is accepted | | `onCallEnd(connection)` | Called when a call ends | | `onInterrupt(connection)` | Called when user interrupts during playback | If startup is rejected or fails, the server sends the client back to `idle` and calls `onCallEnd()` so cleanup logic can run. ### Pipeline Hooks Intercept and transform data at each pipeline stage. Return `null` to skip the current utterance. | Method | Receives | Can skip? | | ------------------------------------------ | --------------- | --------- | | `afterTranscribe(transcript, connection)` | STT text | Yes | | `beforeSynthesize(text, connection)` | Text before TTS | Yes | | `afterSynthesize(audio, text, connection)` | Audio after TTS | Yes | ```typescript export class MyAgent extends VoiceAgent { // Filter out short/noise transcripts afterTranscribe(transcript: string, connection: Connection) { if (transcript.length < 3) return null; // skip return transcript; } // Add SSML or modify text before TTS beforeSynthesize(text: string, connection: Connection) { return text.replace(/\bAI\b/g, "A.I."); // improve pronunciation } } ``` ### Convenience Methods | Method | Description | | -------------------------- | -------------------------------------------- | | `speak(connection, text)` | Synthesize and send audio to one connection | | `speakAll(text)` | Synthesize and send audio to all connections | | `forceEndCall(connection)` | Programmatically end a call | | `saveMessage(role, text)` | Persist a message to conversation history | | `getConversationHistory()` | Retrieve conversation history from SQLite | ### Configuration Options Pass options to `withVoice()` as the second argument: ```typescript const VoiceAgent = withVoice(Agent, { historyLimit: 20, // Max messages loaded for context (default: 20) audioFormat: "mp3", // Audio format sent to client (default: "mp3") sampleRate: 16000, // Sample rate (Hz) for raw pcm16 payloads (default: 16000) maxMessageCount: 1000 // Max messages in SQLite (default: 1000) }); ``` ## Server API: `withVoiceInput` `withVoiceInput(Agent)` adds STT-only voice input — no TTS, no LLM, no response generation. Use this for dictation, search-by-voice, or any UI where you need speech-to-text without a conversational agent. ```typescript import { Agent } from "agents"; import { withVoiceInput, WorkersAINova3STT } from "@cloudflare/voice"; const InputAgent = withVoiceInput(Agent); export class DictationAgent extends InputAgent { transcriber = new WorkersAINova3STT(this.env.AI); onTranscript(text: string, connection: Connection) { console.log("User said:", text); } } ``` ### `onTranscript(text, connection)` Called after each utterance is transcribed. Override this to process the transcript. ### Hooks `withVoiceInput` supports the same lifecycle hooks as `withVoice`: - `beforeCallStart(connection)` — return `false` to reject - `onCallStart(connection)`, `onCallEnd(connection)`, `onInterrupt(connection)` - `createTranscriber(connection)` — override for runtime model switching - `afterTranscribe(transcript, connection)` — filter or transform transcripts If startup is rejected or fails, the server sends the client back to `idle` and calls `onCallEnd()` so cleanup logic can run. It does **not** have TTS hooks (`beforeSynthesize`, `afterSynthesize`) or `onTurn`. ## Client API: React Hooks ### `useVoiceAgent` Wraps `VoiceClient` for `withVoice` agents. Manages connection, mic capture, playback, silence detection, and interrupt detection. ```tsx import { useVoiceAgent } from "@cloudflare/voice/react"; const selectedSpeakerId = "default"; const { status, // "idle" | "listening" | "thinking" | "speaking" transcript, // TranscriptMessage[] — conversation history interimTranscript, // string | null — real-time partial transcript metrics, // VoicePipelineMetrics | null audioLevel, // number (0–1) — current mic RMS level isMuted, // boolean connected, // boolean — WebSocket connected error, // string | null outputDeviceError, // string | null — non-fatal speaker routing issue startCall, // () => Promise endCall, // () => void toggleMute, // () => void sendText, // (text: string) => void — bypass STT sendJSON, // (data: Record) => void lastCustomMessage // unknown — last non-voice message from server } = useVoiceAgent({ agent: "MyAgent", // Required: Durable Object class name name: "default", // Instance name (default: "default") host: window.location.host, // Host to connect to outputDeviceId: selectedSpeakerId, // Optional audiooutput device ID enabled: true // Set false to delay connecting until prerequisites are ready }); ``` Use `enabled: false` when the app must wait for async connection prerequisites, such as a user-scoped capability token. While disabled, the hook does not create or connect a `VoiceClient`, returns the idle/disconnected state, and action callbacks such as `startCall()`, `sendText()`, and `sendJSON()` are safe no-ops. When `enabled` changes to `true`, the hook connects with the current options. The first enable is treated as an initial connection, so `onReconnect` only fires for later connection identity changes while the hook remains enabled. #### Output Device Selection Pass `outputDeviceId` to route assistant playback to a selected speaker when the browser supports `HTMLMediaElement.setSinkId()`: ```tsx const [outputDeviceId, setOutputDeviceId] = useState("default"); const voice = useVoiceAgent({ agent: "MyAgent", outputDeviceId }); ``` Use a `MediaDeviceInfo.deviceId` from `navigator.mediaDevices.enumerateDevices()` where `kind === "audiooutput"`. `"default"` and `undefined` use the system default output. Browsers without sink selection support continue playing through the default output and set `outputDeviceError` when a non-default output is requested. Device labels may be blank until the user grants microphone permission, so refresh device lists after `startCall()` if you show a speaker picker. #### Tuning Options | Option | Type | Default | Description | | -------------------- | -------- | ------- | ------------------------------------------------ | | `silenceThreshold` | `number` | `0.04` | RMS below this is silence | | `silenceDurationMs` | `number` | `500` | Silence duration before `end_of_speech` (ms) | | `interruptThreshold` | `number` | `0.05` | RMS to detect speech during playback | | `interruptChunks` | `number` | `2` | Consecutive high-RMS chunks to trigger interrupt | Changing tuning options triggers a client reconnect (the connection key includes them). ### `useVoiceInput` Lightweight hook for dictation / voice-to-text. Accumulates user transcripts into a single string. ```tsx import { useVoiceInput } from "@cloudflare/voice/react"; function Dictation() { const { transcript, // string — accumulated text from all utterances interimTranscript, // string | null — current partial transcript isListening, // boolean audioLevel, // number (0–1) isMuted, // boolean error, // string | null start, // () => Promise stop, // () => void toggleMute, // () => void clear // () => void — clear accumulated transcript } = useVoiceInput({ agent: "DictationAgent" }); return (