---
title: "Voice Input"
description: "Voice dictation in the agent chat composer — Builder Gemini, BYOK providers, and browser Web Speech fallback."
---

# Voice Input

Every agent-native app has a microphone in the chat composer. Click it, talk, and your words get transcribed into the prompt. Useful on mobile, useful for long prompts, useful when your hands are on something else.

The framework handles all of this automatically. Builder-connected users get Builder-hosted Gemini Flash-Lite by default; otherwise users can bring their own provider key or fall back to browser speech recognition.

## How it works {#how-it-works}

The composer's voice button records audio in the browser, then picks a provider:

1. **Builder Gemini Flash-Lite (default when Builder is connected).** The browser POSTs audio to `/_agent-native/transcribe-voice`, which proxies through Builder.io using Gemini Flash-Lite. No Google API key required.
2. **BYOK cloud providers.** Users can choose Google Gemini, Groq Whisper, or OpenAI Whisper from Settings. The route resolves user-scoped encrypted secrets before shared deployment credentials.
3. **Browser Web Speech API (fallback).** If no server provider is available, the composer can use the browser's built-in speech recognition. Works in Chromium-based browsers (Chrome, Edge, Arc) and Safari. Less accurate; streams live.

Provider choice is stored in application state under `voice-transcription-prefs` so the user can force `"auto"` (default — picks the best available provider), `"builder-gemini"`, `"gemini"`, `"groq"`, `"openai"`, or `"browser"` in the sidebar settings. `"builder"` is a legacy alias that resolves to `"builder-gemini"` — new code should write `"builder-gemini"` directly.

<Diagram id="doc-block-1q55wvc" title="Voice transcription provider fallback" summary={"The composer records audio, then walks server providers in order, dropping to the browser Web Speech API only when no server provider is available."}>

```html
<div class="diagram-voice">
  <div class="diagram-node">
    Mic button<br /><small class="diagram-muted">records webm/opus</small>
  </div>
  <div class="diagram-arrow diagram-muted" aria-hidden="true">&rarr;</div>
  <div class="diagram-card col">
    <div class="diagram-pill accent">1 &middot; Builder Gemini</div>
    <small class="diagram-muted">default when Builder connected</small>
    <div class="diagram-pill">2 &middot; BYOK cloud</div>
    <small class="diagram-muted"
      >Gemini &middot; Groq &middot; OpenAI Whisper</small
    >
  </div>
  <div class="diagram-arrow diagram-warn" aria-hidden="true">&darr;</div>
  <div class="diagram-box diagram-warn" data-rough>
    3 &middot; Browser Web Speech<br /><small class="diagram-muted"
      >fallback on 400 &middot; streams live</small
    >
  </div>
</div>
```

```css
.diagram-voice {
  display: flex;
  align-items: center;
  gap: 12px;
  flex-wrap: wrap;
}
.diagram-voice .col {
  display: flex;
  flex-direction: column;
  gap: 6px;
  padding: 14px;
}
.diagram-voice .diagram-arrow {
  font-size: 22px;
  line-height: 1;
}
```

</Diagram>

The route is **same-origin only** — cross-site POSTs are rejected so an attacker can't burn transcription credits from an external page.

## Enabling Providers {#enabling-providers}

Builder is the easiest path: connect Builder.io from Settings and the default provider becomes Builder Gemini Flash-Lite. For BYOK providers, add the matching key in Settings → API Keys.

### Per-user (recommended for SaaS)

The user sets their own key via the agent sidebar settings UI. It's stored as a user-scoped encrypted secret (via `readAppSecret`). Each user pays for their own transcription; zero cost to the host.

### Shared (for internal tools)

Set `GEMINI_API_KEY`, `GROQ_API_KEY`, or `OPENAI_API_KEY` as an environment variable, or have a workspace admin add it in Settings → API Keys (stored as a shared, encrypted workspace secret). Every user's transcription hits the shared key.

<Callout id="doc-block-1jgl027" tone="info">

**Credential resolution order:** the route checks the user's own encrypted secret first, then the shared deployment key. A power user with their own key always overrides the shared one. If neither exists, the route returns a 400 the composer recognizes and silently falls back to browser Web Speech.

</Callout>

## Realtime speech mode {#realtime-speech-mode}

Beyond one-shot dictation, the composer also offers a live, hands-free conversation mode. When the mic opens, a popover lets you choose **Realtime speech** or **Editable dictation**. Realtime speech uses a connected Builder account automatically (same as dictation), or an `OPENAI_API_KEY` if Builder isn't connected — connects the browser to `gpt-realtime-2.1` over WebRTC. The chat collapses into a speech orb in the bottom-end corner; click the orb to show or hide chat without ending the conversation. The orb stays visible above the chat even if the chat opens automatically, and its compact waveform reacts to your voice and the assistant's voice.

The speech model uses the same authenticated Agent Native tools as chat, so it can navigate the app and call actions while it talks. Approval-gated actions remain paused until you approve them. Only completed user and assistant text turns are appended to the original chat thread. Ending voice mode opens that chat so you can continue over text; audio and interim transcript fragments are never stored. A realtime session lasts at most 60 minutes.

## For developers {#for-developers}

The rest of this page is the transcription route's request/response contract and how to customize or replace it — useful if you're building a custom input surface or an on-prem provider, not required to use voice input day to day.

## The route {#route}

<Endpoint id="doc-block-16oybs0" title="Voice transcription route" method="POST" path="/_agent-native/transcribe-voice" summary="Transcribe a recorded audio clip into prompt text" auth="Active session (Better Auth cookie). Same-origin only." params={[
  {
    "name": "audio",
    "in": "body",
    "type": "file",
    "required": true,
    "description": "The recorded clip, webm/opus by default. Max 25 MB."
  },
  {
    "name": "provider",
    "in": "body",
    "type": "string",
    "required": false,
    "description": "Optional override, e.g. gemini, groq, openai, builder."
  }
]} request={{
  "contentType": "multipart/form-data"
}} responses={[
  {
    "status": "200",
    "description": "Transcription succeeded",
    "example": "{ \"text\": \"reply to Sara that I'll be there by 3\" }"
  },
  {
    "status": "400",
    "description": "No server provider configured — the composer recognizes this and falls back to Web Speech",
    "example": "{ \"error\": \"no_provider\" }"
  }
]}>

The composer POSTs the recorded clip here; the route resolves a provider and returns the transcribed text. You should not call this directly.

</Endpoint>

You don't need to call this directly — the composer does. If you're building a custom input surface, first reuse the shared composer/voice client pieces from `@agent-native/core/client`. Treat this route as the low-level transport boundary for custom helpers that need to send multipart audio.

## Customizing the provider {#customizing}

The provider field is a plain application-state key, so the agent can change it on request (`"use the browser speech recognizer instead"`). If you're building a template with different requirements — say, an on-prem Whisper deployment — swap the route handler by registering your own `transcribe-voice` route before the framework mounts the default.

## What's next

- [**Drop-in Agent**](/docs/drop-in-agent) — the composer that exposes the voice button
- [**Onboarding**](/docs/onboarding) — registering provider keys as setup steps
- [**Security & Data Scoping**](/docs/security) — how encrypted secrets are stored per user