# Voice TTS Flow — Architecture & Implementation Guide This document describes the voice synthesis system in omnius: how it's built, how all the pieces connect, and how to add new voice features following the same patterns. --- ## System Overview ``` ┌─────────────┐ ┌──────────────────┐ ┌──────────────┐ │ interactive │────▶│ VoiceEngine │────▶│ System Audio │ │ .ts │ │ voice.ts │ │ (afplay/ │ │ │ │ │ │ paplay) │ │ ┌────────┐ │ │ ┌────────────┐ │ └──────────────┘ │ │ Agent │──┤ │ │ ONNX Model │ │ │ │ Events │ │ │ │ Session │ │ ┌──────────────┐ │ └────────┘ │ │ └────────────┘ │────▶│ WebSocket │ │ │ │ ┌────────────┐ │ │ Clients │ │ ┌────────┐ │ │ │ MLX Audio │ │ │ (voice- │ │ │Emotion │──┤ │ │ Backend │ │ │ session.ts) │ │ │Context │ │ │ └────────────┘ │ └──────────────┘ │ └────────┘ │ └──────────────────┘ │ │ ▲ │ ┌────────┐ │ │ │ │Narrate │──┘ ┌────────┴────────┐ │ │ Engine │ │ Model Registry │ │ └────────┘ │ glados/over- │ └─────────────┘ │ watch/kokoro │ └─────────────────┘ ``` ## File Map | File | Purpose | Lines | |------|---------|-------| | `packages/cli/src/tui/voice.ts` | VoiceEngine class + narration engine | ~2260 | | `packages/cli/src/tui/voice-session.ts` | WebSocket streaming + cloudflared tunnel | ~885 | | `packages/cli/src/tui/interactive.ts` | Wiring: instantiation, events, narration calls | ~2300 | | `packages/cli/src/tui/commands.ts` | `/voice` command handler | ~1850 | | `packages/cli/src/tui/render.ts` | `renderInfo`/`renderWarning` + content-write hook | ~670 | | `packages/cli/tests/voice-narration.test.ts` | Narration unit tests | varies | | `packages/cli/tests/voice-session.test.ts` | Session unit tests | varies | --- ## 1. Model Registry All voice models are defined in `VOICE_MODELS` (voice.ts, top of file): ```typescript const VOICE_MODELS: Record = { glados: { id: "glados", backend: "onnx", onnxUrl: "...", configUrl: "..." }, overwatch:{ id: "overwatch", backend: "onnx", onnxUrl: "...", configUrl: "..." }, kokoro: { id: "kokoro", backend: "mlx", mlxModelId: "mlx-community/Kokoro-82M-bf16", mlxVoice: "af_heart" }, // ... more kokoro voices }; ``` **To add a new model:** 1. Add an entry to `VOICE_MODELS` with the appropriate `backend` field 2. For ONNX: provide `onnxUrl` + `configUrl` (Piper ONNX format) 3. For MLX: provide `mlxModelId` (Hugging Face repo) + `mlxVoice` + `mlxLangCode` 4. The model is immediately available via `/voice ` --- ## 2. VoiceEngine Lifecycle ### Instantiation A single `VoiceEngine` instance is created in `interactive.ts:1359`: ```typescript const voiceEngine = new VoiceEngine(); ``` ### Startup If the user previously enabled voice (persisted in settings): ```typescript if (savedSettings.voice) { voiceEngine.toggle(); // Enable TTS if (savedSettings.voiceModel) voiceEngine.setModel(savedSettings.voiceModel); // Load specific model } ``` ### toggle() Flow ``` toggle() ├─ If disabling: set enabled=false, killPlayback(), done └─ If enabling: ├─ MLX model? │ └─ ensureMlxAudio() → pip install mlx-audio └─ ONNX model? ├─ ensureRuntime() → npm install onnxruntime-node + phonemizer ├─ ensureModel(id) → download .onnx + .json from GitHub └─ loadSession() → create InferenceSession set enabled=true, ready=true ``` ### setModel(id) Flow ``` setModel(id) ├─ Validate id exists in VOICE_MODELS ├─ Reset: session=null, config=null, ready=false └─ If currently enabled: └─ Load the new model (same as toggle enable path) ``` ### Shutdown ```typescript voiceEngine.dispose(); // in interactive.ts cleanup ``` --- ## 3. Synthesis Pipeline ### ONNX Path (glados, overwatch) ``` text │ ├── chunkText() Split >200 char text on newlines + sentence boundaries │ ├── For each chunk: │ ├── textToPhonemes() espeak-ng WASM phonemization │ ├── phonemesToIds() Map phonemes → integer IDs via config.phoneme_id_map │ │ Format: BOS → PAD → (phoneme + PAD)* → EOS │ ├── Build tensors: input (int64), input_lengths (int64), scales (float32) │ ├── session.run() ONNX inference → Float32 audio samples │ └── Concatenate with 180ms silence gaps between sentences │ ├── Apply volume scaling (0.0–1.0 multiplier per sample) ├── Apply pitch shift (linear-interpolation resampling) │ ├── Stream PCM to WebSocket clients (if onPCMOutput wired) ├── Write WAV to temp file ├── Play via system command (afplay / paplay / pw-play / aplay) └── Delete temp file ``` ### MLX Path (kokoro, kokoro:af_heart, etc.) ``` text │ ├── Clean markdown (* removal) ├── Build Python command: │ python3 -c "from mlx_audio.tts import generate; generate.main([...])" │ --model mlx-community/Kokoro-82M-bf16 │ --text "..." │ --voice af_heart │ --lang_code a │ --audio_path /tmp/omnius-mlx-{ts}.wav │ ├── execSync() with 60s timeout │ (fallback: python3 -m mlx_audio.tts.generate CLI) │ ├── Apply volume scaling (rewrite WAV PCM samples) ├── Stream PCM to WebSocket clients (parse WAV header for sample rate) ├── Play via system command └── Delete temp file ``` --- ## 4. Narration Engine The narration system generates context-aware spoken descriptions of agent activity. It lives in voice.ts below the VoiceEngine class (~line 1268+). ### Personality Levels ``` 1 = minimal "Reading file.ts" 2 = brief "Reading file.ts" 3 = conv "Let me take a look at file.ts" 4 = chatty "Alright, let's crack open file.ts" 5 = theatrical "Alright, let's crack open file.ts and see what we're working with" ``` Mapped from the agent's personality preset: ```typescript { concise: 1, balanced: 3, verbose: 4, pedagogical: 5 } ``` ### describeToolCall(toolName, args, level, emotion) Called from `interactive.ts` on every `tool_call` event: ```typescript if (voice?.enabled) { const desc = describeToolCall(event.toolName, event.toolArgs, vLevel, emoCtx); voice.speakSubordinate(desc, emoCtx); // 55% volume, 0.92x pitch } ``` **Variant pools** — each tool has 3 tiers of phrasings: - `FILE_READ_VARIANTS.terse` / `.conv` / `.chatty` - `FILE_WRITE_VARIANTS`, `FILE_EDIT_VARIANTS`, `GREP_VARIANTS`, etc. - `SHELL_VARIANTS` — categorized by shell command type (git, npm, test, etc.) **Context modifiers** applied based on narration state: - After errors: prefix with "Okay, " or "Right, " - Same file again: "Back to " or "Still working on " - Revisiting a file: "Coming back to " or "Revisiting " - Progress beats every 8 tools: "Making good progress. " ### describeToolResult(toolName, success, level, content, emotion) Called on every `tool_result` event. Generates success/failure descriptions: - Success: "Got it", "Done", "Found it" - Failure: "That didn't work", "Hit a snag" **Content-aware extraction** via `extractResultDigest()`: - ETH balances, test results, error messages, wallet addresses, file paths ### describeTaskComplete(summary, complete, level) Announces when the agent finishes a task: - "Task complete", "Got it done", "All set" ### Narration State ```typescript interface NarrationState { toolCount: number; // Total tools this session toolCounts: Record; // Per-tool counts consecutiveErrors: number; // Reset on success totalErrors: number; lastTool: string; lastFile: string; filesSeen: Set; // All files visited lastVariantIdx: Record; // Avoid repeats lastResultDigest: string; // Last tool result summary } ``` `resetNarrationContext()` is called at the start of each task. ### pick(key, variants) Selects a random variant from a pool, avoiding the last-used index for that key. Ensures you never hear the same phrasing twice in a row. --- ## 5. Emotion Modulation The emotion engine provides valence-arousal context to the voice: ```typescript interface VoiceEmotionContext { valence: number; // -1 (sad) to +1 (happy) arousal: number; // 0 (calm) to 1 (activated) label: string; // "excited", "focused", etc. emoji: string; } ``` ### Pitch Bias `emotionToPitchBias(emotion)` converts emotion to a pitch adjustment: ``` pitch_bias = valence × 0.6 + (arousal - 0.5) × 0.4 clamped to [-0.10, +0.10] ``` - **Excited** (high valence + high arousal) → voice pitch rises - **Dejected** (low valence + low arousal) → voice pitch drops Applied in `speak()` and `speakSubordinate()`: ```typescript speak(text, emotion): pitchFactor = 1.0 + pitchBias speakSubordinate(text): pitchFactor = 0.92 + pitchBias (lower base pitch) ``` ### Emotion Coloring At personality >= 3, ~30% of narrations get emotion-colored prefixes: - Excited: "Feeling good about this" - Stressed: "Pushing through" - Calm: "Nice and steady" - Subdued: "Being careful here" --- ## 6. Queue & Playback ### Speech Queue ```typescript private speakQueue: SpeakItem[] = []; ``` Items are queued FIFO. `drainQueue()` processes them sequentially: 1. Pop item from front 2. Synthesize + play to completion 3. 250ms silence gap 4. Next item Queue overflow protection: if > 30 items backed up, clear the queue. ### Volume Levels ``` speak() → volume 1.0 (full) speakSubordinate() → volume 0.55 (reduced for tool narration) ``` ### Playback WAV temp file → system audio command: - macOS: `afplay` - Linux: `paplay` → `pw-play` → `aplay` (tries in order) - Windows: PowerShell `Media.SoundPlayer` 15-second safety timeout per playback. `killPlayback()` sends SIGTERM. --- ## 7. WebSocket Voice Session `VoiceSession` (voice-session.ts) enables real-time voice interaction via browser: ``` Browser Client ←──WebSocket──→ VoiceSession ←──PCM──→ VoiceEngine (mic) binary PCM (server) onPCMOutput (TTS) (speaker) binary PCM ``` ### Setup ``` start() ├── Start HTTP server (serves HTML single-page app) ├── Start WebSocket server (ws) ├── Launch cloudflared tunnel for public URL └── Wire VoiceEngine.onPCMOutput → broadcast to all clients ``` ### Audio Format - 16kHz, 16-bit, mono PCM (Int16Array) - Binary WebSocket frames - Echo cancellation: suppress mic input while TTS is playing ### Frontend Embedded HTML with: - Braille waveform animator (color-coded: idle/listening/speaking) - WebAudio API for mic capture + speaker playback - Transcript view (user + agent messages) - Start/stop mic button --- ## 8. Settings Persistence Voice settings are saved per-project or globally via `resolveSettings()`: ```typescript { voice: boolean, // Enabled/disabled voiceModel: string // Model ID ("glados", "kokoro:af_heart", etc.) } ``` The `/voice` command handler in commands.ts: ```typescript case "voice": if (arg) { ctx.voiceSetModel(arg); // /voice kokoro save({ voice: true, voiceModel: arg }); } else { ctx.voiceToggle(); // /voice (toggle on/off) save({ voice: isOn }); } ``` --- ## 9. Content-Write Hook (TUI Safety) Voice operations can trigger `renderInfo()` / `renderWarning()` messages asynchronously. To prevent these from overwriting the TUI input area, a global content-write hook brackets all render calls with scroll-region management: ```typescript // render.ts export function renderInfo(message: string): void { _contentWriteHook?.begin(); // → statusBar.beginContentWrite() process.stdout.write(`ℹ ${message}\n`); _contentWriteHook?.end(); // → statusBar.endContentWrite() } ``` Registered once in interactive.ts after StatusBar activation: ```typescript setContentWriteHook({ begin: () => statusBar.beginContentWrite(), end: () => statusBar.endContentWrite(), }); ``` **Rule:** Never use raw `process.stdout.write()` in voice.ts for user-facing messages. Always use `renderInfo()` / `renderWarning()` / `renderError()` so the content hook routes output to the scroll region above the status bar and input area. --- ## 10. How to Add a New Voice Feature ### Adding a New ONNX Voice Model 1. Host the `.onnx` + `.onnx.json` files (Piper format) on a public URL 2. Add entry to `VOICE_MODELS`: ```typescript myvoice: { id: "myvoice", label: "My Voice", backend: "onnx", onnxUrl: "https://...", configUrl: "https://...", }, ``` 3. Done. `/voice myvoice` works immediately. ### Adding a New MLX Voice 1. Find the Hugging Face model ID (must be MLX-compatible) 2. Add entry to `VOICE_MODELS`: ```typescript "newmodel:voice_name": { id: "newmodel:voice_name", label: "New Model (MLX)", backend: "mlx", mlxModelId: "mlx-community/NewModel", mlxVoice: "voice_name", mlxLangCode: "a", onnxUrl: "", configUrl: "", }, ``` 3. Done. `/voice newmodel:voice_name` works on macOS Apple Silicon. ### Adding a New TTS Backend 1. Add a new `backend` value to the `VoiceModel` interface 2. Add a method like `ensureNewBackend()` for installation 3. Add a method like `synthesizeWithNewBackend()` for synthesis 4. Gate in `toggle()`, `setModel()`, `synthesizeAndPlay()`, `synthesizeToBuffer()`, `synthesizeToPCM()` 5. Follow the pattern: the backend outputs a WAV file, then existing `playWav()` handles playback ### Adding New Narration Variants 1. Find the relevant variant pool (e.g., `FILE_READ_VARIANTS`) 2. Add new strings to the `terse`, `conv`, and `chatty` tiers 3. The `pick()` function automatically rotates through variants ### Adding Emotion-Aware Features 1. Receive `VoiceEmotionContext` from the emotion engine 2. Use `emotionToPitchBias()` for pitch modulation 3. Use `emotionColor()` for spoken prefixes 4. Valence drives tone (happy/sad), arousal drives energy (calm/activated) --- ## Quick Reference: Key Functions | Function | Location | Purpose | |----------|----------|---------| | `VoiceEngine.toggle()` | voice.ts | Enable/disable TTS | | `VoiceEngine.speak()` | voice.ts | Queue speech (full volume) | | `VoiceEngine.speakSubordinate()` | voice.ts | Queue speech (55% volume) | | `VoiceEngine.synthesizeAndPlay()` | voice.ts | Core synthesis + playback | | `VoiceEngine.synthesizeWithMlx()` | voice.ts | MLX backend synthesis | | `VoiceEngine.ensureRuntime()` | voice.ts | Install ONNX runtime | | `VoiceEngine.ensureMlxAudio()` | voice.ts | Install mlx-audio pip package | | `describeToolCall()` | voice.ts | Generate tool narration text | | `describeToolResult()` | voice.ts | Generate result narration text | | `describeTaskComplete()` | voice.ts | Generate completion narration | | `emotionToPitchBias()` | voice.ts | Emotion → pitch modulation | | `pick()` | voice.ts | Random variant selection (no repeats) | | `setContentWriteHook()` | render.ts | Register TUI scroll-region safety | | `VoiceSession.start()` | voice-session.ts | Start WebSocket voice server |