# Voice Wake / Push-to-Talk - Status: BLOCKED

**Date:** 2026-03-13
**Status:** Disabled - shipping without voice features
**Blocked by:** macOS TCC (Transparency, Consent, and Control) repeated permission prompts

## Summary

Voice wake (trigger phrase detection) and push-to-talk are implemented using the macOS 26.0+ `SpeechAnalyzer` / `SpeechTranscriber` APIs. The core speech recognition pipeline works correctly in isolation but is unusable in the app due to macOS repeatedly prompting for microphone permission on non-sandboxed apps.

## What Works

- **Speech recognition pipeline:** Standalone test confirms `AVAudioEngine` + `SpeechAnalyzer` + `SpeechTranscriber` correctly captures audio, converts buffer formats, and produces transcription results.
- **Trigger phrase matching:** When recognition works, trigger phrases ("Hey Sophia") are detected and matched.
- **Permission grant flow:** Microphone and speech recognition permissions can be granted (1 prompt each via `AVCaptureDevice.requestAccess` and `SFSpeechRecognizer.requestAuthorization`).
- **Push-to-talk UI:** Mic button in chat input bar with hold-to-talk gesture.

## The Problem

On **non-sandboxed macOS apps**, the system generates repeated microphone permission prompts (10-20+) under these conditions:

1. **Every `engine.prepare()` + `engine.start()` cycle** triggers a new TCC prompt, even if permission was already granted.
2. **Every `AVCaptureDevice.authorizationStatus(for: .audio)` call** can trigger a TCC prompt.
3. **Recognition session restarts** (required because `SFSpeechRecognizer` segments end after ~15s of silence) each trigger prompts if the engine is stopped/restarted.
4. Even with a single shared `AVAudioEngine` that is never stopped, the prompts persist -- suggesting `SpeechAnalyzer.start()` or internal Speech framework calls also trigger TCC.

This results in:

- 10-20 microphone permission dialogs appearing in rapid succession
- ~50% speech recognition accuracy (likely due to audio session interruptions from the prompts)
- Poor user experience making the feature unusable

## Approaches Tried

### 1. SharedAudioEngine (consumer pattern)

- Single `AVAudioEngine` instance broadcasting buffers to registered consumers
- **Result:** Crashed on `AVAudioEngine.prepare()` -- the actor-based initialization didn't properly set up the audio graph (`inputNode != nullptr` assertion failure)

### 2. Per-service engines (matching Swabble pattern)

- Each service creates its own `AVAudioEngine` following the pattern: `inputNode` access -> `installTap` -> `prepare()` -> `start()`
- **Result:** Works for a single session but each engine creation triggers TCC prompts

### 3. Engine reuse across restarts

- One `AVAudioEngine` created on first use, reused across stop/start cycles
- `stop()` removes tap but doesn't stop engine; `start()` reinstalls tap and only calls `prepare()`/`start()` if engine isn't running
- **Result:** Still triggers repeated prompts -- the prompts appear to come from the Speech framework itself, not our engine management

### 4. Static shared engine (never destroyed)

- `nonisolated(unsafe) static var sharedEngine` on `SpeechPipeline` actor
- Engine created once, never stopped, never destroyed
- **Result:** Only 1 engine creation in logs, but prompts still appear (10-20)

### 5. Aggressive permission caching

- `VoicePermissionGate.checkStatusOnce()` -- only calls `AVCaptureDevice.authorizationStatus` once, caches forever
- Removed all `authorizationStatus` checks from hot paths
- **Result:** Prompts still appear

### 6. `SFSpeechRecognizer.requestAuthorization` crash fix

- The callback fires on a random dispatch queue; resuming a `@MainActor` continuation from it caused `dispatch_assert_queue_fail` (SIGTRAP)
- **Fix:** Wrapped in `Task.detached` to avoid actor isolation issues
- **Result:** Fixed the crash but didn't fix the prompts

### 7. Pause/resume pattern for test

- `VoiceWakeService.pause()` / `resume()` with `isPaused` flag to prevent auto-restart during tester use
- **Result:** Correct behavior (only 1 engine in logs) but prompts persist

## Root Cause Analysis

The repeated TCC prompts appear to originate from within Apple's Speech framework (`SpeechAnalyzer` / `SpeechTranscriber`) rather than from our `AVAudioEngine` management. Evidence:

1. Our logs show only **1 engine creation** and **1 `engine.start()` call**, yet 10-20 prompts appear.
2. The prompts appear specifically when `SpeechAnalyzer.start(inputSequence:)` is called or when `SpeechTranscriber` begins processing.
3. The original working OpenClaw app (in Swabble) may have been sandboxed or run under different TCC rules.

This is likely a macOS TCC behavior specific to non-sandboxed apps where the Speech framework's internal microphone access checks each trigger a new consent dialog.

## Possible Solutions (Not Yet Tried)

1. **Sandbox the app** -- Sandboxed apps get a single permission prompt. This is the most likely fix but requires entitlements changes and may affect other features.
2. **Use `SFSpeechRecognizer` with `SFSpeechAudioBufferRecognitionRequest` instead of `SpeechAnalyzer`** -- The older API may not trigger the same TCC behavior. However, our earlier attempts with this API had issues with recognition never producing results when manually feeding buffers.
3. **Use a temporary entitlement or TCC profile** for development.
4. **File a Feedback with Apple** -- The behavior of repeated TCC prompts for an already-authorized non-sandboxed app appears to be a bug.

## Files Involved

### Core Services

- `apps/macos/Sources/SOPHIAClaw/Services/VoiceWakeService.swift` - Main voice wake service + SpeechPipeline actor
- `apps/macos/Sources/SOPHIAClaw/Services/VoiceWakeTester.swift` - Test harness for trigger phrase detection
- `apps/macos/Sources/SOPHIAClaw/Services/PushToTalkService.swift` - Hold-to-talk in chat
- `apps/macos/Sources/SOPHIAClaw/Services/MicLevelMonitor.swift` - Mic level visualization
- `apps/macos/Sources/SOPHIAClaw/Services/VoicePermissionGate.swift` - Permission request/cache
- `apps/macos/Sources/SOPHIAClaw/Services/BufferConverter.swift` - Audio format conversion
- `apps/macos/Sources/SOPHIAClaw/Services/VoiceLog.swift` - Debug file logger

### UI Integration

- `apps/macos/Sources/SOPHIAClaw/UI/Settings/SimplePreferences.swift` - Voice settings tab
- `apps/macos/Sources/SOPHIAClaw/UI/Settings/SettingsView.swift` - Alternative settings view
- `apps/macos/Sources/SOPHIAClaw/UI/Chat/ConversationView.swift` - PTT button in chat
- `apps/macos/Sources/SOPHIAClaw/ViewModels/ConversationViewModel.swift` - TTS readback

### Reference Implementation

- `Swabble/Sources/SwabbleCore/Speech/SpeechPipeline.swift` - Working standalone pipeline
- `Swabble/Sources/SwabbleCore/Speech/BufferConverter.swift` - Buffer conversion

## Current State (Disabled)

Voice wake and push-to-talk are disabled in the UI. The code remains in place but:

- `swabbleEnabled` is set to `false` in config
- The voice settings section is hidden
- VoiceWakeService observer does not auto-start
- PushToTalk buttons are hidden in chat

To re-enable for development, set `swabbleEnabled: true` in `~/.sophiaclaw/app.json`.