# Voice Features Test Plan

## Overview

This document provides a comprehensive test plan for validating the voice wake and push-to-talk functionality in the SOPHIAClaw macOS app.

## Pre-Test Setup

### Environment Requirements

- macOS 14.0+ (Sonoma) or macOS 15.0+ (Sequoia)
- Microphone access (built-in or external)
- Quiet testing environment
- SOPHIAClaw app built from `fix-voice-issues` branch

### Reset Permissions (Clean State)

Before testing, reset permissions to ensure clean state:

```bash
tccutil reset Microphone ai.sophiaclaw
tccutil reset SpeechRecognition ai.sophiaclaw
```

### Build the App

```bash
cd apps/macos
swift build
```

---

## Test Suite 1: Permission Flow

### Test 1.1: Single Permission Prompt

**Objective**: Verify that permissions are requested only once (not 10+ times)

**Steps**:

1. Reset permissions (see above)
2. Launch SOPHIAClaw app
3. Open Settings → Voice tab
4. Toggle "Enable voice wake" ON

**Expected Results**:

- [ ] ONLY ONE permission dialog appears for Microphone access
- [ ] ONLY ONE permission dialog appears for Speech Recognition
- [ ] No additional permission prompts after granting both
- [ ] Voice wake starts successfully after permissions granted

**Success Criteria**: Exactly 2 permission prompts total (1 for mic, 1 for speech)

### Test 1.2: Permission Denial Handling

**Objective**: Verify graceful handling when permissions are denied

**Steps**:

1. Reset permissions
2. Open Settings → Voice tab
3. Toggle "Enable voice wake" ON
4. Click "Don't Allow" on microphone permission

**Expected Results**:

- [ ] Voice wake toggle automatically turns OFF
- [ ] Error message displayed: "Microphone access is required..."
- [ ] No crash or freeze
- [ ] App remains responsive

### Test 1.3: Partial Permission Grant

**Objective**: Verify handling when only one permission is granted

**Steps**:

1. Reset permissions
2. Grant microphone permission but deny speech recognition
3. Try to enable voice wake

**Expected Results**:

- [ ] Voice wake fails to start gracefully
- [ ] Appropriate error message shown
- [ ] Toggle resets to OFF position

---

## Test Suite 2: Voice Wake Detection

### Test 2.1: Basic Wake Word Detection

**Objective**: Verify voice wake responds to trigger words

**Steps**:

1. Enable voice wake in Settings (default trigger: "Sophia")
2. Ensure microphone is active (check mic level indicator)
3. Say trigger word clearly: "Hey Sophia"
4. Wait for response

**Expected Results**:

- [ ] Chime plays (confirmation sound)
- [ ] Notification posted (chat window should open)
- [ ] Voice wake pauses temporarily (5 seconds)
- [ ] Voice wake resumes automatically after pause

**Accuracy Target**: 85%+ detection rate

### Test 2.2: Multiple Trigger Words

**Objective**: Verify all configured trigger words work

**Steps**:

1. Add multiple triggers: ["Sophia", "Hey Sophia", "Okay Sophia"]
2. Test each trigger word individually
3. Test 5 times per trigger word

**Expected Results**:

- [ ] All trigger words are recognized
- [ ] Consistent detection across multiple attempts
- [ ] Detection logged in Console app

### Test 2.3: False Positive Prevention

**Objective**: Verify app doesn't wake on similar-sounding words

**Steps**:

1. Say similar-sounding words: "Sofia", "Sophie", "Fiji", "phobia"
2. Monitor for false detections

**Expected Results**:

- [ ] No false detections on similar words
- [ ] App only wakes on exact trigger word matches

### Test 2.4: Voice Wake Recovery

**Objective**: Verify automatic restart after errors

**Steps**:

1. Enable voice wake
2. Disconnect microphone (if using external mic)
3. Reconnect microphone
4. Wait 5 seconds
5. Say trigger word

**Expected Results**:

- [ ] Voice wake detects disconnection
- [ ] Automatic retry with exponential backoff
- [ ] Voice wake resumes after mic reconnected
- [ ] No manual intervention required

---

## Test Suite 3: Push-to-Talk

### Test 3.1: Basic Recording

**Objective**: Verify push-to-talk records and transcribes speech

**Steps**:

1. Open chat window
2. Click and hold microphone button (or use hotkey)
3. Speak clearly: "What's the weather today?"
4. Release button

**Expected Results**:

- [ ] Recording indicator shows while holding
- [ ] Live transcript updates in real-time
- [ ] Final transcription appears in chat input after release
- [ ] Recording stops immediately on release

### Test 3.2: Voice Mode (Text-to-Speech)

**Objective**: Verify voice mode reads assistant responses

**Steps**:

1. Enable "Voice Mode" in settings
2. Send a message via push-to-talk
3. Wait for assistant response

**Expected Results**:

- [ ] Assistant response is spoken aloud
- [ ] Markdown formatting stripped (no \*\*, ##, etc.)
- [ ] Speech stops when new message sent
- [ ] Stop button works to interrupt speech

### Test 3.3: Retry on Failure

**Objective**: Verify retry logic for failed recordings

**Steps**:

1. Start push-to-talk recording
2. Disconnect microphone mid-recording
3. Reconnect microphone

**Expected Results**:

- [ ] Recording retries automatically (up to 3 attempts)
- [ ] Exponential backoff between retries
- [ ] Error message shown if all retries fail
- [ ] No app crash

---

## Test Suite 4: Edge Cases & Stress Testing

### Test 4.1: Rapid Toggle

**Objective**: Verify stability with rapid enable/disable

**Steps**:

1. Enable voice wake
2. Rapidly toggle ON/OFF 10 times
3. Monitor Console for errors

**Expected Results**:

- [ ] No crashes
- [ ] No memory leaks
- [ ] State remains consistent
- [ ] No duplicate AVAudioEngine instances

### Test 4.2: Simultaneous Features

**Objective**: Verify voice wake and push-to-talk don't conflict

**Steps**:

1. Enable voice wake
2. Start push-to-talk recording
3. While recording, say voice wake trigger word

**Expected Results**:

- [ ] Push-to-talk takes priority
- [ ] Voice wake doesn't interrupt recording
- [ ] Both features work independently

### Test 4.3: Long-Running Session

**Objective**: Verify stability over extended period

**Steps**:

1. Enable voice wake
2. Leave running for 30 minutes
3. Periodically test trigger word detection

**Expected Results**:

- [ ] No memory growth (check Activity Monitor)
- [ ] Consistent detection throughout session
- [ ] No automatic disable after time period
- [ ] CPU usage remains low (<5%)

### Test 4.4: Audio Session Interruption

**Objective**: Verify recovery from audio interruptions

**Steps**:

1. Enable voice wake
2. Start a FaceTime call
3. End FaceTime call
4. Test voice wake

**Expected Results**:

- [ ] Voice wake pauses during call
- [ ] Voice wake resumes after call ends
- [ ] No manual restart required

---

## Test Suite 5: Performance Metrics

### Test 5.1: Recognition Latency

**Objective**: Measure time from speech to transcription

**Steps**:

1. Use stopwatch or screen recording
2. Start push-to-talk
3. Speak single word
4. Stop recording
5. Measure time until transcription appears

**Expected Results**:

- [ ] First partial result: <500ms
- [ ] Final transcription: <2 seconds

### Test 5.2: CPU Usage

**Objective**: Verify low CPU impact

**Steps**:

1. Enable voice wake
2. Open Activity Monitor
3. Monitor SOPHIAClaw CPU usage
4. Let run for 5 minutes

**Expected Results**:

- [ ] Idle CPU: <2%
- [ ] Active listening CPU: <5%
- [ ] Recording CPU: <10%

### Test 5.3: Memory Usage

**Objective**: Verify stable memory footprint

**Steps**:

1. Enable voice wake
2. Open Activity Monitor
3. Record initial memory usage
4. Use voice features for 10 minutes
5. Record final memory usage

**Expected Results**:

- [ ] Memory growth: <10MB over 10 minutes
- [ ] No memory leaks
- [ ] Memory releases when voice wake disabled

---

## Console Logs to Monitor

During testing, monitor these log messages in Console app:

Filter: `subsystem:ai.sophiaclaw category:voicewake`

Expected logs:

- `Starting voice wake listening`
- `Voice wake is now listening`
- `Wake word detected: <transcript>`
- `Voice wake stopped`
- `Too many retry attempts` (if errors occur)

---

## Success Criteria Summary

### Critical (Must Pass)

- [ ] No app crashes during any test
- [ ] Exactly 2 permission prompts total
- [ ] 85%+ wake word detection accuracy
- [ ] Voice wake auto-recovers from errors
- [ ] Push-to-talk transcribes speech correctly

### Important (Should Pass)

- [ ] <500ms latency for first transcription
- [ ] <5% CPU usage during idle
- [ ] No memory leaks over 30 minutes
- [ ] Graceful handling of permission denials
- [ ] Recovery from audio interruptions

### Nice to Have (Optional)

- [ ] 95%+ wake word detection in quiet environment
- [ ] <100ms latency for first transcription
- [ ] Support for custom trigger words
- [ ] Voice activity detection (VAD)

---

## Known Limitations

1. **macOS Version**: Voice features require macOS 26.0+ (via @available check)
2. **Background Processing**: Voice wake may pause when app is in background (macOS limitation)
3. **Hardware**: Requires working microphone (built-in or external)
4. **Network**: Speech recognition may use cloud services for better accuracy

---

## Reporting Issues

If you encounter issues during testing:

1. **Capture Console logs**: Filter by `ai.sophiaclaw` subsystem
2. **Note reproduction steps**: Exact sequence of actions
3. **Check permissions**: System Settings → Privacy & Security → Microphone/Speech Recognition
4. **Test environment**: Note background noise level, microphone type, macOS version
5. **File issue**: Include logs, reproduction steps, and expected vs actual behavior

---

## Test Results Template

Copy this template for each test run:

```
Date: ___________
macOS Version: ___________
Branch: fix-voice-issues
Build: ___________

Test Suite 1: Permission Flow
- Test 1.1: [ ] PASS [ ] FAIL - Notes: ___________
- Test 1.2: [ ] PASS [ ] FAIL - Notes: ___________
- Test 1.3: [ ] PASS [ ] FAIL - Notes: ___________

Test Suite 2: Voice Wake Detection
- Test 2.1: [ ] PASS [ ] FAIL - Detection rate: ___%
- Test 2.2: [ ] PASS [ ] FAIL - Notes: ___________
- Test 2.3: [ ] PASS [ ] FAIL - False positives: ___
- Test 2.4: [ ] PASS [ ] FAIL - Notes: ___________

Test Suite 3: Push-to-Talk
- Test 3.1: [ ] PASS [ ] FAIL - Notes: ___________
- Test 3.2: [ ] PASS [ ] FAIL - Notes: ___________
- Test 3.3: [ ] PASS [ ] FAIL - Notes: ___________

Test Suite 4: Edge Cases
- Test 4.1: [ ] PASS [ ] FAIL - Notes: ___________
- Test 4.2: [ ] PASS [ ] FAIL - Notes: ___________
- Test 4.3: [ ] PASS [ ] FAIL - Memory growth: ___MB
- Test 4.4: [ ] PASS [ ] FAIL - Notes: ___________

Test Suite 5: Performance
- Test 5.1: Latency: ___ms
- Test 5.2: Idle CPU: ___%
- Test 5.3: Memory growth: ___MB

Issues Found:
1. ___________
2. ___________

Overall Status: [ ] PASS [ ] NEEDS WORK
```