# pi-vo implementation plan

`pi-vo` is a pi extension that speaks short voice-over summaries of what the agent is doing. It must **not** read the assistant response verbatim. It should produce brief status narration such as “I’m reading the project files”, “I edited two files and am running tests”, and “Done — the scaffold and plan are ready.”

This document closes the design choices so implementation can proceed without more product questions.

## 1. Fixed decisions

| Topic | Decision |
| --- | --- |
| Package shape | Same style as `pi-idea`: pi package with `package.json` and `pi.extensions: ["./extensions"]`. Entry point is `extensions/index.ts`. |
| Extension scope | User-installed package, loadable with `pi install /home/nikita/dev/pi-vo` or `pi -e /home/nikita/dev/pi-vo`. |
| Default behavior | Enabled in interactive TUI mode when a usable TTS provider is configured. Silent with a clear status/notification if no provider is available. Disabled in `print`/`json` modes unless forced by config/env. |
| What gets spoken | Generated short summaries from pi lifecycle events, not raw assistant output. Final assistant text can be reduced to one short sentence only when safe and useful. |
| Summary source | Phase 1 uses deterministic templates from `agent_start`, `tool_execution_start`, `tool_execution_end`, `turn_end`, and `agent_end`. No extra LLM call by default. |
| TTS provider default | OpenAI-compatible speech endpoint first, configured by `OPENAI_API_KEY`, model `gpt-4o-mini-tts`, voice `alloy`. Provider, model, and voice are configurable. |
| Offline/fallback providers | Add local `piper` CLI provider and platform TTS fallback (`say`, `espeak`, `termux-tts-speak`) after the OpenAI provider. Platform fallback is marked non-AI but useful for development. |
| Playback | Generate audio to temp files and play them sequentially via available player (`mpv`, `ffplay`, `afplay`, `paplay`, `aplay`, or Termux media/tts where applicable). Never overlap narration by default. |
| Privacy | Only send the final short narration string to the TTS provider. Never send full prompts, full command lines with possible secrets, tool outputs, diffs, or full assistant messages to TTS. |
| Noise control | Max 180 characters per utterance, debounce noisy tool events, dedupe repeated summaries, and enforce a minimum interval between spoken updates. |
| Commands | `/vo`, `/vo on`, `/vo off`, `/vo test [text]`, `/vo stop`, `/vo provider <name>`, `/vo voice <name>`, `/vo config`, `/vo mute [seconds]`, `/vo summary minimal|normal|verbose`. |
| Persistent config | Global JSON config at `~/.config/pi-vo.json`, with env vars as overrides. Session-local state is in memory only unless later needed. |
| Error handling | TTS/playback failures never break the agent. They update status and optionally notify once per failure kind. |
| Shutdown | `session_shutdown` stops playback, clears timers, and deletes temp files owned by current session where safe. |

## 2. User experience target

### Normal flow

1. User starts pi with pi-vo loaded.
2. Footer shows `vo: ready` when TTS is available, or `vo: setup` when configuration is missing.
3. User sends a prompt.
4. pi-vo speaks short updates while the agent works:
   - “Starting on your request.”
   - “Reading project files.”
   - “Editing package and README.”
   - “Running tests.”
   - “Done — scaffold created and plan written.”
5. If the agent is very active, pi-vo coalesces updates instead of speaking every tool call.

### Command examples

```text
/vo
/vo on
/vo off
/vo test Pi voice-over is ready.
/vo provider openai
/vo voice alloy
/vo summary minimal
/vo mute 60
/vo stop
/vo config
```

## 3. Runtime architecture

```text
Pi lifecycle events
  ↓
Event summarizer
  - maps tools/actions to safe short phrases
  - aggregates noisy bursts
  - never includes raw outputs/secrets
  ↓
Narration policy
  - enabled/muted/mode checks
  - min interval + dedupe
  - priority handling
  ↓
Speech queue
  - FIFO, one active utterance
  - stop/clear support
  - AbortSignal-aware
  ↓
TTS provider
  - openai-compatible remote speech
  - piper local CLI
  - system fallback
  ↓
Playback backend
  - mpv/ffplay/afplay/paplay/aplay/etc.
```

## 4. File layout to implement

```text
pi-vo/
  package.json
  README.md
  docs/
    IMPLEMENTATION_PLAN.md
  extensions/
    index.ts              # extension entry and event wiring
    config.ts             # load/save config, env overrides
    summarizer.ts         # event → short safe narration text
    queue.ts              # speech queue, debounce, mute, stop
    providers/
      types.ts            # TTS provider interfaces
      openai.ts           # OpenAI-compatible speech endpoint
      piper.ts            # local piper CLI backend
      system.ts           # say/espeak/termux fallback
    playback.ts           # audio player discovery and process lifecycle
    sanitize.ts           # path/command/assistant text sanitization
    commands.ts           # /vo command parsing and help
```

Initial implementation can keep modules inside `extensions/index.ts` until the code becomes uncomfortable, but the final shape above is preferred.

## 5. Config schema

Config path: `~/.config/pi-vo.json`.

```ts
type PiVoConfig = {
  enabled: boolean;                 // default true
  provider: "openai" | "piper" | "system" | "auto"; // default "auto"
  model: string;                    // default "gpt-4o-mini-tts" for openai
  voice: string;                    // default "alloy"
  speed?: number;                   // provider-specific, default 1.0
  summaryLevel: "minimal" | "normal" | "verbose"; // default "normal"
  maxChars: number;                 // default 180
  minIntervalMs: number;            // default 2500
  toolDebounceMs: number;           // default 1200
  speakAgentStart: boolean;         // default true
  speakToolProgress: boolean;       // default true
  speakErrors: boolean;             // default true
  speakFinal: boolean;              // default true
  modes: Array<"tui" | "rpc" | "print" | "json">; // default ["tui"]
  player?: string;                  // optional explicit playback command
  openaiBaseUrl?: string;           // default https://api.openai.com/v1
};
```

Environment overrides:

| Env var | Meaning |
| --- | --- |
| `PI_VO_ENABLED=0|1` | Force enable/disable. |
| `PI_VO_PROVIDER=openai|piper|system|auto` | Provider choice. |
| `PI_VO_VOICE=<voice>` | Voice name. |
| `PI_VO_MODEL=<model>` | TTS model. |
| `PI_VO_PLAYER=<cmd>` | Playback command. |
| `PI_VO_FORCE=1` | Allow non-TUI modes. |
| `OPENAI_API_KEY` | OpenAI TTS API key. |
| `OPENAI_BASE_URL` | Optional OpenAI-compatible base URL. |

## 6. Event mapping details

### Hooks to register

| Hook | Purpose |
| --- | --- |
| `session_start` | Load config, discover provider/player, set status. |
| `agent_start` | Reset per-agent aggregation and optionally say a start phrase. |
| `tool_execution_start` | Add a safe description of the tool action to the current burst. |
| `tool_execution_end` | Track success/error counts; speak error summaries when useful. |
| `turn_end` | Speak a coalesced progress summary for the turn. |
| `agent_end` | Speak a final short completion summary and flush queue. |
| `session_shutdown` | Stop audio, clear timers, cleanup. |

### Tool summary templates

| Tool | Spoken summary rule |
| --- | --- |
| `read` | “Reading `<basename>`” or grouped “Reading project files.” |
| `write` | “Writing `<basename>`.” |
| `edit` | “Editing `<basename>`.” |
| `bash` | Categorize command: tests, install, git, search, build, server, or generic “running a command”. Do not speak full command arguments. |
| `grep` / search | “Searching the codebase.” |
| `find` / `ls` | “Inspecting project files.” |
| unknown tool | “Using `<toolName>`.” |

### Aggregation examples

- Multiple `read` calls inside debounce window: “Reading project files.”
- `edit` on `package.json` + `README.md`: “Editing package and README.”
- `bash` command matching `npm test`, `pnpm test`, `pytest`, etc.: “Running tests.”
- Command error: “The test command failed; checking the error.”
- Final with edits/tests: “Done — edited two files and ran tests.”

## 7. Sanitization rules

1. Limit every utterance to `maxChars` after sanitization.
2. Do not speak raw bash commands except safe first-token categories.
3. Redact likely secrets with patterns for API keys, bearer tokens, passwords, `.env` values, and long base64/hex strings.
4. For file paths, prefer basename or last two path segments.
5. For assistant final text, use at most the first sentence after removing code blocks, URLs with query strings, and markdown tables.
6. If sanitization produces empty text, skip the utterance.

## 8. TTS provider design

```ts
type SpeechRequest = {
  text: string;
  voice: string;
  model?: string;
  speed?: number;
  signal?: AbortSignal;
};

type SpeechAudio = {
  kind: "file";
  path: string;
  mimeType: string;
};

interface TtsProvider {
  name: string;
  isAvailable(): Promise<{ ok: true } | { ok: false; reason: string }>;
  synthesize(request: SpeechRequest): Promise<SpeechAudio | { kind: "spoken-directly" }>;
}
```

### OpenAI-compatible provider

- Availability: `OPENAI_API_KEY` exists or config has explicit key in a later version.
- Request:
  - `POST ${baseUrl}/audio/speech`
  - Headers: `Authorization: Bearer ${OPENAI_API_KEY}`, `Content-Type: application/json`
  - JSON: `{ model, voice, input: text, speed }`
  - Save response bytes as `.mp3` in `os.tmpdir()/pi-vo/`.
- Abort: pass `signal` to `fetch`.
- Errors: throw typed error with status and short body, but never include API key.

### Piper provider

- Availability: `piper` executable found and model path configured.
- Input text via stdin; output wav temp file.
- Good offline AI fallback.

### System provider

- Availability checks in order:
  - macOS `say`
  - Linux `espeak` / `spd-say`
  - Termux `termux-tts-speak`
- It may speak directly rather than returning an audio file.
- Mark in status as `vo: system` so user knows it is not the preferred AI TTS path.

## 9. Playback design

```ts
type PlaybackHandle = {
  done: Promise<void>;
  stop(): void;
};

interface AudioPlayer {
  name: string;
  play(file: SpeechAudio, signal?: AbortSignal): PlaybackHandle;
}
```

Discovery order:

1. Configured `player` command.
2. `mpv --no-terminal --really-quiet <file>`.
3. `ffplay -nodisp -autoexit -loglevel error <file>`.
4. `afplay <file>` on macOS.
5. `paplay <file>` / `aplay <file>` for WAV on Linux.

Queue behavior:

- One active synthesis/playback at a time.
- `stop` aborts current synth fetch/process and kills active player.
- Queue max length defaults to 3; new low-priority utterances replace older pending low-priority ones.
- High priority error/final utterances can clear stale progress utterances.

## 10. `/vo` command behavior

| Command | Behavior |
| --- | --- |
| `/vo` | Show status: enabled, provider, voice, player, muted/unmuted, last error. |
| `/vo help` | Show help. |
| `/vo on` | Set `enabled: true`, save config, update status. |
| `/vo off` | Set `enabled: false`, save config, stop queue, update status. |
| `/vo test [text]` | Speak provided text or “pi voice-over is ready.” |
| `/vo stop` | Abort playback and clear queue. |
| `/vo mute [seconds]` | Mute indefinitely or for N seconds. |
| `/vo provider <name>` | Set provider and re-run availability checks. |
| `/vo voice <name>` | Set voice. |
| `/vo summary <level>` | Set summary verbosity. |
| `/vo config` | Show config path and redacted effective config. |

## 11. Implementation phases

### Phase 0 — bootstrap (done by current scaffold)

- Create `/home/nikita/dev/pi-vo`.
- Add pi package manifest.
- Add extension entry point that loads and exposes a placeholder `/vo` command.
- Add this implementation plan.

### Phase 1 — deterministic narration without audio

Goal: prove event mapping and command UX before TTS.

Tasks:

1. Implement config loading/saving and effective config with env overrides.
2. Implement `/vo` commands except real audio (`test` can use `ctx.ui.notify` initially).
3. Implement summarizer functions and unit-ish pure function tests via simple Node script or `npm test` later.
4. Wire lifecycle hooks to generate narration strings, but route them to debug notifications/status only.
5. Validate no event handler can block agent progress for long.

Acceptance:

- `/vo` shows config/status.
- During a pi turn, status/debug messages show reasonable short summaries.
- No raw tool output or full command lines appear in generated narration.

### Phase 2 — OpenAI TTS + playback queue

Goal: `/vo test` speaks through AI-generated speech.

Tasks:

1. Implement OpenAI-compatible provider.
2. Implement temp file management.
3. Implement player discovery and playback.
4. Implement queue with abort, stop, max length, and dedupe.
5. Wire `/vo test` to real queue.

Acceptance:

- With `OPENAI_API_KEY`, `/vo test hello` speaks.
- `/vo stop` immediately stops current playback and clears pending items.
- Missing API key gives one actionable notification and `vo: setup` status.

### Phase 3 — live voice-over during agent activity

Goal: speak useful progress summaries automatically.

Tasks:

1. Connect lifecycle events to queue.
2. Add tool burst debounce and final summaries.
3. Implement error priority summaries.
4. Tune defaults so it is helpful but not chatty.
5. Clear timers and audio on `session_shutdown`.

Acceptance:

- A normal coding task produces 2–5 short spoken updates, not dozens.
- Tool-heavy bursts are coalesced.
- Final spoken summary is short and not verbatim output.

### Phase 4 — local providers and polish

Goal: make pi-vo robust outside the happy path.

Tasks:

1. Add `piper` provider.
2. Add system provider fallback.
3. Add provider auto-selection and richer `/vo config` diagnostics.
4. Add mute timer and summary verbosity tuning.
5. Add README install/use docs.

Acceptance:

- `provider: auto` selects the best available provider.
- Users can develop without OpenAI using local/system fallback.
- Diagnostics explain exactly what is missing.

### Phase 5 — packaging and release readiness

Tasks:

1. Finalize README with install, config, privacy, troubleshooting.
2. Run local smoke tests with `pi -e /home/nikita/dev/pi-vo`.
3. Ensure package has only needed runtime dependencies. Keep pi packages in `peerDependencies`.
4. Add version, repository metadata, and changelog notes.
5. Commit and prepare npm/github publishing if desired.

Acceptance:

- `pi install /home/nikita/dev/pi-vo` loads the extension.
- `pi -e /home/nikita/dev/pi-vo` works for quick tests.
- No TypeScript/runtime errors at extension load.

## 12. Test plan

Manual smoke tests:

1. Start without `OPENAI_API_KEY`:
   - Expect `vo: setup` and useful `/vo` diagnostics.
2. Start with `OPENAI_API_KEY`:
   - `/vo test pi voice-over is ready` speaks.
3. Run a small pi task that reads/edits files:
   - Expect short activity summaries.
4. Run a task with a failing test command:
   - Expect one short error narration.
5. Use `/vo mute 10`:
   - No speech for 10 seconds, then auto-unmute.
6. Use `/vo stop` during playback:
   - Audio stops quickly.
7. Exit/reload:
   - No orphan player process remains.

Security/privacy checks:

- Bash command with `TOKEN=secret` never speaks the secret.
- `.env` edits/reads are summarized only as “editing an environment file” or skipped.
- Full assistant answer is not sent to TTS.

## 13. Known future enhancements

- Optional AI summarizer mode for richer final summaries, explicitly opt-in because it sends extra context to a model.
- Per-project config in `.pi/pi-vo.json` after trust checks.
- Voice profiles by model/task type.
- “Interrupt old progress when final arrives” setting.
- OSC or native desktop notification integration alongside speech.