# Pi `/voice` implementation handoff

**Native implementation added and installed globally:** `/voice` now runs inside Pi's terminal with microphone/speaker handling, automatic Pi-owned delegation, meters and mute controls—no browser. Run `/reload` in an existing Pi session, then `/voice`. See [IMPLEMENTATION.md](IMPLEMENTATION.md) for controls, setup and validation, including the [real native round-trip result](evidence/native-result.json). Entry point: [`extensions/voice.ts`](extensions/voice.ts). The original research below is preserved unchanged.

Investigation date: 2026-10-03. Tested with Codex CLI 0.160.0 on this Linux machine and Node.js v24.21.0.

## Objective and result

Build a Pi extension that starts a spoken conversation with `/voice`, while the existing Pi session remains responsible for coding, tools, and project context. The user wants to reuse the voice access available through their existing Codex account sign-in, without requiring a separately billed OpenAI API key.

An external Node.js test client successfully started a Codex app-server WebRTC voice session using the machine's existing ChatGPT authentication. No API key was supplied. This establishes that the installed Codex runtime can provide an authenticated voice connection to an external client. It does not establish a finished Pi integration or a supported public subscription-voice API.

Start with the proven Codex app-server connection. Do not replace it with an API-key-based GPT-Live implementation unless the user requests that change.

## Evidence preserved here

- [Original probe](evidence/original-probe.mjs): the exact final connection test, copied from `/tmp`.
- [Final observed results](evidence/result.json): sanitized event names, connection state, account type, and audio statistics.
- [Speech input](evidence/test-input.wav): the public prerecorded sample used in the test. Source: [Vosk example audio](https://github.com/alphacep/vosk-api/blob/master/python/example/test.wav).
- [Installed protocol schemas](schemas/): selected experimental schemas generated by Codex 0.160.0. These are more relevant to reproducing the test than GitHub's moving `main` branch.

No OAuth credentials, authorization headers, account email, or account identifier are included.

## What the test established

| Observation | Evidence and scope |
|---|---|
| Existing account authentication worked | `account/read` reported `type: chatgpt` and `plan: plus`. The stored sign-in had OAuth tokens and no stored API key. |
| No API key was supplied | The parent environment had no `OPENAI_API_KEY`; the probe also removed it and `OPENAI_API_BASE` from the child environment. |
| An external app-server client worked | The client initialized experimental API access, then created an ephemeral thread. |
| WebRTC negotiation succeeded | The service returned an SDP answer; connection and ICE states became `connected`, and the data channel became `open`. |
| Client-managed handoffs were accepted | The start request included `clientManagedHandoffs: true`. No actual delegated coding request was tested. |
| Externally supplied text was accepted | `thread/realtime/appendSpeech` returned successfully, and the browser received `session.context.appended`. |
| Speech input reached the service | An 8.308-second prerecorded sample played through a running browser AudioContext into the outgoing audio track. Input-transcript events followed. Their text was not retained. |
| Output-transcript events arrived | Two output fragments were ` voice connection` and ` verified.`. |
| Incoming audio packets arrived | The final inbound RTP report counted 944 packets and 14,210 bytes. |
| The test shut down | The probe requested realtime stop, received a closed notification, and terminated its app-server and Chromium children. |

The service accepted account authentication without a separately supplied API key. We did not inspect billing records, subscription quota consumption, or entitlement rules. Do not describe this as unlimited or free voice access.

## Limits that matter for implementation

**Audible speech is unverified.** The inbound report showed `totalAudioEnergy: 0` and `totalSamplesReceived: 0`. Packets alone do not establish decoded speech or successful playback. The headless probe deliberately used a silent output sink. Test normal speaker playback and inspect the negotiated tracks/codecs before claiming an end-to-end voice round trip.

**The output transcript is not proof of a response to the sample.** The probe appended speakable text before playing the sample. Output-transcript events appeared before the first input-transcript event. The log therefore cannot establish that the reply was caused by the sample, or that the entire sample was understood accurately.

**Pi was not involved in the test.** There is no Pi extension, coding handoff, mid-task steering, interruption test, or shared-session context test in these artifacts.

**Client-managed handoffs need further investigation.** The flag's schema says it controls automatic forwarding of Codex responses. Do not assume that it disables Codex's own backend execution or redirects requests to Pi. Verify delegation ownership before submitting any real coding request, to avoid running the same task twice.

**Third-party subscription voice is experimental.** Technical acceptance on this account and version does not prove documented support for arbitrary clients, other accounts, or future releases.

## Reproduce the successful connection

The archived probe is a research script, not production extension code. It expects `codex` on PATH, `/usr/bin/chromium`, a modern Node.js with global `fetch` and `WebSocket`, and an existing Codex account sign-in.

It uses fixed `/tmp` paths and debugging port 19347. To reproduce its original layout, run from this directory:

```bash
cp evidence/original-probe.mjs /tmp/pi-voice-probe.mjs
cp evidence/test-input.wav /tmp/pi-voice-test-input.wav
node /tmp/pi-voice-probe.mjs
```

Inspect the console and `/tmp/pi-voice-probe-result.json`. The archived script catches errors and can exit with status zero even when a test fails. Look for `kind: failure`, connection state, and service errors instead of relying on its exit status.

The first sandboxed attempt failed because Codex could not initialize its SQLite runtime under `~/.codex`. The successful tests ran with permission to access that runtime and the network. A local Pi process needs those ordinary runtime permissions. This was a local sandbox failure, not a rejected voice entitlement.

The probe creates a separate ephemeral thread and browser profile. It does not attach to the user's active voice conversation or access the microphone. Its Chromium `--no-sandbox` flag and debugging setup were test conveniences; do not copy them into the extension's normal browser launch.

## Proven app-server request sequence

Spawn a dedicated child process with stdin/stdout JSON messages:

```bash
codex app-server --stdio -c features.realtime_conversation=true
```

Remove API-key environment variables for this experiment. Let Codex manage its existing credentials and refresh behavior rather than reading or exporting OAuth tokens into browser code.

Send one JSON object per line. Match replies by `id`, and process notifications independently. The tested initialization was:

```json
{"id":1,"method":"initialize","params":{"clientInfo":{"name":"pi_voice_probe","version":"0.1.0"},"capabilities":{"experimentalApi":true}}}
{"method":"initialized"}
{"id":2,"method":"account/read","params":{"refreshToken":false}}
```

Then create a thread:

```json
{"id":3,"method":"thread/start","params":{"cwd":"/tmp","ephemeral":true,"sandbox":"read-only","approvalPolicy":"never","baseInstructions":"Connection test only. Do not execute tools or start coding tasks."}}
```

Keep the returned `result.thread.id`. The thread settings above are only the probe settings, not a permission policy for Pi's real coding work.

In the browser, create an `RTCPeerConnection`, an audio track, and a data channel named `oai-events`. Create an offer, set the local description, and wait for ICE gathering. The probe used a synthetic track with no ICE servers configured. For the product, obtain microphone input after the user starts voice.

Send this request, substituting the actual thread ID and SDP offer:

```json
{
  "id": 4,
  "method": "thread/realtime/start",
  "params": {
    "threadId": "<thread-id>",
    "outputModality": "audio",
    "version": "v3",
    "transport": {"type": "webrtc", "sdp": "<offer-sdp>"},
    "clientManagedHandoffs": true,
    "includeStartupContext": false,
    "prompt": "This is a brief voice connection test. When you hear any speech, reply with only: voice connection verified. Do not delegate work or use tools."
  }
}
```

The RPC reply alone is not connection success. Wait for `thread/realtime/sdp`, read `params.sdp`, and set the browser's remote description to an SDP answer. Handle `thread/realtime/error` during this wait. Confirm the peer connection and data channel states.

These calls were also accepted:

```json
{"id":5,"method":"thread/realtime/appendSpeech","params":{"threadId":"<thread-id>","text":"Pi voice connection test succeeded."}}
{"id":6,"method":"thread/realtime/stop","params":{"threadId":"<thread-id>"}}
```

`appendText` was accepted in an earlier run, but only context-append events were observed. Neither text append method should be assumed to force an immediate spoken response.

On exit, stop realtime, close the peer connection, stop microphone tracks, close AudioContexts, and terminate the dedicated child process. Keep failures and timeouts visible in Pi's status UI.

## Pi extension design to investigate

Proposed structure: Pi `/voice` command, a local bridge process, a localhost browser microphone/speaker UI, and the authenticated Codex app-server. The browser handles WebRTC media. The bridge handles app-server RPC and messages between the voice conversation and the existing Pi session.

Pi's extension documentation exposes command registration, message injection, and event subscriptions. Use the installed Pi version's exported types to confirm exact signatures. See [Pi extension docs](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/extensions.md).

Suggested behavior:

- `/voice` starts or opens the voice UI; `/voice stop` closes it. Register a command handler for `voice` and parse its arguments.
- Start long-lived resources from the command, and release them on session shutdown or extension reload.
- Keep Pi responsible for tool execution and its existing permission decisions.
- Send confirmed delegated requests into Pi with `sendUserMessage`. Use steering or follow-up delivery deliberately when Pi is already working.
- Observe Pi's message and tool events to provide concise progress and final results to voice. Do not speak raw tool output or every streamed token.
- Preserve delegation IDs and deduplicate events. Transcript fragments are not separate user requests.
- Keep browser-accessible routes local, and keep account credentials inside Codex. The browser only needs media negotiation and bridge messages.

The connection experiment establishes the transport component. The routing component remains a design hypothesis.

## Next implementation checkpoints

1. Reproduce the connection against the installed Codex version and generated schemas.
2. Verify actual microphone input and audible speaker output in a regular browser. Capture input/output transcripts and meaningful audio statistics.
3. Trigger one harmless delegation and inspect both browser events and app-server notifications. Establish whether Codex starts a backend turn despite `clientManagedHandoffs`.
4. Determine the exact V3 delegation completion/update messages and who owns sending them. A generic speech append may not complete a pending delegation.
5. Connect a trivial Pi request to that delegation, return its result, and prove the voice conversation speaks it. Confirm one Pi turn and no duplicate Codex coding turn.
6. Test steering during a Pi task, interruption of speech, session shutdown, and recovery from a disconnected voice service.

If stock app-server cannot leave backend work to Pi, investigate a small change to the open-source runtime or a different connection boundary. Do not silently substitute Codex for Pi's coding agent.

## Source map

These GitHub links point at moving `main`, not a captured commit of the installed binary. Recheck them and regenerate local schemas when versions change.

| Source | What to inspect |
|---|---|
| [Codex provider configuration](https://github.com/openai/codex/blob/main/codex-rs/model-provider-info/src/lib.rs) | Account-authenticated backend routing. |
| [Codex authenticated client](https://github.com/openai/codex/blob/main/codex-rs/core/src/client.rs) | WebRTC call creation and retention of matching control-connection authentication. |
| [Voice call endpoint](https://github.com/openai/codex/blob/main/codex-rs/codex-api/src/endpoint/realtime_call.rs) | Backend and public-API request shapes and endpoint tests. |
| [Realtime protocol](https://github.com/openai/codex/blob/main/codex-rs/app-server-protocol/src/protocol/v2/realtime.rs) | Experimental start, transport, append, and stop parameters. |
| [Realtime conversation implementation](https://github.com/openai/codex/blob/main/codex-rs/core/src/realtime_conversation.rs) | Separate WebSocket/WebRTC paths, handoff routing, and lifecycle. The standalone WebSocket branch still requires an API key; the successful test used WebRTC. |
| [V3 event parser](https://github.com/openai/codex/blob/main/codex-rs/codex-api/src/endpoint/realtime_websocket/protocol_frameless_bidi.rs) | Transcript events and parsing of `delegation.created` events targeting `client`. This parser does not establish who executes the delegation. |
| [Public GPT-Live docs](https://developers.openai.com/api/docs/guides/live) | Separately documented API-key route and client delegation architecture. |
| [ChatGPT-plan usage limitations](https://developers.openai.com/siwc/token-sharing-open-source/preview-limitations) | Restrictions on the public plan-sharing flow. These do not describe every first-party Codex voice connection. |

Generate the current protocol locally:

```bash
codex app-server generate-json-schema --experimental --out /tmp/pi-voice-current-schema
```

## Suggested prompt for the Pi implementation agent

> Read this handoff, its archived probe, results, and installed-version schemas. Build a Pi `/voice` extension that reuses the tested Codex account-authenticated WebRTC connection. First verify audible playback and delegation ownership. Keep the existing Pi session as the coding backend. Do not assume clientManagedHandoffs disables Codex execution, and do not require an OpenAI API key. Report what is tested versus inferred, and stop each test session cleanly.