# gpt-realtime

A native macOS voice companion plus a reusable Node.js library, CLI, and browser
bridge for `gpt-realtime-2.1`. It supports live speech, streaming transcripts,
server-side WebSockets, browser WebRTC session minting, typed tools, and
injectable Codex or API-key auth.

> The package name is historical: this targets **GPT Realtime 2.1**, not the
> original GPT-2 text model.

## What is included

- `RealtimeClient`: a server-side WebSocket client for text, PCM16 audio,
  transcripts, interruption, raw events, reasoning, and function tools.
- `RealtimeTranscriptionClient`: a dedicated server-side streaming
  speech-to-text client for `gpt-realtime-whisper`, with item-aware deltas and
  manual audio commits.
- `ToolRegistry`: register tools once and reuse them in WebSocket clients and
  browser sessions.
- `RealtimeSessionService`: mint short-lived `ek_...` credentials without
  exposing the real credential to a browser.
- `TranscriptionSessionService`: mint short-lived transcription credentials
  with the same auth, timeout, and secret-handling infrastructure.
- `createRealtimeHandler` / `createRealtimeServer`: embed the browser bridge in
  another Node project or run the included voice UI.
- `createCodexAuthProvider`: read a Codex `auth.json` file on every connection,
  so refreshed credentials are picked up without restarting.
- `gpt-realtime` / `gptrt`: `serve`, `chat`, and `auth` commands.
- Native SwiftUI/AppKit companion: an animated menu-bar orb and a floating,
  compact top-right conversation panel. It defaults to `gpt-live-1-boulder-alpha` over
  bidirectional WebRTC, with `gpt-realtime-2.1` available from the menu.

Imports are side-effect free. Importing the package does not read auth, create
directories, start a server, or enable shell execution.

## Install and build

```bash
pnpm install
pnpm build
```

Install from npm:

```bash
pnpm add gpt-realtime
# or: npm install gpt-realtime
```

The Node package is ESM-only and requires Node.js 22 or newer. The native app
requires macOS 13 or newer plus Xcode Command Line Tools (`xcode-select --install`)
for its one-time local build.

## Native macOS conversation

On macOS, `chat` now opens the native menu-bar experience instead of printing
responses and writing WAV files:

```bash
npx gpt-realtime chat
```

On first run, Swift builds and ad-hoc signs a small local application in
`~/Library/Caches/gpt-realtime/macos/GPT Realtime.app`. macOS asks for microphone
access once. The app then:

- places an animated multicolor voice orb in the status bar;
- opens a glass conversation panel at the top-right of the active display;
- streams microphone PCM audio directly to the Realtime WebSocket when GPT
  Realtime is selected;
- streams echo-cancelled microphone and speaker audio over WebRTC when GPT Live
  is selected (the default);
- waits through natural speech pauses before completing your turn;
- pauses microphone upload during assistant playback to prevent echo loops;
- plays assistant audio as it arrives, without creating WAV files;
- displays live user and assistant transcripts;
- handles GPT Live client delegations natively, including live web search,
  desktop-window discovery, exact-window/named-app/full-screen visual
  inspection, and Bash tool calls, then returns tool output to the voice session;
- lets you mute, interrupt, clear, retry, hide, or quit; typed messages are
  available when GPT Realtime is selected;
- can run Bash commands in a configured working directory;
- pauses for an explicit Approve/Reject decision before any command that may
  edit files or system state;
- disconnects and releases the microphone whenever the panel is hidden.

Only one app instance may run at a time, preventing duplicate microphone
listeners and overlapping Realtime connections.

The first visual screen inspection asks for macOS Screen Recording permission.
If macOS requests it, enable **GPT Realtime** under **System Settings → Privacy
& Security → Screen & System Audio Recording**, then restart the app. Captures
are resized for model input and their temporary local files are deleted
immediately after encoding.

The conversation panel can be resized by dragging any edge or corner; transcript
text reflows between compact and roomy layouts. Click the status-bar orb to
reopen or hide the panel. Right-click the orb to toggle between **GPT Live 1**
and **GPT Realtime**, clear the conversation, or quit. Configuration is passed
through the same command:

```bash
npx gpt-realtime chat \
  --service gpt-live \
  --live-model gpt-live-1-boulder-alpha \
  --live-voice cove \
  --auth /path/to/auth.json \
  --model gpt-realtime-2.1 \
  --voice marin \
  --shell full \
  --cwd ~/Documents \
  --instructions "Keep responses brief."
```

The native macOS app enables Bash in `full` mode by default. A narrow set of
read-only inspection commands runs immediately. Commands outside that set,
including commands containing shell operators, scripts, and commands that may
write files, are shown in the app and do not run until you choose **Approve and
Run**. Use `--shell readonly` to deny those commands completely or `--shell
none` to disable Bash. The app runs approved commands with `/bin/bash -lc` in
the directory selected by `--cwd` (default: `~/Documents`). Each command that
actually runs is recorded as a **BASH** entry in the chat transcript together
with its standard output, standard error, and nonzero exit status.

This capability is part of the macOS target. iOS applications cannot launch
`/bin/bash` under the iOS application sandbox.

The previous terminal workflow remains available explicitly:

```bash
npx gpt-realtime terminal
npx gpt-realtime terminal --no-audio
```

Use `npx gpt-realtime mac --build-only` to prepare the application bundle
without opening it.

## CLI

```bash
# Validate ~/.codex/auth.json without printing the token
pnpm auth:check

# Browser voice app at http://127.0.0.1:8787
pnpm start

# Experimental GPT-Live Boulder app at http://127.0.0.1:8787/gpt-live

# Explicit auth, workspace, and unrestricted shell tool
pnpm start -- --auth /path/to/auth.json --cwd ./workspace --shell full

# Native macOS menu-bar conversation
pnpm chat

# Legacy terminal REPL
tsx src/bin.ts terminal --no-audio

# Installed package equivalents
gptrt serve --shell readonly --cwd ./files
gptrt terminal --no-audio
gptrt auth --auth /path/to/auth.json
```

`serve` binds to `127.0.0.1` by default. Its shell tool defaults to `readonly`;
use `--shell none` to disable it or `--shell full` only for a trusted local
workflow. Passing `--terminal` or `--no-audio` to `chat` selects the terminal
REPL; `--shell` and `--cwd` now configure the native macOS app directly.

Run `gptrt help` for all flags. You can pass a prompt with `--instructions` or
`--instructions-file ./files/me.md`.

### Experimental GPT-Live browser demo

After `pnpm start`, open
[http://127.0.0.1:8787/gpt-live](http://127.0.0.1:8787/gpt-live). This page
connects `gpt-live-1-boulder-alpha` over WebRTC using the Codex/ChatGPT
`quicksilver=v2` call-creation route. The local Node server reads Codex auth
from `~/.codex/auth.json`; the browser receives only the SDP answer and never
receives the access token.

GPT-Live Boulder and this backend route are experimental rather than part of
the public Realtime API, so their availability and wire contract can change.
The demo supports live microphone/speaker audio, transcripts, and two tools
through GPT-Live client delegation: `run_shell` and `web_search`. `serve`
enables Bash in `readonly` mode by default and exposes a live tool inventory at
`GET /gpt-live/tools`. A server-side Realtime delegation agent selects the
appropriate function tool, translates spoken shell work into a structured
`run_shell` call, and uses `web_search` for current public information. Tool
inputs and results appear in the transcript and the concise result returns with
`delegation.context.append`. Use
`--shell full` only in a trusted local workspace, or `--shell none` to disable
Bash. Keep the server bound to its default localhost address.

## Import in a Node project

### Text session with automatic custom tools

```ts
import {
  RealtimeClient,
  ToolRegistry,
  createCodexAuthProvider,
} from "gpt-realtime";

const tools = new ToolRegistry().register<{ orderId: string }, unknown>({
  type: "function",
  name: "get_order",
  description: "Look up an order by id.",
  parameters: {
    type: "object",
    properties: { orderId: { type: "string" } },
    required: ["orderId"],
    additionalProperties: false,
  },
  execute: async ({ orderId }) => database.orders.find(orderId),
});

const client = new RealtimeClient({
  auth: createCodexAuthProvider({ path: "/path/to/auth.json" }),
  model: "gpt-realtime-2.1",
  reasoningEffort: "low",
  instructions: "Be concise and verify orders with the tool.",
  tools,
});

client.on("text.delta", (text) => process.stdout.write(text));
client.on("tool.result", (call, result) => console.log(call.name, result));
client.on("server.error", console.error);
client.on("client.error", console.error);

await client.connect();
client.sendUserText("Where is order A-123?");
```

When a registered function is called, `RealtimeClient` parses its arguments,
awaits the handler, sends `function_call_output`, and requests the next model
response. Set `autoExecuteTools: false` to handle `tool.call` yourself, then use
`sendToolOutput()` and `createResponse()`.

### Remote MCP servers and built-in connectors

Function tools run in your application. MCP tools and built-in connectors run
inside the Realtime service and can be combined with the same local registry:

```ts
import { RealtimeClient, defineMcpTool } from "gpt-realtime";

const docs = defineMcpTool({
  type: "mcp",
  server_label: "openai_docs",
  server_url: "https://developers.openai.com/mcp",
  allowed_tools: ["search_openai_docs", "fetch_openai_doc"],
  require_approval: "never",
});

const client = new RealtimeClient({ auth, tools, sessionTools: [docs] });
client.on("mcp.event", (event) => console.log(event.type));
client.on("mcp.approval", (request) => {
  // Apply your own policy/UI before approving an action.
  client.respondToMcpApproval(String(request.id), true);
});
```

For a managed connector, use `connector_id` instead of `server_url` and supply
the end user's OAuth token in `authorization`. Keep `allowed_tools` narrow.
`defineMcpTool()` rejects ambiguous URL/connector and authorization shapes.
You can also attach MCP tools to one turn with
`client.createResponse({ tools: [docs] })`.

### Pass auth directly

```ts
import { RealtimeClient, createApiKeyAuth } from "gpt-realtime";

const withApiKey = new RealtimeClient({
  auth: createApiKeyAuth(process.env.OPENAI_API_KEY!),
});

const withCustomProvider = new RealtimeClient({
  auth: async () => ({
    accessToken: await secrets.get("openai-token"),
    accountId: "optional-chatgpt-account-id",
  }),
});
```

The auth provider is invoked on each connection or session mint. Credentials
stay server-side. Never put an API key or Codex access token in browser code.

### Embed browser WebRTC support

```ts
import {
  RealtimeSessionService,
  createRealtimeHandler,
  createCodexAuthProvider,
} from "gpt-realtime";

const sessions = new RealtimeSessionService({
  auth: createCodexAuthProvider(),
  defaultInstructions: "You are the voice assistant for Acme.",
  tools,
  sessionTools: [docs], // optional hosted MCP servers/connectors
  safetyIdentifier: hashInternalUserId(user.id),
});

const handler = createRealtimeHandler({
  sessions,
  tools,
  publicDir: false, // expose API routes only in an existing server
});

// Mount `handler` in a Node HTTP server, or call:
// createRealtimeServer({ sessions, tools }).listen(8787, "127.0.0.1");
```

The handler provides:

- `GET /session?model=...&voice=...` — mint an ephemeral browser credential.
- `GET /transcription-session?language=en&delay=low` — mint an ephemeral
  `gpt-realtime-whisper` transcription credential.
- `GET /models` — return the configured model catalog.
- `POST /gpt-live/session?voice=cove&instructions=...` — proxy an SDP offer to
  the experimental GPT-Live Boulder call-creation route.
- `GET /gpt-live/tools` — list the delegation agent's enabled tools and count.
- `POST /gpt-live/delegate` — run a natural-language task through the
  server-side tool delegation agent.
- `POST /tools/:name` — execute a registered server-side tool.
- Static UI routes when `publicDir` is enabled.

If your framework uses Fetch `Request`/`Response` objects rather than Node's
`IncomingMessage`/`ServerResponse`, call `RealtimeSessionService.mint()` and
`ToolRegistry.execute()` directly from framework-native routes.

## Audio and low-level control

Realtime PCM audio is 24 kHz, mono, signed 16-bit. In an audio client:

```ts
const client = new RealtimeClient({ audio: true, voice: "marin", auth });
client.on("audio.delta", (pcmBuffer) => playback.write(pcmBuffer));
client.on("transcript.delta", (delta, role) => console.log(role, delta));

await client.connect();
client.appendAudio(pcmChunk);
client.commitAudio();       // manual turn mode only
client.createResponse();
client.cancelResponse();    // interruption
```

Use `sendRaw()` as an escape hatch for newer client events while keeping the
rest of the package abstractions.

## Streaming transcription with GPT-Realtime-Whisper

Use the dedicated transcription client when you need live text without an
assistant response:

```ts
import {
  RealtimeTranscriptionClient,
  createCodexAuthProvider,
} from "gpt-realtime";

const client = new RealtimeTranscriptionClient({
  auth: createCodexAuthProvider(),
  language: "en",
  delay: "low", // minimal, low, medium, high, or xhigh
  include: ["item.input_audio_transcription.logprobs"],
});

client.on("transcript.delta", ({ itemId, delta }) => {
  process.stdout.write(`[${itemId}] ${delta}`);
});
client.on("transcript.completed", ({ itemId, transcript }) => {
  console.log(`\n[${itemId}] ${transcript}`);
});
client.on("server.error", console.error);
client.on("client.error", console.error);

await client.connect();
client.appendAudio(pcm16Mono24kChunk);
client.commitAudio();
```

`connect()` resolves only after the service accepts the transcription session
update. GPT-Realtime-Whisper requires 24 kHz mono PCM, does not support
transcription prompts or server turn detection, and therefore requires
`commitAudio()` at each boundary. These constraints are validated before a
network request. Completion ordering across different turns is not guaranteed,
so delta and completion events expose `itemId` and `contentIndex`.

For browser or mobile WebRTC clients, mint an ephemeral secret server-side:

```ts
import { TranscriptionSessionService } from "gpt-realtime";

const sessions = new TranscriptionSessionService({
  auth: createCodexAuthProvider(),
  defaultLanguage: "en",
  defaultDelay: "low",
});

const secret = await sessions.mint();
// Return only `secret.value` and non-sensitive session metadata to the client.
```

## Security and auth notes

- Codex auth loading is read-only; this package never refreshes or writes the
  Codex credential store. Run `codex login` when the access token expires.
- ChatGPT/Codex OAuth credentials working with an endpoint does not make that
  flow a stable public authentication contract. For production applications,
  prefer an OpenAI API key or another officially supported server credential.
- The browser receives only a short-lived ephemeral secret.
- Shell execution is opt-in at library level. Even `readonly` is a convenience
  allowlist, not a strong security sandbox; do not expose the demo server to an
  untrusted network.
- For applications with end users, set a stable, privacy-preserving
  `safetyIdentifier` on the session service/client.

## Project layout

| Path | Responsibility |
| --- | --- |
| `src/index.ts` | Stable public exports |
| `src/auth.ts` | Injectable auth and read-only Codex loader |
| `src/tools.ts` | Tool definitions, handlers, and registry |
| `src/realtime.ts` | Server WebSocket client |
| `src/socket.ts` | Shared authenticated WebSocket transport |
| `src/session.ts` | Ephemeral client-secret service |
| `src/transcription.ts` | GPT-Realtime-Whisper client and session service |
| `src/server.ts` | Embeddable Node HTTP routes and static UI |
| `src/exec.ts` | Explicitly-created optional shell tool |
| `src/bin.ts` | CLI entry point |
| `src/macos.ts` | Native companion build/launch bridge |
| `macos/GPTRealtimeMac` | Swift menu-bar app, Realtime audio, and panel UI |
| `examples/custom-tools.ts` | Custom-tool integration example |
| `test/library.test.ts` | Auth, registry, and session contract tests |

## Verify

```bash
pnpm typecheck
pnpm test
pnpm build
pnpm pack --dry-run
swift test --package-path macos/GPTRealtimeMac
```

The Realtime GA flow uses `/v1/realtime`, `/v1/realtime/client_secrets`, and
`/v1/realtime/calls`. The browser uses WebRTC; the Node client uses WebSocket.
See OpenAI's current [Realtime overview](https://developers.openai.com/api/docs/guides/realtime),
[WebSocket guide](https://developers.openai.com/api/docs/guides/realtime-websocket),
and [Realtime tools/MCP guide](https://developers.openai.com/api/docs/guides/realtime-mcp).