<!--
  Canonical spec for fabric-harness voice-telephony connectors.
  Shipped with @fabric-harness/sdk.

  Raw GitHub URL:
    https://raw.githubusercontent.com/Fabric-Pro/fabric-harness/main/packages/sdk/connector-spec/voice-telephony.md
-->

# fabric-harness Voice-Telephony Connector Spec

This document is the contract for bridging a telephony provider (Twilio Media Streams, Vonage, Plivo, your in-house Asterisk / FreeSWITCH gateway) to a fabric-harness `VoiceSession` so phone calls can drive an LLM voice agent.

If you are an AI coding agent reading this to build a connector for a user, follow this document literally and produce a single TypeScript file that exports a factory function returning an HTTP/WS handler that bridges the provider's audio stream to a `VoiceSession`.

---

## High-Level Shape

A telephony bridge has three responsibilities:

1. **Inbound audio**: receive provider audio frames, resample to PCM 16-bit little-endian at 24kHz mono, and call `voice.sendAudio(frame)`.
2. **Outbound audio**: subscribe to `voice.events()`, take each `audio_delta`, resample to the provider's expected format, and write back to the provider's stream.
3. **Lifecycle**: handle DTMF, hangup, transfer, and barge-in (cancel the current LLM response when the caller speaks).

The connector does NOT do any LLM work — fabric-harness owns the model. The connector only handles audio I/O and call control.

---

## Imports You Will Use

All from `@fabric-harness/sdk`:

- `OpenAIRealtimeVoiceProvider` — the LLM-side connection
- `VoiceSession` — what the bridge drives
- `VoiceEvent` — what the bridge consumes

---

## High-Level Flow

```ts
// connectors/twilio-bridge.ts
import { OpenAIRealtimeVoiceProvider, type VoiceSession } from '@fabric-harness/sdk';

export interface TwilioBridgeOptions {
  apiKey: string;             // OpenAI Realtime key
  voice?: string;
  instructions?: string;
}

export function createTwilioBridge(options: TwilioBridgeOptions) {
  const provider = new OpenAIRealtimeVoiceProvider({ apiKey: options.apiKey });

  // Twilio sends a WebSocket per call to your <Stream> endpoint.
  return async function handleTwilioWs(ws: WebSocket) {
    const voice = await provider.connect({
      audioFormat: 'g711_ulaw',     // Twilio's native format — no resample needed!
      voice: options.voice,
      instructions: options.instructions,
    });

    // Twilio → fabric-harness
    ws.addEventListener('message', (ev) => {
      const msg = JSON.parse(typeof ev.data === 'string' ? ev.data : '');
      if (msg.event === 'media' && msg.media?.payload) {
        // Twilio sends base64 μ-law 8kHz; OpenAI accepts g711_ulaw directly.
        const audio = base64ToBytes(msg.media.payload);
        void voice.sendAudio(audio);
      } else if (msg.event === 'stop') {
        void voice.close();
        ws.close();
      }
    });

    // fabric-harness → Twilio
    void (async () => {
      for await (const event of voice.events()) {
        if (event.type === 'audio_delta') {
          ws.send(JSON.stringify({
            event: 'media',
            media: { payload: bytesToBase64(event.audio) },
          }));
        }
      }
    })();
  };
}
```

The user wires that handler behind their HTTP framework (Hono, Express, etc.) and points Twilio's `<Stream url="wss://...">` TwiML at it.

---

## Audio Format Cheat Sheet

| Provider | Native format | OpenAI Realtime `audioFormat` |
|---|---|---|
| Twilio Media Streams | μ-law 8kHz mono | `'g711_ulaw'` (no resample) |
| Vonage Voice API | linear PCM 16-bit 16kHz | `'pcm16'` (resample 16k → 24k) |
| Plivo Audio Stream | μ-law 8kHz mono | `'g711_ulaw'` |
| Browser WebRTC | Opus 48kHz | `'pcm16'` (decode + resample) |
| WebSocket from native CLI | bring-your-own | `'pcm16'` (recommended) |

When the provider's native format matches one of OpenAI Realtime's accepted formats (`g711_ulaw`, `g711_alaw`, `pcm16`), set `audioFormat` accordingly and skip the resample. **This is the cheapest and lowest-latency path.**

---

## DTMF + Hangup

Most providers send DTMF as a separate event (not as audio). Surface it to the LLM by either:

- Sending a `voice.sendText('Caller pressed 1')` for the LLM to interpret, OR
- Using a `dtmf_received` tool the LLM can call when it asks for a numeric choice.

On hangup, call `voice.close()` and your underlying WS / SIP session.

---

## Barge-in

The OpenAI Realtime server-side VAD detects when the caller starts speaking and emits an `input_audio_buffer.speech_started` event. fabric-harness translates this into a session-internal cancel — your bridge doesn't need to do anything special unless you want to suppress outbound audio while the caller is talking. To do that, listen for the SDK event and stop forwarding `audio_delta` to the provider until the next `response.created`.

---

## Cost

Realtime audio is billed per audio token. Fabric Harness records `audioInputTokens` /
`audioOutputTokens` on `usage` and rolls them into `costUsd`. Use `gpt-realtime` as the stable
general-purpose model or `gpt-realtime-mini` when cost and latency matter more than maximum quality.

---

## Checklist Before Submitting

- [ ] Single TypeScript file, no inline test code, exports a factory.
- [ ] Accepts the provider's native audio frame format and forwards to `voice.sendAudio`.
- [ ] Subscribes to `voice.events()` and writes `audio_delta` back to the provider.
- [ ] Handles the provider's hangup/stop signal by calling `voice.close()`.
- [ ] DTMF surfaced as either `voice.sendText` or a tool call (document which).
- [ ] No hardcoded secrets — keys flow through options.
- [ ] Imports use `@fabric-harness/sdk` (not subpaths like `/dist/...`).
- [ ] Top-of-file comment names the provider, the SDK version targeted, and the audio format used.

If the provider's docs are unclear on any frame format, leave a `// TODO` comment with a specific question rather than guessing.