# Dialog A modular framework for building VoIP-Agent applications. ## Introduction Dialog is an orchestration layer for VoIP-Agent applications. Two _common_ VoIP-Agent models exist today: the Speech-to-Speech (S2S) model and the Speech-to-Text with Text-to-Speech (STT–TTS) model. The S2S model converts spoken input into spoken output, while the STT–TTS model first converts speech into text, which is processed by an Agent; the Agent’s textual response is then converted back into speech. Both approaches involve tradeoffs. Dialog adopts the STT–TTS model. It orchestrates communication between the VoIP, STT, TTS, and Agent components. The framework provides concrete implementations of VoIP, STT, and TTS classes, along with abstract Agent classes designed for subclassing. ### Features - Extensible, modular framework - Concrete implementations for VoIP, STT, and TTS, plus abstract Agent classes for extension - Easily extended - implement any STT, TTS, or VoIP component of your choice - Multithreaded deployments - Event-driven architecture - Isolated state - components exchange data while keeping state ownership local and avoiding dependencies on mutations made by other components ## Table of contents - [Installation](#installation) - [Usage](#usage) - [Examples](#examples) - [Architecture](#architecture) - [Implementations](#implementations) - [Custom Implementations](#custom-implementations) - [Multithreading](#multithreading) - [API](#api) - [Troubleshooting](#troubleshooting) - [Tests](#tests) - [Alternatives](#alternatives) - [Support](#support) ## Installation ### Install from NPM ```bash npm install @far-analytics/dialog --save ``` ### Development Installation These instructions describe how to clone the Dialog repository and build the package. #### Clone the repository. ```bash git clone https://github.com/adamjpatterson/dialog.git ``` #### Change directory into the Dialog repository. ```bash cd dialog ``` #### Install the package dependencies. ```bash npm install ``` #### Build the Dialog package. You can use the `clean:build` script in order to do a clean build. ```bash npm run clean:build ``` Alternatively, you can use the `watch` script in order to watch and build the package. This will build the package each time you make a change to a file in `./src`. If you use the `watch` script, you will need to open a new terminal in order to build and run your application. ```bash npm run watch ``` ### Install Dialog into your package #### Change directory into your package directory and install the package. ```bash npm install --save ``` You should now be able to import Dialog artifacts into your package. ## Usage ### How it works When a call is initiated, a `Gateway` (e.g., a Twilio Gateway) emits a `voip` event. The `voip` handler is called with a `VoIP` instance as its single argument. The `VoIP` instance handles the WebSocket connection that is set on it by the `Gateway`. In the `voip` handler, an instance of an `Agent` is constructed by passing a `VoIP`, `STT`, and `TTS` implementation into its constructor. The agent is started by calling its `activate` method. The `activate` method of the `Agent` instance connects the interfaces that comprise the application. An important characteristic of the architecture is that a _new_ `VoIP`, `STT`, `TTS`, and `Agent` instance is created for every call. This allows each instance to maintain state specific to its call. Excerpted from `./examples/custom_twilio_voip_openai_agent/src/main.ts`. ```ts ... const gateway = new TwilioGateway({ httpServer, webSocketServer, webhookURL: new URL(WEBHOOK_URL), authToken: TWILIO_AUTH_TOKEN, accountSid: TWILIO_ACCOUNT_SID, }); gateway.on("voip", (voip: TwilioVoIP) => { const agent = new TwilioCustomAgent({ voip: voip, stt: new DeepgramSTT({ apiKey: DEEPGRAM_API_KEY, liveSchema: DEEPGRAM_LIVE_SCHEMA }), tts: new CartesiaTTS({ apiKey: CARTESIA_API_KEY, speechOptions: CARTESIA_SPEECH_OPTIONS }), apiKey: OPENAI_API_KEY, system: OPENAI_SYSTEM_MESSAGE, greeting: OPENAI_GREETING_MESSAGE, model: OPENAI_MODEL, twilioAccountSid: TWILIO_ACCOUNT_SID, twilioAuthToken: TWILIO_AUTH_TOKEN, }); agent.activate(); }); ... ``` ## Examples Example implementations are provided in the [examples](https://github.com/adamjpatterson/dialog/tree/main/examples/) subpackages. ### _Custom Twilio VoIP, Deepgram STT, Cartesia TTS, and OpenAI Agent_ In the [Custom Twilio VoIP, Deepgram STT, Cartesia TTS, and OpenAI Agent](https://github.com/adamjpatterson/dialog/tree/main/examples/custom_twilio_voip_openai_agent) example you will create a simple hypothetical Agent that prepends its messages with a timestamp and manages its conversation history. ### _Twilio VoIP, Deepgram STT, Cartesia TTS, and OpenAI Agent (Threading)_ In the [Twilio VoIP, Deepgram STT, Cartesia TTS, and OpenAI Agent (Threading)](https://github.com/adamjpatterson/dialog/tree/main/examples/twilio_voip_openai_agent_threading) example you will run each call session and Agent instance in a worker thread. ## Architecture ### Concepts A Dialog orchestration typically consists of one or more of: an **Agent** component, a **STT** component, a **TTS** component, and a **VoIP** component. ##### Agent An **Agent** component is essential to assembling the external LLM, the **VoIP**, **STT**, and **TTS** implementations into a working whole. Dialog, as the _orchestration layer_, does not provide a concrete **Agent** implementation. Instead you are provided with an interface and abstract class that you can implement or subclass with your custom special tool calling logic. For example, an **Agent** will decide when to transfer a call; if the LLM determines the **User** intent is to be transferred, the **Agent** can carry out this intent by calling the `VoIP.transferTo` method - or it could circumvent the provided call transfer facilities entirely and make a direct call to the VoIP provider (e.g., Twilio, Telnyx, etc.) API. The point here is that very little architectural constraints should be imposed on the Agent; this ensures the extensibility of the architecture. ##### STT The **STT** component transcribes the **User** speech into text. The **STT** emits utterance and VAD events that may be consumed by the **Agent**. ##### TTS The **TTS** component synthesizes the text received from the **Agent** and/or LLM. The **TTS** emits message events that may be consumed by the **Agent**. ##### VoIP The **VoIP** component handles the incoming call, transcriptions, recordings, and streams audio into the **STT**. ### Overview Dialog favors simplicity and accessibility over feature richness. Its architecture should meet all the requirements of a typical VoIP-Agent application where many Users interact with a set of Agents. Although Dialog doesn't presently support concepts like "rooms", the simplicity and extensibility of its architecture should lend to even more advanced implementations. #### State Each component in a Dialog orchestration owns and manages its own state (e.g., VoIP, STT, TTS, or Agent). Components may emit messages and consume messages from other components, and they may hold references to each other; however, components are not designed to rely on, or intentionally mutate, state owned by another component. Implementations commonly create updated state snapshots when incorporating incoming metadata or messages. For example, a VoIP component may emit a `Metadata` object containing information about an incoming call that is consumed by other components; the VoIP implementation incorporates metadata updates into its own state rather than relying on consumers to mutate that state. This separation of concerns ensures that component state remains predictable and easy for a _human_ to reason about. **Likewise, the architecture is expected to be easy for LLMs to consume, as the LLM's attention can be focused on the pattern that is exhibited by the relevant component.** #### Data flow ``` +-------------+ audio (base64) +------------+ transcripts +----------+ text +-------------+ | Twilio | ------------------>| STT | -------------------> | Agent | -------> | TTS | | VoIP | --metadata/events--| (Deepgram | --metadata/events--> | (OpenAI) | | (11Labs or | | (WS in/out) | | or OpenAI) | | | | Cartesia) | +-------------+ +------------+ +----------+ +-------------+ ^ v +----------------------------------------------------------------------------------------------+ audio (base64) ``` ## Implementations Dialog provides example [implementations](https://github.com/adamjpatterson/dialog/tree/main/src/implementations) for each of the artifacts that comprise a VoIP-Agent application. You can use a packaged implementation as-is, subclass it, or implement your own. If you choose to implement a custom implementation, you can use one of the provided [interfaces](https://github.com/adamjpatterson/dialog/tree/main/src/interfaces). ### VoIP #### [Twilio](https://github.com/adamjpatterson/dialog/tree/main/src/implementations/voip/twilio) [↗](https://twilio.com/) - Twilio request validation - Recording status - Transcript status - Speech interruption #### [Telnyx](https://github.com/adamjpatterson/dialog/tree/main/src/implementations/voip/telnyx) [↗](https://telnyx.com/) (coming soon) An implementation similar to Twilio is planned. A placeholder exists under `src/implementations/voip/telnyx/`. ### Speech to text (STT) #### [Deepgram](https://github.com/adamjpatterson/dialog/tree/main/src/implementations/stt/deepgram) [↗](https://deepgram.com/) - Voice activity detection (VAD) events #### [OpenAI](https://github.com/adamjpatterson/dialog/tree/main/src/implementations/stt/openai) [↗](https://openai.com/) - Voice activity detection (VAD) events - Semantic VAD ### Text to speech (TTS) #### [Cartesia](https://github.com/adamjpatterson/dialog/tree/main/src/implementations/tts/cartesia) [↗](https://cartesia.ai/) - Configurable voice #### [ElevenLabs](https://github.com/adamjpatterson/dialog/tree/main/src/implementations/tts/elevenlabs) [↗](https://elevenlabs.io/) - Configurable voice ### Agent #### [OpenAI](https://github.com/adamjpatterson/dialog/tree/main/src/implementations/agent/abstract/openai) [↗](https://openai.com/) - An abstract [Agent implementation](https://github.com/adamjpatterson/dialog/blob/main/src/implementations/agent/abstract/openai/openai_agent.ts) is provided that uses the [OpenAI](https://platform.openai.com/docs/overview) API. ## Custom Implementations Dialog provides concrete `VoIP`, `STT`, and `TTS` implementations and an abstract `Agent` implementation. You can use a provided implementation _as-is_, subclass it, or choose an interface and implement your own. If you plan to implement your own `VoIP`, `STT`, `Agent`, or `TTS`, [interfaces](https://github.com/adamjpatterson/dialog/tree/main/src/interfaces) are provided for each component of the application. ### Custom Agents A custom `Agent` implementation will allow you to facilitate tool calling, conversation history, and other nuances. You can extend the provided `OpenAIAgent` class, as in the example below, or just implement the `Agent` interface. The straight-forward `openai_agent.ts` [implementation](https://github.com/adamjpatterson/dialog/blob/main/src/implementations/agent/abstract/openai/openai_agent.ts) can be used as a guide. #### A custom `Agent` based on `openai_agent.ts`. This hypothetical custom `Agent` implementation adds a timestamp to each user message and maintains conversation history. ```ts import { once } from "node:events"; import { randomUUID } from "node:crypto"; import { log, Message, OpenAIAgent, OpenAIAgentOptions, TwilioMetadata, TwilioVoIP, OpenAIConversationHistory, } from "@far-analytics/dialog"; export interface TwilioCustomAgentOptions extends OpenAIAgentOptions { twilioAccountSid: string; twilioAuthToken: string; system?: string; greeting?: string; } export class TwilioCustomAgent extends OpenAIAgent { protected metadata?: TwilioMetadata; protected twilioAccountSid: string; protected twilioAuthToken: string; protected history: OpenAIConversationHistory; protected transcript: unknown[]; protected system: string; protected greeting: string; constructor(options: TwilioCustomAgentOptions) { super(options); this.twilioAccountSid = options.twilioAccountSid; this.twilioAuthToken = options.twilioAuthToken; this.transcript = []; this.system = options.system ?? ""; this.greeting = options.greeting ?? ""; if (this.system) { this.history = [ { role: "system", content: this.system, }, ]; } else { this.history = []; } } public inference = async (message: Message): Promise => { try { const content = `${new Date().toISOString()}\n${message.data}`; log.notice(`User message: ${content}`); this.history.push({ role: "user", content }); const stream = await this.openAI.chat.completions.create({ model: this.model, messages: this.history, temperature: 1, stream: true, }); const assistantMessage = await this.dispatchStream(message.uuid, stream); log.notice(`Assistant message: ${assistantMessage} `); this.history.push({ role: "assistant", content: assistantMessage }); } catch (err) { this.dispose(err); } }; public updateMetadata = (metadata: TwilioMetadata): void => { if (!this.metadata) { this.metadata = metadata; } else { this.metadata = { ...this.metadata, ...metadata }; } }; public activate = (): void => { super.activate(); this.voip.on("streaming_started", this.dispatchInitialMessage); this.voip.on("streaming_started", this.startDisposal); this.voip.on("metadata", this.updateMetadata); }; public deactivate = (): void => { super.deactivate(); this.voip.off("streaming_started", this.dispatchInitialMessage); this.voip.off("streaming_started", this.startDisposal); this.voip.off("metadata", this.updateMetadata); }; public dispatchInitialMessage = (): void => { const uuid = randomUUID(); this.activeMessages.add(uuid); this.history.push({ role: "assistant", content: this.greeting }); this.dispatchMessage({ uuid: uuid, data: this.greeting, done: true }, false).catch(this.dispose); }; protected startDisposal = (): void => { void (async () => { try { await once(this.voip, "streaming_stopped"); this.dispose(); } catch (err) { log.error(err); } })(); }; } ``` ## Multithreading Dialog provides a simple multithreading implementation you can use. An [example](https://github.com/adamjpatterson/dialog/tree/main/examples/twilio_voip_openai_agent_threading) is provided that demonstrates a multithreaded deployment. A `Worker` is spun up for each call. VoIP events are propagated over a `MessageChannel` using the [Port Peer](https://github.com/adamjpatterson/port-peer) RPC-like facility. This approach ensures that any peculiarity that takes place in handling one call will not interfere with other concurrent calls. Another notable aspect of this approach is that it permits hot changes to the Agent (and the STT and TTS) code without interrupting calls that are already underway - new calls will pick up changes each time a `Worker` is spun up. In the excerpt below, a `TwilioVoIPWorker` is instantiated on each call. Excerpted from `./examples/twilio_voip_openai_agent_threading/src/main.ts`. ```ts const gateway = new TwilioGateway({ httpServer, webSocketServer, webhookURL: new URL(WEBHOOK_URL), authToken: TWILIO_AUTH_TOKEN, accountSid: TWILIO_ACCOUNT_SID, requestSizeLimit: 1e6, }); gateway.on("voip", (voip: TwilioVoIP) => { new TwilioVoIPWorker({ voip, worker: new Worker("./dist/worker.js") }); }); ``` Over in `worker.js` the Agent is instantiated, as usual, except using a `TwilioVoIPProxy` instance that implements the `VoIP` interface. Excerpted from `./examples/twilio_voip_openai_agent_threading/src/worker.ts`. ```ts const voip = new TwilioVoIPProxy(); const agent = new Agent({ voip: voip, stt: new DeepgramSTT({ apiKey: DEEPGRAM_API_KEY, liveSchema: DEEPGRAM_LIVE_SCHEMA, }), tts: new CartesiaTTS({ apiKey: CARTESIA_API_KEY, speechOptions: CARTESIA_SPEECH_OPTIONS, }), apiKey: OPENAI_API_KEY, system: OPENAI_SYSTEM_MESSAGE, greeting: OPENAI_GREETING_MESSAGE, model: OPENAI_MODEL, twilioAccountSid: TWILIO_ACCOUNT_SID, twilioAuthToken: TWILIO_AUTH_TOKEN, }); agent.activate(); ``` ## API Dialog provides building blocks to create real‑time, voice‑driven agents that integrate telephony (VoIP), speech‑to‑text (STT), text‑to‑speech (TTS), and LLM agents. It includes interfaces, utility classes, and concrete implementations for Twilio VoIP, Deepgram STT, OpenAI Realtime STT, ElevenLabs TTS, Cartesia TTS, and an OpenAI‑based agent. The API is organized by component. You can mix and match implementations by wiring them through the provided interfaces. ### Common logging utilities The logging utilities are thin wrappers around `streams-logger` for structured, backpressure‑aware logging. #### log, formatter, consoleHandler, SyslogLevel - log `` An initialized Logger pipeline emitting to the console via the included `formatter` and `consoleHandler`. - formatter `>` Formats log records into human‑readable strings. - consoleHandler `>` A console sink with level set to DEBUG. - SyslogLevel `` The syslog‑style levels exported from `streams-logger`. Use these exports in order to emit structured logs across the library. See `streams-logger` for details on usage and configuration. ### The StreamBuffer class #### new StreamBuffer(options?, writableOptions?) - options `` - bufferSizeLimit `` Optionally specify a maximum buffer size in bytes. **Default: `1e6`** - writableOptions `` Optional Node.js stream options; use to customize highWaterMark, etc. Use a `StreamBuffer` in order to buffer incoming stream chunks into a single in‑memory `Buffer` with an upper bound. If the buffer exceeds the limit, an error is emitted. _public_ **streamBuffer.buffer** - `` The accumulated buffer contents. ### The Mutex class #### new Mutex(options?) - options `` - queueSizeLimit `` A hard limit imposed on all mark queues. `mutex.call` rejects if this limit is exceeded. Use a `Mutex` in order to serialize asynchronous calls by key. _public_ **mutex.call(mark, fn, ...args)** - mark `` A key identifying the critical section. - fn `<(...args: unknown[]) => Promise>` An async function to execute exclusively per key. - ...args `` Arguments forwarded to `fn`. Returns: `>` Acquire the mutex for `mark`, invoke `fn`, and release the mutex, even on error. _public_ **mutex.acquire(mark)** - mark `` A key identifying the critical section. Returns: `>` Wait until the mutex for `mark` is available and acquire it. _public_ **mutex.release(mark)** - mark `` A key identifying the critical section. Returns: `` Release a previously acquired mutex for `mark`. Throws if called without a corresponding acquire. ### Core interfaces These interfaces define the contracts between VoIP, STT, TTS, and Agent components. #### interface Message\ - uuid `` A unique identifier for correlation across components. - data `` The payload: audio (base64) or text, depending on the context. - done `` Whether the message is complete (end of stream/utterance). #### interface Agent - inference `(message: Message) => Promise` Implement the main inference loop for a message. - activate `() => void` Begin wiring events between components. - deactivate `() => void` Remove event wiring. #### interface STT Extends: `EventEmitter` Events (STTEvents): - `"message"`: `[Message]` Emitted when a finalized transcription is available. - `"vad"`: `[]` Emitted on voice activity boundary events (start/stop cues). - `"error"`: `[unknown]` Emitted on errors. Methods: - post `(media: Message) => void` Post audio media into the recognizer (typically base64 payloads). - dispose `() => void` Dispose resources and listeners. #### interface TTS Extends: `EventEmitter` Events (TTSEvents): - `"message"`: `[Message]` Emitted with encoded audio output chunks, and a terminal chunk with `done: true`. - `"error"`: `[unknown]` Emitted on errors. Methods: - post `(message: Message) => void` Post text to synthesize. When `done` is `true`, the provider should flush and emit the terminal chunk. - abort `(uuid: UUID) => void` Cancel a previously posted message stream. - dispose `() => void` Dispose resources and listeners. #### interface VoIP\ Extends: `EventEmitter>` Events (VoIPEvents): - `"metadata"`: `[MetadataT]` Emitted for call/session metadata updates. - `"message"`: `[Message]` Emitted for inbound audio media frames (base64 payloads). - `"message_dispatched"`: `[UUID]` Emitted when a downstream consumer has finished dispatching a message identified by the UUID. - `"transcript"`: `[TranscriptT]` Emitted for transcription webhook updates, when supported. - `"recording_url"`: `[string]` Emitted with a URL for completed recordings, when supported. - `"streaming_started"`: `[]` Emitted when the media stream starts. - `"streaming_stopped"`: `[]` Emitted when the media stream ends. - `"error"`: `[unknown]` Emitted on errors. Methods: - post `(message: Message) => void` Post synthesized audio back to the call/session. - abort `(uuid: UUID) => void` Cancel an in‑flight TTS dispatch and clear provider state if needed. - hangup `() => void` Terminate the call/session, when supported by the provider. - transferTo `(tel: string) => void` Transfer the call to the specified telephone number, when supported. - dispose `() => void` Dispose resources and listeners. ### Twilio VoIP Twilio implementations provide inbound call handling, WebSocket media streaming, call control, recording, and transcription via Twilio. #### new TwilioGateway(options) - options `` - httpServer `` An HTTP/HTTPS server for Twilio webhooks. - webSocketServer `` A WebSocket server to receive Twilio Media Streams. - webhookURL `` The public webhook URL path for the voice webhook (full origin and path). - accountSid `` Twilio Account SID. - authToken `` Twilio Auth Token. - recordingStatusURL `` Optional recording status callback URL. If omitted, a unique URL on the same origin is generated. - transcriptStatusURL `` Optional transcription status callback URL. If omitted, a unique URL on the same origin is generated. - requestSizeLimit `` Optional limit (bytes) for inbound webhook bodies. **Default: `1e6`** - webSocketMessageSizeLimit `` Optional limit (bytes) for inbound WebSocket messages. **Default: `1e6`** Use a `TwilioGateway` in order to accept Twilio voice webhooks, validate signatures, respond with a TwiML `Connect ` response, and manage the associated WebSocket connection and callbacks. On each new call, a `TwilioVoIP` instance is created and emitted. Events: - `"voip"`: `[TwilioVoIP]` Emitted when a new call is established and its `TwilioVoIP` instance is ready. #### new WebSocketListener(options) - options `<{ webSocket: ws.WebSocket, twilioGateway: TwilioGateway, callSidToTwilioVoIP: Map }>` Use a `WebSocketListener` in order to translate Twilio Media Stream messages into `VoIP` events for the associated `TwilioVoIP` instance. This class is managed by `TwilioGateway` and not typically constructed directly. _public_ **webSocketListener.webSocket** - `` The underlying WebSocket connection. _public_ **webSocketListener.startMessage** - `` The initial "start" message, when received. #### new TwilioVoIP(options) - options `` - metadata `` Initial call/stream metadata. - accountSid `` Twilio Account SID. - authToken `` Twilio Auth Token. - recordingStatusURL `` Recording status callback URL. - transcriptStatusURL `` Transcription status callback URL. Use a `TwilioVoIP` in order to send synthesized audio back to Twilio, emit inbound media frames, and control the call (transfer, hangup, recording, and transcription). `TwilioVoIPInterface` is the structural contract for custom Twilio-compatible VoIP implementations. _public_ **twilioVoIP.post(message)** - message `` Post base64‑encoded audio media back to Twilio over the Media Stream. When `done` is `true`, a marker is sent to allow downstream dispatch tracking. Returns: `` _public_ **twilioVoIP.abort(uuid)** - uuid `` A message UUID to cancel. Sends a cancel marker and clears state; when no active messages remain, a `clear` control message is sent. Returns: `` _public_ **twilioVoIP.transferTo(tel)** - tel `` A destination telephone number in E.164 format. Returns: `` Transfer the active call to `tel` using TwiML. _public_ **twilioVoIP.hangup()** Returns: `` End the active call using TwiML. _public_ **twilioVoIP.startTranscript()** Returns: `>` Start Twilio call transcription (Deepgram engine) with `both_tracks`. _public_ **twilioVoIP.startRecording()** Returns: `>` Begin dual‑channel call recording with status callbacks. _public_ **twilioVoIP.stopRecording()** Returns: `>` Stop the in‑progress recording when applicable. _public_ **twilioVoIP.removeRecording()** Returns: `>` Remove the last recording via the Twilio API. _public_ **twilioVoIP.dispose()** Returns: `` Close the media WebSocket and clean up listener maps. ### Twilio types Helper types and type guards for Twilio webhook and Media Stream payloads. - **Body** `>` A generic Twilio form‑encoded body map. - **CallMetadata** Extends `Body` with required Twilio voice webhook fields. - **isCallMetadata(message)** Returns: `` - **RecordingStatus** Extends `Body` with Twilio recording status fields. - **isRecordingStatus(message)** Returns: `` - **TranscriptStatus** Extends `Body` with Twilio transcription status fields. - **isTranscriptStatus(message)** Returns: `` - **TwilioWebSocketMessage** `{ event: "start" | "media" | "stop" | "mark" }` - **StartWebSocketMessage, MediaWebSocketMessage, StopWebSocketMessage, MarkWebSocketMessage** Specific Twilio Media Stream messages. - **isStartWebSocketMessage / isMediaWebSocketMessage / isStopWebSocketMessage / isMarkWebSocketMessage** Type guards for the above. - **TwilioMetadata** `Partial & Partial` A merged, partial metadata shape for convenience. ### Agent abstractions #### abstract OpenAIAgent\\>(options) - options `>` - voip `` The telephony transport. - stt `` The speech‑to‑text provider. - tts `` The text‑to‑speech provider. - apiKey `` OpenAI API key. - model `` OpenAI Chat Completions model identifier. - queueSizeLimit `` A queueSizeLimit to be passed to the implementation's `Mutex` constructor. Extend `OpenAIAgent` in order to build streaming, interruptible LLM agents that connect STT input, TTS output, and a VoIP transport. It cannot be instantiated directly; subclasses implement `inference` to call OpenAI APIs and stream back responses. _public (abstract)_ **openAIAgent.inference(message)** - message `` A transcribed user message to process. Returns: `>` Implement this to call OpenAI and generate/stream the assistant’s reply. _public_ **openAIAgent.post(message)** - message `` Push a user message into the agent. Ignored if `message.data` is empty. The message UUID is tracked for cancellation. Returns: `` _protected_ **openAIAgent.dispatchStream(uuid, stream, allowInterrupt?)** - uuid `` The message correlation identifier. - stream `>` The OpenAI streaming iterator. - allowInterrupt `` Whether to allow VAD‑driven interruption. **Default: `true`** Returns: `>` Stream assistant tokens to TTS. When `allowInterrupt` is `false`, waits for a downstream `"message_dispatched"` before returning. _protected_ **openAIAgent.dispatchMessage(message, allowInterrupt?)** - message `` A pre‑composed assistant message to play via TTS. - allowInterrupt `` Whether to allow VAD‑driven interruption. **Default: `true`** Returns: `>` Dispatch a complete assistant message to TTS with optional interruption handling. _public_ **openAIAgent.abort()** Returns: `` Abort all active messages that are not currently being dispatched; cancels TTS and instructs the VoIP transport to clear state. _public_ **openAIAgent.dispose(err?)** - err `` Optional error to log. Returns: `` Abort any in‑flight OpenAI stream and dispose TTS, STT, and VoIP transports. _public_ **openAIAgent.setTTS(tts)** - tts `` Replacement TTS implementation. Returns: `` Swap the current TTS implementation, updating event wiring. _public_ **openAIAgent.setSTT(stt)** - stt `` Replacement STT implementation. Returns: `` Swap the current STT implementation, updating event wiring. _public_ **openAIAgent.activate()** Returns: `` Wire up `voip` → `stt` (media), `stt` → `agent` (messages, vad), and `tts` → `voip` (audio). Also subscribes to error and dispatch events. _public_ **openAIAgent.deactivate()** Returns: `` Remove event wiring. ### The TwilioVoIPOpenAIAgent class #### new TwilioVoIPOpenAIAgent(options) - options `` Extends `OpenAIAgentOptions` - twilioAccountSid `` Twilio Account SID used for authenticated media fetch. - twilioAuthToken `` Twilio Auth Token used for authenticated media fetch. - system `` Optional system prompt for conversation history. **Default: `""`** - greeting `` Optional initial assistant greeting. **Default: `""`** Use a `TwilioVoIPOpenAIAgent` in order to run an OpenAI‑driven assistant over a Twilio call. It records the call, starts transcription, streams a greeting on connect, collects conversation history, and disposes once recording and transcription are complete. _public_ **twilioVoIPOpenAIAgent.updateMetadata(metadata)** - metadata `` Merge updated Twilio metadata. Returns: `` _public_ **twilioVoIPOpenAIAgent.activate()** Returns: `` Extends `OpenAIAgent.activate()` by wiring Twilio‑specific events (stream start/stop, recording, transcript) and dispatching the initial greeting. _public_ **twilioVoIPOpenAIAgent.deactivate()** Returns: `` Remove Twilio‑specific wiring in addition to base wiring. ### Speech‑to‑Text implementations #### new DeepgramSTT(options) - options `` - apiKey `` Deepgram API key. - liveSchema `` Deepgram live connection options. - queueSizeLimit `` A queueSizeLimit to be passed to the implementation's `Mutex` constructor. Use a `DeepgramSTT` in order to stream audio to Deepgram Live and emit final transcripts. Emits `vad` on speech boundary messages. Automatically reconnects when needed. _public_ **deepgramSTT.post(message)** - message `` Base64‑encoded (PCM/Telephony) audio chunk. Returns: `` _public_ **deepgramSTT.dispose()** Returns: `` Close the underlying connection and remove listeners. #### new OpenAISTT(options) - options `` - apiKey `` OpenAI API key. - session `` Realtime transcription session configuration. - queueSizeLimit `` A queueSizeLimit to be passed to the implementation's `Mutex` constructor. Use an `OpenAISTT` in order to stream audio to OpenAI Realtime STT and emit `message` on completed transcriptions and `vad` on speech boundary events. _public_ **openaiSTT.post(message)** - message `` Base64‑encoded audio chunk. Returns: `` _public_ **openaiSTT.dispose()** Returns: `` Close the WebSocket and remove listeners. ### Text‑to‑Speech implementations #### new ElevenlabsTTS(options) - options `` - voiceId `` Optional voice identifier. **Default: `"JBFqnCBsd6RMkjVDRZzb"`** - apiKey `` ElevenLabs API key. - headers `>` Optional additional headers. - url `` Optional override URL for the WebSocket endpoint. - queryParameters `>` Optional query parameters appended to the endpoint. - timeout `` Optional timeout in milliseconds to wait for finalization when `done` is set. If the timeout elapses, a terminal empty chunk is emitted. **Default: `undefined`** - queueSizeLimit `` A queueSizeLimit to be passed to the implementation's `Mutex` constructor. Use an `ElevenlabsTTS` in order to stream synthesized audio back as it’s generated. Supports message contexts (UUIDs), incremental text updates, flushing on `done`, and cancellation. _public_ **elevenlabsTTS.post(message)** - message `` Assistant text to synthesize. When `done` is `true`, the current context is closed and finalization is awaited (with optional timeout). Returns: `` _public_ **elevenlabsTTS.abort(uuid)** - uuid `` The context to cancel; sends a flush and close if initialized. Returns: `` _public_ **elevenlabsTTS.dispose()** Returns: `` Close the WebSocket. #### new CartesiaTTS(options) - options `` - apiKey `` Cartesia API key. - speechOptions `>` Provider options merged into each request. - url `` Optional override URL for the WebSocket endpoint. **Default: `"wss://api.cartesia.ai/tts/websocket"`** - headers `>` Optional additional headers merged with required headers. - timeout `` Optional timeout in milliseconds to wait for finalization when `done` is set. If the timeout elapses, a terminal empty chunk is emitted. **Default: `undefined`** - queueSizeLimit `` A queueSizeLimit to be passed to the implementation's `Mutex` constructor. Use a `CartesiaTTS` in order to stream synthesized audio chunks for a given context UUID. Supports cancellation and optional finalization timeouts. _public_ **cartesiaTTS.post(message)** - message `` Assistant text to synthesize; when `done` is `true`, the provider is instructed to flush and complete the context. Returns: `` _public_ **cartesiaTTS.abort(uuid)** - uuid `` The context to cancel. Returns: `` _public_ **cartesiaTTS.dispose()** Returns: `` Close the WebSocket and remove listeners. ### Twilio VoIP worker adapter The following classes enable running VoIP handling in a worker thread using the `port-peer` library. #### new TwilioVoIPWorker(options) - options `` - worker `` The target worker thread to communicate with. - voip `` The local `TwilioVoIP` instance whose events and methods will be bridged. Use a `TwilioVoIPWorker` in order to expose `TwilioVoIP` events and actions to a worker thread. It forwards VoIP events to the worker and registers callables that invoke the corresponding `TwilioVoIP` methods. #### new TwilioVoIPProxy() Use a `TwilioVoIPProxy` in order to consume VoIP events and call VoIP methods from inside a worker thread. It mirrors the `VoIP` interface and delegates the work to a host `TwilioVoIP` via the `port-peer` channel. _public_ **twilioVoIPProxy.post(message)** - message `` Post synthesized audio. Returns: `` _public_ **twilioVoIPProxy.abort(uuid)** - uuid `` The context to cancel. Returns: `` _public_ **twilioVoIPProxy.hangup()** Returns: `` _public_ **twilioVoIPProxy.transferTo(tel)** - tel `` A destination telephone number in E.164 format. Returns: `` _public_ **twilioVoIPProxy.startRecording()** Returns: `>` _public_ **twilioVoIPProxy.stopRecording()** Returns: `>` _public_ **twilioVoIPProxy.removeRecording()** Returns: `>` _public_ **twilioVoIPProxy.startTranscript()** Returns: `>` _public_ **twilioVoIPProxy.dispose()** Returns: `` ### OpenAI STT session and message types Helper types for configuring OpenAI Realtime STT sessions and message discrimination. _public_ **Session** - `` - input_audio_format `<"pcm16" | "g711_ulaw" | "g711_alaw">` - input_audio_noise_reduction `{ type: "near_field" | "far_field" }` Optional noise reduction. - input_audio_transcription `{ model: "gpt-4o-transcribe" | "gpt-4o-mini-transcribe", prompt?: string, language?: string }` - turn_detection `{ type: "semantic_vad" | "server_vad", threshold?: number, prefix_padding_ms?: number, silence_duration_ms?: number, eagerness?: "low" | "medium" | "high" | "auto" }` Discriminated unions for WebSocket messages are also provided with type guards: - `OpenAISTTWebSocketMessage` and `isCompletedWebSocketMessage`, `isSpeechStartedWebSocketMessage`, `isConversationItemCreatedWebSocketMessage`, `isInputAudioTranscriptionDeltaWebSocketMessage`. ### OpenAI agent types _public_ **OpenAIConversationHistory** - `<{ role: "system" | "assistant" | "user" | "developer", content: string }[]>` A conversation history array suitable for OpenAI chat APIs. ## Tests The formal test suite is located in `tests/dialog-test`. It covers the deterministic framework components and orchestration behavior; it does not require credentials or network access to Twilio, OpenAI, Deepgram, Cartesia, or ElevenLabs. ### How to run the test suite The test suite requires Node.js `>=20.9.0` and npm `>=10.1.0`. #### Clone the repository. ```bash git clone https://github.com/adamjpatterson/dialog ``` #### Change directory into the root of the repository. ```bash cd dialog ``` #### Install Dialog dependencies. ```bash npm install ``` #### Run the tests. This command performs a clean build of the library, builds the test workspace, and runs the tests with Node.js’s built-in test runner. ```bash npm test ``` #### Build and lint. To run the other standard checks independently: ```bash npm run build npm run lint:all ``` ### Live integration tests The repository also includes two live Twilio integration applications. These are separate from the formal test suite and require access to external services, valid credentials, and a publicly reachable HTTPS endpoint for Twilio callbacks. They may incur provider usage charges. - `tests/test_twilio` runs the Twilio, Deepgram, Cartesia, and OpenAI voice-agent flow in a single Node.js process. - `tests/test_twilio_threading` runs the same flow with the VoIP bridge in the main process and the agent session in a worker thread. Both applications load environment variables from the repository root `.env` file and require at least `TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, `DEEPGRAM_API_KEY`, `CARTESIA_API_KEY`, and `OPENAI_API_KEY`. They also require paths to an HTTPS `KEY_FILE` and `CERT_FILE`, plus a `WEBHOOK_URL` that Twilio can reach. The live test workspaces require Node.js `>=20.19.0`. Run either application from the repository root with: ```bash npm run build npm run test --workspace tests/test_twilio ``` Or run the worker-thread variant with: ```bash npm run build npm run test --workspace tests/test_twilio_threading ``` ## Alternatives There are a lot of great VoIP-Agent orchestration implementations out there. This is a selection of implementations that I have experience with. - [LiveKit](https://livekit.io/) - [Vapi](https://vapi.ai/) - [Retell](https://www.retellai.com/) - [Pipecat](https://www.pipecat.ai/) ## Support For feature requests or issues, please open an [issue](https://github.com/adamjpatterson/dialog/issues) or contact the authors. - [Adam Patterson](https://github.com/adamjpatterson)