# @trelis/converse

The browser SDK for the [Converse realtime voice API](https://converse.trelis.com/docs/api/) —
the simplest way to build a realtime voice experience on the web. It owns microphone capture, echo
cancellation, streaming playback, interruptions and reconnects; Converse runs the conversation loop.

```sh
npm install @trelis/converse
```

Keep your persistent `ck_…` API key on your backend. Exchange it for a scoped browser credential
with `POST /api/v1/session-keys`, then connect from a user gesture:

```js
import { ConverseClient } from '@trelis/converse';

const credential = await fetch('/voice/session', {
  method: 'POST',
  credentials: 'same-origin',
}).then((response) => {
  if (!response.ok) throw new Error(`Voice credential failed: ${response.status}`);
  return response.json();
});

const client = new ConverseClient({
  url: 'wss://converse.trelis.com/ws',
  sessionId: credential.session_id,
  apiKey: credential.api_key,
  mode: { kind: 'converse' },
});

startButton.addEventListener('click', async () => {
  await client.unlockAudio();
  await client.connect();
  await client.startMic();
});
```

`startMic()` resolves only after the AudioWorklet delivers an actual audio frame. A frame containing
all zero samples is valid silence; only receiving no frames is a stalled capture. The SDK waits a
bounded period, fully releases the track, worklet, and `AudioContext`, then reacquires once. If the
replacement also stalls, the promise rejects with `error.code === 'capture_stalled'` and
`error.retryable === false`.

Render capture status from the lifecycle events; do not add another retry or a timer that treats an
opened device as ready:

```js
client.addEventListener('warming_up', () => showMicStatus('Preparing microphone…'));
client.addEventListener('recovering', ({ detail }) => {
  showMicStatus(detail.code === 'capture_stalled'
    ? 'Reconnecting microphone…'
    : 'Updating microphone…');
});
client.addEventListener('listening', () => showMicStatus('Listening'));
client.addEventListener('failed', ({ detail }) => {
  showMicError(detail.code, detail.error);
});
```

`warming_up`, `listening`, `recovering`, and `failed` are emitted as typed events and through the
catch-all `event` listener. `startMic()` accepts `deviceId`; the constructor accepts
`inputDeviceId` and `captureStartupTimeoutMs` (default 2000 ms). Device management is generic:

```js
const inputs = await client.getInputDevices();
await client.setInputDevice(inputs[0].deviceId); // restarts an active capture safely

client.addEventListener('devices_changed', ({ detail }) => {
  renderInputPicker(detail.devices, detail.device_id);
});
client.addEventListener('input_device_changed', ({ detail }) => {
  selectInput(detail.device_id); // null means follow the system default
});
```

The SDK listens for `navigator.mediaDevices.devicechange` while its microphone is active. It
restarts capture when the selected input disappears or the system-default input changes, and emits
the refreshed audio-input list. Call `setInputDevice(null)` to return to the system default.
`stopMic()` removes the listener and is a cancellation barrier: when it resolves, even an earlier
non-abortable `getUserMedia()` request has settled and any late track has been released.

A `silent_mic` event means the server has received sustained digital silence or an unusually
low startup signal. Treat it as a nonfatal warning: keep the session live while prompting the
user to check the selected input, hardware mute, and browser and operating-system permissions.
The event includes `reason`, `duration_ms`, and the observed PCM16 `peak`.

### Browser support

The same Browser SDK runs across platforms; there is no separate Chrome SDK. Its API compatibility
targets are Chromium (including Chrome, Brave and Edge), Firefox and WebKit (including Safari and
current iOS builds of Chrome and Brave). This is not a production-certification matrix: browser and
device combinations still require the physical validation described below. WebSocket is the
default transport everywhere. Experimental WebRTC automatically falls back to WebSocket on WebKit
so the SDK-owned echo-cancellation path remains in the audio loop.

Browser automation cannot validate acoustic echo cancellation, physical output level or Bluetooth
routing. Certify those on real devices when adding a new browser/engine version, especially an iOS
build using Apple's alternative-browser-engine entitlement.

Playback stays at unity gain. The SDK does not add a limiter, boost output, select a physical
speaker, or manipulate `navigator.audioSession`: browsers and operating systems own maximum device
volume and routing, and mobile browsers may attenuate playback while microphone capture is active.
Keeping capture active preserves barge-in; applications that require guaranteed speaker routing
need a native media integration.

Each inbound JSON frame is emitted once under its typed event name and once under the catch-all
`event` name. Choose one subscription style for a given handler; registering it on both will render
the same transcript twice.

Automatic WebSocket reconnect resumes conversation context and deferred jobs with the latest server
token. To preserve the same bounded resume opportunity across a full page reload, persist the SDK's
opaque state in tab-scoped storage and import it into the replacement client:

```js
const storageKey = 'converse-resume-state';
const clientOptions = {
  url: 'wss://converse.trelis.com/ws',
  sessionId: credential.session_id,
  apiKey: credential.api_key,
  mode: originalMode, // reconstruct the same options used before reload
};

let savedResumeState = null;
try {
  savedResumeState = JSON.parse(sessionStorage.getItem(storageKey) || 'null');
} catch {
  sessionStorage.removeItem(storageKey);
}

const client = new ConverseClient(clientOptions);
if (savedResumeState !== null) {
  try {
    client.importResumeState(savedResumeState);
  } catch {
    sessionStorage.removeItem(storageKey);
  }
}

client.addEventListener('resume_state', ({ detail }) => {
  if (detail.state) sessionStorage.setItem(storageKey, JSON.stringify(detail.state));
  else sessionStorage.removeItem(storageKey);
});
```

`exportResumeState()` returns the current versioned state (or `null`), and
`importResumeState(state)` may install it before `connect()`. Reconstruct the same client options on
the new page: explicit mode fields override stashed configuration, so relying on constructor defaults
can change the resumed session. Treat the value like a short-lived credential: prefer `sessionStorage` over cross-tab or long-lived storage, and never send it anywhere
except back to Converse through the SDK. The server accepts it only during its short resume window
and only for the identity that created it. The SDK rotates the saved state after every successful
connection and clears it after an intentional end or `resume_failed`. If a persisted token has
expired, `connect()` rejects and emits `resume_failed`; offer a deliberate fresh start.

A terminal `resume_failed` event stops automatic retries; end local capture and let the caller
deliberately start a fresh session.

Text sessions use the ordinary Converse model, instructions, tools and events without opening a
microphone or audio pipeline:

```js
const textClient = new ConverseClient({
  url: 'wss://converse.trelis.com/ws',
  sessionId: credential.session_id,
  apiKey: credential.api_key,
  mode: { kind: 'converse', modality: 'text' },
});
await textClient.connect();
textClient.sendText('What is the weather like?');
```

`sendText(text)` commits one user turn and returns whether it was written to the live connection. (In a voice session `sendText` remains the user-role `injectContext` shorthand and returns its acknowledgement promise.)
The usual `asr`, `turn`, `text_delta`, `utterance`, and `done` events follow without audio. Text
mode is WebSocket-only; microphone and caller-owned audio methods reject.

Hosts can also add silent application context to either modality. Set `reply: true` when the model
should proactively announce the update:

```js
client.injectContext('Claude Code finished. Tell the user briefly.', {
  role: 'context',
  reply: true,
  messageId: 'job-42-complete',
});
```

`injectContext()` returns a promise resolving to the broker's correlated
`{type: 'inject_context_ack', message_id, accepted, ...}`. Its role defaults to `'context'`; use
`'user'` only when the text should be represented as a real user turn. `reply` defaults to false.

Browser setup and events are in the [browser guide](https://converse.trelis.com/docs/api/browser/).
Wire-level playback and tools are in the [WebSocket guide](https://converse.trelis.com/docs/api/websocket/).

Tool declarations pass through in `mode.tools`. `expected_duration` says what the caller should
hear: `"instant"` for fast lookups (the caller hears the answer directly), `"seconds"` for anything
that takes more than about a second (the assistant acknowledges first, then answers when the result
arrives), `"long"` for agent or batch jobs; leave it out and Converse learns from observed results.
`status_label` supplies a short user-safe name for pending work:

```js
mode: {
  kind: "converse",
  tools: [{
    name: "lookup_order",
    description: "Look up an order by ID.",
    parameters: { type: "object", properties: { order_id: { type: "string" } } },
    read_only: true,
    expected_duration: "instant",
    status_label: "order lookup",
  }],
}
```

For host jobs that should outlive the current voice turn — a coding-agent task, report generation,
anything the caller should not wait on — declare the tool as a background job:

```js
mode: {
  kind: "converse",
  tools: [{
    name: "run_task",
    description: "Run a long coding task and report when done.",
    parameters: { type: "object", properties: { instruction: { type: "string" } } },
    deferred: true,          // job may outlive the voice turn
    deferred_timeout: 7200,  // seconds the detached job may run
    notify_on_complete: true, // speak up when the result lands, even mid-topic
    status_label: "coding task",
  }],
}
```

Acknowledge the individual call to release the voice turn, then continue the work in the background:

```js
client.sendToolDeferred(event.detail.id, {
  handle: `cc-${event.detail.id}`, statusLabel: "Claude Code task",
});
```

The handle names *that call*, not your worker — mint a fresh one each time (deriving it from the
call id, as above, is the simplest way) and check the acknowledgement. Re-using a live handle is
rejected with `{ accepted: false, reason: "handle_in_use" }`, and a rejected defer is not a
deferral: the call stays on the ordinary tool timeout and will expire while you believe it is
running in the background. Route several calls to one long-lived worker with your own
handle-to-worker map instead.

The conversation carries on while the job runs: the agent narrates the hand-off, the caller can talk
about something else, and when the host reports the result the broker voices it as a completion
turn (`notify_on_complete`). The user can interrupt that narration or cancel the pending job at any
point, and reconnects resume pending jobs via the latest server token. See the
[background tools guide](https://converse.trelis.com/docs/api/background-tools/) for the full
lifecycle, including progress updates and the delivery-failure contract.

A running job that discovers it needs a mid-call decision raises it with
`sendToolPartialResult(id, content, { interaction: { id, prompt, options } })` — the broker asks
the user by voice at the next opportunity, with a wire-visible lifecycle (`tool_job_narration`,
tracked via `narrationState`/`interactionState`). If the decision gets made elsewhere first (e.g.
clicked in your own UI) or newer intent makes it moot, close it without completing the call:

```js
const ack = await client.sendToolInteractionUpdate(
  event.detail.id, "overwrite-1", "resolved", { note: "approved in the IDE" });
// ack.applied === false carries a stable reason (e.g. "already_closed") for late duplicates.
```

Built-in web search remains bridge-first. A server `tool_cancel` can represent timeout/discard or
an explicit user request to cancel pending work; hosts should stop that call promptly.

### WebRTC transport (experimental)

Experimental: the API is stable, but this transport is newly shipped and still being hardened on
real networks; `ws` remains the default and recommended fallback.

Pass `transport: 'webrtc'` to `ConverseClient` to carry the call over WebRTC (UDP) instead of the
default WebSocket — more resilient to jitter and packet loss on weak networks. `ws` remains the
default. Safari/WebKit falls back to `ws` automatically. See the
[browser guide's WebRTC section](https://converse.trelis.com/docs/api/browser/#webrtc).

Trelis-authored SDK code is licensed under the [Apache License 2.0](LICENSE). The bundled AEC
module contains components under their own terms; see [NOTICE](NOTICE) and
[THIRD_PARTY_LICENSES](THIRD_PARTY_LICENSES/README.md). These licenses do not apply to the hosted
Converse service, its models, or its server-side implementation.

### Ambience: background music and the thinking sound

```js
const client = new ConverseClient({ url, sessionId, apiKey, mode, ambience: 'thinking' });
client.setAmbience('continuous');   // switch live; 'off' | 'thinking' | 'continuous'
```

`ambience` (default `'thinking'`; pass `'off'` to opt out) plays a soft generative bed through the
SDK's own player:

- `'continuous'`: under the whole call from the first reply onward (it never leads), so the silence
  between turns feels connected.
- `'thinking'`: silent, except while Converse is blocking on a tool result with nothing to say (the
  server's `working` event - client tools, `web_search`, `think_deeply`). After ~1.5 s it fades in;
  it fades out again under the first syllables of the reply. A caller waiting on a slow backend
  hears "still working" instead of dead air, and nothing else.

Pass an object to tune the envelope: `{ mode: 'thinking', afterS: 1.5, fadeInS: 1.5, fadeOutS: 0.3,
level: 1 }` (`level` is a linear multiplier on the bed's built-in peak of about -21 dBFS). The bed is
mixed into the same scheduled chunks as reply audio, so it is inside the echo canceller's far-end
reference, including the SDK's WASM canceller on WebKit/iOS; it is never queued ahead of a reply and
never counts toward barge `discarded_ms`. WebSocket transport only: over webrtc the SDK player is not
in the audio path, so the local ambience stays silent and the server-mixed `mode.background_audio`
bed is the option there.

## Repository development

`package.json` is the version source and `CHANGELOG.md` records compatibility changes. Production
uses the checked-in copy under `web/vendor/converse/`.

Before release, run:

```sh
npm run check
npm run pack:check
```