# Gateway Architecture This document owns public-ingress and channel webhook architecture. The repo-level architecture index lives in [`/ARCHITECTURE.md`](../ARCHITECTURE.md). ## Ingress Boundary Architecture — Gateway-Only Public Ingress All external webhook endpoints (Telegram, Twilio, WhatsApp, OAuth) and persistent channel connections (Slack Socket Mode) are handled exclusively by the gateway. The runtime never exposes webhook ingress. The runtime-proxy explicitly blocks forwarding of `/webhooks/*` paths. This boundary is enforced at three layers: 1. **Gateway routing:** The gateway's `fetch()` handler matches `/webhooks/*` paths to dedicated handlers before the runtime-proxy fallthrough. Webhook traffic is validated (signature checks, payload limits) and forwarded to internal runtime endpoints. 2. **Runtime-proxy guard:** The runtime-proxy handler rejects any `/webhooks/*` path with a 404, preventing accidental forwarding of webhook traffic to the runtime even if a new webhook path is added to the gateway without a dedicated handler. 3. **Runtime server:** The runtime HTTP server does not register any `/webhooks/telegram` routes. Direct Twilio webhook routes (`/webhooks/twilio/*`) return 410 with a `GATEWAY_ONLY` error code, and call WebSocket upgrades are restricted to private network peers. ``` Internet | +-- Twilio ----------> Gateway POST /webhooks/twilio/* --> Runtime /v1/internal/twilio/* +-- OAuth Provider ---> Gateway GET /webhooks/oauth/callback --> Runtime /v1/internal/oauth/callback +-- MCP Auth Server --> Gateway GET /oauth/client-metadata.json +-- Telegram --------> Gateway POST /webhooks/telegram --> Runtime /v1/channels/inbound +-- Slack ------------> Gateway (Socket Mode WebSocket) --> Runtime /v1/channels/inbound | +-- Tunnel (ngrok, Cloudflare, etc.) | v Gateway (http://127.0.0.1:7830) | +-- Dedicated /v1/health --> Runtime /v1/health +-- /v1/integrations/twilio/* --> Runtime (Twilio control-plane proxy) +-- /v1/channels/readiness* --> Runtime (channel readiness proxy) +-- Runtime proxy /v1/* --> Runtime (http://127.0.0.1:7821) +-- /webhooks/* --> BLOCKED (404, never forwarded to runtime) ``` ### STT Route Proxying (Assistant-Scoped Rewrite) Clients send speech-to-text transcription requests through the gateway to the daemon's STT service. Clients POST to the assistant-scoped path `/v1/assistants/:assistantId/stt/transcribe`, which the gateway's runtime proxy rewrites to the flat daemon path `/v1/stt/transcribe`. This follows the same assistant-scoped rewrite pattern used by other client-facing endpoints (feature flags, privacy config, etc.). The request carries base64-encoded WAV audio and a MIME type. The daemon resolves the configured STT provider via `resolveBatchTranscriber()` and returns the transcribed text. Clients use the response to implement a service-first strategy: the service transcription takes precedence when available, with a client-local fallback when the service returns 503 (not configured) or fails. | Client path (gateway) | Daemon path (after rewrite) | Method | | ----------------------------------- | --------------------------- | ------ | | `/v1/assistants/:id/stt/transcribe` | `/v1/stt/transcribe` | POST | **Key source files:** | File | Purpose | | ----------------------------------------------- | ------------------------------------------------------------------------- | | `gateway/src/http/routes/runtime-proxy.ts` | Assistant-scoped path rewriting (`/v1/assistants/:id/...` → `/v1/...`) | | `assistant/src/runtime/routes/stt-routes.ts` | Daemon HTTP endpoint: validates audio, resolves transcriber, returns text | | `clients/web/src/domains/chat/voice/stt-api.ts` | Web client: POSTs audio to the gateway, returns a typed result | ### STT Streaming WebSocket Proxy Clients open WebSocket connections through the gateway to the daemon's real-time STT streaming endpoint for conversation chat message capture. The gateway authenticates the downstream client using an edge JWT (actor principal required), or a velay-attested caller when the client arrived through the gateway's velay tunnel, then opens an upstream WebSocket connection to the daemon's `/v1/stt/stream` endpoint with a short-lived gateway service token. This keeps the daemon's WebSocket endpoint unreachable from the public internet while allowing authenticated clients to stream audio for real-time transcription. **Config-authoritative model:** The runtime always resolves the streaming transcriber from the assistant config, regardless of any `provider` query parameter. Dictation reads its own role (`services.stt.roles.dictation`, falling back to `services.stt.provider`). The `provider` parameter is optional compatibility metadata: when supplied and it disagrees with the provider that role resolves to, the runtime logs a mismatch warning for operator visibility. **Client path:** `wss:///v1/stt/stream?mimeType=[&provider=][&sampleRate=]` **Query parameters:** | Parameter | Required | Description | | ------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mimeType` | Yes | MIME type of the audio being streamed (e.g. `audio/webm;codecs=opus`) | | `provider` | No | Optional STT provider identifier (any ID in the daemon's STT provider catalog, e.g. `deepgram`, `deepgram-flux`, `google-gemini`). Forwarded as compatibility metadata, and the gateway does not validate it: the runtime resolves the transcriber from config, not from this parameter. | | `sampleRate` | No | Sample rate in Hz (e.g. `16000`). Passed through to the daemon. | | `token` | No | Edge JWT (alternative to `Authorization: Bearer` header for WS upgrades) | **Auth model:** STT streaming is an authenticated, assistant-scoped path. The client presents a valid edge JWT with an actor principal; service tokens are rejected. A client that reached the gateway through its velay tunnel (a managed pod, or a locally hosted assistant dialled from the mobile app) carries no edge JWT: velay validated the browser's token and injected `X-Velay-*` headers, and the gateway admits that attestation when it has a velay tunnel configured (`acceptsVelayAttestation` in `guardian-pin.ts`) and the request carries the process-local bridge proof. Live voice and the watch stream take the same velay path and additionally pin the caller to the bound guardian; dictation accepts any valid actor. When `runtimeProxyRequireAuth` is globally disabled (dev bypass), the upgrade proceeds without token validation. **Proxy behavior:** The gateway buffers up to 100 downstream messages while the upstream connection to the daemon is being established. If the buffer overflows, the downstream connection is closed with code 1008 (policy violation). Once the upstream connection opens, buffered messages are flushed in order. All subsequent messages are forwarded bidirectionally: client audio frames flow upstream, daemon session events (JSON text frames: `ready`, the transcript and turn-boundary events of the daemon's `SttStreamServerEvent` union, and `error` / `closed`) flow downstream. The gateway forwards these opaquely and needs no change when the daemon adds an event type. When either side closes, the other side is closed with the same code/reason. **Key source files:** | File | Purpose | | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `gateway/src/http/routes/stt-stream-websocket.ts` | WebSocket upgrade handler (`createSttStreamWebsocketHandler`) and proxy handlers (`getSttStreamWebsocketHandlers`) | | `gateway/src/index.ts` | Route registration: wires upgrade handler to the gateway's Bun HTTP server | | `assistant/src/runtime/http-server.ts` | Daemon-side WebSocket upgrade at `/v1/stt/stream`, session creation and registry | | `assistant/src/stt/stt-stream-session.ts` | Runtime session orchestrator: drives the `StreamingTranscriber` from the WebSocket | | `clients/web/src/domains/chat/voice/dictation-stream.ts` | Web client: opens the gateway WebSocket, parses transcript events, reports failures | ### Assistant Feature Flags API The gateway exposes a REST API for reading and mutating assistant feature flags. Assistant feature flags are assistant-scoped, declaration-driven booleans that can gate any assistant behavior. Skill availability is one consumer, but not a required coupling (see [`assistant/ARCHITECTURE.md`](../assistant/ARCHITECTURE.md) for resolver and skill enforcement details). **Unified registry loader:** The gateway loads the unified feature flag registry from `meta/feature-flags/feature-flag-registry.json` (bundled copy at `gateway/src/feature-flag-registry.json`) via `loadFeatureFlagDefaults()` in `gateway/src/feature-flag-defaults.ts`. Only flags with `scope: "assistant"` are used for the API. The registry is loaded once and cached for the lifetime of the process. Invalid entries are skipped with a warning. The `isFlagDeclared()` helper validates that a flag key exists in the registry before allowing writes. **Endpoints (GET/PATCH contract):** | Method | Path | Description | | ------ | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GET | `/v1/feature-flags` | List all declared assistant feature flags from the defaults registry, merged with persisted values from the feature flag store. Returns `{ flags: FeatureFlagEntry[] }` where each entry has `key`, `enabled`, `defaultEnabled`, and `description`. | | PATCH | `/v1/feature-flags/:key` | Set a single assistant feature flag. Body: `{ "enabled": true\|false }`. Key must be a simple kebab-case flag key declared in the defaults registry. Writes to `~/.vellum/protected/feature-flags.json`. | **Unified registry:** All declared feature flags and their default values are defined in the unified registry at `meta/feature-flags/feature-flag-registry.json` (bundled copy at `gateway/src/feature-flag-registry.json`). The gateway loads this registry on startup via `gateway/src/feature-flag-defaults.ts`, filtering to `scope: "assistant"` flags. Labels come from the registry. The GET endpoint merges persisted overrides with explicit remote values and registry defaults to produce the full flag list. Declared flags missing from a remote snapshot fall back to their registry `defaultEnabled`; undeclared keys are rejected by writes and fail closed in direct resolver calls. The PATCH endpoint validates that the target flag key exists in the registry before accepting a write. Only declared keys are exposed by this API. **Flag key format:** The canonical key format is simple kebab-case (e.g., `browser`, `ces-tools`). Only keys matching this pattern and declared in the registry are accepted by the PATCH endpoint; other patterns are rejected with 400. All writes use the canonical format and are stored in the protected feature flag store (`~/.vellum/protected/feature-flags.json`). **Storage:** Flag overrides are persisted in `~/.vellum/protected/feature-flags.json` (local) or `GATEWAY_SECURITY_DIR/feature-flags.json` (Docker). The store uses a versioned JSON format (`{ version: 1, values: Record }`). Remote platform snapshots are persisted separately in `feature-flags-remote.json`; local overrides win over remote values, explicit remote values win over registry defaults, and missing remote values fall back to registry defaults. The gateway writes atomically (temp file + rename, 0o600 permissions). The daemon's config watcher monitors the protected directory and hot-reloads flag changes, so flag mutations take effect on the next session or tool resolution without a restart. **Token separation (authentication boundary):** The assistant feature flags API uses scope-based JWT auth. The gateway issues JWTs with `feature_flags.read` and `feature_flags.write` scopes to control access. | Operation | Required scope | | ------------------------------ | --------------------- | | `GET /v1/feature-flags` | `feature_flags.read` | | `PATCH /v1/feature-flags/:key` | `feature_flags.write` | The assistant daemon does not read or distribute a feature-flag token. All feature-flag auth flows go through the gateway's scoped JWT mechanism. **Protected feature flag store:** The canonical storage for assistant feature flag overrides is `~/.vellum/protected/feature-flags.json` (local) or `GATEWAY_SECURITY_DIR/feature-flags.json` (Docker). The store is managed by `gateway/src/feature-flag-store.ts` and uses a versioned JSON format with `Record` values keyed by canonical flag keys (simple kebab-case, e.g., `browser`). The gateway's PATCH handler writes exclusively to this store. The daemon's resolver reads it with highest priority, falling back to the defaults registry. Undeclared keys are ignored by the resolver. **Key source files:** | File | Purpose | | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `gateway/src/http/routes/feature-flags.ts` | GET and PATCH handlers; key format validation; delegates to feature-flag-store for persistence | | `gateway/src/feature-flag-store.ts` | File-backed persistence: `readPersistedFeatureFlags()`, `writeFeatureFlag()`, atomic writes to protected directory | | `gateway/src/feature-flag-defaults.ts` | `loadFeatureFlagDefaults()` — loads the shared defaults registry; `isFlagDeclared()` — validates flag keys | | `gateway/src/config.ts` | `readOrGenerateFeatureFlagToken()` — token provisioning; `featureFlagToken` config field | | `gateway/src/index.ts` | Route registration, auth enforcement (dual-token for GET, flag-token-only for PATCH), token file watcher | | `meta/feature-flags/feature-flag-registry.json` | Unified feature flag registry (repo root) — all declared flags with scope, label, default values, and descriptions | | `gateway/src/feature-flag-registry.json` | Bundled copy of the unified registry for compiled binary resolution | ### Channel Verification Session Control-Plane Proxy Channel verification session endpoints are exposed directly by the gateway and forwarded to runtime integration handlers for dedicated auth handling. This keeps assistant skills and user-facing tooling on gateway URLs only. **Forwarded endpoints:** | Method | Path | | ------ | ------------------------------------------ | | POST | `/v1/channel-verification-sessions` | | DELETE | `/v1/channel-verification-sessions` | | POST | `/v1/channel-verification-sessions/resend` | | GET | `/v1/channel-verification-sessions/status` | | POST | `/v1/channel-verification-sessions/revoke` | | POST | `/v1/guardian/refresh` | The `/v1/guardian/refresh` endpoint is the only public ingress for rotating JWT access + refresh token credentials. Clients must call this through the gateway; the runtime endpoint is not directly exposed. The gateway validates the caller's JWT and forwards to the runtime, which handles refresh token validation, rotation, and replay detection (see [`assistant/ARCHITECTURE.md`](../assistant/ARCHITECTURE.md) for the JWT auth lifecycle). **Authentication boundary:** - Gateway validates the caller's JWT bearer token. - Gateway forwards requests to runtime with a minted JWT (`gateway_service_v1` scope profile). - Upstream 4xx/5xx responses are passed through, while connection errors return `502` and timeouts return `504`. **Key source files:** | File | Purpose | | --------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | `gateway/src/http/routes/channel-verification-session-proxy.ts` | Channel verification session proxy handlers and upstream forwarding | | `gateway/src/index.ts` | Route registration and JWT auth enforcement for `/v1/channel-verification-sessions/*` | ### Runtime Health Proxy Runtime health is exposed directly by the gateway at `GET /v1/health` and forwarded to the runtime's `GET /v1/health` endpoint for dedicated auth handling. **Authentication boundary:** - Gateway validates the caller's JWT bearer token. - Gateway forwards the request to runtime with a minted JWT (`gateway_service_v1` scope profile). - Upstream 4xx/5xx responses are passed through, while connection errors return `502` and timeouts return `504`. **Key source files:** | File | Purpose | | ------------------------------------------------- | --------------------------------------------------------------- | | `gateway/src/http/routes/runtime-health-proxy.ts` | Runtime health proxy handler and upstream forwarding | | `gateway/src/index.ts` | Route registration and bearer-auth enforcement for `/v1/health` | ### Telegram + Contacts Control-Plane Proxies Telegram integration setup/config endpoints and contacts/invites endpoints are also exposed directly by the gateway for dedicated auth handling. Contact endpoints are gateway-native — the gateway DB is the ACL source of truth, with a best-effort identity/info mirror to the assistant DB; search-filtered list reads relay through the runtime (which owns filter/search + info) and get an authoritative gateway ACL overlay. Invite endpoints are gateway-native — they run against the gateway DB's `ingress_invites` table with no runtime round-trip (the outbound-call relay is the one exception: the gateway validates its invite row, then delegates the provider call to the assistant). All contact paths are registered flat only (`/v1/contacts...`, `/v1/contact-channels...`); the gateway registers no assistant-scoped contact variants. Clients emit assistant-scoped URLs (`/v1/assistants/{id}/...`), and the deployment boundary strips the prefix before the gateway routes the request: cloud's Django `RuntimeProxyView` strips it for every route, and the self-hosted web client's `rewriteForSelfHostedIngress` (`clients/web/src/lib/api-interceptors.ts`) flattens the contact family the same way. **Forwarded Telegram endpoints:** | Method | Path | | --------------- | ------------------------------------ | | GET/POST/DELETE | `/v1/integrations/telegram/config` | | POST | `/v1/integrations/telegram/commands` | | POST | `/v1/integrations/telegram/setup` | **Contact endpoints (gateway-native):** | Method | Path | | ---------- | ----------------------------------------------- | | GET/POST | `/v1/contacts` | | GET/DELETE | `/v1/contacts/:contactId` | | POST | `/v1/contacts/merge` | | POST | `/v1/contacts/prompt/submit` | | POST | `/v1/contacts/guardian/channel` | | PATCH | `/v1/contact-channels/:contactChannelId` | | POST | `/v1/contact-channels/:contactChannelId/verify` | **Gateway-native invite endpoints:** | Method | Path | | -------- | ------------------------------------- | | GET/POST | `/v1/contacts/invites` | | DELETE | `/v1/contacts/invites/:inviteId` | | POST | `/v1/contacts/invites/:inviteId/call` | | POST | `/v1/contacts/invites/redeem` | **Authentication boundary:** - Gateway validates the caller's JWT bearer token. - Native contact/invite endpoints are served from the gateway DB after that bearer-auth check; runtime-relayed requests (Telegram control plane, contact search reads) reach the runtime with a minted JWT (`gateway_ingress_v1` or `gateway_service_v1` scope profile). - Upstream 4xx/5xx responses are passed through, while connection errors return `502` and timeouts return `504`. **Key source files:** | File | Purpose | | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `gateway/src/http/routes/telegram-control-plane-proxy.ts` | Telegram control-plane proxy handlers and upstream forwarding | | `gateway/src/http/routes/contacts-control-plane-proxy.ts` | Gateway-native contact handlers (with runtime search relay) plus gateway-native invite handlers (mint, list, revoke, redeem, call relay) | | `gateway/src/index.ts` | Route registration and bearer-auth enforcement for `/v1/integrations/telegram/*` and `/v1/contacts*` | ### Twilio Control-Plane Proxy Twilio integration setup/config endpoints are exposed directly by the gateway and forwarded to runtime handlers for dedicated auth handling. This keeps skills and clients on gateway URLs exclusively. **Forwarded endpoints:** | Method | Path | | ------ | ------------------------------------------- | | GET | `/v1/integrations/twilio/config` | | POST | `/v1/integrations/twilio/credentials` | | DELETE | `/v1/integrations/twilio/credentials` | | GET | `/v1/integrations/twilio/numbers` | | POST | `/v1/integrations/twilio/numbers/provision` | | POST | `/v1/integrations/twilio/numbers/assign` | | POST | `/v1/integrations/twilio/numbers/release` | **Authentication boundary:** - Gateway validates the caller's JWT bearer token. - Gateway forwards requests to runtime with a minted JWT (`gateway_ingress_v1` or `gateway_service_v1` scope profile). - Upstream 4xx/5xx responses are passed through, while connection errors return `502` and timeouts return `504`. **Key source files:** | File | Purpose | | ------------------------------------------------------- | ------------------------------------------------------------------------------ | | `gateway/src/http/routes/twilio-control-plane-proxy.ts` | Twilio control-plane proxy handlers and upstream forwarding | | `gateway/src/index.ts` | Route registration and bearer-auth enforcement for `/v1/integrations/twilio/*` | ### Channel Readiness Proxy Channel readiness endpoints are exposed directly by the gateway and forwarded to runtime handlers for dedicated auth handling. **Forwarded endpoints:** | Method | Path | | ------ | -------------------------------- | | GET | `/v1/channels/readiness` | | POST | `/v1/channels/readiness/refresh` | **Authentication boundary:** - Gateway validates the caller's JWT bearer token. - Gateway forwards requests to runtime with a minted JWT (`gateway_ingress_v1` or `gateway_service_v1` scope profile). - Upstream 4xx/5xx responses are passed through, while connection errors return `502` and timeouts return `504`. **Key source files:** | File | Purpose | | ---------------------------------------------------- | ---------------------------------------------------------------------------- | | `gateway/src/http/routes/channel-readiness-proxy.ts` | Channel readiness proxy handlers and upstream forwarding | | `gateway/src/index.ts` | Route registration and bearer-auth enforcement for `/v1/channels/readiness*` | ### Channel Binding Lifecycle (Lane Separation) Each channel (desktop, Telegram, etc.) operates in its own **lane**: conversations created by an external channel are never displayed in the desktop conversation list, and desktop conversations are never exposed to external channels. The `channelBinding` metadata on a conversation is used solely for routing inbound/outbound messages within that lane and for filtering conversations during desktop conversation restoration. Channel bindings follow a three-phase lifecycle: 1. **Bind** — An inbound message from an external channel (e.g., Telegram chat) arrives at the gateway, which normalizes it and forwards it to the runtime's `/v1/channels/inbound` endpoint. The runtime creates or reuses a conversation, establishing the channel binding (`sourceChannel` metadata on the conversation). 2. **Route**: Subsequent messages on the same external chat are routed to the same conversation via the channel binding. Slack and Telegram are thread-scoped: a message that arrives in a Slack thread (including every message in a Slack agent DM, which Slack always delivers in a thread) or a Telegram topic resolves to that thread's own conversation, keyed on the chat plus the thread id (`assistant/src/persistence/delivery-crud.ts`, `buildScopedConversationKey`); a thread-less message resolves to the chat's base conversation. Replies from the assistant go out through the daemon's channel transport for that channel (`assistant/src/messaging/providers`), which calls the provider's API directly; they never pass back through the gateway. The desktop client filters out channel-bound conversations during conversation restoration (`ConversationRestorer`) so they never appear in the desktop conversation list. 3. **Rebind** — If a message arrives on an external chat whose conversation was previously deleted, the channel inbound handler treats it as a new conversation and establishes a fresh binding. The external chat ID is reused, but the conversation is new. ### Configuration The public URL where the gateway is reachable is configured via: | Source | Description | | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ingress.publicBaseUrl` (workspace config) | Canonical public ingress URL for Telegram webhooks, OAuth callbacks and client metadata, email callbacks, generic JSON webhooks, Twilio webhooks, and Twilio WebSocket URLs | | `ingress.publicBaseUrlManagedBy` (workspace config) | Ownership marker used when Velay published `ingress.publicBaseUrl`; lets the gateway clear stale Velay-managed URLs without disturbing manual URLs | ### Tunnel-Agnostic Setup To expose the gateway for external callbacks during local development: 1. **Start your tunnel** service (ngrok, Cloudflare Tunnel, or any similar tool), pointing it at the local gateway: `http://127.0.0.1:7830` 2. **Set the public URL** provided by the tunnel as `ingress.publicBaseUrl` in the Settings UI (Public Ingress section) The assistant runtime reads this URL via the centralized `public-ingress-urls.ts` module and uses it to construct webhook and callback URLs automatically. Ngrok and custom tunnels remain supported for every ingress surface, including Twilio fallback. ### Velay Twilio Ingress Velay is a platform-managed tunnel for assistant-hosted HTTP and WebSocket traffic. When it is active, Velay publishes the registered public assistant URL to `ingress.publicBaseUrl` and marks it with `ingress.publicBaseUrlManagedBy: "velay"`. When `VELAY_BASE_URL` is present in the gateway environment, the gateway creates `VelayTunnelClient` but starts it only after Twilio setup has been started in the workspace. The Twilio setup skill writes `twilio.setupStarted: true` at the beginning of setup so the tunnel can open while credentials and phone-number selection are still in progress. On boot, existing Twilio credentials or existing `twilio.accountSid` / `twilio.phoneNumber` config also count as prior setup. Before credential-backed startup side effects run, the gateway clears any stale Velay-managed `ingress.publicBaseUrl`; if setup has not started, it does this without opening a tunnel. The client registers with Velay over `GET /v1/register` using the assistant API key, then receives a `registered` frame containing a public assistant URL such as `https://velay.vellum.ai/`. If Vellum platform credentials change while the tunnel is already running or waiting on backoff, the gateway asks the client to reconnect with fresh credentials instead of waiting for process restart. The gateway writes the registered URL to `ingress.publicBaseUrl`. When the tunnel disconnects, it clears that value only if the Velay ownership marker is still present and the URL still matches what the tunnel published, leaving manual URLs intact. Velay forwards both HTTP request frames and WebSocket frames into the local gateway loopback listener: ```text Public Velay HTTPS/WSS URL → Velay tunnel session → Gateway Velay bridge → Gateway loopback listener (http://127.0.0.1:) → Existing gateway route handlers ``` The HTTP bridge can carry normal JSON requests and health checks, so it is useful for local bridge smoke tests. Velay-managed `ingress.publicBaseUrl` changes are tagged with `publicBaseUrlManagedBy` so gateway side effects can skip unrelated webhook reconciliation while still refreshing Twilio phone-number webhooks. While `velay-webhooks` is on, the Telegram reconcile runs on that change too, because the published URL is then the address Telegram is meant to point at; email callback re-registration stays suppressed either way. Local platform smoke-test flow: 1. In `vellum-assistant-platform`, run `vel up velay`. 2. Ensure vembda passes the environment-appropriate `VELAY_BASE_URL` into assistant gateway containers. 3. Start or complete Twilio setup in the workspace so the gateway is allowed to connect the tunnel. 4. Re-hatch or restart the assistant so the gateway receives the new environment. 5. Confirm gateway logs show `Velay tunnel connected` and `Velay tunnel registered`. 6. Verify HTTP forwarding by requesting `${VELAY_PUBLIC_BASE_URL}//healthz` and `${VELAY_PUBLIC_BASE_URL}//schema`. For MCP OAuth testing, request `${VELAY_PUBLIC_BASE_URL}//oauth/client-metadata.json` and confirm its `redirect_uris` entry points at the same public assistant URL. When validating a JSON webhook route under active development, POST a small JSON body through the same Velay public URL and confirm it reaches the loopback gateway. 7. Verify Twilio WebSocket forwarding with a synthetic local WebSocket client against `${VELAY_PUBLIC_BASE_URL}//webhooks/twilio/media-stream//`, then with a real Twilio call after the gateway has registered with Velay. ### Webhook Ingress Route Registry The whole `/webhooks/` namespace used to be reachable through the tunnel as one wildcard. The registry replaces that with a per-assistant allowlist: an assistant answers exactly the subpaths it has claimed, and nothing else. It is gated on the `velay-webhooks` feature flag; with the flag off every rule below is the pre-registry one. **Three admission layers**, outermost first. Each is narrower than the one in front of it, and none replaces the others: 1. **Velay edge rules.** Every WebSocket upgrade to Velay carries the `X-Vellum-Velay-Allowed-Paths` header, a JSON array of Go RE2 patterns built in `gateway/src/velay/allowed-paths.ts`. Velay compiles them and drops any request matching none of them before it enters the tunnel. With the flag on, the array carries one exact-match rule per registered row in place of the `^/webhooks/` wildcard. 2. **Bridge re-validation.** `isAllowedVelayHttpPath` (`gateway/src/velay/bridge-utils.ts`) consults the registry again as the frame arrives, so an edge rule that has gone stale still admits nothing the gateway has stopped claiming. 3. **Route table and per-route authentication.** An admitted path must still match a registered gateway route, and each provider route runs its own check: Telegram's `secret_token`, Twilio's HMAC-SHA1 signature, a plugin webhook's token. **Where the registry lives.** One `webhook_ingress_routes` row per path in `gateway.sqlite`, which sits in `GATEWAY_SECURITY_DIR` (the PVC in a managed deployment), so claims survive a pod restart. A row holds the origin-relative path, the owning `type` (`telegram`, `twilio_voice`, `plugin`, and so on), an optional `source` naming the instance within that type, and `last_registered_at`, refreshed on every re-registration so a path nothing claims any more is visible as a stale row. Rows never expire on their own; `unregister_webhook_route` removes one. **Who claims what.** The daemon claims over IPC (`register_webhook_route`, `gateway/src/ipc/webhook-route-handlers.ts`) for plugin webhooks and its own callback registration. The gateway claims in-process for the channels it owns end to end: Telegram in `telegram/webhook-manager.ts`, Twilio in `twilio/webhook-sync.ts`. Registration is gated on the flag at every call site; revocation and listing are not, so an operator can always see and remove what was claimed while the flag was on. Where no claim is available, because the flag is off, no tunnel URL is published yet, or the write fails, the caller registers a Django-hosted callback route instead (`POST /v1/internal/gateway/callback-routes/register/`), which is the pre-registry behavior and stays the fallback. **Propagation and its window.** Registering a row that changes the advertised set fires `onWebhookIngressRoutesChanged`, which asks the tunnel client to reconnect so the new rules ride the next upgrade header. That reconnect is debounced 5s and deferred while the tunnel is carrying traffic, so for a short window Velay is still enforcing the previous rules and drops a just-claimed path at the edge. Nothing retries a request the edge dropped, so a path is claimed before its URL is handed to a provider, never after. **Twilio's static entries.** `^/webhooks/twilio/` stays a prefix rule at the edge and in the bridge, because the media-stream path carries call state in its segments (`/webhooks/twilio/media-stream//`) that an exact-match row cannot express. The voice and status paths are exact, and the Twilio webhook sync claims them so the registry describes them. Those claims are bookkeeping toward narrowing the static prefix down to the media-stream subtree. ### URL Builders All public-facing URLs are constructed by `assistant/src/inbound/public-ingress-urls.ts`: | Function | URL Pattern | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `getPublicBaseUrl()` | Resolves the canonical base URL from `ingress.publicBaseUrl` in workspace config or module-level state (assistant-side; the gateway reads via `ConfigFileCache`) | | `getTwilioVoiceWebhookUrl()` | `${base}/webhooks/twilio/voice?callSessionId=...`, using `ingress.publicBaseUrl` | | `getTwilioStatusCallbackUrl()` | `${base}/webhooks/twilio/status`, using `ingress.publicBaseUrl` | | `getTwilioMediaStreamUrl()` | `ws(s)://.../webhooks/twilio/media-stream`, using `ingress.publicBaseUrl` (per-call path segments appended at TwiML build time) | | `getOAuthCallbackUrl()` | `${base}/webhooks/oauth/callback` | | `getMcpOAuthClientMetadataUrl()` | `${base}/oauth/client-metadata.json` | | `getTelegramWebhookUrl()` | `${base}/webhooks/telegram` | ### Telegram Messaging Flow Telegram messages follow three paths through the system: ``` Inbound (user → assistant): Telegram → Gateway POST /webhooks/telegram → verify secret → normalize → route → Runtime POST /v1/assistants/:id/channels/inbound (replyCallbackUrl = ${gatewayInternalBaseUrl}/deliver/telegram[?threadId=]) Outbound reply (assistant → user, triggered by inbound): Daemon Telegram transport → Telegram Bot API sendMessage/sendPhoto/sendDocument/sendChatAction Outbound proactive (assistant → user, messaging tool or POST /v1/channels/send): Daemon Telegram transport → Telegram Bot API sendMessage ``` The gateway does not serve `/deliver/telegram`. The `replyCallbackUrl` it attaches to the inbound forward is an addressing token: the daemon resolves its path to the channel's transport (`channelForCallback` in `assistant/src/messaging/providers/callback-routing.ts`) and reads per-channel parameters from its query (the Telegram transport reads `threadId` to reply into the same topic). The daemon never dials the URL's host and port, which come from `gatewayInternalBaseUrl` (`http://127.0.0.1:${GATEWAY_PORT}`, `gateway/src/config.ts`). The gateway's own replies to invite and verification codes it intercepts at ingress go out the same way: `deliverVerificationReply` (`gateway/src/verification/reply-delivery.ts`) hands the reply and the inbound message's callback URL to the daemon's IPC-only `deliver_gateway_reply` method, which resolves the transport from that URL as it does for any reply. The gateway never sends those replies to a provider itself. The daemon's Telegram transport (`assistant/src/messaging/providers/telegram-bot/`) reads the bot token from credential storage and calls the Bot API itself. The gateway sends to Telegram on its own only for notices it composes while handling the webhook, before or instead of a runtime turn: the `/start` acknowledgement, the "not fully set up" routing-rejection notice, a setup-hiccup notice when the forward fails, and the denial text the runtime returns when ingress ACL rejects the sender. These go through `sendTelegramReply` in `gateway/src/telegram/send.ts`. **Bot-account limitations:** The Telegram Bot API only supports sending messages to chats that have previously interacted with the bot. Bots cannot enumerate chats, read message history, or search messages. A future MTProto user-account session track may lift some of these restrictions. ### Channel Approval Flow When the assistant requires tool-use confirmation during a channel session (e.g., Telegram), the approval flow intercepts the run and surfaces an interactive prompt to the user. This approval-aware path is always active when orchestrator + callback context are available. Guardian enforcement (fail-closed denial for unknown actors, explicit approval prompts for side effects, guardian-routed approval prompts) applies consistently to non-guardian/unverified actors. **State machine:** ``` Running → NeedsConfirmation → Running → Completed | Failed ``` The run transitions to `NeedsConfirmation` when the agent loop emits a `confirmation_request` event. It returns to `Running` after the user submits a decision (approve/reject), and eventually reaches `Completed` or `Failed`. **Telegram button round-trip:** ``` Runtime detects needs_confirmation → runtime builds approval prompt + UI metadata → daemon Telegram transport sends the prompt with an inline keyboard (buttons: Approve once, Approve always, Reject) → user clicks button → Telegram callback_query → gateway normalizes callback_query into inbound event (callbackData field) → runtime parses callback data (format: apr::) → runtime applies decision to pending run → run resumes ``` **Deliver-once guard:** Both the main poll (`processChannelMessageWithApprovals`) and the post-decision poll (`schedulePostDecisionDelivery`) race to deliver the final assistant reply when a run reaches terminal state. The `claimRunDelivery()` function in `delivery-channels.ts` uses an in-memory `Set` to ensure at-most-one delivery per run. The first caller to claim the run ID proceeds with delivery; the other silently skips it. This is an in-memory guard — sufficient because both racing pollers execute within the same process. **Fail-closed prompt delivery:** All approval prompt delivery paths auto-deny the run if the prompt cannot be delivered, preventing silent `needs_confirmation` hangs: - **Standard (self-approval):** `deliverApprovalPrompt()` failure triggers `handleChannelDecision(reject)`. - **Guardian-routed:** Delivery failure to the guardian's chat marks the approval record `denied`, rejects the run, and notifies the requester. - **Unverified channel (no guardian binding):** Sensitive actions are auto-denied immediately without attempting prompt delivery. **Stale callback blocking:** When inbound `callbackData` does not match any pending approval (e.g., a button from an old prompt), the runtime returns `stale_ignored` and does not process the payload as a regular message. This prevents stale button presses from triggering unrelated agent loops. **Conversational approval turn:** When a text message arrives while an approval is pending (e.g., non-Telegram channels or user typing a reply instead of clicking a button), a **conversational approval turn** is run via `runApprovalConversationTurn()` from `approval-conversation-turn.ts`. The conversational engine uses LLM structured output (native `tool_use`) to classify user intent as: `keep_pending` (reply without deciding), `approve_once`, `approve_always`, or `reject`. Non-decision messages receive a natural assistant reply and the run stays pending — no reminder spam. The engine fails closed: any model failure returns `keep_pending` with a deterministic fallback asking the user to try again. Callback/button handling remains deterministic and unchanged. The `supportsInlineOptions` channel capability (`channelSupportsInlineOptions()`) determines whether to send the structured `promptText` (for channels whose adapter renders inline buttons — Telegram, WhatsApp, Slack) or the `plainTextFallback` string (for all other channels). **Guardian-aware routing:** When a guardian binding exists for the channel, the approval flow resolves the sender's actor role (`guardian` vs `non-guardian`). Non-guardian actors have `forcePromptSideEffects` set on the session so all side-effect tools trigger approval prompts regardless of existing allow rules. Approval prompts for non-guardian actions are routed to the guardian's delivery chat (not the requester's chat), and a guardian request (kind `tool_approval`) is recorded in the gateway-owned `guardian_requests` table via `guardian_requests_create`. When the guardian approves or denies, the decision commits through `guardian_requests_decide`, is applied to the underlying run, and the requester's chat is notified of the outcome. Guardian actors follow the standard approval flow. Guardian approval follow-ups also use the conversational engine with role-specific context; `approve_always` is downgraded to `approve_once` for guardian approvals since permanent allow-rules require guardian authority. All guardian state (bindings, challenges, guardian requests) is scoped to the `(assistantId, channel)` pair -- the `assistantId` parameter flows through `handleChannelInbound`, `validateAndConsumeVerification`, `isGuardian`, and `getGuardianBinding`. **Proactive expiry sweep:** The daemon runs a periodic sweep every 60 seconds (`assistant/src/runtime/routes/guardian-expiry-sweep.ts`) that reads a bounded batch of past-deadline pending requests (`guardian_requests_list_expired_pending`, read-only), fans out card withdrawals and requester expiry notices per row, and only then confirms each with the per-request `guardian_requests_expire` CAS. The status flip is the receipt that the side effects ran: a lost IPC response or a daemon crash leaves the row pending and listed for the next round. Expired approvals close without waiting for follow-up traffic from either party, and a decision arriving past the deadline loses to expiry atomically (the decide CAS guards on `expires_at`). **Gateway-origin ingress contract:** The JWT token exchanged during gateway-to-runtime authentication proves gateway origin (via the `aud=vellum-daemon` claim). No separate header is required. **Key modules:** | Module | Purpose | | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `assistant/src/runtime/approval-conversation-turn.ts` | Conversational approval turn engine: LLM-based intent classification (structured output) for pending approval follow-ups, with fail-closed safety | | `assistant/src/runtime/approval-message-composer.ts` | Centralized approval message composition: layered source selection (assistant preface → deterministic fallback) for all approval/guardian/verification user-facing copy | | `assistant/src/runtime/channel-approvals.ts` | Orchestration: detect pending confirmations, build prompts (including guardian-aware prompts), apply decisions, plain-text fallback selection | | `assistant/src/runtime/channel-approval-types.ts` | Shared types: actions, prompts, UI metadata, decisions | | `assistant/src/runtime/routes/channel-routes.ts` | Integration point: approval interception, actor role resolution, guardian approval routing, deliver-once guard, fail-closed prompt delivery | | `assistant/src/runtime/channel-verification-service.ts` | Guardian binding lookups: `isGuardian()`, `getGuardianBinding()` | | `assistant/src/memory/delivery-channels.ts` | `claimRunDelivery()`: in-memory deliver-once guard for terminal reply idempotency | | `assistant/src/channels/gateway-guardian-requests.ts` | Typed daemon client for the gateway-owned `guardian_requests` lifecycle (`guardian_requests_create` / `_decide` / `_list_expired_pending` / `_expire`) | | `assistant/src/runtime/gateway-client.ts` | `deliverApprovalPrompt()`: hands the approval prompt to the channel transport the callback URL names | | `assistant/src/messaging/providers/telegram-bot/send.ts` | `buildInlineKeyboard()`: renders approval actions as Telegram inline buttons | | `gateway/src/telegram/normalize.ts` | `callback_query` normalization into `GatewayInboundEvent` (private chats, groups, and supergroups; drops callbacks without data) | ### Approval Message Composer The `approval-message-composer.ts` module provides a centralized system for generating approval lifecycle messages across channels. It uses a layered source selection: 1. **Assistant preface** — reuses existing assistant text (macOS parity) 2. **Deterministic fallback** — scenario-specific templates with required semantic content All approval/guardian/verification user-facing copy routes through this composer. Deterministic behavior (action IDs, callback payloads) remains separate and unchanged. A guard test (`approval-hardcoded-copy-guard.test.ts`) scans the route/service files for banned hard-coded copy literals to prevent regressions. ### Channel Guardian Security The guardian system adds a cryptographic trust layer for channel-based interactions. A **guardian** is the designated human authority for a given `(assistantId, channel)` pair. Once verified, the guardian's identity gates sensitive actions by non-guardian users and provides an approval pathway. All guardian operations (bindings, challenges, approval requests) are scoped by `assistantId` -- verifying as guardian on one assistant does not grant guardian status on another. #### Canonical Assistant ID Scoping The daemon uses the fixed constant `DAEMON_INTERNAL_ASSISTANT_ID` (`"self"`) from `runtime/assistant-scope.ts` for all assistant-scoped storage keys. Public/external assistant IDs are a gateway/platform concern and never leak into daemon scoping logic. The `normalizeAssistantId()` function in `util/platform.ts` exists solely for the gateway layer to canonicalize inbound requests before proxying -- daemon and runtime code must not call it or accept an `assistantId` parameter for scoping. #### Guardian Verify Code Parsing The inbound message handler (`inbound-message-handler.ts`) accepts verification codes as bare-code replies: - **Bare code**: A 6-digit numeric code sent as the entire message body. This is the primary flow — the user is shown a verification code in setup UI and sends that code in-channel as a plain message. #### Explicit Rebind Policy Creating a new guardian challenge when a binding already exists for the `(assistantId, channel)` pair requires explicit `rebind: true` in the HTTP request. Without it, the daemon returns `already_bound` to the caller. This prevents accidental guardian replacement -- the desktop UI must explicitly acknowledge that it is replacing an existing guardian before a new challenge is issued. Minting is all it permits on a text channel: a code never swaps the guardian's linked identity when it is redeemed (see below), so changing it means revoking the existing binding and then verifying again. An outbound phone verification is the exception: it is guardian-initiated by design, and its code replaces the bound number (`applyPhoneGuardianBindingGatewayWrites` in `gateway/src/verification/session-service.ts`). #### Guardian Takeover Prevention Text-channel redemption (`applyGuardianSideEffects` in `gateway/src/verification/text-verification.ts`) rejects a guardian code when any active guardian binding on the channel belongs to a _different_ external user: the code is spent, the sender is told it was invalid or expired, and they are made neither guardian nor contact. This prevents an attacker who intercepts a verification code from hijacking an established guardian binding. Same-user re-verification (e.g., re-verifying after a session timeout) is allowed, since the external user ID matches the existing binding. A guardian who revoked their binding can verify the same account again: their own revoked row is reactivated when it belongs to the guardian contact, its stored address is exactly the redeeming one, and no other identity is linked on the channel. A revoked row that belonged to any other contact stays revoked. A blocked row is never reactivated. #### Guardian Verification Flow A challenge-response handshake binds a Telegram user as the guardian: ```mermaid sequenceDiagram participant User as Guardian Candidate participant Desktop as Desktop Client participant TG as Telegram participant GW as Gateway participant Daemon as Daemon (Runtime) Desktop->>Daemon: POST /v1/channel-verification-sessions (action: create_session) Daemon->>Daemon: Generate random secret, hash (SHA-256), store challenge (10min TTL) Daemon-->>Desktop: Return secret + instruction Desktop-->>User: Display verification code User->>TG: TG->>GW: POST /webhooks/telegram (webhook secret validated) GW->>GW: Verify webhook secret, normalize update GW->>Daemon: POST /v1/channels/inbound (JWT auth) Daemon->>Daemon: Verify JWT auth Daemon->>Daemon: Hash secret, find pending challenge, validate expiry Daemon->>Daemon: Consume challenge (replay prevention) Daemon->>Daemon: Revoke existing binding (if any) Daemon->>Daemon: Create new guardian binding (active) Daemon->>GW: Reply via callback URL GW->>TG: sendMessage: "You are now the guardian" ``` The raw secret is shown only once in the desktop UI and must be sent by the user in-channel to complete verification. (Outbound session creation via `POST /v1/channel-verification-sessions` with a `destination` separately sends template messages/calls with the code.) Only the SHA-256 hash is persisted. Challenges expire after 10 minutes. Consumed challenges cannot be reused. Rate limiting (5 invalid attempts per 15-minute window, 30-minute lockout) protects against brute-force attacks. #### Inbound Message Decision Chain The channel inbound handler (`inbound-message-handler.ts`) evaluates incoming messages through a strict decision chain. Ingress ACL enforcement runs first, before guardian role resolution and message processing: ```mermaid flowchart TD MSG["Inbound message arrives
POST /channels/inbound"] --> GW_CHECK{"JWT auth
valid?"} GW_CHECK -- No --> REJECT_403["403 Unauthorized"] GW_CHECK -- Yes --> HAS_SENDER{"actorExternalId
present?"} HAS_SENDER -- Yes --> ACL_LOOKUP["Look up ingress member
by (channel, userId/chatId)"] HAS_SENDER -- No --> RECORD["Record inbound event"] ACL_LOOKUP --> HAS_MEMBER{"Member record
found?"} HAS_MEMBER -- No --> DENY_MEMBER["Deny: not_a_member"] HAS_MEMBER -- Yes --> STATUS_CHECK{"Member status
= active?"} STATUS_CHECK -- No --> DENY_STATUS["Deny: member_{status}"] STATUS_CHECK -- Yes --> POLICY_CHECK{"Member policy?"} POLICY_CHECK -- deny --> DENY_POLICY["Deny: policy_deny"] POLICY_CHECK -- allow --> RECORD RECORD --> VERIFY_CHECK{"Guardian verify
code?"} VERIFY_CHECK -- Yes --> VERIFY["Validate challenge
→ create guardian binding"] VERIFY_CHECK -- No --> ROLE_RESOLVE["Resolve actor role
(trust-context-resolver)"] ROLE_RESOLVE --> APPROVAL_INTERCEPT["Approval interception
+ message processing"] ``` This ordering ensures that ingress ACL decisions are finalized before any agent processing occurs. Guardian verification code replies are intercepted after ACL enforcement but before the agent loop, so they never trigger inference. #### Actor Role Resolution When a message arrives on a channel, the runtime resolves the sender's role. Role _classification_ runs unconditionally. Guardian enforcement (`forcePromptSideEffects`, fail-closed denial, guardian approval routing) applies to non-guardian/unverified actors whenever orchestrator + callback context are available: - **Guardian**: `actorExternalId` matches the binding's `guardianExternalUserId` (DB column) for the `(assistantId, channel)` pair. Self-approval is handled through the same approval-aware channel flow. - **Non-guardian**: A known sender who is not the guardian. Side-effect tools are forced through the confirmation flow (`forcePromptSideEffects`), and approval prompts are routed to the guardian's chat instead of the requester's chat. - **Unverified channel**: No guardian binding exists for the channel, or `actorExternalId` is absent. Sensitive actions are auto-denied immediately (fail-closed). This prevents unverified senders from self-approving actions or bypassing guardian enforcement by omitting identity data. #### Sensitive Action Gating (Non-Guardian Approval) When a non-guardian user triggers a tool requiring confirmation, the approval is escalated to the guardian: ```mermaid sequenceDiagram participant NG as Non-Guardian User participant TG as Telegram participant GW as Gateway participant Daemon as Daemon (Runtime) participant Guardian as Guardian (Telegram DM) NG->>TG: Message triggers tool use TG->>GW: POST /webhooks/telegram GW->>Daemon: POST /v1/channels/inbound (JWT auth) Daemon->>Daemon: Detect non-guardian, set forcePromptSideEffects Daemon->>Daemon: Tool needs confirmation → create GuardianApprovalRequest Daemon->>TG: sendMessage via Telegram transport (approval prompt + inline keyboard) TG-->>Guardian: Approval prompt Daemon->>TG: sendMessage via Telegram transport (requester notification) TG-->>NG: "Waiting for guardian approval..." Guardian->>TG: Approve / Deny (callback_query or text) TG->>GW: POST /webhooks/telegram (callback_query) GW->>Daemon: POST /v1/channels/inbound (JWT auth) Daemon->>Daemon: Validate guardian identity, update approval decision Daemon->>Daemon: Apply decision to pending run Daemon->>TG: sendMessage via Telegram transport (outcome notification) TG-->>NG: "Guardian approved/denied your request" Daemon->>TG: sendMessage via Telegram transport (confirmation) TG-->>Guardian: Confirmation of decision ``` Approval state lives in the gateway's `guardian_requests` table (kind `tool_approval`), with per-surface card deliveries in `guardian_request_deliveries`. Each request records the requester, guardian, tool name, risk level, and decision outcome; decisions commit atomically via the gateway's `guardian_requests_decide` IPC route. **Key modules:** | Module | Purpose | | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `gateway/src/db/guardian-request-store.ts` | Gateway-side store for `guardian_requests` / `guardian_request_deliveries`; first-writer-wins decision via atomic status CAS | | `assistant/src/channels/gateway-guardian-requests.ts` | Daemon-side client: typed wrappers over the gateway's `guardian_requests_*` IPC routes | | `assistant/src/channels/gateway-verification-sessions.ts` | Guardian binding types and verification-session relay to the gateway store | | `assistant/src/runtime/channel-verification-service.ts` | Challenge creation/validation, guardian identity checks (`isGuardian()`, `getGuardianBinding()`) -- all accept `assistantId` | | `assistant/src/runtime/trust-context-resolver.ts` | Actor role classification: guardian / non-guardian / unverified_channel based on binding state + sender identity | | `assistant/src/runtime/routes/inbound-message-handler.ts` | Ingress ACL enforcement, verification-code intercept, actor role resolution | | `assistant/src/runtime/routes/guardian-expiry-sweep.ts` | Proactive 60s expiry sweep: gateway CAS-expiry plus card-withdrawal and requester-notice fan-out | | `assistant/src/calls/guardian-dispatch.ts` | Cross-channel ASK_GUARDIAN dispatch: creates gateway guardian requests, fans out to mac/telegram, records deliveries | | `assistant/src/calls/guardian-action-sweep.ts` | Expiry notice delivery to guardian destinations (vellum conversation message or direct channel reply) | ### Ingress Membership The ingress membership system extends the guardian security model to support controlled cross-user access. External users interact with the assistant through channels (Telegram, WhatsApp) under an invite-based membership system with per-member access policies. #### Ingress Membership ACL The channel inbound handler (`inbound-message-handler.ts`) enforces an access control layer between message receipt and agent processing. The ACL runs at the top of the handler, before guardian role resolution or verification-code interception (see [Inbound Message Decision Chain](#inbound-message-decision-chain) for the full ordering): 1. When `actorExternalId` is present, the handler looks up the sender in the `contacts` table via `findContactChannel` by `(channelType, externalUserId)` or `(channelType, externalChatId)`. 2. If no member record exists, the message is denied (`not_a_member`). 3. If a member exists but is not `active` (e.g., `revoked`, `blocked`), the message is denied. 4. If the member's `policy` is `deny`, the message is rejected. If `allow`, the message proceeds to normal processing. **Invite-based onboarding:** Invite tokens are minted by the gateway via the invite HTTP API and stored SHA-256 hashed on the gateway DB's `ingress_invites` row -- the raw token is returned exactly once at creation time. External users redeem invites by sending the token as a channel message, which atomically creates a member record with `active` status and `allow` policy. **Relationship to guardian verification:** Guardian verification and ingress contact management are independent systems. Guardian verification establishes who controls the assistant on a channel (the trust anchor for approvals). Ingress contacts control who can interact with the assistant. #### SQLite Tables **Assistant DB** (`assistant.db` — owner of contact identity/info; ACL fields are a best-effort mirror of the gateway DB): | Table | Purpose | | ------------------ | ----------------------------------------------------------------- | | `contacts` | Contact records with role, relationship, and per-contact metadata | | `contact_channels` | Channel bindings per contact with access policy (allow/deny) | **Gateway DB** (`gateway.sqlite` — canonical owner of invites and contact auth/authz): | Table | Purpose | | ------------------ | ------------------------------------------------------------------------------------ | | `contacts` | Contact auth/authz: id, display_name, role, principal_id | | `contact_channels` | Channel bindings with policy, status, external IDs, verification state | | `ingress_invites` | Canonical invite store — token/code hashes, expiry, use counts, voice/display fields | The gateway's `ingress_invites` table is the sole invite store: mint, list, revoke, and redemption all run gateway-natively, and the daemon relays its invite surfaces here over IPC. The gateway data migrations `m0007`/`m0009` reference `assistant_ingress_invites` — a legacy assistant table absent from the current assistant schema — only as a one-time backfill source. The gateway declares `contacts` and `contact_channels` tables and exposes them via IPC (`list_contacts`, `get_contact`, `get_contact_by_channel`, `get_channels_for_contact`). The gateway DB is the canonical owner of contact ACL (role, principal, channel status/policy/verification): contact writes land gateway-first, with identity/info fields mirrored to the assistant DB best-effort. #### Key Modules | Module | Purpose | | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | `gateway/src/http/routes/contacts-control-plane-proxy.ts` | Gateway-native invite lifecycle (mint, list, revoke, redeem) shared by HTTP and IPC | | `gateway/src/ipc/invite-handlers.ts` | IPC routes relaying the daemon's invite surfaces to the native functions | | `gateway/src/verification/invite-redemption.ts` | Redemption engine — validation, atomic claim, ACL activation | | `assistant/src/contacts/contact-store.ts` | Contact and channel lookups (findContactChannel, guardian bindings) | | `assistant/src/contacts/contacts-write.ts` | Contact and channel identity/info writes (upsert, redemption info mirror — ACL is gateway-owned) | | `assistant/src/ipc/routes/invite-ipc-routes.ts` | `invite_redeemed` info mirror — local contact/channel identity upsert | | `assistant/src/runtime/routes/inbound-message-handler.ts` | ACL enforcement point -- member lookup, policy check | | `gateway/src/db/contact-store.ts` | Gateway-side ContactStore — contact/channel reads and invite CRUD | | `gateway/src/ipc/contact-handlers.ts` | IPC route handlers for contact reads | ### Telegram Credential Flow In desktop deployments, Telegram bot tokens are stored in secure storage (CES HTTP API when available, or the encrypted file store at `~/.vellum/protected/keys.enc`) and never in plaintext config files. When deploying the gateway standalone, operators may also supply credentials via environment variables (`TELEGRAM_BOT_TOKEN`, `TELEGRAM_WEBHOOK_SECRET`). ``` Entry points: 1. Skill-based setup (chat): assistant credentials prompt → token stored in secure storage → telegram_config HTTP (action: set) — daemon reads token from secure storage 2. Desktop Settings UI (macOS): SettingsStore.saveTelegramToken(token) → telegram_config HTTP (action: set, botToken: token) — token passed directly Both paths converge at: → Daemon handler validates token via Telegram getMe API → setSecureKeyAsync("credential/telegram/bot_token", token) → upsertCredentialMetadata("telegram", "bot_token", {}) → Stores bot username in config at telegram.botUsername → Auto-generates webhook secret if missing → setSecureKeyAsync("credential/telegram/webhook_secret", secret) → upsertCredentialMetadata("telegram", "webhook_secret", {}) → On storage failure: rolls back bot_token + metadata, returns error → If webhook secret already exists: upserts metadata anyway (self-heal) → Credential watcher detects storage change → Invalidates credential cache, triggers side-effect callback → Webhook reconciliation registers webhook with Telegram ``` The `telegram_config` HTTP endpoint supports three actions: - **`get`** — returns connection status (`hasBotToken`, `botUsername`, `connected`, `hasWebhookSecret`) without exposing secret values - **`set`** — validates the bot token against the Telegram API, stores it in secure storage, auto-generates a webhook secret if none exists (with rollback on failure), and self-heals webhook_secret metadata if it already exists. The gateway's credential watcher detects the storage change and triggers webhook reconciliation automatically - **`clear`** — deregisters the webhook by calling Telegram's `deleteWebhook` API directly (while the token is still available), then deletes the bot token and webhook secret from both secure storage and credential metadata. The gateway's credential watcher detects the storage change and updates its readiness state automatically The gateway reads Telegram credentials via its `credential-reader` module (`gateway/src/credential-reader.ts`), which uses a CES-first fallback strategy: it tries the CES HTTP API first (when `CES_CREDENTIAL_URL` is configured), then falls back to the encrypted file store (`~/.vellum/protected/keys.enc`). ### Webhook Reconciliation On startup, the gateway automatically reconciles the Telegram webhook registration: 1. Reads Telegram credentials (bot token, webhook secret) from secure storage via the credential reader, and resolves the URL to register (see the tiers below) 2. Calls `getWebhookInfo` to log the current registration state 3. Unconditionally calls `setWebhook` with the expected URL, secret, and allowed updates (idempotent — Telegram does not expose the current secret via `getWebhookInfo`, so a compare-then-set approach would miss secret rotations) **Which URL Telegram is pointed at.** `resolveExpectedTelegramWebhookUrl` (`gateway/src/telegram/webhook-manager.ts`) resolves it, and has to agree tier for tier with `hasWebhookRoutingConfigured` in `assistant/src/config/webhook-routing.ts`, because that derivation is what the daemon reports to the user: - A **platform pod** with `velay-webhooks` off registers a Django-hosted callback route and never consults ingress. A pod's `ingress.publicBaseUrl` is written by the Velay tunnel client and cleared when the tunnel drops, while Telegram keeps delivering to whatever was last registered, so resolving through that address alone would leave the pod pointed at a dead one. - A **platform pod** with the flag on claims `/webhooks/telegram` in the webhook ingress route registry and points Telegram at `/webhooks/telegram`. The claim comes first because both admission layers in front of the route consult the registry. No published URL, or a claim that fails, falls back to the Django callback route. - **Everyone else** uses a configured `ingress.publicBaseUrl` when there is one (a self-hosted tunnel) and a Django callback route otherwise. An explicit `ingress.enabled: false` deregisters the webhook instead; platform pods are exempt from that flag, having no self-owned ingress to disable. Reconciliation re-runs when the credential watcher sees Telegram or `vellum` credentials change, when the config file watcher sees an ingress change, on system wake, and when `velay-webhooks` flips. The flag flip and the tunnel publishing its URL are what move a pod between the two addresses, in either direction, so a gradual flag rollout migrates assistants without a restart. Those triggers arrive in bursts, because a tunnel refresh clears `ingress.publicBaseUrl` and republishes it moments later, so `reconcileTelegramWebhook` serializes them: one reconciliation runs, everything that arrives during it collapses into a single rerun, and that rerun re-reads config so the last `setWebhook` is the one holding the settled address. No daemon involvement is needed for any of it, and manual webhook registration is not required. ### Routing A gateway process fronts exactly one daemon, so routing needs no configuration: any inbound event carrying a routable identity resolves to the local assistant (`LOCAL_ASSISTANT_ID`). The only event routing drops is one with neither a conversation nor an actor id, which has nothing to route on. Whether a resolved event is _admitted_ is a separate decision, made against the channel's admission policy floor — see "Channel Trust Classification & Admission Policy" in `gateway/CLAUDE.md`. Routing answers "which assistant", admission answers "is this sender allowed". Optional `routingEntries` in workspace config still map specific conversation/actor ids to explicit assistant ids, ahead of the local fallback. ### Slack Channel (Socket Mode) The Slack channel enables inbound and outbound messaging via Slack's Socket Mode API. Unlike Telegram and WhatsApp which use HTTP webhook ingress, Slack uses a persistent WebSocket connection — no public ingress URL is required. **Connection lifecycle** (`apps.connections.open`): 1. The gateway calls `POST https://slack.com/api/apps.connections.open` with the Slack app-level token (`xapp-...`) to obtain a WebSocket URL. 2. A `SlackSocketModeClient` opens the WebSocket and maintains a single active connection. 3. On connection, the client resets its reconnect counter. On close or error, it schedules a reconnect with capped exponential backoff (1s base, 30s max) plus 0-50% jitter. 4. Slack may send a `disconnect` envelope to request a reconnect (e.g., server rotation). The client handles this by closing the current socket and reconnecting immediately (attempt counter reset to 0). **Event processing** (inbound): 1. Every Socket Mode envelope is ACKed immediately by echoing `{ envelope_id }` back on the WebSocket — this is required by Slack regardless of whether the event is processed. 2. `events_api` envelopes are admitted per event kind (`processEventPayload` in `socket-mode.ts`): `app_mention` events; every message in a DM or group DM (`message.im`, `message.mpim`); unmentioned replies in a thread the assistant is already part of (mention-then-thread mode); and edits, deletes, and reactions scoped to those rooms. The bot's own echoes are dropped by a single self-filter before routing. Interactive payloads (button presses) are handled separately; slash commands are ACKed and ignored. 3. Events are deduplicated by a compound key in the SQLite-backed `slack_seen_events` table: every event records its Slack `event_id`, and message-shaped events additionally record `msg:${channel}:${ts}` so the live and reconnect-replay paths dedup symmetrically. Entries TTL out after 24h; a periodic cleanup sweep evicts expired rows. 4. The `normalizeSlackAppMention()` function strips leading bot-mention tokens (`<@U...>`) from the message text and produces a `GatewayInboundEvent` with `sourceChannel: "slack"`, using the Slack channel ID as `conversationExternalId` and the sender's user ID as `actorExternalId`. 5. Routing uses the standard `resolveAssistant()` chain (conversation_id -> actor_id -> default/reject). Events that cannot be routed are dropped. 6. The normalized event is forwarded to the runtime via `POST /v1/channels/inbound` with a `replyCallbackUrl` of `/deliver/slack?channel=`, plus `threadTs` for a threaded message or `messageTs` for a thread-less message. **Egress:** The gateway does not serve `/deliver/slack`; for the daemon the callback URL only addresses the reply (the gateway's intercepted-code replies reach the same transport through the daemon, see Telegram Messaging Flow). The daemon's Slack transport (`assistant/src/messaging/providers/slack/`) calls the Slack Web API itself with the bot token: `chat.postMessage` and `chat.update` for whole messages, and `chat.startStream` / `chat.appendStream` / `chat.stopStream` for streamed replies. It reads `threadTs` from the callback URL to reply in the thread and `messageTs` to anchor the busy indicator on a thread-less message. **Credential management:** The Slack channel requires two tokens: | Token | Format | Purpose | | --------- | ---------- | ------------------------------------------------------------------------------------ | | App token | `xapp-...` | Used for `apps.connections.open` to establish the Socket Mode WebSocket connection | | Bot token | `xoxb-...` | Used for `chat.postMessage` to send outbound messages and for `auth.test` validation | Both tokens are stored in secure storage (`credential/slack_channel/app_token`, `credential/slack_channel/bot_token`) via the assistant's Slack channel config endpoints (see `assistant/ARCHITECTURE.md`). The gateway reads them via its `credential-reader` module using the same CES-first fallback strategy as Telegram credentials. **Auto-reconnect behavior:** The Socket Mode client auto-reconnects on any WebSocket close or error. The backoff schedule is: `min(1000 * 2^attempt, 30000)` + random jitter. After a successful connection, the attempt counter resets. Slack-initiated disconnects (envelope `type: "disconnect"`) trigger an immediate reconnect with no backoff. **Reconnect catch-up:** Slack [does not buffer Socket Mode events](https://api.slack.com/apis/socket-mode#events) for disconnected clients, so any @mention or DM that arrives during a reconnect window is lost on the wire. The client recovers these by maintaining a persistent high-watermark (`slack_last_seen_ts`) of the latest accepted event timestamp and, on every WebSocket `open`, fetching a bounded slice of [`conversations.history`](https://api.slack.com/methods/conversations.history) and [`conversations.replies`](https://api.slack.com/methods/conversations.replies) since that watermark. Recovered messages are wrapped in synthetic Socket Mode envelopes and dispatched through the same `processEventPayload` path as live events, so filter, dedup, normalize, and routing logic stay in one place. Catch-up is scoped to channels the gateway can reasonably reach: routing entries with a Slack-shaped conversation ID (`C…` / `D…` / `G…`), tracked active threads (`slack_active_threads`), and previously-seen DM channels (`contact_channels` rows of type `slack`). It is bounded by max-lookback (1h), per-channel limit (50), and concurrency (4); on a 429 the cycle aborts immediately and resumes on the next reconnect. Brand-new mentions in unrouted, never-engaged channels remain unrecoverable here — the daemon's existing inbound-triggered Slack backfill (`triggerSlackThreadBackfillIfNeeded` and `tryBackfillSlackDmIfCold`) hydrates context once the next live event arrives. **General principle — stateful-stream transports require catch-up-on-reconnect:** Any persistent-stream transport that does not buffer events for disconnected clients (Slack Socket Mode, similar WebSocket gateways) must combine the WebSocket with a HTTP catch-up path on reconnect. A persisted high-watermark + bounded replay through the shared event pipeline is the standard pattern; without it, every transient disconnect silently drops events. **Key modules:** | Module | Purpose | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | `gateway/src/slack/socket-mode.ts` | `SlackSocketModeClient`: WebSocket lifecycle, ACK, dedup, auto-reconnect, reconnect catch-up | | `gateway/src/slack/slack-web.ts` | `conversations.history` / `conversations.replies` helpers for reconnect catch-up | | `gateway/src/slack/message-normalizer.ts` | Normalizers per event family (`normalizeSlackAppMention()`, DM, group DM, channel message) with bot-mention stripping | | `gateway/src/index.ts` | Builds the `/deliver/slack` callback URL (channel, thread ts, message ts) the daemon's Slack transport replies to | **What the ingress does not do:** it never forwards the bot's own posts to the daemon (except a deletion of one, which the daemon records), and it never reads history on the daemon's behalf beyond the bounded reconnect catch-up above; the daemon's inbound-triggered backfill hydrates context. --- ## AI Phone Calls — Twilio Voice The Calls subsystem supports both **outbound** and **inbound** voice calls via Twilio. Every call connects over Twilio Media Streams (`` TwiML): the daemon performs speech-to-text on the raw mu-law audio (streaming when the configured `services.stt` provider supports it, batch otherwise) and synthesizes speech via the configured `services.tts` provider, transcoded to mu-law frames. The assistant uses an LLM-driven conversation loop to speak in real time. Voice is a first-class channel with its own per-call conversation (outbound key: `asst:${assistantId}:voice:call:${callSessionId}`, inbound key: `asst:${assistantId}:voice:inbound:${callSid}`). When the AI needs guardian input during a call, it dispatches ASK_GUARDIAN requests cross-channel to mac/telegram via the guardian dispatch engine. Answer resolution uses first-writer-wins semantics -- the first channel to respond provides the answer, and remaining channels receive a "already answered" notice. ### Outbound Call Flow ```mermaid sequenceDiagram participant User as User (Chat UI) participant CallStore as CallStore (SQLite) participant TwilioProvider as TwilioProvider participant TwilioAPI as Twilio REST API participant Gateway as Gateway (public) participant Routes as twilio-routes.ts (runtime) participant WS as MediaStreamCallSession (WebSocket) participant Ctrl as CallController participant Bridge as voice-session-bridge participant RunOrch as RunOrchestrator participant Session as Session / AgentLoop participant LLM as Anthropic Claude participant State as CallState (Notifiers) participant GuardianDispatch as GuardianDispatch participant Mac as Mac Desktop participant TG as Telegram participant CallDomain as CallDomain User->>Session: call_start tool Session->>CallStore: createCallSession() Session->>TwilioProvider: initiateCall() TwilioProvider->>TwilioAPI: POST /Calls.json TwilioAPI-->>TwilioProvider: { callSid } Session->>CallStore: updateCallSession(providerCallSid) TwilioAPI->>Gateway: POST /webhooks/twilio/voice Gateway->>Gateway: validateTwilioWebhookRequest() Gateway->>Routes: forward to runtime /v1/internal/twilio/voice-webhook (+ params, originalUrl) Routes->>CallStore: getCallSession() Routes->>Routes: credential preflight (STT + TTS readiness) Routes-->>Gateway: TwiML (Connect/Stream media-stream connect) Gateway-->>TwilioAPI: TwiML response TwilioAPI->>Gateway: WebSocket /webhooks/twilio/media-stream/{callSessionId}/{token} Gateway->>WS: proxy WS to runtime /v1/calls/media-stream WS->>WS: start event (streamSid, callSid) WS->>WS: routeSetup() → setup outcome (CallSetupFlow for interactive outcomes) WS->>Ctrl: new CallController() Ctrl->>State: registerCallController() loop Conversation turns TwilioAPI->>WS: media frames (mu-law audio) WS->>WS: daemon STT (streaming or batch) → final transcript WS->>Ctrl: handleCallerUtterance(transcript) Ctrl->>Bridge: startVoiceTurn() Bridge->>RunOrch: startRun(conversationId, content, {sourceChannel: 'phone', eventSink}) RunOrch->>Session: route to session pipeline Session->>LLM: agent loop (tools, memory, skills) LLM-->>Session: text tokens (streaming) Session-->>Bridge: eventSink.onTextDelta() Bridge-->>Ctrl: onTextDelta callback Ctrl->>WS: sendTextToken() (daemon TTS → mu-law media frames) Ctrl->>CallStore: recordCallEvent() end alt ASK_GUARDIAN pattern detected Ctrl->>CallStore: createPendingQuestion() Ctrl->>GuardianDispatch: dispatchGuardianQuestion() GuardianDispatch->>Mac: notification_conversation_created SSE GuardianDispatch->>TG: sendMessage via notification pipeline Note over Mac,TG: First channel to respond wins Mac/TG->>Routes: guardian answer Routes->>CallDomain: answerCall() CallDomain->>Ctrl: handleUserAnswer() Ctrl->>Bridge: startVoiceTurn([USER_ANSWERED: ...]) end alt END_CALL pattern detected Ctrl->>WS: endSession() Ctrl->>CallStore: updateCallSession(completed) Ctrl->>State: fireCallCompletionNotifier() end TwilioAPI->>Gateway: POST /webhooks/twilio/status Gateway->>Gateway: validateTwilioWebhookRequest() Gateway->>Routes: forward to runtime /v1/internal/twilio/status (+ params) Routes->>CallStore: updateCallSession(status) ``` ### Inbound Call Flow Inbound calls are triggered when someone dials the assistant's Twilio phone number. The gateway resolves which assistant owns the number, the runtime bootstraps a session keyed by CallSid, and the call setup flow gates the call (guardian verification, invite redemption, or name capture + guardian approval wait, as routed by `routeSetup`) before handing off to the CallController. ```mermaid sequenceDiagram participant Caller as External Caller participant TwilioAPI as Twilio participant Gateway as Gateway (public) participant Routes as twilio-routes.ts (runtime) participant CallDomain as CallDomain participant CallStore as CallStore (SQLite) participant WS as MediaStreamCallSession + CallSetupFlow (WebSocket) participant GuardianSvc as ChannelGuardianService participant Ctrl as CallController participant Bridge as voice-session-bridge participant RunOrch as RunOrchestrator participant Session as Session / AgentLoop participant LLM as Anthropic Claude Caller->>TwilioAPI: Dials assistant phone number TwilioAPI->>Gateway: POST /webhooks/twilio/voice (no callSessionId) Gateway->>Gateway: validateTwilioWebhookRequest() Gateway->>Gateway: resolveAssistantByPhoneNumber(config, To) alt Phone number matches an assistant Gateway->>Gateway: assistantId resolved else No phone number match Gateway->>Gateway: resolveAssistant(From, From) — fallback routing chain Gateway->>Gateway: use LOCAL_ASSISTANT_ID end Gateway->>Routes: forward to runtime /v1/internal/twilio/voice-webhook (+ params, originalUrl, assistantId) Routes->>CallDomain: createInboundVoiceSession(callSid, from, to, assistantId) CallDomain->>CallStore: getOrCreateConversation(voice:inbound:${callSid}) CallDomain->>CallStore: createCallSession() — task=null for inbound Routes->>Routes: credential preflight (STT + TTS readiness; Say + Hangup when not ready) Routes-->>Gateway: TwiML (Connect/Stream media-stream connect) Gateway-->>TwilioAPI: TwiML response TwilioAPI->>Gateway: WebSocket /webhooks/twilio/media-stream/{callSessionId}/{token} Gateway->>WS: proxy WS to runtime /v1/calls/media-stream WS->>WS: start event (streamSid, callSid) WS->>WS: detect isInbound (`session.initiatedFromConversationId == null`) WS->>WS: routeSetup() → setup outcome alt Pending voice guardian challenge exists WS->>WS: CallSetupFlow enters collecting_code state WS->>Caller: TTS "Please enter your six-digit verification code" loop DTMF / spoken digit attempts (max 3) Caller->>WS: DTMF digits or spoken digits WS->>GuardianSvc: validateAndConsumeVerification(code) alt Code matches GuardianSvc-->>WS: success + guardian binding created WS->>Ctrl: new CallController() + initial greeting else Code incorrect + attempts remaining WS->>Caller: TTS "That code was incorrect. Please try again." else Max attempts exceeded WS->>Caller: TTS "Verification failed. Goodbye." WS->>CallStore: updateCallSession(failed) WS->>WS: endSession() end end else No pending guardian challenge WS->>Ctrl: new CallController() + initial greeting end Ctrl->>Bridge: startVoiceTurn([CALL_OPENING]) Bridge->>RunOrch: startRun(conversationId, [CALL_OPENING], {sourceChannel: 'phone', eventSink}) RunOrch->>Session: route to session pipeline Note over Session: Session runtime assembly injects
voice channel context + system prompt Session->>LLM: agent loop (initial greeting turn) LLM-->>Session: receptionist-style greeting Session-->>Bridge: eventSink.onTextDelta() Bridge-->>Ctrl: onTextDelta callback Ctrl->>WS: sendTextToken() (daemon TTS → mu-law frames to caller) loop Conversation turns Caller->>WS: media frames (mu-law audio) WS->>WS: daemon STT (streaming or batch) → final transcript WS->>Ctrl: handleCallerUtterance(transcript) Ctrl->>Bridge: startVoiceTurn() Bridge->>RunOrch: startRun(conversationId, content, {sourceChannel: 'phone', eventSink}) RunOrch->>Session: route to session pipeline Session->>LLM: agent loop (tools, memory, skills) LLM-->>Session: text tokens (streaming) Session-->>Bridge: eventSink.onTextDelta() Bridge-->>Ctrl: onTextDelta callback Ctrl->>WS: sendTextToken() (daemon TTS → mu-law media frames) Ctrl->>CallStore: recordCallEvent() end ``` **Inbound vs. outbound detection**: The media-stream server determines call direction by checking `session.initiatedFromConversationId`. Outbound calls are initiated from an existing conversation (`initiatedFromConversationId` set). Inbound calls are bootstrapped from Twilio webhooks and therefore have `initiatedFromConversationId == null`. **Inbound system prompt**: The session pipeline (via voice-session-bridge) generates system prompts appropriate for the voice channel context. For inbound calls, this produces a receptionist-style prompt that greets the caller warmly and helps them with what they need. **Guardian voice verification gate**: When a pending voice guardian challenge exists (created via the desktop UI), inbound callers must enter a six-digit code via DTMF or by speaking the digits before the call proceeds. Up to 3 attempts are allowed. On success, a guardian binding is created and the call transitions to normal flow. On failure, the call ends with a "Verification failed" message. This allows guardians to verify their identity over voice before being granted channel access. ### Key Components | File | Role | | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `assistant/src/calls/call-store.ts` | CRUD operations for call sessions, call events, and pending questions in SQLite via Drizzle ORM | | `assistant/src/calls/call-domain.ts` | Shared domain functions (`startCall`, `getCallStatus`, `cancelCall`, `answerCall`, `relayInstruction`) used by both tools and HTTP routes | | `assistant/src/calls/guardian-dispatch.ts` | Cross-channel dispatch engine: fans out ASK_GUARDIAN questions to mac/telegram, creates server-side guardian conversations, manages deliveries | | `gateway/src/db/guardian-request-store.ts` | Gateway-side store for guardian requests and deliveries; first-writer-wins resolution via atomic status CAS (`guardian_requests_decide`) | | `assistant/src/calls/guardian-action-sweep.ts` | Expiry notices for expired guardian requests, sent to all delivery destinations | | `assistant/src/calls/call-domain.ts:createInboundVoiceSession()` | Creates or reuses a voice session for an inbound call keyed by CallSid (idempotent replay protection) | | `assistant/src/runtime/channel-verification-service.ts` | Channel verification session lifecycle: create session with six-digit code, find pending sessions, validate and consume on match | | `assistant/src/calls/call-state-machine.ts` | Deterministic state transition validator with allowed-transition table and terminal-state enforcement | | `assistant/src/calls/call-recovery.ts` | Startup reconciliation of non-terminal calls: fetches provider status and transitions stale sessions | | `assistant/src/calls/twilio-provider.ts` | Twilio Voice REST API integration (initiateCall, endCall, getCallStatus) using direct fetch, no Twilio SDK dependency | | `assistant/src/calls/twilio-routes.ts` | HTTP webhook handlers: voice webhook (returns `` TwiML, enforces the credential preflight with `` + `` when not ready), status callback | | `assistant/src/calls/media-stream-server.ts` | WebSocket handler for Twilio Media Streams; manages one MediaStreamCallSession per call, runs `routeSetup`, and drives interactive setup outcomes through `CallSetupFlow` | | `assistant/src/calls/call-setup-flow.ts` | Transport-agnostic call setup flow: verification, invite-redemption, name-capture, and unverified-caller sub-flows over DTMF/spoken input | | `assistant/src/calls/guardian-wait-controller.ts` | Guardian access-request wait orchestration: hold messaging, heartbeats, status polling, consultation timeout, callback handoff | | `assistant/src/calls/media-stream-stt-session.ts` | Daemon-side STT for media-stream audio: streaming transcriber (utterance-boundary finals) with batch + VAD turn-detection fallback | | `assistant/src/calls/telephony-credential-preflight.ts` | Combined STT + TTS credential-readiness resolver gating inbound TwiML and outbound call placement | | `assistant/src/calls/call-controller.ts` | Session-backed voice controller: routes voice turns through the daemon session pipeline via voice-session-bridge, detects ASK_GUARDIAN and END_CALL control markers | | `assistant/src/calls/voice-session-bridge.ts` | Bridge between the voice call controller and the daemon session/run pipeline: wraps RunOrchestrator.startRun() with voice-specific defaults, translating agent-loop events into callbacks for real-time TTS streaming | | `assistant/src/calls/call-state.ts` | Notifier pattern (Maps with register/unregister/fire helpers) for cross-component communication: question notifiers, completion notifiers, and controller registry | | `assistant/src/calls/call-constants.ts` | Config-backed constants: max call duration, user consultation timeout, silence timeout, denied emergency numbers | | `assistant/src/calls/voice-provider.ts` | Abstract VoiceProvider interface for provider-agnostic call initiation | | `assistant/src/calls/twilio-config.ts` | Twilio credential and configuration resolution from secure key store and environment | | `assistant/src/calls/types.ts` | TypeScript type definitions: CallSession, CallEvent, CallPendingQuestion, CallStatus, CallEventType | | `gateway/src/http/routes/twilio-voice-webhook.ts` | Gateway route: validates Twilio signature, forwards voice webhook to runtime | | `gateway/src/http/routes/twilio-status-webhook.ts` | Gateway route: validates Twilio signature, forwards status callback to runtime | | `gateway/src/http/routes/twilio-media-websocket.ts` | Gateway route: WebSocket proxy for Media Streams frames between Twilio and runtime (all calls) | | `gateway/src/twilio/validate-webhook.ts` | Twilio webhook validation: HMAC-SHA1 signature verification, payload size limits, fail-closed when auth token missing | ### Call State Machine All call status transitions are validated by a deterministic state machine (`call-state-machine.ts`). Terminal states are immutable — once a call reaches `completed`, `failed`, or `cancelled`, no further transitions are permitted. ``` initiated ──> ringing ──> in_progress ──> waiting_on_user ──> in_progress (loop) │ │ │ │ │ │ │ ├──> completed │ │ │ ├──> failed │ │ │ └──> cancelled │ │ ├──> completed │ │ ├──> failed │ │ └──> cancelled │ ├──> completed │ ├──> failed │ └──> cancelled ├──> completed ├──> failed └──> cancelled ``` The `validateTransition(current, next)` function is called by `updateCallSession()` in the call store. Same-state transitions (no-ops) are always valid. Invalid transitions are rejected with an explanatory reason string. ### Cross-Channel Guardian Consultation When the LLM emits `[ASK_GUARDIAN: question]` during a voice call, the controller creates a pending question and calls `dispatchGuardianQuestion()` on the guardian dispatch engine. The dispatch engine handles the full cross-channel fan-out: 1. **Request creation**: A `guardian_requests` row (kind `pending_question`) is created gateway-side via `guardian_requests_create` with a unique 6-character hex request code, the question text, a `pending` status, and an expiry timestamp. 2. **Delivery fan-out via notification pipeline**: The guardian dispatch calls `emitNotificationSignal()` and uses the same notification decision + broadcaster path as every other producer. - **Vellum**: Conversation pairing happens in the notification broadcaster. The resulting `notification_conversation_created` event surfaces the conversation in the desktop UI. - **Telegram**: Delivery is handled by channel adapters selected by the notification decision and guarded by configured bindings. - Guardian dispatch records `guardian_request_deliveries` (through the `guardian-delivery-recorder.ts` single write sink) from pipeline delivery results. It also uses the per-dispatch `onConversationCreated` callback so vellum delivery rows are created as soon as conversation pairing occurs (without waiting for slower channels). 3. **Answer resolution**: The first channel to respond wins. Decisions commit via the gateway's `guardian_requests_decide` route, which performs an atomic `WHERE status = 'pending'` compare-and-set on the `guardian_requests` table -- only the first writer succeeds in transitioning the request to a decided status. The winning answer text and deciding identity are recorded on the request row. 4. **Stale responses**: Channels that lose the race (respond after another channel has already answered) receive a "already answered" notice informing them that the question was resolved by another channel. 5. **Request-code disambiguation**: When a guardian has multiple pending requests across concurrent calls, they prefix their answer with the 6-character hex request code to indicate which question they are answering. This allows unambiguous routing even when questions arrive on the same channel in quick succession. 6. **Expiry sweep**: The daemon's `guardian-expiry-sweep.ts` runs a periodic 60-second sweep that lists past-deadline pending requests (`guardian_requests_list_expired_pending`), runs each row's side effects, and confirms each with the per-request `guardian_requests_expire` CAS. Expiry notices are sent to all delivery destinations associated with each expired request (`guardian-action-sweep.ts`). 7. **One store, many kinds**: Voice questions share the gateway's `guardian_requests` store with every other guardian request, distinguished by `kind` — `tool_approval` and `tool_grant_request` rows gate specific tool invocations, `pending_question` rows carry free-form questions from voice calls that require human input to continue the conversation, and `access_request` rows gate unknown-sender access. #### macOS Notification + Deep-Link Flow When a guardian question is dispatched while the macOS app is backgrounded, the Swift client posts a native `UNUserNotificationCenter` notification from the generic `notification_intent` payload (`NOTIFICATION_INTENT` category). The deep-link metadata includes the paired conversation ID, so tapping the notification routes directly to the guardian conversation. ### SQLite Tables The three call tables live in `$VELLUM_WORKSPACE_DIR/data/db/assistant.db` alongside existing tables; guardian requests and their deliveries live in the gateway DB (the assistant-side copies were dropped by gateway migration m0016): - **`call_sessions`** — One row per call (inbound or outbound). Tracks conversation association, provider info (Twilio CallSid), phone numbers, task description (null for inbound calls), status lifecycle (`initiated` -> `ringing` -> `in_progress` -> `waiting_on_user` -> `completed`/`failed`), and timestamps. For inbound calls, the session is keyed by CallSid via `createInboundVoiceSession()` with idempotent replay protection. Foreign key to `conversations(id)` with cascade delete. - **`call_events`** — Append-only event log for each call session. Event types include `call_started`, `call_connected`, `caller_spoke`, `assistant_spoke`, `user_question_asked`, `user_answered`, `call_ended`, `call_failed`. For voice prompts, `caller_spoke` payloads include speaker context (`speakerId`, `speakerLabel`, `speakerConfidence`, `speakerSource`) when available. Foreign key to `call_sessions(id)` with cascade delete. Includes a unique index on `(call_session_id, dedupe_key)` for callback idempotency. - **`call_pending_questions`** — Tracks questions the AI asks the user during a call (via the `[ASK_GUARDIAN: ...]` pattern). Status lifecycle: `pending` -> `answered`/`expired`/`cancelled`. Foreign key to `call_sessions(id)` with cascade delete. - **`guardian_requests`** (gateway DB) — Guardian requests of every kind, including one row per ASK_GUARDIAN question from a voice call (kind `pending_question`). Tracks question/activity text, request code (6-char hex), status lifecycle (`pending` -> `approved`/`denied`/`expired`/`cancelled`), answer text, deciding identity, and expiry timestamp. - **`guardian_request_deliveries`** (gateway DB) — Per-surface delivery tracking for guardian requests. One row per (request, surface) pair. Tracks delivery status, destination channel, and destination conversation/chat/message IDs — the addressing that lets a delivered card be withdrawn or matched to an inbound reaction/reply. ### Gateway Twilio Webhook Ingress Internet-facing Twilio callbacks terminate at the gateway, which validates signatures before forwarding to the runtime. This keeps the runtime behind the gateway's bearer-auth boundary. | Gateway Route | Validates | Forwards To (Runtime) | | ---------------------------------------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------- | | `POST /webhooks/twilio/voice` | HMAC-SHA1 signature, payload size | `POST /v1/internal/twilio/voice-webhook` (JSON: `{ params, originalUrl, assistantId? }`) | | `POST /webhooks/twilio/status` | HMAC-SHA1 signature, payload size | `POST /v1/internal/twilio/status` (JSON: `{ params }`) | | `WS /webhooks/twilio/media-stream//` | WebSocket upgrade | `WS /v1/calls/media-stream` (bidirectional proxy) — Media Streams path | In gateway-fronted deployments, the TwiML WebSocket URL (returned by the voice webhook) points to the gateway's `/webhooks/twilio/media-stream//` endpoint rather than directly to the runtime. The gateway proxies frames bidirectionally between Twilio and the runtime, preserving close and error semantics for proper cleanup. **Media Streams handshake metadata:** Twilio Media Streams does not reliably preserve URL query parameters across the WebSocket upgrade, so handshake metadata (`callSessionId` and auth `token`) is encoded as **URL path segments** (primary transport). The gateway also supports legacy query-parameter-based handshake as a fallback for backward compatibility. The metadata extractor in `twilio-media-websocket.ts` resolves values from path segments first, falling back to query parameters. Signature validation is **fail-closed**: if the Twilio auth token is not configured, all webhook requests are rejected with `403`. Missing or invalid `X-Twilio-Signature` headers are also rejected with `403`. Payload size is capped by `maxWebhookPayloadBytes` (checked via both `Content-Length` header and actual body size). **Webhook base URL resolution:** Public ingress URL construction is centralized in `public-ingress-urls.ts`: - Twilio voice/status/media-stream URLs use `ingress.publicBaseUrl`. - Velay registration publishes its public assistant URL to `ingress.publicBaseUrl` with `ingress.publicBaseUrlManagedBy: "velay"`. - Telegram webhooks, OAuth callbacks, email callbacks, and normal JSON webhook URLs also use `ingress.publicBaseUrl`; Velay-managed URL changes are tagged so unrelated reconciliation can be skipped when appropriate. With `velay-webhooks` on, the Telegram reconcile stops skipping them, because a pod's Velay URL is then the address it registers. - Module-level assistant state remains a fallback for legacy tunnel start/stop flows. All webhook paths (`/webhooks/twilio/voice`, `/webhooks/twilio/status`, `/webhooks/telegram`, `/webhooks/oauth/callback`, etc.) are appended automatically. For **inbound Twilio signature validation** at the gateway, URL reconstruction now supports multiple candidates in order: 1. `ConfigFileCache.getString("ingress", "publicBaseUrl")` (if configured) 2. Forwarded public URL headers from the tunnel/proxy (`X-Forwarded-Proto` + `X-Forwarded-Host`/`X-Original-Host` fallbacks) 3. Raw request URL (always included as the final fallback) This makes ingress URL updates smoother in local tunnel workflows because Twilio webhooks can continue validating immediately. For Telegram, the config file watcher detects ingress URL changes and triggers webhook reconciliation directly, so neither channel requires a gateway restart. ### Runtime HTTP Endpoints | Method | Path | Description | | ------ | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | POST | `/v1/calls/start` | Initiate a new outgoing call (gated by `calls.enabled` config) | | POST | `/v1/calls/twilio/voice-webhook` | Twilio voice webhook (direct access; **blocked with 410 in gateway-only mode**) | | POST | `/v1/internal/twilio/voice-webhook` | Internal voice webhook used by gateway; accepts JSON `{ params, originalUrl, assistantId? }`, creates inbound session or returns TwiML | | GET | `/v1/calls/:callSessionId` | Get call status, including any pending question | | POST | `/v1/calls/:callSessionId/cancel` | Cancel an active call | | POST | `/v1/calls/:callSessionId/answer` | Answer a pending question via HTTP (alternative to in-conversation bridge) | | POST | `/v1/calls/:callSessionId/instruction` | Relay a steering instruction to an active call's controller (alternative to in-conversation bridge) | | POST | `/v1/internal/twilio/status` | Internal status callback used by gateway; accepts JSON `{ params }` | | WS | `/v1/calls/media-stream` | Media Streams WebSocket (raw audio from Twilio, daemon-side STT/TTS) — all calls | ### Tools | Tool | Description | | ------------- | ---------------------------------------------------------------------------------------- | | `call_start` | Initiates an outgoing phone call to a specified number with an optional task description | | `call_status` | Retrieves the current status of a call session | | `call_end` | Terminates an active call | Both tools and HTTP routes delegate to the same domain functions in `call-domain.ts` (`startCall`, `getCallStatus`, `cancelCall`, `answerCall`, `relayInstruction`), ensuring consistent validation and behavior. Inbound calls do not use tools — they are initiated by the external caller and bootstrapped automatically by the voice webhook and media-stream server. ### Control Markers The CallController detects two special markers in the LLM's response text: - **`[ASK_GUARDIAN: question]`** — The AI needs to consult the guardian. The controller creates a pending question, notifies the session via `fireCallQuestionNotifier`, puts the caller on hold, and waits for a guardian answer (timeout configured via `calls.userConsultTimeoutSeconds`). - **`[END_CALL]`** — The AI has determined the call's purpose is fulfilled. The controller sends a goodbye, closes the call session, and marks the call as completed. Both markers are stripped from the TTS output so the callee never hears the raw control text. ### Call Recovery on Startup When the daemon restarts, any calls left in non-terminal states (initiated, ringing, in_progress, waiting_on_user) may be stale. The `reconcileCallsOnStartup()` function in `call-recovery.ts` runs during daemon lifecycle initialization and handles each recoverable session: 1. **No provider SID** — The call never connected. It is transitioned to `failed` with an explanatory `lastError`. 2. **Has provider SID** — The actual status is fetched from Twilio via `provider.getCallStatus()`. If the provider reports a terminal state (completed, failed, busy, no-answer, canceled), the session is transitioned accordingly. If the call is still active on the provider side, it is left for subsequent webhooks to handle. 3. **Provider fetch failure** — If the provider API call fails, the session is transitioned to `failed` with the error message recorded in `lastError`. 4. **Pending questions** — Any pending questions for sessions that transition to a terminal state are expired. Malformed or unprocessable provider callback payloads are logged as dead-letter events via `logDeadLetterEvent()` for investigation. ### Calls Configuration Call behavior is controlled via the `calls` config block in the assistant configuration (`config/schema.ts`). All values have sensible defaults and are validated via Zod: | Field | Type | Default | Description | | --------------------------------- | ------- | ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `calls.enabled` | boolean | `true` | Master toggle for the calls feature. When `false`, call routes return 403 and tools return errors. | | `calls.provider` | enum | `'twilio'` | Voice provider to use (currently only Twilio is supported). | | `calls.maxDurationSeconds` | int | `3600` | Maximum allowed duration per call. | | `calls.userConsultTimeoutSeconds` | int | `120` | How long to wait for a user answer before timing out a pending question. | | `calls.disclosure.enabled` | boolean | `true` | Whether the AI should disclose it is an AI at the start of the call. | | `calls.disclosure.text` | string | _(default disclosure prompt)_ | The disclosure instruction included in the system prompt. | | `llm.callSites.callAgent.model` | string | _(unset; falls back to the resolved call-site default)_ | Optional override for the LLM model used in voice call conversations. | | `services.stt.provider` | enum | `'deepgram'` | Global STT provider. Every boundary falls back to it, and a consumer with a `services.stt.roles.` override (`liveVoice`, `telephony`, `dictation`, `watch`, `batch`) uses that instead. The daemon transcribes media-stream call audio with whichever provider the `telephony` role resolves to (streaming when supported, batch otherwise). | | `services.stt.language` | string | `multi` | Spoken language for transcription: a BCP-47 code pins one language, `multi` (the schema default) enables code-switching on providers that support it. Per-language TTS voices are configured via `services.tts.providers..languageVoices`. | | `services.tts.provider` | enum | `'elevenlabs'` | Active TTS provider for speech synthesis (catalog-driven; see [TTS Provider Abstraction](../assistant/ARCHITECTURE.md#tts-provider-abstraction-servicestts)). | | `services.tts.providers..*` | object | _(per-provider defaults)_ | Provider-specific settings block. One block per catalog entry (e.g. `elevenlabs`, `fish-audio`). | ### Caller Identity Resolution When a call is initiated, the system resolves the caller identity (which phone number to show as the caller ID). **Implicit calls (no explicit `callerIdentityMode`) always use `assistant_number`** — the assistant's Twilio number. `user_number` mode is only used when explicitly requested per call. The identity mode is validated against Twilio's eligible caller IDs before the call is placed. The resolved identity mode and source are persisted in the `call_sessions` table for auditability. The resolution is performed by `resolveCallerIdentity()` in `call-domain.ts`: 1. **Per-call override** — If `callerIdentityMode` is provided in the call input and `calls.callerIdentity.allowPerCallOverride` is enabled, the requested mode is used (source: `per_call_override`). 2. **Implicit default** — Otherwise, `assistant_number` is always used (source: `implicit_default`). There is no configurable default mode — this is a strict policy. 3. **User number lookup** — For `user_number` mode (explicit only), the number is resolved from (in priority order): `calls.callerIdentity.userNumber` config (source: `user_config`) or the `credential/twilio/user_phone_number` secure key (source: `secure_key`). 4. **Eligibility check** — User numbers are verified against the Twilio API to confirm they can be used as an outbound caller ID. Both the resolved mode and source are logged at info level on success, and rejections are logged at warn level. ### Voice and TTS Configuration Voice and TTS settings are configurable via the `calls.voice` and `services.tts` config blocks — they are not hardcoded. The active TTS provider is determined by `services.tts.provider` (default: `"elevenlabs"`). Provider-specific settings (voice ID, model, tuning parameters) are read from `services.tts.providers.`. The call mode (`native-twilio` or `synthesized-play`) is resolved from the canonical provider catalog via `resolveCallStrategy()` in `tts-call-strategy.ts` — it reads the provider's declared `callMode` rather than inferring behavior from runtime capabilities. For `native-twilio` providers (e.g. ElevenLabs), spoken text sent via `sendTextToken()` is re-synthesized through daemon TTS on the media-stream transport — collapsing the callMode split is a documented deferred follow-up. For `synthesized-play` providers (e.g. Fish Audio), the assistant synthesises audio via the provider's HTTP API and delivers it through the audio store / `sendPlayUrl()` path. On the media-stream transport, call playback requires audio the daemon can transcode to mu-law: each TTS catalog entry declares `mediaStreamPlayback.outputFormat`, `resolveTelephonyTtsCapability()` combines that with credential availability, and the call TTS resolver falls back to a credentialed playable provider rather than producing silence. For full details on the catalog-driven TTS architecture, provider catalog, call strategy abstraction, and the provider-add checklist, see the [TTS Provider Abstraction](../assistant/ARCHITECTURE.md#tts-provider-abstraction-servicestts) section in the assistant architecture docs. ### Telephony STT: Daemon-Side Media-Stream Transcription Telephony STT is unified under `services.stt`: calls resolve the `telephony` role when it is set and `services.stt.provider` otherwise. The voice webhook in `twilio-routes.ts` emits `` TwiML pointing to the gateway's media-stream proxy (`/webhooks/twilio/media-stream//`); the gateway forwards raw audio frames to the daemon's media-stream server, which transcribes them itself: - **Streaming** (default) — when `calls.voice.telephonyStreaming` is enabled and the configured provider resolves a streaming transcriber, audio is decoded (mu-law → PCM16, 8 kHz → 16 kHz) and fed to the provider's realtime adapter; replies trigger only on utterance-boundary finals and barge-in fires from local energy VAD. - **Batch fallback** — otherwise turns are segmented with the energy-based VAD turn detector and transcribed via the provider's batch API. A credential preflight (`resolveTelephonyCredentialReadiness()`) requires a credentialed telephony-capable STT provider and a media-stream-playable TTS provider before any call connects: inbound not-ready calls get `` setup-required copy plus ``, and outbound placement fails before dialing. See `docs/internal-reference.md` for a provider-specific troubleshooting matrix. ---