# Chat

The chat endpoint enables multi-turn conversations with an agent. Each call is synchronous — it blocks until the agent finishes its response (including any tool calls).

---

## POST /agents/:name/chat

Send a message to an agent and receive a reply.

**Path parameters**

| Param | Description |
|-------|-------------|
| `name` | Agent name |

**Request body**

```json
{
  "message": "What files are in the current directory?",
  "sessionId": "sess_abc123"
}
```

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `message` | string | ✓* | User message content. *Required unless `continue: true`. |
| `sessionId` | string | | Existing session ID to continue a conversation. Omit to start a new session. Required when `continue: true`. |
| `sse` | boolean | | `true` to enable SSE streaming — the response is `text/event-stream` instead of JSON. |
| `fireAndForget` | boolean | | `true` to dispatch the turn detached and respond `202` as soon as the session id is known — the turn keeps running server-side with no client attached. See [Fire-and-forget mode](#fire-and-forget-mode) below. Mutually exclusive with `sse`. |
| `continue` | boolean | | `true` to resume an existing session without adding a new user message. Requires `sessionId`. Useful for re-triggering an interrupted run. |
| `attachments` | array | | Multimodal content to include with the message. See [Attachments](#attachments) below. |
| `overrides` | object | | Per-call LLM-parameter overrides. Highest precedence in the resolver chain (`perCall > session > agent`). See [Per-call overrides](#per-call-overrides) below. |

**Response**

```json
{
  "sessionId": "sess_4f3a1b9c2d8e7f01",
  "message": {
    "role": "assistant",
    "content": "Here are the files in the current directory:\n- README.md\n- package.json\n..."
  },
  "tokenUsage": {
    "input": 1240,
    "output": 183,
    "cache": 0,
    "cost": 0.0021
  },
  "toolCalls": [
    { "name": "list_dir", "durationMs": 12, "success": true },
    { "name": "read_file", "durationMs": 8, "success": true }
  ]
}
```

| Field | Type | Description |
|-------|------|-------------|
| `sessionId` | string | Session ID (pass this back for multi-turn) |
| `message.role` | string | Always `"assistant"` |
| `message.content` | string | Agent reply text (last iteration's response) |
| `tokenUsage.input` | integer | Total input tokens for this turn |
| `tokenUsage.output` | integer | Total output tokens for this turn |
| `tokenUsage.cache` | integer | Cached tokens (if provider supports it) |
| `tokenUsage.cost` | number | Estimated cost in USD |
| `toolCalls` | array | Summary of tools called: `{ name, durationMs, success }` |

**Error responses**

| Code | Condition |
|------|-----------|
| `400 VALIDATION_ERROR` | `message` missing/not a string, `continue: true` without `sessionId`, `attachments` is not an array, an attachment exceeds the 8 MB inline-base64 cap, or an `overrides` field fails validation. `details.field` identifies which field failed. |
| `400 MODE_NOT_SUPPORTED` | Agent does not have `chat.enabled: true` |
| `400 SESSION_CLOSED` | The provided `sessionId` belongs to a closed session |
| `400 MODEL_MODALITY_UNSUPPORTED` | The chat call routes through the claude-cli engine and includes `attachments`. The Claude SDK's MCP surface does not consume inline multimodal content blocks, so the attachment is rejected with `details.attachmentType` (the first attachment's `type`). Switch the agent to a model on the openai engine to use attachments. |
| `404 AGENT_NOT_FOUND` | No agent with that name |
| `500 LLM_ERROR` | The provider failed repeatedly (4 consecutive attempts). Previously this surfaced as a *successful* response with `message.content: null`; it is now a real error. |
| `500 MAX_ITERATIONS` / `500 MAX_DURATION` | The chat turn exhausted its iteration or duration limit before producing a final reply. Also previously a silent empty success. |

> **Budget breaches are NOT 400s.** When the loop breaches `max_tokens` or `max_wall_seconds` mid-run, the runtime cancels the loop cooperatively and emits the `session.budget_exceeded` WS event. The chat endpoint then returns the same shape it returns for any other cancel: SSE clients see `error { code: "CANCELLED" }` with `details.reason: "budget_exceeded"`; HTTP-JSON clients see `done.cancelled: true` with `done.cancelReason: "budget_exceeded"`. The only place where `400 BUDGET_EXCEEDED` is the literal HTTP response code is the spawn-depth pre-check in `agent_spawn` (see [Multi-Agent → Depth and Budget](../guide/09-multi-agent.md#depth-and-budget)).

---

## Per-call overrides

Pass `overrides` to swap LLM parameters for this single chat call without modifying the session row. The resolver merges them as the highest-precedence layer:

```
perCall (this call's overrides)  >  session row  >  agent.json defaults
```

```json
{
  "message": "Re-summarise the file in three bullets.",
  "sessionId": "sess_4f3a1b9c2d8e7f01",
  "overrides": {
    "model": "anthropic/claude-opus-4-5",
    "temperature": 0.2,
    "max_tokens": 1024,
    "reasoning": { "effort": "high", "max_tokens": 4000 }
  }
}
```

| Field | Type | Description |
|-------|------|-------------|
| `model` | string | LLM model identifier for this call only. |
| `temperature` | number | Sampling temperature (0–2). |
| `max_tokens` | integer | Output-token cap (positive integer). |
| `reasoning` | object | Engine-blind reasoning config: `{ effort, max_tokens? }`. See [05-sessions.md](05-sessions.md) "Reasoning unification" for the routing table (effort enum + cross-engine caps). The legacy bare-string form (`"reasoning": "high"`) and legacy `effort`/`thinking` keys are no longer accepted. |

**Validation.** Unknown fields (including legacy `effort`/`thinking`), agent-identity fields (`tools`, `permissions`, `mcpServers`, `modes`, `skills`, `disallowedTools`), and `top_p` are rejected with `400 VALIDATION_ERROR`. `details.field` identifies which field failed. Same whitelist as `PATCH /sessions/:id`.

**Persistence.** Per-call overrides are **not stored** — the session row stays untouched. To make a setting sticky, use `PATCH /sessions/:id` instead.

**Inspection.** `GET /sessions/:id/config` returns the resolved values without per-call overrides applied (only effective + per-session overrides + agent defaults).

---

## Attachments

Include multimodal content alongside a `message` by passing an `attachments` array. Attachments are assembled into the LLM request only — they are **not stored** in the session database (only the plain text `message` is persisted).

```json
{
  "message": "What's in this image?",
  "attachments": [
    { "type": "image", "url": "https://example.com/photo.jpg" },
    { "type": "image", "mediaType": "image/png", "data": "<base64>" },
    { "type": "audio", "format": "wav", "data": "<base64>" },
    { "type": "video", "url": "https://example.com/clip.mp4" },
    { "type": "video", "mediaType": "video/mp4", "data": "<base64>" },
    { "type": "file", "filename": "report.pdf", "mediaType": "application/pdf", "data": "<base64>" },
    { "type": "file", "filename": "report.pdf", "url": "https://example.com/report.pdf", "mediaType": "application/pdf" }
  ]
}
```

**Attachment fields**

| Field | Required | Description |
|-------|----------|-------------|
| `type` | ✓ | `"image"`, `"audio"`, `"video"`, or `"file"` |
| `data` | one of `data`/`url` | Base64-encoded content. Required for `audio`; optional for others. |
| `url` | one of `data`/`url` | Public URL. **Not supported for `audio`** — only base64 is accepted for audio. |
| `mediaType` | for `image`/`video`/`file` with `data` | MIME type, e.g. `image/png`, `video/mp4`, `application/pdf` |
| `format` | for `audio` | Audio format string: `"wav"`, `"mp3"`, `"aac"`, `"flac"`, `"ogg"`, `"m4a"` |
| `filename` | for `file` | Original filename (informational, passed to the document block) |

**Type support matrix**

| Type | URL input | Base64 input |
|------|-----------|--------------|
| `image` | ✓ | ✓ |
| `audio` | ✗ (base64 only) | ✓ |
| `video` | ✓ | ✓ |
| `file` | ✓ | ✓ |

When both `url` and `data` are present, `url` takes precedence. Audio attachments with no `data` field are silently skipped (no error). Unknown `type` values are also silently skipped.

The server does not validate whether the agent's model supports multimodal input — pass-through to the LLM API, which will return an error if not supported.

---

## @-mention expansion

When the `message` contains `@`-mentions that resolve to real paths in the workspace, the server auto-attaches the referenced content to the turn so the model sees it without an extra tool round-trip. This is applied **at the HTTP chat route only** — it covers both engines, but never rewrites agent-to-agent or wake-seeded messages.

Two syntaxes are recognised:

- `@relative/path` or `@/absolute/path` — bare token (trailing sentence punctuation like `,`, `.`, `)` is stripped).
- `@"path with spaces"` — quoted token for paths containing spaces.

Tokens that don't resolve to an existing path (emails, decorators, `@mentions`) are ignored. For each resolved mention the server appends a hidden block to the message:

````
<veil-attachments>
System: contents of the @-mentioned paths above, auto-attached for reference. This block is hidden from the user's chat display.

## @src/config.js (file, 1.2 KB)
```
…file contents…
```

## @docs (directory, 7 entries)
- api/
- guide/
- README.md (3.1 KB)
</veil-attachments>
````

- **Files** are inlined in a fenced block. Binary files and files larger than 2 MB are described by name/size only (not inlined).
- **Directories** are listed (up to **20** entries; a `… and N more entries` footer is added beyond that).
- UIs strip the `<veil-attachments>` block on display; it is part of the persisted user message.

**Limits:** up to **8** mentions per message, **50 000** chars per file, **150 000** chars total across all mentions.

**Security:** `.veil/auth.json` is always denied — mentions are resolved through `realpath` first, so a symlink pointing at `auth.json` is rejected too. A message that already contains a `<veil-attachments>` block (e.g. a resend/edit) is passed through unchanged.

---

## Fire-and-forget mode

Set `"fireAndForget": true` to dispatch a turn without waiting for it. The server responds `202 Accepted` as soon as the session id is known; the agent turn then runs to completion server-side with no client attached. This backs the `veil send` CLI command.

**Request** — `POST /agents/assistant/chat`
```json
{
  "message": "Kick off the nightly report",
  "fireAndForget": true
}
```

Pass an existing `sessionId` to continue a conversation, or omit it to start a new session.

**Response** `202 Accepted`
```json
{ "accepted": true, "sessionId": "sess_4f3a1b9c2d8e7f01" }
```

For a **new** session the response waits only until the session row is created (bounded by a 10 s guard, after which `sessionId` may be `null`); for an **existing** session the id is returned immediately. Poll `GET /sessions/:id/messages` (or subscribe to `GET /sessions/:id/stream`) to observe the detached turn. `fireAndForget` takes precedence over `sse`.

---

## SSE Streaming Mode

Set `"sse": true` in the request body to receive a real-time `text/event-stream` response. The connection stays open across all LLM iterations (including tool call rounds) and closes only when the full turn is complete.

> **WebSocket parity.** Even when `sse: false` (default JSON-mode response), live `session.stream` events are still fanned out on the WebSocket bus (`/ws`) for any subscriber. JSON-mode callers who don't subscribe to the WS see no chunk noise; WS clients still get the live UX. This also applies to tool-driven sub-agent turns (`agent_spawn` / `agent_message` / `async_inform`) — see [WebSocket → Trigger-path parity](09-websocket.md).

**Design principle:** Events are split into two categories — _inference_ events carry transient streaming data not needed for state; _message_ events carry the exact schema returned by `GET /sessions/:id/messages` so clients can update their local state directly without any conversion.

**Event types**

| Event | Category | Description |
|-------|----------|-------------|
| `inference.chunk` | Inference | One streaming token fragment from the current LLM call |
| `inference.tool` | Inference | Tool name detected in stream before arguments are complete — lets UI show a loading indicator early |
| `thinking.chunk` | Inference | Streaming reasoning / extended-thinking text from the model (best-effort — only fires when the provider streams thinking content; see [Thinking streams](#thinking-streams) below) |
| `tool.chunk` | Inference | Live stdout/stderr fragments from a tool while it runs (e.g. `bash` streaming output before the call completes) |
| `message` | State | An assistant or tool message, identical to the session messages API schema |
| `done` | State | Turn complete — includes the updated session record and turn-level stats |
| `error` | — | Emitted instead of `done` if the run fails |

---

**`inference.chunk`**
```json
{ "content": "Here are the" }
```

**`inference.tool`** — fires once per tool call as soon as the tool name appears in the stream (before arguments are complete)
```json
{ "name": "sleep" }
```

---

### Thinking streams

**`thinking.chunk`** — streams the model's reasoning / extended-thinking text as it arrives. Same shape as `inference.chunk`, separate channel. Engine-blind: fires identically on both the openai and claude-cli engines.

```
event: thinking.chunk
data: {"content":"Let me work this out step by step. First, "}
```

**Engine coverage:**

| Engine | When it fires | Volume |
|--------|---------------|--------|
| openai | When the provider streams reasoning text. Field-name variants handled: `delta.reasoning_content` (DeepSeek native, some Chinese providers) and `delta.reasoning` (OpenRouter normalized field, used for DeepSeek-R1, GLM-4.5, o1-line via OR, Anthropic-via-OpenRouter with `reasoning.effort` set). One event per delta. | Often hundreds of events per turn. |
| claude-cli | When the SDK emits `thinking_delta` blocks, gated by `thinking: { type: "enabled", budget_tokens: N }`. One event per delta. | Variable — short thinking (~tens of tokens) consolidates to 1 event; longer reasoning streams across many deltas (verified 3+ chunks ~400ms apart on heavier prompts). |

**When it doesn't fire:**
- The provider doesn't stream reasoning (most non-thinking models).
- `reasoning.effort` is `'none'` (or unset) on Claude — yielding `thinking.type='disabled'` on the SDK.
- The reasoning happens but in a format the runtime doesn't recognize (we cover the two common variants above; other custom shapes are silently dropped — the full reasoning still lands on the persisted message as `thinking_content` at end-of-turn).

**Final-message accumulation.** All chunks are also accumulated and persisted to the session as `message.thinking_content` on the assistant turn. So thinking-aware clients can either:
- Render live (subscribe to `thinking.chunk` for streaming UX), or
- Render at end-of-turn only (read `thinking_content` from the `message` event or from `GET /sessions/:id/messages`).

**Enabling it on a chat call:**

```json
{
  "message": "What's 17 × 23? Reason briefly first.",
  "sse": true,
  "overrides": {
    "model": "deepseek/deepseek-r1",
    "reasoning": { "effort": "high" }
  }
}
```

The same `reasoning` shape works on Claude-engine agents — `effort: "high"` maps to SDK `effort: "high"`; pair with `max_tokens` to set the SDK's `thinking.budgetTokens` budget:

```json
{
  "message": "Plan the refactor.",
  "sse": true,
  "overrides": {
    "reasoning": { "effort": "max", "max_tokens": 5000 }
  }
}
```

---

**`message`** — session-API-compatible message object, augmented with inference metadata for assistant turns

*Assistant turn (text only):*
```json
{
  "id": 5,
  "session_id": "sess_4f3a1b9c2d8e7f01",
  "role": "assistant",
  "content": "Here are the files:\n- README.md",
  "tool_calls": null,
  "tool_call_id": null,
  "thinking_content": null,
  "thinking_tokens": null,
  "created_at": "2025-03-02T10:00:04.000Z",
  "finishReason": "stop",
  "iteration": 1,
  "tokenUsage": { "input": 1240, "output": 38, "cache": 0, "cost": 0.0003 }
}
```

*Assistant turn (with extended thinking):*
```json
{
  "id": 5,
  "session_id": "sess_4f3a1b9c2d8e7f01",
  "role": "assistant",
  "content": "The answer is 42.",
  "tool_calls": null,
  "tool_call_id": null,
  "thinking_content": "Let me reason through this step by step...",
  "thinking_tokens": 512,
  "created_at": "2025-03-02T10:00:04.000Z",
  "finishReason": "stop",
  "iteration": 1,
  "tokenUsage": { "input": 1240, "output": 38, "cache": 0, "cost": 0.0003 }
}
```

*Assistant turn (with tool calls):*
```json
{
  "id": 3,
  "session_id": "sess_4f3a1b9c2d8e7f01",
  "role": "assistant",
  "content": null,
  "tool_calls": [{ "id": "call_abc", "type": "function", "function": { "name": "list_dir", "arguments": "{\"dir\": \".\"}" } }],
  "tool_call_id": null,
  "thinking_content": null,
  "thinking_tokens": null,
  "created_at": "2025-03-02T10:00:02.000Z",
  "finishReason": "tool_calls",
  "iteration": 1,
  "tokenUsage": { "input": 980, "output": 22, "cache": 0, "cost": 0.0002 }
}
```

*Tool result:*
```json
{
  "id": 4,
  "session_id": "sess_4f3a1b9c2d8e7f01",
  "role": "tool",
  "content": "README.md (1.2KB)\npackage.json (544B)",
  "tool_calls": null,
  "tool_call_id": "call_abc",
  "created_at": "2025-03-02T10:00:03.000Z"
}
```

`finishReason` on assistant turns: `"stop"` = no tools followed | `"tool_calls"` = continued with tool use | `"length"` = token limit hit

---

**`done`**
```json
{
  "session": {
    "id": "sess_4f3a1b9c2d8e7f01",
    "agent_name": "assistant",
    "mode": "chat",
    "status": "active",
    "model": "claude-3-5-sonnet",
    "total_input_tokens": 6842,
    "total_output_tokens": 124,
    "cost": 0.0089,
    "message_count": 5,
    "created_at": "2025-03-02T10:00:00.000Z",
    "updated_at": "2025-03-02T10:00:05.000Z"
  },
  "agentName": "assistant",
  "model": "claude-3-5-sonnet",
  "iterations": 3,
  "durationMs": 4210,
  "tokenUsage": { "input": 6842, "output": 124, "cache": 0, "cost": 0.0089 },
  "toolCalls": [
    { "name": "sleep", "durationMs": 1003, "success": true }
  ],
  "cancelled": false,
  "cancelReason": null
}
```

`cancelled` is `true` when `POST /sessions/:id/cancel` (or the agent's own `agent_control` `stop` action, or an upstream cancel propagation) interrupted the loop. `cancelReason` carries the cancel source string (e.g. `"user_cancel"`); both fields are absent on uninterrupted turns.

**`error`**
```json
{ "error": "Cancelled by user", "code": "CANCELLED" }
```

```json
{ "error": "LLM API error 429: rate limit", "code": "INTERNAL_ERROR" }
```

`code: "CANCELLED"` is emitted instead of `done` when the loop was cancelled before producing a final assistant message. SSE consumers should treat `CANCELLED` as terminal — the stream closes immediately after.

---

**Behavior notes**
- `inference.chunk` fires for every LLM iteration across the full turn — all chunks are streamed in order.
- `inference.tool` fires during streaming as soon as the tool name is known, before arguments finish — use this for tool loading indicators.
- Each `message` event (role `assistant` or `tool`) matches what `GET /sessions/:id/messages` returns, so clients can append them directly to local state without schema conversion.
- `done.session` is a snapshot of the session record after the turn — clients can use it to update session state without a separate `GET /sessions/:id` call.
- The response `Content-Type` is `text/event-stream`; use `EventSource` or an SSE client library.

**JavaScript example**
```js
const response = await fetch('http://localhost:5050/agents/assistant/chat', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ message: 'List files', sse: true }),
});

const messages = [];
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
let pendingEvent = null;

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split('\n');
  buffer = lines.pop();
  for (const line of lines) {
    if (line.startsWith('event: ')) { pendingEvent = line.slice(7).trim(); continue; }
    if (line.startsWith('data: ') && pendingEvent) {
      const data = JSON.parse(line.slice(6));
      if (pendingEvent === 'inference.chunk') process.stdout.write(data.content);
      if (pendingEvent === 'inference.tool')  console.log(`[tool] ${data.name}…`);
      if (pendingEvent === 'message')         messages.push(data);  // directly usable
      if (pendingEvent === 'done')            console.log('session:', data.session);
      pendingEvent = null;
    }
  }
}
```

**curl example**
```bash
curl -X POST http://localhost:5050/agents/assistant/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "List files", "sse": true}' \
  --no-buffer
```

---

## Multi-turn Conversation Example

Start a new conversation, then continue it:

```bash
# Turn 1 — new session
curl -X POST http://localhost:5050/agents/assistant/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "My name is Alice."}'

# Response includes sessionId, e.g. "sess_4f3a1b9c2d8e7f01"

# Turn 2 — continue the same session
curl -X POST http://localhost:5050/agents/assistant/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "What is my name?", "sessionId": "sess_4f3a1b9c2d8e7f01"}'

# Agent remembers: "Your name is Alice."
```

---

## Notes

- The agent's full message history (including tool call messages) is persisted in SQLite and replayed on each turn.
- If the agent uses tools (e.g. `read_file`, `bash`), those calls happen **before** the reply is returned — the response always contains the final text response.
- To inspect which tools were called during a chat turn, fetch the session messages after the call: `GET /sessions/:id/messages`.
- Sessions are scoped to a workspace (`instanceFolder`). A `sessionId` from one server instance will not be found by another.
- `thinking_content` and `thinking_tokens` are populated only when the agent's model supports extended thinking and it is enabled (via `reasoning` on the session or agent config — non-`'none'`/`'auto'` effort, or any `max_tokens` budget). They are `null` otherwise.
- `attachments` are ephemeral — they are assembled into the LLM message in-memory for the current turn only. They do not appear in `GET /sessions/:id/messages` (only the plain text `message` is stored).
- Using `continue: true` skips injecting a new user message and resumes the session from its current state. Useful for re-running after an error or resuming after a manual tool result edit.