# §8 — Multimodal Attachments Implementation Plan

## Summary

Add `attachments` support to `POST /agents/:name/chat` so the Studio can send images,
audio, video, and files alongside a user message. The attachment content is assembled into
an OpenRouter-compatible multi-part user message for the LLM call; only the plain text
message is stored in the DB (no base64 blobs in SQLite).

---

## OpenRouter Content Block Formats (verified)

| VeilCLI type | OpenRouter block type | URL | Base64 |
|---|---|---|---|
| `image` | `image_url` | ✅ | ✅ `data:<mediaType>;base64,<data>` |
| `audio` | `input_audio` | ❌ **base64 only** | ✅ raw base64 + `format` field |
| `video` | `video_url` | ✅ | ✅ `data:<mediaType>;base64,<data>` |
| `file` | `document` | ✅ (in `source.url`) | ✅ (in `source.data`) |

Audio is the only type that does NOT support URL input.

---

## Request Contract

`POST /agents/:name/chat` — new optional field `attachments`:

```json
{
  "message": "What's in this image?",
  "sessionId": "sess_abc",
  "sse": true,
  "attachments": [
    { "type": "image", "mediaType": "image/png", "data": "<base64>" },
    { "type": "image", "url": "https://example.com/photo.jpg" },
    { "type": "audio", "format": "wav", "data": "<base64>" },
    { "type": "video", "mediaType": "video/mp4", "data": "<base64>" },
    { "type": "video", "url": "https://example.com/clip.mp4" },
    { "type": "file", "filename": "report.pdf", "mediaType": "application/pdf", "data": "<base64>" },
    { "type": "file", "filename": "report.pdf", "url": "https://example.com/report.pdf", "mediaType": "application/pdf" }
  ]
}
```

### Attachment field reference

| Field | Required | Description |
|-------|----------|-------------|
| `type` | ✅ | `"image"`, `"audio"`, `"video"`, or `"file"` |
| `data` | one of `data`/`url` | Base64-encoded content. Audio requires this; others accept either. |
| `url` | one of `data`/`url` | Public URL. Not supported for audio. |
| `mediaType` | for `image`/`video`/`file` with `data` | MIME type: `image/png`, `video/mp4`, `application/pdf` etc. |
| `format` | for `audio` | Format string: `"wav"`, `"mp3"`, `"aac"`, `"flac"`, `"ogg"`, `"m4a"` |
| `filename` | for `file` | Original filename (informational, passed to document block) |

---

## Files to Change

1. **`api/routes/chat.js`** — extract `attachments` from body, validate, pass to `runChat`
2. **`core/router.js`** — add `buildAttachmentBlock` + `buildUserContent` helpers, use in `runChat`

No DB changes. No migration. No loop changes.

---

## Implementation Detail

### `chat.js` changes

Extract and forward:
```js
const { message, sessionId, sse = false, continue: continueFlag = false, attachments } = req.body;
```

Validate:
```js
if (attachments !== undefined && !Array.isArray(attachments)) {
  return sendError(res, 400, 'VALIDATION_ERROR', '"attachments" must be an array');
}
```

Pass to `runChat`:
```js
runChat({ agentName: name, message, sessionId, continueFlag, attachments, cwd, settings, ... })
```

### `router.js` changes

Add two helpers before `runChat`:

```js
const VALID_AUDIO_FORMATS = new Set(['wav', 'mp3', 'aiff', 'aac', 'ogg', 'flac', 'm4a', 'pcm16']);

function buildAttachmentBlock(a) {
  switch (a.type) {
    case 'image': {
      const url = a.url || `data:${a.mediaType};base64,${a.data}`;
      return { type: 'image_url', image_url: { url } };
    }
    case 'audio': {
      // Audio requires base64 — no URL support on OpenRouter
      if (!a.data) return null;
      const format = VALID_AUDIO_FORMATS.has(a.format) ? a.format : 'wav';
      return { type: 'input_audio', input_audio: { data: a.data, format } };
    }
    case 'video': {
      const url = a.url || `data:${a.mediaType};base64,${a.data}`;
      return { type: 'video_url', video_url: { url } };
    }
    case 'file': {
      if (a.url) {
        return { type: 'document', source: { type: 'url', url: a.url } };
      }
      return { type: 'document', source: { type: 'base64', media_type: a.mediaType || 'application/pdf', data: a.data } };
    }
    default:
      return null;
  }
}

function buildUserContent(message, attachments) {
  if (!attachments || attachments.length === 0) return message;
  const parts = [{ type: 'text', text: message }];
  for (const a of attachments) {
    const block = buildAttachmentBlock(a);
    if (block) parts.push(block);
  }
  return parts;
}
```

Update `runChat` signature:
```js
async function runChat({ agentName, message, sessionId, continueFlag = false, attachments, cwd, settings, ... })
```

Update the user message section in `runChat`:
```js
if (!continueFlag || message) {
  const userContent = buildUserContent(message, attachments);
  messages.push({ role: 'user', content: userContent });   // multi-part to LLM
  db.addMessage({ sessionId: sid, role: 'user', content: message }); // plain text to DB
  eventBus.emit('event', { type: 'chat.user_message', ... });
}
```

---

## Key Design Decisions

### Why plain text in DB, multi-part to LLM?
Base64-encoded images/video/audio can be megabytes each. Storing them in SQLite
would bloat the messages table and make `GET /sessions/:id/messages` responses
enormous. Multi-part content is assembled in-memory per LLM call only.

### Why null-check audio `data`?
Audio is the only type that does NOT support URL input (OpenRouter docs). If a caller
sends `{ type: "audio", url: "..." }` without `data`, we silently skip the block
rather than sending a malformed request to the LLM API. The text message still goes through.

### Why no model validation?
The server does not check whether the current agent's model supports multimodal input.
Pass through and let the LLM API return an error if not — the caller is responsible for
knowing their model's capabilities.

### Why no DB schema changes?
Attachments are ephemeral per-request. Storing base64 blobs in SQLite is not feasible
at scale. If persistent attachment history is ever needed, that requires a dedicated
blob store (object storage) — not in scope here.

---

## Edge Cases

| Case | Handling |
|------|----------|
| `attachments: []` | `buildUserContent` returns plain string — no array wrapping |
| Audio with `url` only, no `data` | `buildAttachmentBlock` returns `null`, block is skipped |
| Unknown `type` in attachment | `buildAttachmentBlock` returns `null`, block is skipped |
| `message` is empty string + `continue:true` | `buildUserContent` called with `""` — returns `""`, no array. That's fine since no message is added anyway. |
| Image with both `url` and `data` | `url` takes precedence (checked first in `a.url ||`) |
| File with no `mediaType` and no `url` | Defaults to `"application/pdf"` for `media_type` |