# AI Completion Test API

An **OpenAI-compatible fake provider** that replays pre-recorded LLM chats.
Point any OpenAI-compatible client at it and it serves the nearest recorded
exchange — full streaming, reasoning deltas, tool-call deltas, and usage
frames — indistinguishable from a real provider until you notice the replies
repeat.

Its purpose: drive the **whole VeilCLI harness** (agent loop, tools,
sessions, sub-agents, compaction, cancellation) through real production-shaped
traffic in tests, with **zero per-run cost and full determinism**. Record once
against a real model; replay forever.

```
┌────────────┐   /v1/chat/completions   ┌──────────────┐   replay   ┌─────────────┐
│  VeilCLI │ ───────────────────────► │  fake API    │ ─────────► │ recordings/ │
│  (any      │ ◄─ SSE: reasoning, tool ─ │ (this server)│   record   │  *.json     │
│  OAI client)│     calls, usage         │              │ ─proxy───► │ OpenRouter  │
└────────────┘                          └──────────────┘            └─────────────┘
```

## Quick start

```bash
# Replay the recorded scenario suite (fast, offline, free):
npm run test:advanced

# Run the fake API standalone (replay mode) on :5098:
npm run test:fake-api

# Re-record everything against real OpenRouter (uses .veil/auth.json key):
node test/ai-completion-test-api/run-scenarios.js --mode record

# Record / replay a subset:
node test/ai-completion-test-api/run-scenarios.js --mode replay --scenarios sub-agents,fork-sanity

# Run the suite in PARALLEL (each slot gets its own fake-api + veil + DB):
node test/ai-completion-test-api/run-scenarios.js --mode replay --jobs 4
```

## How it works

### Recording (`--mode record`)
The server reverse-proxies `/chat/completions` to a real upstream (OpenRouter),
streams the response back to the client **byte-for-byte untouched**, and in
parallel assembles the stream into a stored exchange (`content`, `reasoning`,
`tool_calls`, `usage`, `finish_reason`). Each exchange is one
`recordings/<scenario>/NNNN.json` file: the verbatim request + the assembled
response.

### Replay (`--mode replay`, default)
For each incoming request the server scores every recorded exchange and serves
the best one. Two complications are solved so recordings replay in a *different*
workspace than they were recorded in:

1. **Workspace-path templating** — recorded tool-call args (e.g.
   `write_file` paths) contain the record-time workspace absolute path. At
   record time it's tokenized to `<WORKSPACE>`; at replay time the live
   workspace path is substituted back in. (`lib/record.js`)

2. **Dynamic ID remapping** — a recorded `agent_spawn` minted a sessionId that
   a later `agent_message` referenced. On replay those ids are different. The
   matched recording's history is aligned against the incoming conversation to
   learn `recordedId → replayId`, and the served response is rewritten with
   that map. This is what makes multi-agent flows replayable. (`lib/remap.js`)

### Matching (`lib/match.js`)
Blends two signals:
- **Continuity (dominant)** — which recorded exchange's own history shares the
  most assistant-message fingerprints with the incoming conversation (i.e. is
  the genuine *next step* of a known flow). An exchange whose own response is
  already in the incoming history is penalized to avoid loops.
- **Similarity** — last-user-message, conversation-tail, system-prompt, and
  request-shape (role sequence) trigram overlap, plus offered-tools and model.

Volatile substrings (ids, timestamps, absolute paths, ports) are normalized
before comparison (`lib/normalize.js`), so "list the files" matches a recording
of "show me the files in this directory".

## Scenarios (`scenarios/index.js`)

The **same driver code** runs in record and replay modes (only the provider
endpoint differs), which guarantees replayed requests look like recorded ones.
Each scenario defines agents, a `run(ctx)` that drives the veil REST API, and
an `assert(ctx, artifacts)` that runs in **both** modes.

**Core flows (recorded):**

| Scenario | Proves |
|----------|--------|
| `simple-chat` | basic chat completion + streaming |
| `multi-turn` | session memory across turns |
| `sub-agents` | `agent_spawn` + `agent_message` orchestration (ID remapping) |
| `compaction` | `/sessions/:id/compact` + summary injection |
| `midturn-injection` | second message injected into a running session |
| `cancellation` | `/sessions/:id/cancel` mid-tool, session usable after |
| `async-inform-fanout` | `agent_spawn(async_inform)` ×2 → wake.js delivers both replies to an **idle** parent (the scariest path) |
| `concurrent-injection-and-fork` | two concurrent injections + fork + reset(409) on one busy session (entry-lock) |
| `dangling-toolcall-recovery` | failed tool → error fed back → model recovers in the same turn |

**Session surgery (recorded):**

| Scenario | Proves |
|----------|--------|
| `fork-sanity` | fork at message N: child remembers pre-fork facts ONLY, original untouched, both usable |
| `delete-message-and-continue` | delete one message → counters reconcile (DB oracle C1), session still usable |
| `trim-and-continue` | trim after anchor → trimmed content provably invisible to the model next turn |
| `stop-continue-flag` | cancel mid-tool, then `continue:true` (no new message) resumes from the reconciled history |
| `model-patch-mid-session` | `PATCH /sessions/:id {model}` → next turn runs on the new model |
| `agent-name-patch-busy` | `PATCH {agent_name}` → 409 while a turn runs, 200 after, next turn runs as the new agent |
| `agentmd-reload-next-turn` | AGENT.md rewritten on disk mid-session → system prompt rebuilt on the NEXT turn |

**Misbehaving-model chaos (fault-injected via `POST /__fault`, replay-only):**

| Scenario | Fault | Proves |
|----------|-------|--------|
| `bad-empty-response` | `empty_response` | content:null turn doesn't crash; session stays usable |
| `bad-truncated-stream` | `truncated_stream` | stream cut with no terminal frame → partial returned, no hang |
| `bad-dropped-connection` | `dropped_connection` | socket killed mid-stream → retried → **full recovery** |
| `bad-error-in-200` | `error_in_200` | 200-with-error-body detected as error → retried → recovery |
| `bad-http-503` | `http_503` | 503 → retried → recovery |
| `bad-malformed-tool-args` | `malformed_tool_args` | truncated tool-args JSON → error fed back, turn completes |

Chaos scenarios ride the existing recordings (the fault mutates/cuts the served
response, one-shot — the NEXT request serves normally, which is exactly what
proves recovery). `expectFaults: true` on a scenario downgrades the event/request
oracle findings the injected malformation legitimately causes.

If `npm run test:advanced` is green, the full harness is working end-to-end.
Library recorded against `moonshotai/kimi-k2.5` (~$0.10 total). The two chaos
bugs this suite caught on day one: dropped connections were NOT retried
(`isRetriableError` missed undici's "terminated"), and an error-in-200 body on
the streaming path silently became an empty "successful" reply. Both fixed in
`llm/provider.js` / `llm/client.js`.

## Oracle stack — the diagnostic core

Each scenario run is checked by **five independent oracles** over the same
execution. Adding a scenario gets all five for free. The insight that drives
this: outcome assertions ("did the file get created?") miss bugs where the
agent does the right thing but the harness mis-reports it. The oracles inspect
the three surfaces only a record/replay rig with control of the endpoint can
see — what the harness *sends* the model, what it *broadcasts*, what it
*persists*.

1. **Event firehose** (`lib/event-capture.js`) — connects to veil's `/ws`,
   records every event, checks: streamed-but-never-finalized, tool.start/end
   pairing, task lifecycle reaches terminal, no duplicate session.created /
   double chat.user_message, error frames carry a code.
2. **Outgoing-request validator** (`lib/request-validator.js`) — the fake API
   validates every `/chat/completions` request the harness *sends*: tool-call
   pairing (no orphan calls/results — the #1 provider-400 cause), no null tool
   content, valid role alternation, no top-level `cache_control`, no double
   user messages, system-prompt presence.
3. **DB-integrity oracle** (`lib/db-oracle.js`) — 33 SQL invariants over the
   SQLite after each run: no task left non-terminal, no dangling tool pairs,
   message_count reconciles, compact_size in bounds, no orphan/unanswered
   agent_messages, subscriptions delivered, referential integrity. Derived
   from the June-2026 bug history — each fixed bug implies a state invariant.
4. **Log scrape** — veil stderr for crashes / unhandled rejections (split
   from known-benign warnings).
5. **Process-leak check** — orphan shell processes after shutdown.

**Isolated HOME per scenario.** veil's global config+DB lives at
`~/.veil/data.db` (via `os.homedir()`), shared across workspaces. Each
scenario runs under its own `HOME=<workspace>/.home`, so (a) the user's real
DB is never polluted, and (b) each scenario gets a pristine DB the integrity
oracle can assert *in full*.

A scenario can PASS its assertions while an oracle finds a structural problem —
that is flagged loudly and fails the suite.

### Real bugs this stack found (all fixed)
- tool-input-**validation-failure** path emitted no bus `tool.end` → UI stuck spinner.
- **per-tool drain** re-emitted injected `USER_SENTINEL` messages → double `chat.user_message`.
- **openai `runChat` had no post-teardown wake** (the claude engine did) → an
  async task-subscription / async_inform notice that lands *during* the
  subscriber's turn was stranded forever. Found by `task-subscribe-notify` via
  a timing race (71s-timeout vs 12s-pass split across recordings).

### Latent findings (reported, not fixed — would need risky migration surgery)
- migration `003` references `token_budget` (added in `011`), so it **fails on
  every fresh DB** and is skipped; `ensureColumn` recovers fully (benign).
- `resolveBudget` reads `agentConfig.budget.*` but the agent.json schema
  forbids a `budget` key — agent-level budgets are unreachable via config.

## Event-firehose capture (`--capture-events`, on by default)

Every scenario run also connects a WebSocket client to veil's `/ws` firehose
(which broadcasts every `eventBus` event) and records the **full event stream**
to `_events/<scenario>.json`. A structural analyzer (`lib/event-capture.js`)
then checks invariants that outcome-based assertions can't see — the events a
real UI (Studio, the live feed) consumes:

- **streamed-but-never-finalized** — a session that emitted `*.chunk` frames
  must emit a terminal `done`/`error` (else the UI is stuck "streaming").
- **tool.start / tool.end pairing** — an unbalanced count means a tool never
  reported completion on the bus (stuck spinner).
- **task lifecycle** — any task that emits `task.status` must reach a terminal
  state; nothing after terminal.
- **duplicate `session.created`**, **double `chat.user_message`** (injection
  double-persist), **error frames missing a code**, **late chunks after done**.

Disable with `--no-capture`. Violations are printed inline and summarized; a
scenario can PASS its assertions while still having event violations, which is
flagged loudly.

This pass found two real harness bugs the assertions missed (both fixed):
1. the tool-input-**validation-failure** path yielded `tool.end` to the loop
   but never emitted it on the bus → UI tool stuck running;
2. the **per-tool drain** re-emitted/re-persisted injected `USER_SENTINEL`
   messages (the iteration-start drain was guarded but this second drain
   wasn't) → `chat.user_message` fired twice.

## Recording a new scenario

1. Add a scenario to `scenarios/index.js` (agents + `run` + `assert`).
2. `node run-scenarios.js --mode record --scenarios <name>` — records against
   the real model and asserts the live behavior.
3. `node run-scenarios.js --mode replay --scenarios <name>` — confirms it
   replays in a fresh workspace.
4. Commit `recordings/<name>/`.

Recordings are committed; scratch workspaces (`record-workspaces/`,
`../advanced-test-projects/`) are gitignored.

## Notes & limits

- **OpenAI engine only.** The claude-cli engine talks to the Claude SDK
  subprocess, not HTTP, so it can't be faked this way.
- Tools run **for real** on replay — only the LLM is faked. `cancellation` and
  `midturn-injection` therefore really execute their `sleep` commands, so those
  scenarios are slower (~30s / ~7s) than the instant ones.
- `garbage-tool-args` recordings are **synthesized** (`synthesize-garbage.js`)
  from the real `builder` system prompt, since a real model won't reliably emit
  malformed JSON on demand.
- v1 library was recorded against `moonshotai/kimi-k2.5` for **~$0.04 total**.

## Control endpoints (both modes)

`POST /__fault {type, remaining?, afterChunks?}` arms a one-shot fault (see
`lib/fault.js` for the full type list); `DELETE /__fault` disarms. Faults are
auto-disarmed on `/__workspace` so chaos never leaks across scenarios.

`GET /health` · `GET /models` · `GET /__recordings` ·
`POST /__scenario {scenario}` · `POST /__workspace {path}` · `POST /__reload`