---
name: eval-fakes
description: "Use when an eval needs an MCP server the Recipe declares: what a fake is, the layout of fakes/, its contract and scenarios, the rules every runner binds it by, and the checks for drift from the live server."
---

# eval-fakes

<!-- rules to carry (eval-fakes, from introspection-operator docs/migration-rules.json): all carried. The contract-test rules moved to `fakes/write`, R-fakes-ci-job-no-model to `compare/ci`. -->

A fake stands in for one MCP server the Recipe declares, so a case runs with no real credentials and no customer data. `@introspection-ai/mcp-fakes` serves it: it validates every call against the server's captured tool schemas, checks every answer against the fake's contract, and keeps the state a verifier reads.

## What this step covers

The `fakes/` package, the contract files, scenarios, the rules every runner binds the fakes by, and the drift checks. Writing one fake (entities, the manifest, answers, handlers) and its contract tests is `fakes/write`. Binding the fakes in a runner is `author/vitest-fakes` or `author/harbor-fakes`; the CI job is in `compare/ci`. Generating a fake from platform evidence is not covered.

## What it does not cover

Which servers a case needs and how much of each to fake are the loops' judgment: build-draft builds the first fakes, improve extends them for a hypothesis. Build only what a case needs.

## Versions

- `@introspection-ai/mcp-fakes` 0.3.0 or later (0.5.0 current), with CLI 0.59.0 or later.
- Node 22.18 or later, which runs the fakes' TypeScript directly.

## Lay out fakes/

```text
fakes/
  package.json          private, "type": "module"; depends on @introspection-ai/mcp-fakes and typebox;
                        devDependency @modelcontextprotocol/client 2.3.1 for the contract tests
  package-lock.json     committed; the sidecar installs from it
  mcp/
    index.ts            every fake, imported statically
    <server-id>/
      index.ts          exports manifest, contract, register, observed
      tools.ts          written by `introspection bindings mcp capture`
      observed.ts       written by `introspection bindings mcp observe`, where there are traces
      answers.ts        each tool's successful answer, by hand
      contract.ts       contractOf(tools, answers)
      entities.ts       the stored things and the State type
      handlers.ts       one SDK callback per tool the Recipe allows
      manifest.json     server, capture, entities, settings, per-tool reads and writes
  scenarios/
    index.ts            SCENARIOS and world(name, clock)
    base.ts             the world every scenario starts from
    <scenario>.ts       one module per situation
  tests/
    fakes.test.js       checkRecipeFakes
    setup.js            the recording's global setup
    <server-id>.test.js the fake's contract tests
    responses.jsonl     every answer the contract tests received, committed
```

<!-- rule: R-fake-dir-is-server-id -->

The fake's directory name is the only key to the server id the Recipe declares (`lightfield-app`, not `lightfield`). Capture, observe, the host's routes and both runners use it; no manifest field maps it.

<!-- rule: R-fake-erasable-ts-static-import -->

Fakes are TypeScript that Node runs with no build: erasable syntax only (`import type`, no `enum`), imports with the `.ts` extension, and every fake imported statically in `mcp/index.ts`, or it is not served.

<!-- rule: R-pnpm-workspace-root-devdep -->

Add `fakes` to `pnpm-workspace.yaml` (`packages: [fakes]`), since pnpm ignores a `workspaces` field in `package.json`. Add `@introspection-ai/mcp-fakes` to the root `devDependencies` too, because the evals import it.

<!-- rule: R-fake-never-deployed -->

A fake is never deployed: list `fakes/` in `.recipeignore`.

<!-- rule: R-root-fakes-declared-eval-path -->

Fakes live at the Recipe root `fakes/`, and keep `fakes/` in `.recipeignore`. From CLI 0.68.0 (recipe-check 0.19.0) that is the default; with an older CLI, declare `"pi": { "evals": { "fakes": "fakes" } }` so recipe-check treats `fakes/` as an eval path.

## The contract

<!-- rule: R-contract-files-written-by-cli -->

`tools.ts` and `observed.ts` are written by `introspection bindings mcp capture <id>` and `introspection bindings mcp observe <id>` only: never by hand or by a model, and never relayed through a reply. `answers.ts` is the one contract file written by hand.

`capture` reads `initialize` and `tools/list` from the project's binding of the server and never calls a tool; it needs a login and a binding. When the server isn't bound yet, bind it and capture before writing the fake; don't stand in for `tools.ts`. `observe` derives each tool's observed call and answer shapes from live traces.

<!-- rule: R-refusal-stop-not-edit-tools -->

If registration refuses a tool (it declares `taskSupport` `optional` or `required`, or a default below the top level), stop and report the server, the tools and the paths. Never edit `tools.ts` to get past it.

## Scenarios

A scenario is the world the agent meets: every fake's starting state, including the conditions under which a call fails. `base.ts` builds the base world from one clock; each other scenario is one module that changes it. `world(name, clock)` builds a named world.

<!-- rule: R-scenario-conditions-not-outcomes -->

A scenario sets conditions, never a tool-call sequence or an expected outcome. One module per scenario covers every fake the situation concerns, and the assertions stay in the verifier.

<!-- rule: R-scenario-only-changes-state -->

A scenario only changes state: no handler, no response text. A failure is state the handler reads. Only review enforces this.

<!-- rule: R-unknown-scenario-refused -->

`world()` throws on an unknown name; `checkRecipeFakes` fails on it ("an unknown scenario is refused"). Never default a runner's scenario to a made-up name such as `default`.

<!-- rule: R-scenario-never-default-name -->

Pass the case's own scenario to the harness, or none for the base world, never a name `world()` doesn't know: it throws. An eval run never writes `fakes/tests/responses.jsonl`; only the contract tests do.

<!-- rule: R-scenario-never-in-prompt -->

The harness chooses the scenario, from the case's input or the implementation's environment. It never reaches the prompt or a tool argument.

<!-- rule: R-labels-clock-synthetic-data -->

Name records with `labelId('<label>')` and find them by name in checks. Compute every date from the one clock. Seed synthetic data on reserved domains such as `example.test`; a response copied from a real server is customer data until replaced.

<!-- rule: R-failure-before-effect-default -->

A failure happens before the effect (nothing changed, a retry is safe) unless a case needs it after the effect (committed but errored, so a blind retry repeats it).

<!-- rule: R-types-not-interfaces -->

Declare entity, `State` and `World` types with `type`, not `interface`, so a state is assignable to a run's JSON artifacts.

<!-- rule: R-fake-serves-start-state -->

A scenario is identified by its name plus the Recipe commit. Dates vary by day, so each fake also serves the state it started with, and the runner keeps it beside the results.

```ts
// fakes/scenarios/blocked.ts: updating the Acme deal is rejected
import { labelId } from "@introspection-ai/mcp-fakes";
import type { World } from "./index.ts";

export default function blocked(world: World) {
  world.crm.failUpdate = labelId("acme-growth");
}
```

## Bind the fakes for a run

<!-- rule: R-eval-never-falls-back-live -->

A fake is not a binding: an eval binds the declared server id to the fake and never falls back to the live server.

<!-- rule: R-every-required-server-bound -->

Only the servers the harness returns are bound; a binding file the Recipe ships is not read. Declare a fake for every server the Recipe requires, even one no case exercises, or the run fails at configuration.

<!-- rule: R-reset-mutable-state-per-case -->

Reset mutable fixture state for every case. Shared state lets one candidate's side effects show up in another's result.

To try the agent against the fakes by hand, `introspection local --scenario <name> [-p "<request>"]` starts `mcp-fakes serve` from `fakes/`, binds every fake (`--fakes <id>` to choose), and writes their state to `.introspection/local/<scenario>/state.json`. Add `.introspection/local/` to `.gitignore`.

## Check for drift

<!-- rule: R-live-drift-checks-not-ci -->

CI runs only offline checks: the type check, the fakes' tests and `checkRecipeFakes`. `introspection bindings mcp capture <id> --check` and `observe <id> --check` (against the recorded window, without `--lookback`) need a member login, so run them by hand or on a schedule.

## Commands

- `introspection bindings mcp capture <id> [--check]`
- `introspection bindings mcp observe <id> [--check] [--examples]`
- `introspection local --scenario <name> [--fakes <id>] [-p "<request>"]`
- `cd fakes && pnpm test`

## References

Each is a file in `references/`; outside an Operator session, `introspection evals fakes/<reference>` prints it.

- [`write`](references/write.md): one fake's entities, the manifest, answers, handlers, the code-mode surface, and its contract tests.

## Next

- Build one fake with `fakes/write`.
- Bind them in your runner with `author/vitest-fakes` or `author/harbor-fakes`.
- The CI job is in `compare/ci`.