---
name: eval-simulate
description: "Builds the fakes that simulate a Recipe's MCP servers and HTTP APIs in evals, with no real credentials or customer data: the fakes/ layout, captured contracts, scenarios, binding rules and drift checks. Use when an eval needs a binding faked."
metadata:
  min-cli-version: "0.73.0"
  next: ""
---

# eval-simulate

<!-- rules to carry (eval-simulate, from introspection-operator docs/migration-rules.json): all carried. The contract-test rules moved to `simulate/write`, R-fakes-ci-job-no-model to `compare/ci`. -->

A fake simulates one binding the Recipe uses, so a case runs with no real credentials and no customer data. [`@introspection-ai/eval-simulate`](https://github.com/introspection-org/introspection-cli/tree/main/integrations/eval-simulate) serves the Recipe's fakes: one for each MCP server the Recipe declares, and one for each HTTP API its code calls directly (an API fake). It validates every call against the binding's captured contract (an MCP server's tool schemas, an API's trimmed OpenAPI document), checks every answer against it, and keeps the state a verifier reads.

Writing one fake (entities, the manifest, answers, handlers, the code-mode surface) and its contract tests is in [write](references/write.md). Binding the fakes in a runner is `author/vitest-fakes` or `author/harbor-fakes`; the CI job is in `compare/ci`. Generating a fake from platform evidence is not covered. Outside an Operator session, `introspection evals simulate/<reference>` prints a reference.

Which servers a case needs and how much of each to fake are the loops' judgment: build-draft builds the first fakes, improve extends them for a hypothesis. Build only what a case needs.

## 1. Versions

- `@introspection-ai/eval-simulate` 0.6.0 or later, imported from its `/mcp` entry points (`@introspection-ai/eval-simulate/mcp`, `/mcp/checks`, `/mcp/verify`, `/mcp/answers`, `/mcp/testing`), with CLI 0.74.0 or later, which finds it in `fakes/`.
- Node 22.18 or later, which runs the fakes' TypeScript directly.

## 2. Lay out fakes/

The full tree, file by file, is in [layout](references/layout.md): `fakes/package.json` and its committed lockfile, `mcp/<server-id>/` for each MCP fake, `api/<api-id>/` for each API fake, `scenarios/` and `tests/`.

<!-- rule: R-fake-dir-is-server-id -->

The fake's directory name is the only key to the server id the Recipe declares (`lightfield-app`, not `lightfield`). Capture, observe, the host's routes and both runners use it; no manifest field maps it.

<!-- rule: R-fake-erasable-ts-static-import -->

Fakes are TypeScript that Node runs with no build: erasable syntax only (`import type`, no `enum`), imports with the `.ts` extension, and every fake imported statically in `mcp/index.ts`, or it is not served.

<!-- rule: R-pnpm-workspace-root-devdep -->

Add `fakes` to `pnpm-workspace.yaml` (`packages: [fakes]`), since pnpm ignores a `workspaces` field in `package.json`. Add `@introspection-ai/eval-simulate` to the root `devDependencies` too, because the evals import it.

<!-- rule: R-fake-never-deployed -->

A fake is never deployed: list `fakes/` in `.recipeignore`.

<!-- rule: R-root-fakes-declared-eval-path -->

Fakes live at the Recipe root `fakes/`, and keep `fakes/` in `.recipeignore`. From CLI 0.68.0 (recipe-check 0.19.0) that is the default; with an older CLI, declare `"pi": { "evals": { "fakes": "fakes" } }` so recipe-check treats `fakes/` as an eval path.

## 3. Capture the contract

<!-- rule: R-contract-files-written-by-cli -->

`tools.ts` and `observed.ts` are written by `introspection bindings mcp capture <id>` and `introspection bindings mcp observe <id>` only: never by hand or by a model, and never relayed through a reply. `answers.ts` is the one contract file written by hand.

`capture` reads `initialize` and `tools/list` from the project's binding of the server and never calls a tool; it needs a login and a binding. When the server isn't bound yet, bind it and capture before writing the fake; don't stand in for `tools.ts`. `observe` derives each tool's observed call and answer shapes from live traces.

<!-- rule: R-refusal-stop-not-edit-tools -->

If registration refuses a tool (it declares `taskSupport` `optional` or `required`, or a default below the top level), stop and report the server, the tools and the paths. Never edit `tools.ts` to get past it.

```sh
introspection bindings mcp capture <id> [--check]
introspection bindings mcp observe <id> [--check] [--examples]
```

## 4. Simulate an API binding

An API binding is an HTTP API the Recipe's own code calls, such as a provider calling Microsoft Graph, rather than an MCP server the agent calls. Its fake lives in `fakes/api/<api-id>/`, bound by host, and its contract is captured, never written:

```sh
eval-simulate capture <api-id> --spec <url|file> [--check]
```

Use a service emulator where one covers the API. How to capture, bind and check one is in [api](references/api.md).

## 5. Write the scenarios

A scenario is the world the agent meets: every fake's starting state, including the conditions under which a call fails. `base.ts` builds the base world from one clock; each other scenario is one module that changes it. `world(name, clock)` builds a named world.

<!-- rule: R-scenario-conditions-not-outcomes -->

A scenario sets conditions, never a tool-call sequence or an expected outcome. One module per scenario covers every fake the situation concerns, and the assertions stay in the verifier.

<!-- rule: R-scenario-only-changes-state -->

A scenario only changes state: no handler, no response text. A failure is state the handler reads. Only review enforces this.

<!-- rule: R-unknown-scenario-refused -->

`world()` throws on an unknown name; `checkRecipeFakes` fails on it ("an unknown scenario is refused"). Never default a runner's scenario to a made-up name such as `default`.

<!-- rule: R-scenario-never-default-name -->

Pass the case's own scenario to the harness, or none for the base world, never a name `world()` doesn't know: it throws. An eval run never writes `fakes/tests/responses.jsonl`; only the contract tests do.

<!-- rule: R-scenario-never-in-prompt -->

The harness chooses the scenario, from the case's input or the implementation's environment. It never reaches the prompt or a tool argument.

<!-- rule: R-labels-clock-synthetic-data -->

Name records with `labelId('<label>')` and find them by name in checks. Compute every date from the one clock. Seed synthetic data on reserved domains such as `example.test`; a response copied from a real server is customer data until replaced.

<!-- rule: R-failure-before-effect-default -->

A failure happens before the effect (nothing changed, a retry is safe) unless a case needs it after the effect (committed but errored, so a blind retry repeats it).

<!-- rule: R-types-not-interfaces -->

Declare entity, `State` and `World` types with `type`, not `interface`, so a state is assignable to a run's JSON artifacts.

Keep every fake's state JSON, as [scenarios](references/scenarios.md) says.

<!-- rule: R-fake-serves-start-state -->

A scenario is identified by its name plus the Recipe commit. Dates vary by day, so each fake also serves the state it started with, and the runner keeps it beside the results.

## 6. Bind the fakes for a run

<!-- rule: R-eval-never-falls-back-live -->

A fake is not a binding: an eval binds the declared server id to the fake and never falls back to the live server.

<!-- rule: R-every-required-server-bound -->

Only the servers the harness returns are bound; a binding file the Recipe ships is not read. Declare a fake for every server the Recipe requires, even one no case exercises, or the run fails at configuration.

<!-- rule: R-reset-mutable-state-per-case -->

Reset mutable fixture state for every case. Shared state lets one candidate's side effects show up in another's result.

To try the agent against the fakes by hand:

```sh
introspection local --scenario <name> [--fakes <id>] [-p "<request>"]
```

It starts `eval-simulate serve` from `fakes/`, binds every fake (`--fakes <id>` to choose), and writes their state to `.introspection/local/<scenario>/state.json`. Add `.introspection/local/` to `.gitignore`.

## 7. Check for drift

<!-- rule: R-live-drift-checks-not-ci -->

CI runs only offline checks: the type check, the fakes' tests and `checkRecipeFakes`. `introspection bindings mcp capture <id> --check` and `observe <id> --check` (against the recorded window, without `--lookback`) need a member login, so run them by hand or on a schedule.

```sh
cd fakes && pnpm test
```

## Done when

- Every server the Recipe requires has a fake, served from a captured contract, and `cd fakes && pnpm test` passes.
- Each case's scenario sets only conditions, and the fakes are bound for the run, never falling back to the live server.
- Hand back to the step that asked for the fakes.