---
name: eval-run
description: "Runs an eval suite, or part of one, with introspection eval run, and reads what it measured in run.json: scores, incomplete cases, trials, retries. Also replays a production conversation. Use whenever a step runs or reads evals."
metadata:
  min-cli-version: "0.82.0"
  next: ""
---

# eval-run

<!-- rules to carry (eval-run, from introspection-operator docs/migration-rules.json):
     - [ ] R-vitest-result-evidence-paths: the native report's paths are runner-specific and the body names no runner; stated in author/vitest ("Read the result"), so the rule's target should move to eval-author/vitest.
-->

`introspection eval run` runs a suite or one implementation, on whichever runner the path belongs to, and writes one `run.json`. Every later step reads `run.json`, not the runner's own files. Runs against a pushed commit or a runtime version are in [hosted](references/hosted.md). Outside an Operator session, `introspection evals run/<reference>` prints a reference.

When a loop runs evals, on which runner, how many trials it can afford and what result passes its gate are the loop's judgment: build-review, deploy-test, improve-propose and the optimize steps decide that.

## 1. Run

```sh
introspection eval run -p <suite-or-implementation> [--agent <name>] [--trials <n>]
introspection eval view [<run-id>] [-r <runner>]
```

The runner is inferred from `-p`; pass `--runner` only when the path is ambiguous. `eval view` opens the runner's native viewer on the latest run, or the one named.

<!-- rule: R-setup-harbor-when-selected -->

Install a runner's toolchain with `introspection setup --target <runner>` once that runner is chosen, not while designing cases. Authoring cases alone needs no containers.

<!-- rule: R-no-direct-pi-invocation -->

Resolve the Runtime and the Recipe through the CLI and the runner's Recipe harness. Never invoke Pi directly, build session JSONL by hand, wrap `introspection local`, or copy the agent for the eval.

<!-- rule: R-direct-vitest-no-run-record -->

Running a test framework directly, outside `introspection eval run`, writes no run record and supports no replay or trials. Use `introspection eval run` for anything you compare.

<!-- rule: R-other-runner-ask-first -->

When the repository already carries a suite for another runner, keep it and ask before running it. Never run it as an automatic preflight or gate.

## 2. Run part of a suite

Everything after `--` goes to the runner unchanged, so a run can select cases with the runner's own filters:

```sh
introspection eval run -p <suite> -- <runner filters>
```

Each runner's filters are in [filters](references/filters.md).

A filtered run is for a working loop, never a measurement of the suite. `run.json` holds only the cases that ran, `eval compare` reports a case left out as `removed`, not regressed, and `eval baseline promote` refuses the run (`eval-compare`). Measure the suite with a run of all of it.

## 3. Run Harbor locally

With no `-r`, Harbor runs each task in local Docker, on the task's own image. Which adapter to use, and how a local run relates to a plane run, are in [harbor-local](references/harbor-local.md).

## 4. Run at a commit

A local run measures the working tree. To measure a commit that isn't checked out (the commit a version was built from, the commit production serves, a baseline commit), run in a checkout of that commit of its own, so `run.json` records it clean:

```sh
git worktree add <scratch folder>/<slug>-<short sha> <commit>
cd <scratch folder>/<slug>-<short sha>
<install with the Recipe's package manager, from its lockfile>   # pnpm install --frozen-lockfile, say
introspection eval run -p <suite> [--trials <n>]
git worktree remove <scratch folder>/<slug>-<short sha>          # once no later step needs its runs
```

Keep the worktree, outside the repository, until every step that reads its runs is done, as [at-commit](references/at-commit.md) says.

<!-- rule: R-agent-flag-yaml-name-single -->

`--agent` selects a Recipe agent by the `name` its YAML declares, and takes one value, so a baseline and a candidate agent are two invocations. With a dirty workspace, keep the candidate's diff with the results: `run.json` records the commit and dirty state, but no hash of the agent definition.

## 5. Read run.json

Each run is stored in `eval-runs/<run-id>/`: `run.json`, and the runner's native files beside it. [The run.json reference](references/run-json.md) has a full example, every field, `result.completion_reason` and the cost fields.

<!-- rule: R-run-json-case-complete -->

`result.cases[]` holds each case's `id`, `score`, named `scores`, `usage`, `fingerprint` and `complete`. A failing score is a complete measurement. An error sets `complete: false`, with the kind in `error` and the cause in `error_message`. Cost is per case; duration is per run only (`started_at`, `completed_at`).

## 6. A broken measurement is not a failing agent

`complete: false` means the case wasn't measured: setup, the harness, the agent run or the verifier broke. It counts as neither a pass nor a fail, and every mean in the run leaves it out. Fix the cause; never read it as the agent's score. What each `error` kind means is in [the run.json reference](references/run-json.md#error-kinds).

<!-- rule: R-read-reward-and-details -->

Compare runs by score. For each score below 1, quote the failing criteria and their reasoning (`criteria[]` in `run.json`, from the verifier's `reward-details.json`), and the verifier's output when a check raised.

<!-- rule: R-task-path-structure-diagnosis -->

When a run finds no cases at `-p`, compare the path with the implementation layout in `author/<runner>` and name the missing component and how to fix it; don't report only that setup failed.

<!-- rule: R-eval-run-exit-zero -->

`eval run` exits 0 whenever the runner wrote its report, even when cases failed; it exits non-zero only when the harness itself failed. Never treat its exit code as the result.

<!-- rule: R-keep-reports-in-files -->

Print the run's status, the failed case ids and short error context first. Keep full JSON, JSONL, trajectories and native reports in files rather than printing them.

## 7. Trials

`--trials N` runs the suite N times as one trial set: N `run.json`, each recording the set's id (its first run's) as `trial.set_id`. `eval compare` and `eval baseline promote` take the whole set by any of its run ids, scoring each case by its mean and the trials it passed (`eval-compare`). Never count trials by hand.

```sh
introspection eval run -p <suite> --trials 3
introspection eval compare <base-run-id> <run-id> -o table   # each case: passed 0/3 -> 3/3
```

A case that passes some trials and fails others is flaky: fix the case, the fixture or the Recipe before trusting a comparison on it. To find one, compare two trials of the set with each other: `introspection eval compare <trial-1-run-id> <trial-2-run-id> --single-trial --query "cases[?change!='unchanged'].id"`.

<!-- rule: R-replay-model-and-trial-count -->

The evaluated model is the agent's `ai.model` (falling back to `model.name`). Start with one trial; go to three only when attempts conflict, and only within the approved budget.

## 8. Retries and resuming

<!-- rule: R-no-paid-rerun-unchanged-failure -->

Don't repeat an unchanged setup or auth failure as a paid attempt, rerun a trial that reached the agent beyond the budget, or rerun a valid failure to get a pass.

<!-- rule: R-retry-errors-infrastructure-only -->

`--retry-errors N` reruns only infrastructure failures, never an agent failure or a low score, and never a `reward_invalid` or `contract_unsupported` case, which are the verifier's to fix. It lists the reruns in `result.retries` and can't be combined with `--from-conversation`. Only some runners support it; `introspection eval run --help` says which.

To reuse a result or resume an interrupted suite, follow [resume](references/resume.md): which unit to rerun, when results merge, and why `--fresh-steps` makes a different test.

## 9. Replay a production conversation

```sh
introspection eval run --from-conversation <conversation-id> --turn <n> -p <suite-or-implementation>
```

<!-- rule: R-hydration-messages-only -->

`--from-conversation <id> --turn N` hydrates only the completed turns before N and runs turn N's prompt (`--prompt` replaces it). It recreates no files, services or credentials, and the source conversation stays read-only.

<!-- rule: R-from-conversation-context-only -->

`--from-conversation` supplies only conversation context: not the runner, assertions, verifier, fixtures, credentials or services. Select an existing suite with `-p`, and read `introspection eval run --help` first.

A replay is exploration. Making a turn a durable case is `eval-case`.

## Done when

- The run's `run.json` is read: its scores, each failing criterion's reasoning, and every incomplete case's cause.
- Hand back to the step that asked for the run.