--- name: eval-compare description: "Use when comparing two eval runs case by case with introspection eval compare, promoting or checking a suite's baseline, or wiring the comparison into a pull-request check in CI." metadata: min-cli-version: "0.73.0" --- # eval-compare Needs CLI 0.73.0 or later, which compares `run.json` schema 10 records by case id and names moved criteria. `introspection eval compare` lines two runs up case by case and reports what changed. A suite's baseline is one committed run that later runs compare against, and CI runs that comparison as the pull-request gate. ## What this step covers `eval compare`, `eval baseline promote` and `check`, and what each refuses. Reading one run is `eval-run`; the CI job is `compare/ci`. ## What it does not cover What counts as passing, whether a regression blocks a change, and when to replace a baseline are the loops' judgment: build-review promotes the first baseline, improve-collect's regression check compares the served commit with the committed baseline, improve-propose compares a fix with its baseline and promotes the fix's run before it hands over, and the optimize steps compare candidates. ## Compare two runs ```sh introspection eval compare [] # candidate defaults to the latest run of the same runner introspection eval compare --baseline [] # base is the suite's committed baseline.json ``` Each case is `improved`, `regressed`, `unchanged`, `new`, `removed` or `incomplete`. A case not measured in either run is `incomplete` and has no score in the comparison, whatever its verifier wrote; it counts as neither improved nor regressed. - **The join.** Cases are joined by `id` only, so a renamed case reads as removed and new, never silently matched. - **Moved criteria.** A score that changed lists each criterion whose score moved, with both scores and the candidate's reasoning. - **Old records.** `compare` and baselines refuse a record or baseline older than schema 10: run the suite again and promote that run. `eval view` still opens such a record. A record from a newer CLI is refused with "upgrade the CLI". - **One runner per comparison.** Each case runs on one runner, so a comparison pairs two runs of the same runner: two Vitest runs or two Harbor runs. A Recipe with a Vitest and a Harbor suite compares each suite with its own earlier run, never one with the other: improve compares its Harbor runs with the Harbor baseline, and its Vitest regression guard with the Vitest run at the baseline commit. Two runs of different runners share no case, and the report says the runners differ. `compare` refuses two runs in which a case's fingerprint differs, with no override: the case didn't measure the same thing. A fingerprint is per case (CLI 0.67.0 or later), so editing one case's implementation changes only that case; the agent is excluded, so runs of different `--agent` values compare. When only one of the two runs has `case-v1:` fingerprints, the check is skipped and the cases are named. Promote a new baseline in its own change. A fingerprint covers only the implementation's own files (`author/` says which). A change outside them, such as a fakes build context, verifier code shipped another way, the Runtime image, or tool versions resolved at run time, leaves it unchanged. After such a change, promote a new baseline yourself and say so in the pull request. `compare` warns when shared eval code changed between the two runs' commits: `evals/lib/`, `judges/`, an eval lockfile, or a suite's files outside its implementations. It never fails the comparison, and without a commit or with a dirty tree it says it couldn't check. A shared verifier can move every score without the agent changing, so read that warning before reading the scores. To compare two agents, run twice changing only `--agent`, then confirm the snapshot, prompt, turn, suite, model, Recipe revision and trials match before comparing. Prefer a candidate made with `from:` that changes one mechanism. **A run measures its commit, wherever it ran.** A local trial and a hosted trial of the same commit, with the same cases, model and trials, compare with each other and either can be promoted as that commit's baseline. They still run different images (`run/hosted`): when a case differs only between a local and a hosted run, check that first, before calling it the agent's. ## Read the comparison as data `compare -o json` carries `.summary` (case counts by change), `.score.{base,candidate}` (the mean over cases complete in both runs, `.score.incomplete` left out), `.{base,candidate}.cases_passed` (complete with score 1), `.cases_total` and `.cases_incomplete`, `.{base,candidate}.scores` (each named score's mean), and `.cases[]` with `id`, `join`, `change`, the scores, `score_changes[]` and their `criteria[]`. Per-check accuracy is `--query candidate.scores`. `compare` exits 0 whatever the scores are, unless a `--fail-on` condition holds. Gate with `introspection eval compare --baseline --fail-on regressed --fail-on incomplete`. Add `--fail-on score-drop` only when the team chooses to. Record each comparison's provenance: CLI version, judge model, dataset version, Recipe revision, candidate model and trials. For a replay, add the source conversation, transcript hash, turn, prefix count and where the artifacts are. In the pull request, paste the `eval compare --baseline` output with the runner, where it ran, the commit, the model and the trials. With one trial, a difference of a case or two may be variance; say so. A baseline replacement is its own reviewed pull request, with the reason. ## Keep a baseline Commit the suite, run it on the unchanged Recipe, and promote that run in its own commit (`promote` refuses uncommitted changes). Write every commit message as a Conventional Commit: a type (`feat`, `fix`, `docs`, `test`, `chore`) and what changed for the agent's users, as in `feat: ask for the budget before planning`. Then compare with `--baseline` and read the JSON, not the exit code. ```sh introspection eval baseline promote [] [-r ] git add evals//baseline.json && git commit -m "test(evals): baseline for " introspection eval baseline check ``` `promote` writes the run, without machine paths, to `baseline.json` in the suite folder. `promote` refuses a run with a missing, repeated or incomplete case, files changed on disk since the run, files Git doesn't track inside a fingerprint, and `recipe_git_dirty` (`--allow-dirty` only when the user accepts a baseline from uncommitted code). Promote only an unfiltered, complete run, and one trial of a `--trials` batch, never a merge. Never hand-edit `baseline.json`. `eval baseline check` runs nothing and needs no login, Docker or container runner. It covers every committed `evals/*/baseline.json` and fails when a suite's cases changed without a new baseline, which catches a pull request that edits cases and doesn't promote. ## The CI gate Setting up the pull-request gate is `compare/ci`. ## Commands - `introspection eval compare []` - `introspection eval compare --baseline --fail-on regressed --fail-on incomplete` - `introspection eval baseline promote []` - `introspection eval baseline check [-p ]` ## References Each is a file in `references/`; outside an Operator session, `introspection evals compare/` prints it. - [`ci`](references/ci.md): the pull-request gate: the job, its setup and secrets, what CI can record, and the fakes job that runs first.