--- name: eval-compare description: "Use when comparing two eval runs case by case with introspection eval compare, promoting or checking a suite's baseline, or wiring the comparison into a pull-request check in CI." --- # eval-compare Needs CLI 0.67.0 or later, which compares `run.json` schema 10 records, joins older ones by `legacy_id`, names moved criteria and compares runs from two runners. `introspection eval compare` lines two runs up case by case and reports what changed. A suite's baseline is one committed run that later runs compare against, and CI runs that comparison as the pull-request gate. ## What this step covers `eval compare`, `eval baseline promote` and `check`, and what each refuses. Reading one run is `eval-run`; the CI job is `compare/ci`. ## What it does not cover What counts as passing, whether a regression blocks a change, and when to replace a baseline are the loops' judgment: build-review promotes the first baseline, improve-propose compares a fix with its baseline, and the optimize steps compare candidates. ## Compare two runs ```sh introspection eval compare [] # candidate defaults to the latest run of the same runner introspection eval compare --baseline [] # base is the suite's committed baseline.json ``` Each case is `improved`, `regressed`, `unchanged`, `new`, `removed` or `incomplete`. A case not measured in either run is `incomplete` and has no score in the comparison, whatever its verifier wrote; it counts as neither improved nor regressed. - **The join.** Cases are joined by `id`. Against a record written before schema 10, a case left over is joined by its `legacy_id`, and each case's `join` (`id` or `legacy_id`) says which was used. Two schema 10 records join on `id` only, so a renamed case reads as removed and new, never silently matched. - **Moved criteria.** A score that changed lists each criterion whose score moved, with both scores and the candidate's reasoning. - **Old records.** `compare` and baselines refuse a record older than schema 9, which has no per-case results to join; run the suite again. `eval view` still opens such a record. A record from a newer CLI is refused with "upgrade the CLI". - **Two runners.** Comparing a local run with a hosted run of the same cases needs both run ids named. The report says the runners differ, skips the fingerprint check (each runner hashes its own implementation of a case), and names any case whose two implementations report different dimension names. `compare` refuses two runs in which a case's fingerprint differs, with no override: the case didn't measure the same thing. A fingerprint is per case (CLI 0.67.0 or later), so editing one case's implementation changes only that case; the agent is excluded, so runs of different `--agent` values compare. When only one of the two runs has `case-v1:` fingerprints, the check is skipped and the cases are named. Promote a new baseline in its own change. A fingerprint covers only the implementation's own files (`author/` says which). A change outside them, such as a fakes build context, verifier code shipped another way, the Runtime image, or tool versions resolved at run time, leaves it unchanged. After such a change, promote a new baseline yourself and say so in the pull request. `compare` warns when shared eval code changed between the two runs' commits: `evals/lib/`, `judges/`, an eval lockfile, or a suite's files outside its implementations. It never fails the comparison, and without a commit or with a dirty tree it says it couldn't check. A shared verifier can move every score without the agent changing, so read that warning before reading the scores. To compare two agents, run twice changing only `--agent`, then confirm the snapshot, prompt, turn, suite, model, Recipe revision and trials match before comparing. Prefer a candidate made with `from:` that changes one mechanism. A local trial in a substitute image and a hosted trial on the Runtime image measure different images: never compare or baseline one against the other (`run/hosted`). ## Read the comparison as data `compare -o json` carries `.summary` (case counts by change), `.score.{base,candidate}` (the mean over cases complete in both runs, `.score.incomplete` left out), `.{base,candidate}.cases_passed` (complete with score 1), `.cases_total` and `.cases_incomplete`, `.{base,candidate}.scores` (each named score's mean), and `.cases[]` with `id`, `join`, `change`, the scores, `score_changes[]` and their `criteria[]`. Per-check accuracy is `--query candidate.scores`. `compare` exits 0 whatever the scores are, unless a `--fail-on` condition holds. Gate with `introspection eval compare --baseline --fail-on regressed --fail-on incomplete`. Add `--fail-on score-drop` only when the team chooses to. Record each comparison's provenance: CLI version, judge model, dataset version, Recipe revision, candidate model and trials. For a replay, add the source conversation, transcript hash, turn, prefix count and where the artifacts are. In the pull request, paste the `eval compare --baseline` output with the runner, where it ran, the commit, the model and the trials. With one trial, a difference of a case or two may be variance; say so. A baseline replacement is its own reviewed pull request, with the reason. ## Keep a baseline Commit the suite, run it on the unchanged Recipe, and promote that run in its own commit (`promote` refuses uncommitted changes). Then compare with `--baseline` and read the JSON, not the exit code. ```sh introspection eval baseline promote [] [-r ] git add evals//baseline.json && git commit -m "test(evals): baseline for " introspection eval baseline check ``` `promote` writes the run, without machine paths, to `baseline.json` in the suite folder. `promote` refuses a run with a missing, repeated or incomplete case, files changed on disk since the run, files Git doesn't track inside a fingerprint, and `recipe_git_dirty` (`--allow-dirty` only when the user accepts a baseline from uncommitted code). Promote only an unfiltered, complete run, and one trial of a `--trials` batch, never a merge. Never hand-edit `baseline.json`. `eval baseline check` runs nothing and needs no login, Docker or container runner. It covers every committed `evals/*/baseline.json` and fails when a suite's cases changed without a new baseline, which catches a pull request that edits cases and doesn't promote. ## The CI gate Setting up the pull-request gate is `compare/ci`. ## Commands - `introspection eval compare []` - `introspection eval compare --baseline --fail-on regressed --fail-on incomplete` - `introspection eval baseline promote []` - `introspection eval baseline check [-p ]` ## References Each is a file in `references/`; outside an Operator session, `introspection evals compare/` prints it. - [`ci`](references/ci.md): the pull-request gate: the job, its setup and secrets, what CI can record, and the fakes job that runs first.