--- name: eval-compare description: "Compares two eval runs case by case, promotes and checks a suite's baseline, and reads the baseline committed at a commit. Use when measuring a change against a baseline, replacing a baseline, or setting up the pull-request eval gate." metadata: min-cli-version: "0.82.0" next: "" --- # eval-compare `introspection eval compare` lines two runs up case by case and reports what changed. A suite's baseline is one committed run that later runs compare against, and CI runs that comparison as the pull-request gate. Reading one run is `eval-run`. What counts as passing, whether a regression blocks a change, and when to replace a baseline are the loops' judgment: build-review promotes the first baseline, improve-collect reads the baseline committed at the commit it improves, improve-propose compares a fix with its baseline and promotes the fix's run before it hands over, and the optimize steps compare candidates. ## 1. Compare two runs ```sh introspection eval compare [] # candidate defaults to the latest run of the same runner introspection eval compare --baseline [] # base is the suite's committed baseline.json ``` Each case is `improved`, `regressed`, `unchanged`, `new`, `removed` or `incomplete`. A case not measured in either run is `incomplete` and has no score in the comparison, whatever its verifier wrote; it counts as neither improved nor regressed. - **The join.** Cases are joined by `id` only, so a renamed case reads as removed and new, never silently matched. - **Moved criteria.** A score that changed lists each criterion whose score moved, with both scores and the candidate's reasoning. - **Old records.** `compare` and baselines refuse a record or baseline older than schema 10: run the suite again and promote that run. `eval view` still opens such a record. A record from a newer CLI is refused with "upgrade the CLI". - **One runner per comparison.** Each case runs on one runner, so a comparison pairs two runs of the same runner: two Vitest runs or two Harbor runs. A Recipe with a Vitest and a Harbor suite compares each suite with its own earlier run, never one with the other: improve compares its Harbor runs with the Harbor baseline, and its Vitest regression guard with the Vitest run at the baseline commit. Two runs of different runners share no case, and the report says the runners differ. `compare` refuses two runs in which a case's fingerprint differs, with no override: the case didn't measure the same thing. A fingerprint is per case (CLI 0.67.0 or later), so editing one case's implementation changes only that case; the agent is excluded, so runs of different `--agent` values compare. When only one of the two runs has `case-v1:` fingerprints, the check is skipped and the cases are named. Promote a new baseline in its own change. A fingerprint covers only the implementation's own files (`author/` says which). A change outside them, such as a fakes build context, verifier code shipped another way, the Runtime image, or tool versions resolved at run time, leaves it unchanged. After such a change, promote a new baseline yourself and say so in the pull request. `compare` warns when shared eval code changed between the two runs' commits: `evals/lib/`, `judges/`, an eval lockfile, or a suite's files outside its implementations. It never fails the comparison, and without a commit or with a dirty tree it says it couldn't check. A shared verifier can move every score without the agent changing, so read that warning before reading the scores. To compare two agents, run twice changing only `--agent`, then confirm the snapshot, prompt, turn, suite, model, Recipe revision and trials match before comparing. Prefer a candidate made with `from:` that changes one mechanism. **A run measures its commit, wherever it ran.** A local trial and a hosted trial of the same commit, with the same cases, model and trials, compare with each other and either can be promoted as that commit's baseline. They still run different images (`run/hosted`): when a case differs only between a local and a hosted run, check that first, before calling it the agent's. **A `--trials` run compares as one set,** by any of its run ids: each case's mean and trials passed ([json](references/json.md)). `--single-trial` compares runs alone. ## 2. Read and report the result Read `compare -o json`, not the exit code: `compare` exits 0 whatever the scores are, unless a `--fail-on` condition holds. Its fields are in [the JSON reference](references/json.md). Gate with `introspection eval compare --baseline --fail-on regressed --fail-on incomplete --fail-on draft`: a comparison is a measurement, so a draft fails it. Add `--fail-on score-drop` only when the team chooses to. Record each comparison's provenance: CLI version, judge model, dataset version, Recipe revision, candidate model and trials. For a replay, add the source conversation, transcript hash, turn, prefix count and where the artifacts are. In the pull request, paste the `eval compare --baseline` output with the runner, where it ran, the commit, the model and the trials. With one trial, a difference of a case or two may be variance; say so. With a set, give trials passed. A baseline replacement is its own reviewed pull request, with the reason. ## 3. Keep a baseline Commit the suite, with every case in it approved and its verifier written (a baseline is a measurement: `eval-case`, "Approval"), run it on the unchanged Recipe, and promote that run in its own commit (`promote` refuses uncommitted changes). Then compare with `--baseline` and read the JSON, not the exit code. ```sh introspection eval baseline promote [] [-r ] git add evals//baseline.json && git commit -m "test(evals): baseline for " introspection eval baseline check [-p ] ``` `promote` writes the run, without machine paths, to `baseline.json` in the suite folder. `promote` refuses a run with a missing, repeated, incomplete or draft case, files changed on disk since the run, files Git doesn't track inside a fingerprint, and `recipe_git_dirty` (`--allow-dirty` only when the user accepts a baseline from uncommitted code). Promote only an unfiltered, complete run; a `--trials` run is promoted as its whole set, and a set missing a trial is refused. Never hand-edit `baseline.json`. `eval baseline check` runs nothing and needs no login, Docker or container runner. It covers every committed `evals/*/baseline.json` and fails when a suite's cases changed without a new baseline, which catches a pull request that edits cases and doesn't promote. ## 4. Read the baseline at a commit How the approved cases scored at a commit is in the baselines committed there, and `show` reads them without running anything: ```sh introspection eval baseline show --runtime -o json # the commit that version was built from introspection eval baseline show [-p ] # this checkout ``` `show` exits 0 whatever it finds; `check` is the gate. Each suite is `present`, `missing` or `unreadable`, and each case `passing`, `failing` or `unmeasured`, as [the JSON reference](references/json.md) says. When `shared_eval_code` is `changed` or `unchecked`, say so before reading the scores. ## 5. Set up the CI gate The pull-request gate is in [the CI reference](references/ci.md): the job, its setup and secrets, what CI can record, and the fakes job that runs first. Outside an Operator session, `introspection evals compare/` prints a reference. ## Done when - The comparison's JSON is read, and its output, provenance and any variance are in the pull request. - A baseline, when one was promoted, is in its own commit and `eval baseline check` passes. - Hand back to the step that asked for the comparison.