--- name: eval-run description: "Use when running an eval suite with introspection eval run and reading what it measured: run.json and its cases, incomplete cases and reward_invalid, trials, retries, and replaying a production conversation." --- # eval-run Needs CLI 0.67.0 or later, which writes `run.json` schema 10. `introspection eval run` runs a suite or one implementation, on whichever runner the path belongs to, and writes one `run.json`. Every later step reads `run.json`, not the runner's own files. ## What this step covers Running, reading `run.json`, telling a broken measurement from a failing agent, trials, retries, resuming, and replaying a production conversation. Runs against a pushed commit or a runtime version are `run/hosted`. ## What it does not cover When a loop runs evals, on which runner, how many trials it can afford and what result passes its gate are the loop's judgment: build-review, deploy-test, improve-propose and the optimize steps decide that. ## Run ```sh introspection eval run -p [--agent ] [--trials ] introspection eval view [] [-r ] ``` The runner is inferred from `-p`; pass `--runner` only when the path is ambiguous. `eval view` opens the runner's native viewer on the latest run, or the one named. Install a runner's toolchain with `introspection setup --target ` once that runner is chosen, not while designing cases. Authoring cases alone needs no containers. Resolve the Runtime and the Recipe through the CLI and the runner's Recipe harness. Never invoke Pi directly, build session JSONL by hand, wrap `introspection local`, or copy the agent for the eval. Running a test framework directly, outside `introspection eval run`, writes no run record and supports no replay or trials. Use `introspection eval run` for anything you compare. When the repository already carries a suite for another runner, keep it and ask before running it. Never run it as an automatic preflight or gate. `--agent` selects a Recipe agent by the `name` its YAML declares, and takes one value, so a baseline and a candidate agent are two invocations. With a dirty workspace, keep the candidate's diff with the results: `run.json` records the commit and dirty state, but no hash of the agent definition. ## Read run.json Each run is stored in `eval-runs//`: `run.json`, and the runner's native files beside it. ```json { "schema_version": 10, "status": "completed", "candidate": { "agent": "agent", "model": "anthropic/claude-sonnet-4-6", "recipe_git_commit_sha": "4e1d…", "recipe_git_dirty": false }, "trial": { "index": 1, "count": 1 }, "result": { "completion_reason": "completed_with_errors", "cases": [ { "id": "C3", "complete": true, "score": 0.5, "scores": [ { "name": "refund issued", "score": 1.0 }, { "name": "reply", "score": 0.0, "criteria": [ { "name": "no_promised_date", "score": 0.0, "reasoning": "The reply says the money arrives by Friday." } ] } ], "usage": { "agent": { "input_tokens": 18234, "output_tokens": 912, "cost_usd": 0.07 }, "verifier": { "input_tokens": 2100, "output_tokens": 140, "cost_usd": 0.01 } }, "fingerprint": "sha256:…" }, { "id": "C4", "complete": false, "error": "reward_invalid", "error_message": "case C4: the rewards object is invalid: `redline_total` is 7 (number): a score must be between 0 and 1; …", "scores": [], "usage": { "agent": { "input_tokens": 9120, "output_tokens": 455, "cost_usd": 0.03 } } } ] } } ``` The top-level `status` is `completed` whenever the runner finished. The run's outcome is `result.completion_reason`, one of three values: - `stop`: every case is complete. The local runner also requires every case to pass; Harbor reports `stop` for any complete run, whatever its scores. - `scored_failure`: the local runner only, when every case is complete and at least one failed. - `completed_with_errors`: a case is incomplete or, under Harbor, a trial errored or failed in a step. `result.cases[]` holds each case's `id`, `score`, named `scores`, `usage`, `fingerprint` and `complete`. A failing score is a complete measurement. An error sets `complete: false`, with the kind in `error` and the cause in `error_message`. Cost is per case; duration is per run only (`started_at`, `completed_at`). - **`id`** is the case the implementation names, and `legacy_id` the id it had before schema 10. - **`score`** is the verifier's `reward`. `score_derived: true` means the verifier wrote no `reward` and the score is the mean of its dimensions. - **`scores[]`** is one entry per dimension, with `criteria[]` holding each criterion's `name`, `score`, `reasoning` and `status`, and `reasoning` copied up when there is exactly one. A criterion that didn't apply has no score and the status `not_applicable`; that is per criterion, never a whole case. A multi-step implementation labels each score with its `step`. - **`usage.agent`** is the agent's tokens and cost; **`usage.verifier`**, present when a model graded any reward, is the verifier's. A schema 9 record's `usage` reads as `usage.agent`. Read cost per case from `result.cases[].usage.agent.cost_usd`, with `input_tokens` and `output_tokens` beside it, and sum over the cases; add `usage.verifier` for what grading cost. A continued multi-turn run already reports session totals. ## A broken measurement is not a failing agent `complete: false` means the case wasn't measured: setup, the harness, the agent run or the verifier broke. It counts as neither a pass nor a fail, and every mean in the run leaves it out. Fix the cause; never read it as the agent's score. | `error` | What broke | | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `reward_invalid` | The verifier's `reward.json` broke the contract: a value outside 0 to 1, a count, a boolean, a string, or an empty object without details showing that every dimension did not apply. Also a reward file the runner couldn't parse, or that is missing or empty, with the runner's own message (for example `RewardFileNotFoundError`), which earlier CLIs dropped silently. | | `contract_unsupported` | The verifier declares a rewards contract version newer than this CLI reads. Upgrade the CLI. | | `NoScores`, `ScoreMissing`, `AssertionFailure` | The local runner's case recorded no score, a check returned none, or a non-score assertion failed (`author/vitest`). | | anything else | The runner's own error kind, such as an agent timeout or an infrastructure failure. | Compare runs by score. For each score below 1, quote the failing criteria and their reasoning (`criteria[]` in `run.json`, from the verifier's `reward-details.json`), and the verifier's output when a check raised. When a run finds no cases at `-p`, compare the path with the implementation layout in `author/` and name the missing component and how to fix it; don't report only that setup failed. `eval run` exits 0 whenever the runner wrote its report, even when cases failed; it exits non-zero only when the harness itself failed. Never treat its exit code as the result. Print the run's status, the failed case ids and short error context first. Keep full JSON, JSONL, trajectories and native reports in files rather than printing them. ## Trials `--trials N` writes N separate `run.json`, each with `trial: {index, count}`, and nothing aggregates them. Run the batch into an empty, absolute `INTROSPECTION_EVAL_RUNS_DIR` and compute each case's pass rate. Promote one complete trial, never a merged batch. ```sh export INTROSPECTION_EVAL_RUNS_DIR="$PWD/eval-runs/trials-$(date +%s)" introspection eval run -p --trials 3 jq -s '[.[].result.cases[]] | group_by(.id) | map({id: .[0].id, runs: length, passed: (map(select(.complete and .score >= 1)) | length), incomplete: (map(select(.complete | not)) | length)})' "$INTROSPECTION_EVAL_RUNS_DIR"/*/run.json ``` A case that passes some trials and fails others is flaky: fix the case, the fixture or the Recipe before trusting a comparison on it. The evaluated model is the agent's `ai.model` (falling back to `model.name`). Start with one trial; go to three only when attempts conflict, and only within the approved budget. ## Retries and resuming Don't repeat an unchanged setup or auth failure as a paid attempt, rerun a trial that reached the agent beyond the budget, or rerun a valid failure to get a pass. `--retry-errors N` reruns only infrastructure failures, never an agent failure or a low score, and never a `reward_invalid` or `contract_unsupported` case, which are the verifier's to fix. It lists the reruns in `result.retries` and can't be combined with `--from-conversation`. Only some runners support it; `introspection eval run --help` says which. Reuse a result only when everything that defines the measurement still matches. After an infrastructure interruption, keep the valid completed work and rerun only the smallest unit the runner can rerun natively, such as `--retry-errors`, then consolidate. Resume an interrupted suite at the runner's narrowest native rerunnable unit, never a finer one you engineer. Merge results only if nothing that affects behaviour changed between invocations, and keep one result per case and trial, the new replacing the retained. An option that changes what is measured makes a different test. `--fresh-steps` starts a new conversation at every step of a multi-step implementation instead of continuing one: record which ran. **Re-scoring without rerunning is not built yet.** `eval run --score-only `, which would re-score a run's kept outputs once a verifier is fixed, isn't built. Until then, fix the verifier and run the case again. ## Replay a production conversation ```sh introspection eval run --from-conversation --turn -p ``` `--from-conversation --turn N` hydrates only the completed turns before N and runs turn N's prompt (`--prompt` replaces it). It recreates no files, services or credentials, and the source conversation stays read-only. `--from-conversation` supplies only conversation context: not the runner, assertions, verifier, fixtures, credentials or services. Select an existing suite with `-p`, and read `introspection eval run --help` first. A replay is exploration. Making a turn a durable case is `eval-case`. ## References Each is a file in `references/`; outside an Operator session, `introspection evals run/` prints it. - [`hosted`](references/hosted.md): plane trials against a pushed commit (`--commit-sha`) or a runtime version (`--runtime-version`), and how they differ from local trials. Container runner only.