--- name: eval-author description: "Writes an eval implementation and its verifier for one approved case: what to check, which judge grades it, and the reward.json and reward-details.json the CLI reads. Use when adding or changing a case's eval." metadata: min-cli-version: "0.73.0" next: "" --- # eval-author An implementation is how one runner executes a case. A verifier scores one run of it. This step holds for both runners; each runner's mechanics are in its reference. Which cases get an implementation, which runner and what passes are the loops' judgment. build-draft writes the first implementations, on Vitest unless a case needs the container; improve-propose writes the Harbor cases that test a hypothesis. Writing an online judge in `judges/` is improve-classify's. When a case needs the container runner is in [harbor](references/harbor.md). ## 1. Name the case Every implementation names exactly one case from `evals/cases/.md` (`eval-case`). That id becomes the result's `id` in `run.json`. A case has zero or one implementation, on one runner: never write both a Vitest and a Harbor implementation of a case. To measure the same situation on the other runner, write a new case with its own id. A Vitest row without `case` fails the run; a Harbor task without one takes its folder name as the case id. Then: ```sh introspection check ``` What each of its case errors and warnings means is in [checks](references/checks.md). ## 2. Lay out evals/ Each runner's implementations live in their own folder under `evals/`, named in its reference. Code several cases share goes only in `evals/lib/`. Never add an eval key to the package manifest: an eval is versioned in Git beside the Recipe, not declared as a Recipe resource. ```text evals/ cases/.md the case files / implementations, one folder per runner lib/ code several cases import sources/ pinned generators of derived eval data ``` Runs land in `eval-runs//` at the Recipe root. Add `/eval-runs/` to `.gitignore`: they are generated results, not source. Nothing eval-only reaches a running agent. Exclude every eval path in `.recipeignore`; one `evals/` line does it. `package.json#pi.evals` is optional and overrides the default locations. Declaring `"evals": {}` keeps the defaults and turns on recipe-check's layout checks: an implementation folder missing its config file, and eval files outside every eval path. Each eval keeps only authorized fixtures and setup, and resolves every path relative to its suite or task: never a laptop path, a temp folder, an expired upload or a private URL. A reference output is evidence for the verifier only. Keep it with the verifier, never in what the agent receives; the runner reference says which folder reaches the agent. Never call a deployed MCP endpoint from an eval, and never leave a background process or database running after it. A missing local fixture is a setup failure, not something to route around. The fakes are built in `eval-simulate`; bind them with [vitest-fakes](references/vitest-fakes.md) or [harbor-fakes](references/harbor-fakes.md). ## 3. Write the verifier Expected behaviour comes from the approved case, never from what the model produced. Never derive a rubric from the candidate's output. When a case fails, read the run it scored first: fix the Recipe when the agent got it wrong, and change the verifier only when it misjudged a run that did what the case asks, keeping the case's meaning as approved. Pick the verification for each failure mode in this order: deterministic code for exact invariants, then a reviewed semantic check against an approved rubric, then human review for disputed policy. Prefer observable outcomes (the output, structured results, the effect of a tool on fake state) over requiring one internal tool path. Read tool calls through the runner's helper, never from raw events or guessed MCP tool names. Regexes, keywords, substrings, edit distance and exact prose are proxies for meaning. Use them only when the literal text is the approved requirement; otherwise redesign the case so it exposes state a check can inspect. "No state changed" compares the initial and final state of every entity a fake's write map says some tool writes, outbound effects included. A tool's `readOnlyHint` proves nothing. "No write was attempted" is a separate check, named and scored on its own. The runners' write-attempt helpers are attempt checks: name and score one beside the state check. Both helpers raise on an empty set of manifests rather than pass. A fake answers in the real server's format (TOON, prefixed JSON or text), so never parse a tool result as JSON yourself. Check state and each call's name and arguments, and decode an answer with the fakes' contract helpers. An answer outside the fake's contract is the fake's fault: an invalid measurement, not a failed criterion. A tool that states an `outputSchema` can't be read from the agent's evidence yet, so check it through state. Shape each non-trivial verifier around four cases: a clear pass, a clear fail, an acceptable variation that must pass, and a plausible shortcut that must fail. Label those checks synthetic. Set an explicit timeout for each case's model work. When one hits, diagnose whether it was setup, execution or the case's contract; never raise a behavioural deadline to turn a failure into a pass. ## 4. Choose the judge A model-graded check is a judge. Reuse an online judge before defining your own. 1. **Reuse an online judge by name.** An online judge is a definition in the Recipe's `judges/`, written and calibrated on production turns by improve-classify, and run by the platform on live conversations. When one decides what the case asks, call it by its `name` as the runner reference says. 2. **Otherwise define the judge in the suite.** A judge only the eval needs is part of its suite: in Vitest a `createJudge` judge or a judge YAML in `evals/vitest/judges/` ([vitest](references/vitest.md)); in Harbor a RewardKit `[judge]` TOML in the task's `tests/` ([rewardkit](references/rewardkit.md)). **Its model is the agent's own:** the `ai.model` in the agent's `agents/.yaml`, never the model you run on. Never write a judge only the eval needs into `judges/`: the platform grades every judge there, apart from mission judges, on production. The fields are in Sentry vitest-evals' README, RewardKit's docs and `@introspection-ai/recipes`' `docs/recipe-judges.md`. ## 5. Produce the rewards Every verifier produces two objects: - **`reward.json`:** a flat object of scores, each a finite number from 0 to 1. `reward` is the case's score; every other key is one dimension, named for the requirement it checks. A count or total is not a score and makes the case `reward_invalid`. - **`reward-details.json`:** optional, the reasons: each dimension's criteria with `name`, `value` and `reasoning`. Not applicable is per criterion, never a whole case. The full contract, and why a case with only assertions is incomplete, is in [rewards](references/rewards.md). ## 6. Run it ```sh introspection eval create # scaffold a local suite introspection eval run -p # run it, as eval-run says ``` Each runner's layout and mechanics have the same sections, so they read side by side: - [vitest](references/vitest.md) and [vitest-fakes](references/vitest-fakes.md): the local runner, and binding the fakes in it. - [harbor](references/harbor.md) and [harbor-fakes](references/harbor-fakes.md): the container runner, and serving the fakes to its trials. - [rewardkit](references/rewardkit.md): the container runner's verifier library. Outside an Operator session, `introspection evals author/` prints a reference. ## Done when - The implementation names one approved case, and `introspection check` reports no case error. - Its verifier writes a valid `reward.json` with at least one score, from what the case asks. - Hand back to the step that asked for the eval.