--- name: eval-author description: "Use when writing or changing an eval implementation and its verifier: naming the case it measures, choosing what to check, and producing reward.json and reward-details.json that the CLI reads into run.json." --- # eval-author Needs CLI 0.67.0 or later, which reads the rewards contract below and reports a broken one as `reward_invalid`. An implementation is how one runner executes a case. A verifier scores one run of it. This step is written once for both runners; how each runner lays out an implementation and hands its rewards over is in its reference. ## What this step covers Naming the case, choosing checks, the two rewards objects every verifier produces, and the layout of `evals/`. When a case needs the container runner rather than the local one is in `author/harbor`. ## What it does not cover Which cases get an implementation, which runner a loop uses, and what result passes are the loops' judgment. build-draft writes the first implementations; improve-classify writes the one that tests a hypothesis. ## Name the case Every implementation names exactly one case from `evals/cases/.md` (`eval-case`). That id becomes the result's `id` in `run.json`. A case has zero or one implementation per runner. An implementation that names no case keeps the id it had before schema 10 (see the runner reference), and `run.json` records that id as `legacy_id` on every case either way, so `compare` can still join old records. ## Lay out evals/ Each runner's implementations live in their own folder under `evals/`, named in its reference. Code several cases share goes only in `evals/lib/`. Never add an eval key to the package manifest: an eval is versioned in Git beside the Recipe, not declared as a Recipe resource. ```text evals/ cases/.md the case files / implementations, one folder per runner lib/ code several cases import sources/ pinned generators of derived eval data ``` Runs land in `eval-runs//` at the Recipe root. Add `/eval-runs/` to `.gitignore`: they are generated results, not source. Nothing eval-only reaches a running agent. Exclude every eval path in `.recipeignore`; one `evals/` line does it. `package.json#pi.evals` is optional and overrides the default locations. Declaring `"evals": {}` keeps the defaults and turns on recipe-check's layout checks: an implementation folder missing its config file, and eval files outside every eval path. Each eval keeps only authorized fixtures and setup, and resolves every path relative to its suite or task: never a laptop path, a temp folder, an expired upload or a private URL. A reference output is evidence for the verifier only. Keep it with the verifier, never in what the agent receives; the runner reference says which folder reaches the agent. Never call a deployed MCP endpoint from an eval, and never leave a background process or database running after it. A missing local fixture is a setup failure, not something to route around. The fakes are built in `eval-fakes`; bind them with `author/-fakes`. ## Write the verifier Expected behaviour comes from the approved case, never from what the model produced. Never derive a rubric from the candidate's output. Pick the verification for each failure mode in this order: deterministic code for exact invariants, then a reviewed semantic check against an approved rubric, then human review for disputed policy. Prefer observable outcomes (the output, structured results, the effect of a tool on fake state) over requiring one internal tool path. Read tool calls through the runner's helper, never from raw events or guessed MCP tool names. Regexes, keywords, substrings, edit distance and exact prose are proxies for meaning. Use them only when the literal text is the approved requirement; otherwise redesign the case so it exposes state a check can inspect. "No state changed" compares the initial and final state of every entity a fake's write map says some tool writes, outbound effects included. A tool's `readOnlyHint` proves nothing. "No write was attempted" is a separate check, named and scored on its own. The runners' write-attempt helpers are attempt checks: name and score one beside the state check. Both helpers raise on an empty set of manifests rather than pass. A fake answers in the real server's format (TOON, prefixed JSON or text), so never parse a tool result as JSON yourself. Check state and each call's name and arguments, and decode an answer with the fakes' contract helpers. An answer outside the fake's contract is the fake's fault: an invalid measurement, not a failed criterion. A tool that states an `outputSchema` can't be read from the agent's evidence yet, so check it through state. Shape each non-trivial verifier around four cases: a clear pass, a clear fail, an acceptable variation that must pass, and a plausible shortcut that must fail. Label those checks synthetic. Set an explicit timeout for each case's model work. When one hits, diagnose whether it was setup, execution or the case's contract; never raise a behavioural deadline to turn a failure into a pass. ## Produce the rewards Every verifier produces two objects, whichever runner and language it uses. The runner reference says how they reach the CLI. **`reward.json`: scores only.** A flat object. ```json { "reward": 0.5, "tools": 1.0, "output": 0.0 } ``` - Every value is a finite number from 0 to 1. A boolean, a string or a number outside 0 to 1 is rejected. - An empty object `{}` is accepted only when `reward-details.json` shows that every dimension did not apply: the case is then complete with no score. Otherwise `{}` makes the case `reward_invalid`. - `reward` is the case's score. Without it, the CLI uses the plain mean of the other keys and marks the case `score_derived`. - Every other key is one dimension, named for the requirement it checks. - **Any number that is not a score, such as a count or a total, makes the case `reward_invalid`**, with a message naming the case, the key, its value and the rule it broke. Keep such numbers in the verifier's own artifact. recipe-redlining moved its counts to `grade.json` for this reason (recipe-redlining#14). **`reward-details.json`: the reasons, optional.** One entry per dimension, each holding its criteria with `name`, `value`, `reasoning`, and `optional` and `error` for a criterion that may not apply. The CLI copies them into `run.json` as each score's `criteria[]`. A missing or unreadable details file never makes a case incomplete; it only costs the reasoning. **Not applicable is per criterion.** A criterion that didn't apply is `optional: true` with an `error` saying why; the CLI records it with no score and the status `not_applicable`. A dimension that `reward.json` leaves out and whose criteria are all not applicable is recorded the same way. A case has no "not applicable" status: it is complete with its scores, complete with no score when every dimension did not apply (`reward.json` is then `{}`), or incomplete with an error. Every case records at least one score through its verifier. A case whose only checks are assertions records none: it is incomplete, and never counts as a pass or a regression. **The contract version.** Each producer declares the version of this contract it writes; none means 1, the only version today. A version the CLI doesn't know makes the case `contract_unsupported`, and the message says to upgrade the CLI. ## Commands - `introspection eval create `: scaffold a local suite (`author/vitest`). - `introspection eval run -p `: run it (`eval-run`). - `references/.md`: the runner's layout and mechanics. ## References Each is a file in `references/`; outside an Operator session, `introspection evals author/` prints it. Each has the same sections, so they read side by side. - [`vitest`](references/vitest.md): the local runner: dataset rows that name a case, code and judge checks, and the `rewards()` helper. - [`vitest-fakes`](references/vitest-fakes.md): binding the fakes in the local runner with `fakesHarness`, a scenario per case, and scoring their state and answers. - [`harbor`](references/harbor.md): the container runner: task folders, `test.sh`, multi-step tasks, and when a case needs it. - [`harbor-fakes`](references/harbor-fakes.md): serving the fakes to a container trial: the sidecar image, the compose service, `task.toml` bindings and the verifier's view. - [`rewardkit`](references/rewardkit.md): the container runner's verifier library: score folders, criteria, the summary reward, and what the CLI reads from its output.