--- name: eval-case description: "Writes or changes an eval case file, evals/cases/.md, records the owner's approval of it, and adds a case from a production conversation or a local session. Use when a case is created, edited, approved or reopened." metadata: min-cli-version: "0.79.0" next: "" --- # eval-case A case is one situation in the owner's words, in one file: `evals/cases/.md`. Its one implementation, on Vitest or Harbor, names it by that id, and every result in `run.json` is keyed by it. Writing the implementation and its verifier is `eval-author`. Which cases to write, how to get the owner's words, and when a case is ready are the loops' judgment: build-brainstorm writes cases with the owner, improve-classify writes one per hypothesis. This step is the format they share. ## 1. Write the case file Start from [the case file reference](references/case-file.md): a complete example and every field. The front matter holds `id`, `title`, `inputs`, `sources`, `check`, the approval fields and, for a real conversation, `conversation`. The body has Situation, A good result, A bad result and Verification. - **The id is the file name and never changes.** A case keeps its id when edited, and a removed case's id is not reused. Ids are unique across the Recipe, not per suite. - **The case never names a path.** Implementations point to the case, never the other way round. - **`check`** says how the case is judged: `code` (a verifier), `person` (the owner or a reviewer) or `later` (left for improve once real conversations exist). Only `check: code` cases get an implementation. - **Inputs are referenced, never copied.** Record each input's hash and where it came from. Synthetic inputs and fakes you create yourself may be committed under `evals/cases/inputs/`. Record each case's required outcomes, its unacceptable outcomes, and the variation that is still acceptable. A semantic criterion nobody has resolved yet is recorded as needing human review, not guessed. Write the case's input the way a real user would send it: the user's goal, with production's brevity and ambiguity. Never prescribe tools, reveal how it is scored or restate the outcome. Record synthetic provenance in metadata the agent never sees, not in the input. ## 2. Approval The approval fields (`approved_by`, `approved_at`, `approved_in`) stay empty until the owner approves the case. A case with empty approval fields is a draft, and so is a Harbor case whose `tests/test.sh` is still the stub `eval add` wrote. **A draft may run; only a measurement needs approved cases.** Run a draft to try it out: to see it fail, check its verifier or show the owner what it does. A run whose numbers count is a measurement: a baseline, a comparison, the proof that a change works, or anything reported as a case's result. Every case in a measurement is approved, with its verifier written: a draft in it measures nothing the owner approved. The CLI holds you to it: `eval baseline promote` refuses a run with a draft, and `eval compare --fail-on draft` fails on one, each naming the case and what it lacks. So get the approval, or take the draft out of the suite, before a measurement, and record it before the run, not after: `eval approve` edits the case file, which changes the case's fingerprint, so a run from before the approval can't be compared or promoted. A case the owner approved before it was written, such as at improve-classify's gate, is recorded as soon as its body is final. Record an approval with `introspection eval approve`, never by editing the front matter: ```sh introspection eval approve C3 --by "Jane Doe " --in "chat message 01a19d7a" ``` It writes `approved_by` (the git user when `--by` is left out; always a person, never a role), `approved_at` (now), `approved_in` and `approved_hash`, the hash of the body after the front matter. `introspection check` compares that hash, so a case whose body changed after approval is a draft again until it is approved again. Several ids approve several cases from one message. - **`--by` and `--in`:** the person and the message, found as [the approval reference](references/approval.md) says. - **Record only what the owner approved,** in an explicit message: never infer an approval, and never re-stamp a case the message didn't cover. ## 3. Reopen a changed case Per case, the record holds its id, input, the failure mode it tests, the expected outcome, rationale and provenance, how it is verified, who approved it and the dataset version. Reopen approval on any material change or relabel: clear the approval fields (`introspection eval approve --clear`) and get them again. A run derived from production stays exploratory until its expected behaviour and its storage are reviewed and approved as a durable case. Replaying a conversation with `introspection eval run --from-conversation` is exploration, not a case (`eval-run`). ## 4. A case from a real conversation Never copy a production transcript into Git just because it helped. Before committing a production-derived case, confirm it is authorized for repository storage and free of secrets and customer data that can't be committed. ```sh introspection eval add --from-conversation --turn [--runner vitest] [--case-id ] introspection eval add --from-session [--turn ] ``` After the owner approves and the data is cleared, add the case with `introspection eval add --from-conversation --turn `. For Harbor, `` is the suite's folder, `evals/harbor`, never a name of your own. It writes a draft `evals/cases/.md` with the `conversation: { id, task_id, turn }` field, and one implementation naming the case: a Harbor task by default with `--from-conversation`, or a Vitest row with `--runner vitest`. `--case-id` names the case (default the next free `C`). Then write the case file's body from the draft the owner approved. A Harbor task's `tests/test.sh` from `eval add` is a stub that scores every trial 0: write the case's verifier in its place (`eval-author`). When the owner already approved the case, as at improve-classify's gate, record that approval with `eval approve` once the body is final. Until both are done the case is a draft, which may run but measures nothing ("Approval"). `eval add` refuses a case whose file already exists, so keep the draft in the conversation until it has run, and it creates the suite when there is none yet. `--expected pass|fail|not-applicable` also appends the completed trajectory to the same-name judge's working set, for calibrating that judge (`--judge ` when the names differ). One turn is one case. Select it with `--from-conversation --turn ` (1-based), never with a model span id or events spliced from different turns. `--from-session [--turn ]` does the same for a local session file that `introspection local` wrote. `--trace-id` still works but is not the form to use. A case added this way holds the completed history before the selected turn plus that turn's prompt. It omits the completion that was observed, which the agent must regenerate when the case runs. The committed copy is what every later run uses: a case keeps running after the conversation passes retention, and no run needs access to conversations. `introspection eval add --help` lists the source options and what each writes. ## Done when - The case file has its id, inputs, check and the three sections the owner reads. - Its approval is recorded with `eval approve` from the owner's explicit message, or it is left a draft and kept out of every measurement. - Hand back to the step that asked.