--- name: eval-case description: "Use when writing or changing an eval case's goal file, evals/cases/.md, or adding a case from a production conversation or a local session: its front matter, approval, the conversation it came from, and what the case must record." --- # eval-case Needs CLI 0.67.0 or later, whose `run.json` keys results by the case id an implementation names. A case is one situation in the owner's words, in one file: `evals/cases/.md`. Every implementation of the case, on any runner, names it by that id, and every result in `run.json` is keyed by it. ## What this step covers The case file's format, how a case is approved and reopened, and how a case is added from a real conversation. Writing the implementation and its verifier is `eval-author`. ## What it does not cover Which cases to write, how to get the owner's words, and when a case is ready are the loops' judgment: build-brainstorm writes cases with the owner, improve-classify writes one per hypothesis. This step is the format they share. ## The case file ```markdown --- id: C3 title: Counterparty adds an uncapped indemnity inputs: - name: contract.docx sha256: 9f2c… location: owner upload, 2026-11-02 chat sources: [documents] check: code approved_by: Jane Doe approved_at: 2026-11-02T10:42:00Z approved_in: chat message 01a19d7a… conversation: # only for a case from a real conversation id: 019a2b3c-… task_id: 019a2b3c-… # only when the conversation ran as a platform task turn: 3 issue: 019a2b40-… # only when improve records the hypothesis as an issue --- ## Situation The other side's draft adds an indemnity with no cap. ## A good result The indemnity is struck as a tracked change and replaced with mutual, capped wording, with a margin comment giving the reason. ## A bad result The clause is left as is, or rewritten without a tracked change. ## Verification Code checks the tracked change and the comment; a person reviews whether the replacement wording is acceptable. ``` - **The id is the file name and never changes.** A case keeps its id when edited, and a removed case's id is not reused. Ids are unique across the Recipe, not per suite. - **The case never names a path.** Implementations point to the case, never the other way round. - **`check`** says how the case is judged: `code` (a verifier), `person` (the owner or a reviewer) or `later` (left for improve once real conversations exist). Only `check: code` cases get an implementation. - **Inputs are referenced, never copied.** Record each input's hash and where it came from. Synthetic inputs and fakes you create yourself may be committed under `evals/cases/inputs/`. - **`id` and `title`**: the case id, the same as the file name, and a one-line title in the owner's words. - **`sources`** lists what the agent must reach for this case. It is not where the case came from; that is `conversation`. - **`conversation`** (optional) records the production conversation a case came from: its `id`, the `task_id` when it ran as a platform task, and the `turn`. It is provenance only; no run reads the conversation (below). - **`issue`** (optional) names the `/v1/issues` row of the hypothesis the case tests. It is used only if the Operator keeps Issues (introspection-operator decision D2, open), and the issue id is never the case id: a case exists without a login and outlives its hypothesis as a regression test. - **The body** has three sections the owner reads: Situation, A good result, A bad result. A fourth, Verification, records how the case is checked. Record each case's required outcomes, its unacceptable outcomes, and the variation that is still acceptable. A semantic criterion nobody has resolved yet is recorded as needing human review, not guessed. Write the case's input the way a real user would send it: the user's goal, with production's brevity and ambiguity. Never prescribe tools, reveal how it is scored or restate the outcome. Record synthetic provenance in metadata the agent never sees, not in the input. ## Approval The approval fields (`approved_by`, `approved_at`, `approved_in`) stay empty until the owner approves the case. A case with empty approval fields is a draft. A draft with an implementation still runs, and the owner approves it when reviewing the pull request that adds it. Per case, the record holds its id, input, the failure mode it tests, the expected outcome, rationale and provenance, how it is verified, who approved it and the dataset version. Reopen approval on any material change or relabel: clear the approval fields and get them again. A run derived from production stays exploratory until its expected behaviour and its storage are reviewed and approved as a durable case. Replaying a conversation with `introspection eval run --from-conversation` is exploration, not a case (`eval-run`). ## A case from a real conversation Never copy a production transcript into Git just because it helped. Before committing a production-derived case, confirm it is authorized for repository storage and free of secrets and customer data that can't be committed. After the owner approves and the data is cleared, add the case with `introspection eval add --from-conversation --turn `. `--expected pass|fail|not-applicable` also appends the completed trajectory to the same-name judge's working set, for calibrating that judge (`--judge ` when the names differ). One turn is one case. Select it with `--from-conversation --turn ` (1-based), never with a model span id or events spliced from different turns. `--from-session [--turn ]` does the same for a local session file that `introspection local` wrote. `--trace-id` still works but is not the form to use. A case added this way holds the completed history before the selected turn plus that turn's prompt. It omits the completion that was observed, which the agent must regenerate when the case runs. The committed copy is what every later run uses: a case keeps running after the conversation passes retention, and no run needs access to conversations. **Not yet.** Today `eval add` writes only the local runner's implementation (a dataset row) and no case file. Writing a draft `evals/cases/.md` with a `conversation: { id, task_id, turn }` field, `--runner` to choose the implementation, and `--case-id` (default the next free `C`) are not built yet. Until then, write the case file yourself and name its id in the implementation (`eval-author`). ## Commands - `introspection eval add --from-conversation --turn [--case-name ] [--expected pass|fail|not-applicable]` - `introspection eval add --from-session [--turn ]` - `introspection eval add --help` for the source options and what each writes.