# Evals

Golden-set conventions, rubric design, and pass thresholds — the measurement half of eval-first. The loop that applies them is the `prompt-eval` skill; the certification they produce is recorded per the `prompt-library` skill.

## Golden sets

- One set per prompt family at `evals/{family}/` — items in `cases.md` (or `cases/*.md` when large), the rubric in `rubric.md`, the threshold declared at the top of the rubric.
- **An item is input + expected properties**, not input + one blessed output — score what must be true of the answer (fields present, grounded in the supplied context, refusal when the input demands one), never string equality on prose.
- **Edge cases are the set's value**: adversarial inputs, missing-information inputs, inputs that *should* trigger the fallback. `data-generator` drafts them; a human reviews before they join the set — a wrong expected property poisons every future run.
- Sets are append-mostly: production failures become new items the same week they're found. Removing an item is a design change to the family and is recorded like one.

## Rubrics

- Every dimension gets **anchors** — a written example of a 1, a 3, and a 5 — or graders drift; a bare "score helpfulness 1–5" is not a rubric.
- **Mechanical before judgment**: anything checkable in code (format validity, required fields, length bounds, banned content) is checked in code and gates before any graded dimension spends a model call.
- Judgment dimensions are graded at the economy tier against the anchors; grader-vs-human disagreement above ~1 anchor step, recurring, is a rubric defect — fix the anchors, then re-run.
- **Rubric changes re-certify the family.** A rubric edit invalidates every score produced under the old rubric; never tune a rubric and a prompt in the same change.

## Thresholds & determinism

- Each family declares one pass threshold (house default: **0.90** weighted across dimensions; safety-critical dimensions are pass/fail gates, not weighted terms).
- Runs are reproducible: fixed item order, temperature 0 where the task allows, the full set every time — a sampled run is a smoke check, never a certification.
- A certified score always names its context: prompt version, tier, rubric version, date. Scores without all four are not comparable and never quoted.