--- name: eval-rubric description: Author an evaluation rubric for an initiative — the measurable definition of "good" used to judge agent output. Use before any build starts; the eval is written first, paired with the PRD. version: 1.1.0 owner: wawan risk: low category: eval scope: read:okf, write:planning/evals --- # eval-rubric — Author an Evaluation Rubric This skill produces an eval rubric instance for one initiative. In eval-driven PM the rubric **is the spec made measurable** ([eval-driven-pm](/okf/core/concepts/eval-driven-pm.md)): it is authored *before* the build, alongside the [PRD](/skills/prd.skill), and it defines how output will pass through the [three-tier gate](/okf/core/concepts/output-eval.md). > **Format note:** three-level progressive disclosure > ([SKILL-FORMAT](/skills/SKILL-FORMAT.md)). Level 1 frontmatter above is the trigger and > carries only `name` + `description`. Note: the *rubric instances* this skill produces carry their > own frontmatter (including `eval_type`, below) — that is the rubric's frontmatter, not this skill > file's. ## The three-tier gate vocabulary A rubric is written with the gate it serves in mind. Name the tiers explicitly; they are not interchangeable: - **Quality Gate** — *deterministic, automated.* Compile, lint, unit tests. Toolchain-enforced pass/fail. No human judgment. A rubric's Quality-Gate criteria must be machine-checkable. - **Review Gate** — *probabilistic, adversarial.* Does the output satisfy the rubric's criteria? Applied by a separate evaluator agent or by the PM as critic. This is where the weighted dimensions below are scored. Treat its verdict as **uncalibrated** until validated ([evaluator leniency](/okf/core/concepts/output-eval.md)). - **Acceptance Gate** — *subjective, strategic.* Is this the **right** thing? Human PM only. **This is NOT a re-check of the rubric.** The PM's role at the end of every agent run **is** the Acceptance Gate — a strategic-fit decision, not a re-scoring of dimensions. Make this explicit in every rubric: state which criteria belong to which gate, and state that the Acceptance Gate is the PM's and is not graded by the rubric. ## Required frontmatter on the rubric instance Every rubric document carries, at minimum: - **OKF frontmatter** — the file opens with `type: Rubric`, `title`, `timestamp` (repo-wide convention, D63; [frontmatter-format](/okf/core/concepts/frontmatter-format.md)). - **`eval_type: capability | regression`** — - `capability` — tests whether the agent can do something new / harder. Used to push the frontier. - `regression` — guards an already-passing capability against backsliding. Saturated capability rubrics retire into the regression suite. - `initiative_id`, `product_slug`, `pm_slug`, parent `KR`. ## Grading rules 1. **Outcome-only grading.** Criteria check **what the agent produced**, not the path it took. **Do not grade tool call sequences**, intermediate reasoning, or how many steps were taken. A correct outcome by an unexpected route still passes. 2. **Weighted dimensions with partial credit.** Each rubric dimension gets a **weight**. Each is scored (not just pass/fail), the weighted scores are **summed to a total**, and the total is compared against a **pass threshold**. **Partial credit is required** — a rubric that is all-or-nothing per dimension is malformed. See [eval-rubric-template](/okf/core/templates/eval-rubric-template.md) for the scoring math. 3. **Pre-commit the threshold.** The pass threshold is set when the rubric is written, before output exists — not tuned afterward to match what came back. ### Worked shape | Dimension | Weight | Score (0–1) | Weighted | |-----------|-------:|------------:|---------:| | Correctness | 0.40 | … | … | | Completeness vs PRD | 0.30 | … | … | | Conformance (format/spec) | 0.20 | … | … | | Clarity | 0.10 | … | … | | **Total** | **1.00** | | **vs threshold** | ## Sprint contract (pre-run contract) Before any work begins, the generator and evaluator **agree on done-criteria** — this is the contract stage of the [SDLC loop](/okf/core/concepts/sdlc-loop.md). A rubric is incomplete without one. **Reach the criteria by grilling, not assumption** (the [prd skill](/skills/prd.skill)'s step-0 discipline applies here too): where the PM's intent is ambiguous, ask **one question at a time with a recommended answer**, stop at unambiguous — no cap, and a batch questionnaire is not compliant. If a fact can be found by exploring the environment, look it up rather than asking — the decisions are the PM's (D12). **Write criteria for durability:** behavioral contracts and interfaces, never file paths or line numbers (they stale while work waits); each criterion independently verifiable. **Format:** ``` initiative_id: run_id: done_criteria: 1. 2. ... test_method: 1. 2. ... PM_approved: true | false ``` - **P0 (manual):** write the contract to a temp file and get explicit **PM approval** before work starts. `PM_approved` must be `true` to proceed. - **P1+:** the contract is stored in `agent_runs.pre_run_contract`. The contract and the rubric are consistent by construction: every `done_criterion` maps to rubric dimensions, and `test_method` states the gate (Quality/Review) that checks it. **Executable test_methods first (D58).** For feature-lane work, each `test_method` SHOULD be an *executable* check written **before** the build (a test, a script, a reproducible command) — the code-level face of eval-first, and what keeps a fresh repo's Quality Gate from being silently empty. Where an executable check is not feasible, say why in the contract — the deviation is stated, never silent. **Rule-coherence criteria are semantic, not pattern (calibration lesson).** A criterion asserting a rule "holds everywhere" must (a) **enumerate the live-doc set** it covers — every living mirror of a live surface: skills, templates, playbooks, concepts, protocol docs, tool schemas AND their reference/data-model mirrors, UI copy, intake instruments — and (b) set its `test_method` to a **semantic pass** — reading those docs for the rule's *meaning*. A grep sweep may be cited as supporting evidence but can never be the criterion's definition of done: in the health-anchors run, five successive sweep patterns each reported "clean" and were each wrong (stale rules hid in wraps, concat seams, unswept sets, and keyword-less phrasings — see [eval-calibration](/planning/evals/eval-calibration.md)). A semantic pass is a **sanctioned non-executable test_method** — it satisfies the D58 executable-first rule's escape clause by construction; no per-contract justification needed beyond naming the doc set. **Dimensions must be evidence-citable.** The Review Gate scores each dimension with a verbatim quote or precise pointer from the artifact as evidence (an unevidenced dimension is `unverified`, scored ≤ 0.5 — see [evaluator.skill](/skills/evaluator.skill)). Write every dimension as an **observable outcome a quote can prove** — "the dry-run verdict names its exemplars" — never an impression ("the verdict feels thorough"). If you cannot imagine the evidence quote for a dimension, rewrite the dimension ([llm-judge-calibration](/planning/research/llm-judge-calibration.md)). ## Output and placement - Rubric instances are **operational** — they live under `/planning/evals/`, not in the OKF bundle. - Filename: slugged to the initiative, e.g. `p0-eval-rubric.md`. ## Quality bar Before handoff: `eval_type` set; every dimension weighted; weights sum to 1.0; partial credit possible; threshold pre-committed; gate assignment stated per criterion; Acceptance Gate marked as PM-owned and not rubric-graded; any rule-coherence criterion enumerates its live-doc set and tests by semantic pass; every dimension evidence-citable (an observable outcome a quote can prove); sprint contract present with `PM_approved`. ## Level 3 sub-cases - Calibrating a leniency-prone evaluator against [eval-calibration.md](/planning/evals/eval-calibration.md): see `eval-calibration-procedure.md` (author on first need). - Rubrics for non-code deliverables (docs, strategy): see `eval-rubric-nontext.md` (author on first need).