# Calibration Reference — Blind Re-runs · Disposition · The Fleet Ledger The reference for the harness's **only external signal**. Everything else in this harness is self-assessment: collectors grade their own `confidence` and `completeness_pct`, Evaluation weights those self-grades, and Mode 5 scores the result against a checklist the same system authored. That loop can converge on a corpus that is confidently, well-provenanced, and wrong. **Calibration is the one procedure that can tell you whether any of it is true.** It re-derives claims a prior run published, blind, and records what survived. > **The measured position, as of 2026-07-28.** One target has ever been re-run (`emergent`, 2026-06-25 → > 2026-07-28). Of 16 claims re-tested: **7 refuted, 4 corrected, 5 confirmed** — a 44% flat-falsification > rate — and **4 of the 7 refutations came from lanes the prior run had graded `confidence: high`.** The > prior run had self-scored **88/100 "strong"** and `stopped_on: converged`. > > That is **n=1**. It is the entire empirical basis of this harness. Every rule in `.claude/rules/` is > currently a hypothesis with n=0 on any target it did not come from. The purpose of this file is to make > n grow. --- ## Two tiers — pick the one that will actually happen A metric too expensive to run leaves `n` at 1 forever. The cheap tier is the important one. | Tier | Cadence | Cost | What it measures | | --- | --- | --- | --- | | **A — Claim spot-check** | monthly | ~1 session, no full ingestion | the refutation rate, and whether `confidence` tracks truth | | **B — Full blind re-run** | quarterly, or when a rule set changes materially | a complete run | everything Tier A does, plus coverage drift and whether new rules changed behaviour | **Tier A is the default. Do not skip it waiting for capacity to do Tier B** — a monthly spot-check of 15–20 claims across one target produces the regression that matters, and produces it twelve times a year. --- ## Selection — rotate, never choose **The target is selected by rotation, not by judgment.** Take the least-recently-calibrated target in the corpus root; break ties by oldest `captured_at`. Record the selection rule alongside the result. A human (or a run) choosing which target to re-check will choose the one it feels good about. That single substitution destroys the metric, because the sample stops being representative of the fleet and starts being representative of the harness's confidence — which is the very thing under test. --- ## THE BLIND RULE (this is what makes it evidence) **Capture first. Read the prior corpus only after your own observation is written down.** 1. Select the target. **Do not open its corpus.** 2. Draw the claim sample (below) — a list of *questions*, not answers. Someone or something other than the re-deriving agent extracts the questions, or the questions are extracted mechanically from claim anchors, so the sample carries no prior conclusions. 3. **Re-derive each claim from the live product**, from scratch, recording what you observe and the method. 4. **Only then** open the prior corpus and diff. > **This deliberately overrides `discovery.md` Step 00 for calibration runs.** Step 00 tells a run to find a > prior corpus, restore it, and reconcile against it — correct for a normal re-run, where the prior corpus > is a genuine corroboration lane and re-deriving everything from zero is waste. **It is wrong here.** A run > that reads the prior claim first will anchor on it: it will look for confirmation, find it, and report a > confirmation rate that measures its own suggestibility. Anchoring is the specific failure this protocol > exists to avoid, so for a calibration run Step 00 is deferred until after step 3, and the run records > `blind: true`. > > If a run cannot honestly attest that it re-derived before reading, it must record `blind: false` — and > the result is then a **consistency check, not a calibration**, and does not enter the ledger's refutation > rate. **Sampling the claims.** 15–20 per target, drawn to make the regression computable, not to be flattering: - **Stratify by the prior run's stated `confidence`** — roughly equal numbers of `high`, `medium`, `low`. Without this the regression cannot be computed, and the regression is the point. - **Include the load-bearing ones** — claims a README headline rests on, and claims a reader would act on (pricing, limits, guarantees, auth model, what the product can and cannot do). - **Include at least 3 negative claims** ("no public SDK", "no realtime layer", "not supported"). Negatives are the harness's demonstrated weak spot: **5 of the 7 refutations in the one data point were negatives.** - **Exclude** anything whose truth cannot change and cannot be re-checked (a historical funding round). --- ## Disposition — five values, and the fifth is the one the old schema lacked Every sampled claim resolves to exactly one: | Disposition | Meaning | What it says about the harness | | --- | --- | --- | | **confirmed** | independently re-derived, same conclusion | the method worked | | **corrected** | right direction, wrong detail (a count, a version, a name) | the method worked; the reading was imprecise | | **refuted** | the claim was **wrong when it was made** | **the method failed — this is the signal** | | **stale** | the claim was **true when made; the product changed** | the method worked; the corpus has a shelf life | | **unverifiable** | cannot be re-checked now (surface gone, plan changed, gated) | no signal either way; excluded from the rate | > **`refuted` and `stale` are the distinction the original artifact had no slot for, and they mean opposite > things.** A high `refuted` rate means *the instrument is unreliable* and the rules need work. A high > `stale` rate means *the instrument is fine and the corpus decays* — which calls for shorter shelf-lives > and `volatility` tagging, not methodology changes. Conflating them makes the harness fix the wrong > problem. When you cannot tell which it is — no dated evidence either way — record **`refuted`**. The > conservative reading is the one that keeps the harness honest about itself. > > (Origin: `emergent/00-prior-run-reconciliation.md` — a genuinely good artifact whose section A had to be > titled *"REFUTED (prior claim was wrong, not merely stale)"* precisely because the schema gave it nowhere > else to put the distinction.) **The refutation rate** = `refuted ÷ (confirmed + corrected + refuted + stale)`. `unverifiable` is excluded from the denominator, and its count is reported so a shrinking sample is visible. --- ## The regression — the actual question For each calibration, record the disposition **against the prior run's stated confidence for that claim**: | prior confidence | n | confirmed | corrected | refuted | stale | refutation rate | | --- | --- | --- | --- | --- | --- | --- | | high | | | | | | | | medium | | | | | | | | low | | | | | | | **If `high` claims are refuted at a similar rate to `low` claims, the `confidence` field carries no signal and `ingestion.md` §3's method→confidence table is decorative.** That is the single most consequential thing this harness could learn about itself, and it is unknowable without this table. Also record refutation rate **by `method`** once n is large enough — that tells you which of the 17 methods in §3 are over-graded, which is directly actionable (a method's band gets lowered). --- ## Artifacts ### Per calibration — `research//00-prior-run-reconciliation.md` Formalizes the shape `emergent` invented. **This file is part of the output contract** whenever a prior corpus exists. ```markdown --- target: reconciled_at: prior_run_date: prior_run_commit: prior_run_score: tier: A | B # spot-check or full blind re-run blind: true | false # false ⇒ consistency check only, excluded from the rate selection_rule: rotation # or: named-by-user (record it either way) sampled: confirmed: corrected: refuted: stale: unverifiable: refutation_rate: <0-100> by_confidence: # the regression — REQUIRED high: { n: , refuted: } medium: { n: , refuted: } low: { n: , refuted: } --- ## A. REFUTED — wrong when made (claim · what was observed · lane that refuted it · prior confidence) ## B. STALE — true when made, changed (claim · what changed · evidence of the change being NEW) ## C. CORRECTED — right, imprecise ## D. CONFIRMED — independently re-derived ## E. UNVERIFIABLE — and why ## F. What this says about the METHOD (which lanes/methods produced the refutations, and whether a §3 confidence band should be lowered) ``` ### Fleet-level — `research/_calibration/ledger.md` **The metric itself.** One row per calibration, accumulating across targets and time. This is the file to read when asking "does this harness work?" It is never rewritten, only appended to. --- ## What a result CHANGES (otherwise this is just bookkeeping) A calibration that produces a number and no action is theatre. Each outcome routes: | Finding | Action | | --- | --- | | A **method** shows a high refutation rate | lower its band in `ingestion.md` §3, and record the calibration as the origin | | **Negative claims** refute disproportionately | tighten the absence rules (`ingestion.md` §7 rule 10, `discovery.md` Part B) — the `blind_to:` direction | | `high` refutes ≈ as often as `low` | **the confidence field is not carrying signal** — this is a structural finding about the harness, and it goes to the user, not into a rule tweak | | A **rule added after the prior run** would have prevented a refutation | evidence the improvement loop works — record it, it is the only such evidence that exists | | A refutation recurs **after** a rule was added to prevent it | evidence a rule did **not** change behaviour; escalate to the user rather than adding another rule | | High **stale** rate | shorten shelf-life expectations and tag volatile claims; **not** a methodology problem | --- ## How to run one (Tier A, the monthly default) 1. **Select** by rotation from `research/_calibration/ledger.md`. Record the target and the rule. 2. **Extract the sample** mechanically — pull claim anchors from the target's `evaluation/*.md`, stratify by the cited confidence, and write them as **questions with the answers stripped**. 3. **Re-derive blind.** Do not open the corpus. Use whatever lanes the access vector allows; record method and observation per claim. 4. **Diff and disposition** each claim into the five values. 5. **Write** `00-prior-run-reconciliation.md` (append a new dated section if one already exists — never overwrite a prior calibration). 6. **Append one row** to the fleet ledger, and update its running totals. 7. **Route** every finding through the table above. Anything that becomes a rule change goes through the normal Mode-5 approval gate. **Cost/state discipline is unchanged:** a calibration is read-only unless the user authorizes otherwise; a claim that can only be re-derived by spending money or mutating state is recorded `unverifiable` with the reason, never quietly exercised. --- ## How the harness uses this reference Calibration is a **cross-run process**, not a mode of a single run — nothing in Modes 1–5 triggers it, and it must not be folded into them (a run cannot calibrate itself; that is the circularity it exists to break). Mode 5 reads the fleet ledger as *context* when weighing a proposal — a rule contradicted by calibration data should not be promoted — but Mode 5 never writes to it. > **Do not attach calibration to the Mode-5 score.** The moment `refutation_rate` becomes a rubric input, > the run acquires an interest in the number, and the one uncorrupted signal in the system is gone. It > measures the **harness**, not the run. Publish it; never grade with it.