# Test-audit campaign A campaign covers a whole repo or subsystem in one draft MR. Everything in [../SKILL.md](../SKILL.md) applies to every lane: the goal, the value bar, the ledger marks, the eight evidence fields, falsification, and the owner list. This file adds the order of work, the measurements, and the mutation runner. Each phase ends on its done-criterion. After every phase, write its numbers into the report file before the next one starts: a session limit can stop the coordinator itself, and a resumed session must be able to continue from the files alone (`git log`, `git status`, the ledgers and manifests under `/tmp//`). Reviewers and lane agents create their result file as a skeleton first and fill it per lane, so a run cut off by a time cap leaves a partial state instead of nothing (one campaign lost a whole P4 round that way). P3 may start as soon as the P1 file list exists and the last CI pipeline at the pinned SHA is green; B marks are reconciled when the M0 pass/fail list lands. P5 and later never start before the P4 owner checkpoint is answered. **Splitting a large campaign.** Above about 25,000 test lines or 15 lanes, P1 to P4 remain one campaign and P5 to P9 run as sub-campaigns with one draft MR each (measured once: 50k lines, 25 lanes; P3 alone took 2 h 46 min at three agents in parallel, and a 3-hour session cap cut P4 in the middle). Sub-campaigns that supply keepers to others go first; the goal applies to the sum. **Briefs.** A campaign brief (the run's instruction from an operator or a coordinating session) wins over this file wherever it is stricter: when it forbids a pre-check or a local run, that stays forbidden. Headless rules: SKILL.md § Never without the owner. ## Phases **P1 Baseline (M0).** Pin a SHA. Check the scope against the CI configuration, not against the default test script alone: a suite the pipeline runs under another script (`test:shell-suites`, integration jobs) is in scope and its lines count toward the goal, and one the pipeline never runs is reported as such (measured once: 32 lines under `pnpm test`, 7,850 under a second CI script). Check the coverage provider before anything expensive: a runner that answers `MISSING DEPENDENCY` to `--coverage` (vitest without `@vitest/coverage-v8`) makes the goal's coverage clause and the per-file backstop unmeasurable, and a provider is a dependency, so it is an owner decision. Raise it now with three options: (1) a devDependency pinned to the runner's version, (2) a transient install in the offload worktree only, (3) mutations as the only backstop, named as such in the MR. P1 to P4 continue meanwhile with the coverage clause marked not verifiable; P5 waits for the answer (3 of 6 pilot repos lacked the provider, and it surfaced after P4 each time). Record every measurement below and the pass/fail state of every in-scope test file. Baseline-red tests go on their own list as B: they are possible product bugs, not stale tests. Done when every in-scope test file has a recorded result, the coverage provider is confirmed or its owner decision is open, and the M0 table carries its commands and the SHA. **P2 Lanes.** Split the tests along production ownership (the module a test protects), not file-name prefixes. Every test file belongs to exactly one lane; the lane file counts add up to the P1 total. Cross-cutting test files (error-path or contract suites spanning several modules) form their own lane, scheduled after the owner lanes. Budget each lane by lines and declarations: at most about 2,200 test lines or about 150 declarations, whichever comes first (a single unsplittable file may exceed it). Measured over five TypeScript/vitest campaigns (2026-09-24/25): a read-only lane agent takes 10–29 min and 220–500k tokens; Bash suites cost more per line (1,000–1,600 lines, 300–400k). Each lane agent gets one run plus one retry; a third run is the budget overrun that stops the lane (SKILL.md § Stop and success). The declaration count at P2 is a grep estimate; the P3 agent's count is binding, and the runner's own test count stands beside it (a `describe` block is not a declaration, and `numTotalTestSuites` counts blocks, not files). Run at most three agents in parallel: three never hit the account limit (HTTP 429) in five campaigns, five did once. Plan `ceil(lanes / 3)` rounds of about 20 min each. Done when the lane table sums to the total. **P3 Ledger per lane.** One read-only `qa-strategist` per lane, in parallel, all from one shared brief. It reads every assigned test in full, including parameter tables, plus the production owner, its callers, history, and CI routing, and writes one mark with evidence per declaration. For a duplicate whose twin lies in another lane, the default keeper is the file the production module owns: owner file beats cross-cutting file, real boundary beats mocked collaborator, exact assertion beats `typeof`. Mark the other side C and name the keeper; never mark both sides C. When the non-owner twin must stay for a contract of its own, the owner side may be C into it only if the twin runs at a boundary at least as real; otherwise both stay R. Cite a cross-lane keeper by its declaration line, never by an assertion line inside another declaration. The cross-cutting lane receives the owner lanes' ledgers as input. `qa-strategist` has Bash: it writes `/tmp//ledger-.md` itself (heredoc) and answers with the count line, the path, and at most five sentences; the lane brief says that this path and the pre-check copy override the agent definition's report path and read-only Bash rule, and nothing else does. The coordinator assembles `docs/audits/-test-audit-ledger.md`. Suspected product bugs go into the ledger's _Issues_ section; nobody files them yet. Evidence field 8 is written as a patch file under `/tmp//evidence-/`. A lane agent may pre-check that patch, but only under § Pre-checks; nothing else runs under a mutation before P5. Done when every declaration in the lane has a mark and an evidence line. **P4 Layer plan and owner checkpoint.** A different read-only agent (`architect-reviewer`, or a fresh `qa-strategist` that did not write the ledger) takes the ledger as input and looks for redundant layers rather than single tests: which suite is the keeper for each contract, which assertions move into it, which test-only seams become free. One reviewer covers at most eight or nine lanes; above that, several reviewers share one brief and each receives its predecessors' result files. Prefer the real boundary with a fake transport over a mocked collaborator. It re-reads at least 15 % of the C and D declarations against the source, the largest and the weakest-argued first, resolves every cross-lane keeper reference against the owner lane's mark (a keeper must end as R or F; a keeper that is itself C or D, or a wrong line number, invalidates the chain, 16 cases in one campaign), and records every fix in a corrections table (`file:line`, old mark, new mark, evidence); it never edits the lane ledgers in place. P4 ends at an owner checkpoint, written into the report: the goal estimate (§ Goal accounting), every O with its line count, the coverage decision from P1 if still open, and the bundled issue list from the ledgers. When the estimate lies far below the goal, the owner chooses between continuing, a repair-and-close goal (F repairs, N tests, issues, no deletion path), or stopping; the threshold for "far below" is not fixed yet. A headless run takes the brief's answer; without one it stops after P4 with the checkpoint in the report. Issues are filed after this checkpoint, bundled by cause, by the session that owns the run unless the brief delegates it. P4 reviewers run no pre-checks. Done when each lane names its retired files, its keeper per contract, the assertions to carry over, and the seams unlocked, and the checkpoint is answered. **P5 Keepers and falsification.** One `test-writer` per lane with disjoint file scopes repairs every F, absorbs every C into its keeper, and deletes its C and D, all uncommitted in one pass, and writes the lane's mutation manifest (§ Mutation manifest). Changes to shared helpers and fixtures go through one owner, serialised. Then sync and mutate. A lane is ready when every run row is `CAUGHT` (`REF` and `NOCONTRACT` rows are listed only); a `SURVIVED`, `NOPROOF`, or `APPLYFAIL` row sends it back to its agent. The control run therefore sees the deletions, which is the point: three campaigns ran this order and it saved one agent round per lane. The binding proof is the coordinator's run in the offload worktree (§ Pre-checks for what agents may run on their own). Done when every C and F maps to at least one caught mutation id with a restore proof, and every D to one or to a `NOCONTRACT` row with its ledger evidence. **P6 Cutover.** Read the repo's commitlint configuration before the first commit and run commitlint on every lane commit: the repo's config beats the brief and the MR title in SKILL.md (one repo demanded sentence case where the brief said lower case; another had a 100-character subject limit, and the chain had to be rebuilt). Agents never run git write commands (PSA-007); the coordinator commits ready lanes one commit per lane, the owner lanes before the lane that deletes cross-cutting duplicates, so every intermediate commit still carries each contract's proof. Removing a test-only production seam is a separate `code-implementer` task and a separate commit; a D whose evidence field 6 names dead production code (the test is its only caller) lands with or after that removal commit, and a removed code path leaves the coverage denominator instead of counting as a drop. Done when every ready lane is committed and each lane's keepers pass. **P7 Preservation review.** Independent, read-only, one reviewer per group of lanes (`session-reviewer`, or `pr-review-toolkit:pr-test-analyzer` when installed). It compares deleted coverage against the keepers and looks for contracts that lost their only proof, for new assertions that cannot fail, and for any production change outside the seam commits. Each restored contract gets a caught mutation. Commit every lane before P7 starts, and give the reviewers the commit list and the current file state: reviewers cite HEAD line numbers and otherwise take the campaign's own edits for another session's, and one brief called files deleted that still existed. Done when every reported gap is restored or rejected with source evidence. **P8 Final measurement (M1).** Same commands, same gate, same host as M0. Done when the M0/M1 table is complete and per-file coverage is compared. **P9 Hand-off.** Draft MR and follow-up issues as listed in the SKILL.md § Output. Before the push, prove the production change is zero: `git diff --stat origin/main...HEAD -- . ':(exclude)' ':(exclude)docs/audits/'` prints nothing except the seam commits and owner-approved dependency or config changes (a coverage provider from P1), each named in the MR. Count the branch's commits against the pipeline's clone depth (`GIT_DEPTH`): commitlint fails on a shallow clone that cannot reach the merge base (one 20-commit branch against depth 20); an overrun goes into the MR as an owner point, never a squash on your own. Then wait for the MR pipeline in the foreground, blocking, and write MR number, pipeline id and job list into the report; the report is the last thing the run produces (two pilots ended their turn while the pipeline still ran and left no final report). A campaign that outlives many `main` commits merges `main` instead of rebasing; when `main` changed a file the campaign deleted, the deletion stays and the new contract moves into the keeper. **Calibration** (one TypeScript/vitest campaign, 2026-09-24; 5.8k test lines, 138 C/D/F marks): 105 mutations covered the 138 marks and took about 3 minutes on the offload host, each a focused `-t` run (81 of them in 63 s). P7 still found 4 lost proofs (2.9 % of the marks) and the per-file backstop 2 more; two later campaigns put P7's yield at 2.9–4.8 % even after a 45 % P4 sample, so neither replaces the other. The whole campaign took about 2 h wall-clock and about 3.2 M subagent tokens. Re-checked with `mutate.sh` at the MR head: control green, 105 of 105 mutations CAUGHT with a red summary, 11 cross-lane `REF` rows. ## Goal accounting The 20 % goal counts removed test lines. The estimate after P3 and P4 counts only D, C into another file, and the header lines (imports, setup) of retired files. Reshaping inside the same file (C-same, parametrising a table) is net about zero and is reported on its own line; so is mock ballast without a mark, and so is the F/N growth, which eats into the goal (one campaign: about +430 lines). The O sum stands beside the estimate, never inside it. In three of the six pilot repos (tests:src 0.55–0.80, well inside the TV-003 corridor) the evidence rules held the removable share at 2–5 %; a fourth, at 0.62, estimated 26 % and delivered 12.8 % in its first sub-campaign; the one above the corridor (1.88) reached 31 %. A low ratio predicts a short deletion path but does not decide it; that is what the P4 checkpoint is for. ## Pre-checks A pre-check is a mutation run by a lane agent (P3 or P5) on its own, before the coordinator's binding run. One campaign got 90 % of its binding mutations ready-made from lane pre-checks and ran P5 with zero retries; one leaked an in-repo pre-check into a parallel lane for a few seconds; one was halted because its brief forbade pre-checks and the lane prompt never said so. The rules: - Only in a private copy outside the repo: `d=$(mktemp -d /tmp//evidence-/copy.XXXX) && git archive HEAD | tar -x -C "$d"`; in P5, copy the agent's own uncommitted files over it, since `git archive` exports committed files only. Never in the repo, never in any checkout another agent or the runner uses, never against a `node_modules` symlinked into the repo (a package manager will install through the link). - Only for the agent's own patch files, and the result is marked `pre-checked, not binding` in the manifest row or the ledger. The binding proof is the coordinator's `mutate.sh` run in the offload worktree. - Artefacts under `/tmp//evidence-/`, nothing in the campaign root; the coordinator removes the copies (a destructive-command guard blocks `rm -rf` for agents). - The lane prompt repeats these rules verbatim and states that they override the agent definition's read-only Bash rule for the copy only; a brief that forbids pre-checks wins. ## Measurements Every number carries the command that produced it and the SHA it was measured at. Heavy runs (runtime, flakes, coverage) go to the offload host: `offload run -H --job -- `, then `--job --no-sync` for the remaining runs on the same worktree (`--rm` removes it only after a successful run). - Job names carry the repo: `ta--`. Host artefacts live under `/.ta-/`, local ones under `/tmp//`, never in the working tree (a `.sh` helper in the tree turned one repo's lint red) and never under a shared host path (two campaigns overwrote each other's M0 under `/tmp/ta-m0` on the same host). - A sync deletes untracked files in the job worktree, including `.ta-/`, `mutate.sh`, and copied manifests, but does not reset local changes. So: copy `mutate.sh` and the manifests in after every sync, copy results out before the next one, and after every sync check `git status --porcelain` and that the snapshot SHA changed (one campaign ran two rounds with 42 false `SURVIVED` on a stale worktree). A reference worktree created inside the offload clone falls into the runner's include (770 instead of 277 tests): create it elsewhere or remove it right after the measurement. - Measurement and mutation scripts exit non-zero when any run is red, so `--rm` keeps the worktree, and print the per-run summary and the failing test names to stdout: files left in a removed worktree are gone before anyone reads them. - The command is not shell-expanded on the host: `~` and `$HOME` resolve locally or not at all. Pass literal remote paths, or wrap in `bash -c '…'`. - Record both SHAs the host prints (`SHA=… SNAPSHOT=…`). The snapshot SHA differs from the pinned one; the tracked tree must not (`FILES=` lists only untracked files). - `timeout` is not present on every macOS host (rc 127 turns the control run red). Check `command -v timeout || command -v gtimeout` first; otherwise rely on `offload --timeout`. - A suite under about 60 s per run may be measured locally instead, sequentially, under `nice -n 10` and `timeout -k 5 600`: for short suites, offload sync and install cost more than the runs (measured once: 13 min waiting for 3.7 min of runs). M0 and M1 still run on the same host. Mutations (P5) always go to the offload worktree. | Measure | How | |---|---| | Test count | the runner's own count (vitest: `--reporter=json`, field `numTotalTests`; file count `testResults.length`) | | Test lines vs production lines | at M0: line totals of the tracked test files the runner actually includes (vitest `include`, pytest `testpaths`, …) and of the rest, counted separately. Tracked test files outside the include are _dead test files_ only when they are neither a setup file, nor imported by an included test, nor a documented manual or live suite; those are listed as _outside include, live_ with their own total. Dead files count neither in M0 nor toward the goal, and their removal is reported on its own line. M0 to M1: `git diff --numstat ` split by the same globs | | Suite runtime | median of 3 full runs on the measuring host | | Flakes | 5 runs in shuffled order. A run is flaky when its exit code is non-zero or any test fails, runner-level unhandled errors included (exit 1 with `numFailedTests: 0` is a flake). M1 may not have more than M0. Fake timers next to an unmocked network client (ioredis, fetch) are a known cause | | Coverage | total and per production file, on all four metrics (lines, branches, functions, statements); thresholds and config unchanged | | tests:src ratio | per TV-003 (`test-value.md`); by hand when the script below does not know the stack | `scripts/lib/tests-src-ratio.mjs` counts JavaScript and TypeScript (`.mjs`/`.js`/`.cjs`/`.jsx`/`.ts`/`.tsx`/`.mts`/`.cts`; `*.d.ts` on neither side). Its test side is a top-level `tests/` plus co-located tests anywhere: `*.test.`, `*.spec.`, and files under `__tests__/`. Its `--json` envelope prints that definition; quote it with the number. Two limits remain. A nested test directory (`packages/x/tests/`) whose helpers are not named `*.test.*` counts as src. Other stacks (Python, Swift, Go, …) are invisible to it: the envelope then says `reason: "no-tests-found"` (ratio `null`) or measures only the JS/TS code it happens to find. There, `null` or a low ratio means not measurable, never within the corridor. Compute those by hand with the TV-003 recipe: numerator the physical lines of the tracked test files (`git ls-files '/**'`, code files only), denominator every other tracked code file. State both numbers and the globs. Stack recipes (check each flag against the installed version before use): - **TypeScript / vitest:** count `vitest run --reporter=json --outputFile=/count.json` (without `--outputFile` the JSON mixes into stdout; fields `numTotalTests`, per file `testResults[].status`); flakes the same with `--sequence.shuffle --sequence.seed=$i` for `i` in 1..5; coverage `vitest run --coverage --coverage.reporter=json-summary` → `coverage/coverage-summary.json`. Its keys are absolute paths: strip the worktree prefix before comparing M0 and M1. Thresholds stay enforced, so a red threshold fails the run. For a backstop hit, `--coverage.reporter=json` (`coverage/coverage-final.json`) names the uncovered line, branch, or function. - **Python / pytest:** count `--collect-only -q`; coverage `--cov --cov-report=json`; leave `fail_under` as it is. Shuffle only with a plugin that is already installed; otherwise 5 plain runs. - **Swift / XCTest:** count `swift test list`; coverage via `swift test --enable-code-coverage` and `xcrun xccov`. - **Bash harness:** falsify the harness first (known-broken input, non-zero exit, FAIL in the artefact), under `bash`, then count and time it. ## Mutation manifest One directory per lane, `/tmp//mutations//`, holding `manifest.tsv` and the patch files it names. One tab-separated line per mutation; `#` lines and an `id` header are skipped. Every row fills all six columns (an empty column shifts the rest left): `REF`, `NOCONTRACT`, and type-check rows put `-` in `testfiles` and `pattern`. | Column | Content | |---|---| | `id` | unique mutation id, cited by the ledger | | `patch` | patch file in the lane directory: `git diff -- ` of the mutated tree; `EXTERN:` when the contract's mutation lives in another lane's manifest (logged as `REF`, not run); or `NOCONTRACT` for a D that removes no contract (dead production code, a duplicate whose keeper is proven outside this campaign, a test-local copy), logged and not run, with the evidence in the ledger; a keeper proven in another lane's manifest is `EXTERN:` | | `testfiles` | space-separated keeper test files that must go red; `-` when the keeper is the whole command (a type check) | | `pattern` | the runner's name filter for the keeper (vitest `-t`), or `-` for whole files. It is a regex: escape `.`, `(`, `$`; vitest quotes `it.each` `$name` values, so match them with `.?` around the name | | `contract` | one sentence | | `ledger_refs` | the marks it proves (`file:line`, comma-separated); every C, D, and F appears in at least one line | [mutate.sh](mutate.sh) runs from the offload worktree root: `bash mutate.sh `. The host does not see local `/tmp`, so copy the script and the mutations directory there after every sync and pass literal remote paths (§ Measurements). It runs the whole suite once as control and stops on red (exit 2). Per line it applies the patch, runs the focused test, reverses it, and compares the SHA-256 of the touched files before and after; a mismatch aborts at once (exit 3). Log verdicts: `CAUGHT` needs a non-zero exit and the failure summary; `NOPROOF` is a non-zero exit without it (a compile or collection error proves nothing); `SURVIVED` is a green run under the mutation; `APPLYFAIL` is a patch that does not apply; `REF` and `NOCONTRACT` rows are listed and not run. Any verdict other than `CAUGHT`, or no manifest line at all, ends with exit 1. It keeps `.out` next to each patch for the mutation table. Paths with spaces are not supported. Runner defaults: `MUTATE_CMD` is `pnpm exec vitest run` when a `pnpm-lock.yaml` exists, otherwise `node_modules/.bin/vitest run` (never `npx`, which may download); set it explicitly for anything else. Other stacks set `MUTATE_CMD`, `MUTATE_PATTERN_FLAG`, and `MUTATE_RED_RE` (pytest: `-k`, `[0-9]+ failed`). A type checker as keeper (a mutation that changes a shape a fresh object literal must satisfy): `MUTATE_CMD="timeout -k 5 300 pnpm exec tsgo --noEmit"`, `MUTATE_RED_RE='error TS[0-9]+'`, `testfiles=-`, `pattern=-`, measured with 14 caught mutations in one campaign. tsgo catches only what the type graph sees: not `.mjs` under `allowJs: false`, not runtime schemas (`zod`), so those contracts need a test keeper. ## MR body - Goal as stated at the start, and reached or not with the reason; the estimate per § Goal accounting with C-same, ballast, F/N growth and the O sum on their own lines. - M0/M1 table with command and SHA per row. - R/F/C/D/N/B/O counts per lane. - Keeper per contract, and the retired layers. - Mutation table: `file:line`, mutation, red test, restore proof; `NOCONTRACT` rows with their ledger evidence. - Kept false alarms: tests that looked like junk and stayed, with the contract they guard. - Production lines and test lines, separately; dead test files and live files outside the include on their own lines; the zero-production-diff command and its empty output. - The P4 corrections table and the ledger's _Issues_ section. - Owner decisions taken during the campaign, and the open ones: every O with its line count, so a follow-up MR finds the candidates; the coverage-provider decision when it was open; the commit count against the CI clone depth when it overran. - MR number, pipeline id and job list, written after the pipeline finished.