# Benchmark Results

> Methodology and caveats: [METHODOLOGY.md](METHODOLOGY.md). Per-configuration reports (generated by `npm run benchmark -- report`): [vitest](RESULTS-vitest.md) · [saleor / Sonnet](RESULTS-saleor-sonnet.md) · [saleor / Opus](RESULTS-saleor-opus.md) · [saleor-schema](RESULTS-saleor-schema.md).

## Headline

**Across four benchmark configurations — 92 valid ticket pairs on two OSS repos — a doc-wiki wiki did not improve Claude Code's autonomous ticket-fix pass rate.** Baseline arms passed 57/92; wiki arms passed 54/92.

We are publishing this null result in full rather than shipping a favorable-looking slice of it. The harness, ticket manifests, per-run artifacts, and these numbers are the reproducible record; run it on your own repo and argue with the data.

## Summary

A "valid pair" is a ticket where **both** arms (baseline and wiki) completed and were graded. Rates below are over valid pairs; the per-configuration reports additionally show raw per-arm graded counts, which can differ where one arm errored.

| configuration | model | valid pairs | baseline passed | wiki passed | Δ (wiki − baseline) | regime |
|---|---|---|---|---|---|---|
| vitest | claude-sonnet-4-6 | 23 | 12 (52%) | 11 (48%) | −1 | mid-difficulty |
| saleor | claude-sonnet-4-6 | 25 | 22 (88%) | 20 (80%) | −2 | ceiling |
| saleor | claude-opus-4-8 | 25 | 19 (76%) | 22 (88%) | +3 | ceiling (see below) |
| saleor-schema | claude-sonnet-4-6 | 19 | 4 (21%) | 1 (5%) | −3 | floor |
| **total** | | **92** | **57 (62%)** | **54 (59%)** | **−3** | |

Session cost of the published runs: $56.62 (vitest) + $46.47 (saleor/Sonnet) + $55.37 (saleor/Opus) + $53.99 (saleor-schema) = **$212.45**, plus wiki-overlay build costs (billed separately per run; the saleor wiki build was $21.67).

## How to read each configuration

**vitest / Sonnet (−1).** 11 tickets passed both arms, 11 failed both, one flipped to baseline. Within single-run noise; no signal either way.

**saleor / Sonnet (−2, ceiling effect).** The baseline already passes 88% — well-calibrated single-repo OSS tickets with clean prompts are a regime the model largely saturates without help. Only 3 of 25 tickets failed both arms, so the maximum possible lift was +3. The two tickets the wiki *lost* (#19111, #19280) share a mechanism: the wiki arms wrote roughly 40% smaller diffs and under-implemented the fix.

**saleor / Opus (+3, not a real lift).** The only positive cell, and we don't believe it. The three wiki-only wins (#19122, #19228, #19253) include backport duplicates of tickets the baseline passed elsewhere in the same run — i.e. bugs demonstrably solvable without the wiki, where a single baseline run flaked. The ticket miner's PR-as-ticket fallback means the 25 tickets collapse to ~15 unique bugs (7 backport/cherry-pick families), so single-run paired comparison can manufacture spurious wins. Decisively: the 3 genuinely hard tickets (#19114, #19179, #19286) failed **both arms on both models** — the wiki cracked zero tickets that were actually out of the baseline's reach.

**saleor-schema / Sonnet (−3, floor effect).** 24 tickets hand-picked to touch ORM models, migrations, and FK relationships — the regime where a wiki with 21 ER-diagram data-model pages should help most. 19 valid pairs after calibration and one persistent infrastructure failure (#18543 baseline, dropped). 15/19 failed both arms: these are large feature/refactor PRs needing near-complete reproduction to pass their test suites — too hard for a single-shot session regardless of context. On the three tickets the baseline won, the wiki arms attempted real fixes (224/212/627-line diffs) but under-implemented relative to baseline and failed. The wiki's one pass was also a baseline pass.

## What we conclude

1. **No measured autonomous-accuracy lift in any single-repo OSS regime we could construct.** Mid-difficulty, ceiling, and floor regimes all show the wiki at or below baseline; the one positive cell dissolves under duplicate-ticket analysis.
2. **A repeated failure mechanism, not just noise:** on losing tickets, wiki arms consistently wrote *smaller* diffs than baseline. High-level architectural context appears to anchor the agent toward minimal changes where the ticket actually required broad ones.
3. **The regime this measures is narrow.** Single-repo OSS tickets cluster at ceiling (localized bugs the model already solves) or floor (feature PRs no single session solves). What this benchmark does **not** measure: human-in-the-loop workflows (navigation, onboarding, review context), enterprise codebases with tribal knowledge and external context (Jira/Confluence history), and cross-service root-repo setups — the settings doc-wiki was actually built for. We have not yet built a benchmark for those regimes; until we do, we make no quantitative claims about them.

## Provenance

- Run states: `benchmark/runs/<config>/state.json` — **committed** for the four published configurations, so `npm run benchmark -- report --repo <id> --out <file>` regenerates each per-config report from a fresh checkout. Per-ticket artifacts (prompts, diffs, credential-pruned transcripts, grades) remain local-only under the same tree.
- Ticket manifests with every exclusion and sanitization logged: `benchmark/tickets/`.
- Runs used harness defaults: `--max-turns 80`, `--timeout-sec 1800`. Only `--model` varied between the two saleor runs.
- **Reproducing each configuration:** the harness keys run state and reports by config id (`benchmark/runs/<id>/state.json`), so each row has its own committed config: `repos/vitest.yaml`, `repos/saleor.yaml` (Sonnet, the default model), `repos/saleor-opus.yaml` (identical to saleor incl. `wiki_commit`, distinct id — run with `--model claude-opus-4-8`), and `repos/saleor-schema.yaml`. `tickets/saleor-opus.json` is a verbatim copy of `tickets/saleor.json`: the Opus row is the same ticket set under a different model. Historical note: our own Opus run predates the `saleor-opus` id — its state was produced under id `saleor` (after relocating the Sonnet state aside), which is why [RESULTS-saleor-opus.md](RESULTS-saleor-opus.md)'s generated header reads `## saleor`. The committed `runs/saleor-opus/state.json` is that same state with its `repo` field relabeled to the new id so the checkpoint loader accepts it; run outcomes, timestamps, and costs are byte-identical to the original.
- Prompts were adversarially detainted (PR-derived ticket bodies rewritten to issue-style problem statements; originals preserved in `title_pr` / `body_sanitized_pr` fields for audit).
- An earlier (V1) run with weaker controls was withdrawn before publication — post-mortem in [ANALYSIS.md](ANALYSIS.md).