# Methodology

**Claim under test:** a doc-wiki generated wiki in the repo improves Claude Code's autonomous ticket-fix pass rate on real closed issues.

**Design:** paired two-arm runs per ticket (baseline / wiki) — identical container, model (`claude-sonnet-4-6`, pinned full ID), prompt, and flags; the only delta is the presence of the pre-built wiki + `CLAUDE.md` in the checkout (committed before the session so the agent's diff contains only its own work). Grading: the real fix PR's tests, overlaid onto the agent's diff (SWE-bench style). Pass = all overlaid tests pass; a pass that needed the configured single retry is recorded distinctly (`tests-passed-on-retry`).

**Ticket eligibility:** closed issue with a merged linked fix PR touching both test and non-test source, <400 changed lines, natural-language body ≥200 chars, human author, merged after the repo's `ticket_after` floor. The fix PR must include at least one *runnable* test entry point (per-repo `run_patterns`) — test-side support files (configs, fixtures, utils) are overlaid at grade time but never executed directly. Tickets whose runnable tests need toolchains absent from the grade container (per-repo `exclude_test_paths`, e.g. vitest's Playwright-backed `test/browser/**` and `test/ui/**` suites) are excluded up front. The committed `tickets/<repo>.json` is the exact set, including every exclusion and its reason.

**Contamination controls:**
1. *Fix leak:* the wiki is built at `wiki_commit`, verified (`git merge-base --is-ancestor`) to predate every ticket's base commit.
2. *Issue-body leak:* bodies are sanitized (forward issue/PR references, cross-repo and GH-style references, commit SHAs, "fixed by" lines, github URLs stripped); every redaction is logged in the ticket record alongside the verbatim body for audit.
3. *Online-lookup leak:* agent sessions run behind an egress firewall (IPv4 + IPv6) allowing only Anthropic endpoints, with IPs pinned at session start; DNS (port 53) stays open for resolution, so the control targets HTTP(S) lookups — the realistic agent behavior — not deliberate DNS tunneling. Repo installs happen before the firewall comes up. Agent edits to overlay-introduced files (the wiki itself) are excluded from the graded diff — they cannot apply at base_commit and are not part of the fix. *Sidecars:* service containers (Postgres, valkey, etc.) run on a private docker network. The firewall permits RFC1918 ranges (`10/8`, `172.16/12`, `192.168/16`) only when `BENCH_ALLOW_PRIVATE_NET=1` is set (injected automatically when services are configured), so the agent can reach local DB/cache sidecars but public internet — including the real fix on GitHub — remains REJECTED. *Host-surface caveat:* allowing RFC1918 means any host service reachable on a private IP (e.g. an exposed Docker API on the gateway, or an internet-bridging local proxy listening on a private range) becomes reachable from the agent for service-backed repos; operators must not run the benchmark on a host that exposes internet-bridging private-network services. Public GitHub stays blocked regardless — this is the documented tradeoff of the sidecar allowance.
4. *Training-data leak:* tickets postdate `ticket_after` (set from the pinned model's training cutoff); merge dates are published per ticket.

**Calibration (pre-registered):** before any agent runs, each ticket's fix-PR tests must fail on the clean base commit (overlaid from the fix commit — newly-added regression tests are the canonical shape) and pass on the fix commit, with every test path present at the fix commit. Failures are excluded up front, with reasons logged in the committed ticket file.

**Known caveats:**
- Single run per (ticket, arm): no variance estimate per ticket; treat per-repo aggregates, not per-ticket outcomes, as the signal.
- OSS repos ≠ enterprise codebases. The author's enterprise-codebase experience (the README hero number) is an anecdote, not this benchmark's claim; the benchmark's claim is whatever RESULTS.md says.
- Ticket discovery uses GitHub's `closingIssuesReferences` (keyword-linked issues only) — PRs that reference an issue solely in free-text prose are not mined, so the candidate pool understates true fix volume. Selection bias is toward well-linked, process-followed fixes.
- Rebase-merged PRs can make `base_commit` (merge-commit parent) partially contain the fix; the calibration gate excludes them. `merge_parents` on each ticket record flags true merge commits (=2); rebase merges have a single parent and are detectable only via calibration.
- `--max-turns` and container timeout: all published runs used the defaults (80 turns, 1800 s). Only `--model` varied between the two saleor configurations (`claude-sonnet-4-6` vs `claude-opus-4-8`).
- Ticket dedup: the miner's PR-as-ticket fallback admits backport/cherry-pick duplicates of the same underlying bug (on saleor, 25 tickets ≈ 15 unique bugs across 7 backport families). Duplicates make single-run paired comparison noisier than the pair count suggests — a flaked baseline on one family member reads as a "wiki win". RESULTS.md analyzes the affected cells explicitly.
- Headline rates are computed over **valid pairs** (both arms graded); the per-configuration generated reports also print raw per-arm graded counts, which differ where one arm hit an infrastructure error (e.g. saleor-schema #18543 baseline, dropped from the paired tally).

**Reproduction:** see [README.md](README.md). Session cost of the published runs is itemized in [RESULTS.md](RESULTS.md) ($212.45 across the four configurations, plus wiki-overlay builds).