# Benchmark Harness Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Build the reproducible ticket-fix benchmark harness (spec: `docs/superpowers/specs/2026-06-10-benchmark-harness-design.md`) — deterministic TypeScript orchestration around headless Claude Code sessions in Docker, graded by the real fix's tests, piloted on `vitest-dev/vitest`.

**Architecture:** A `benchmark/` top-level directory: pure-logic modules (eligibility filters, sanitization, checkpoint state machine, grade decisions, report rendering) unit-tested with Vitest; side-effecting boundaries (gh API, docker, git) behind one injectable `Runner` interface so every orchestration path is testable with fakes; shell assets (`entrypoint.sh`, `grade.sh`, `init-firewall.sh`) do the in-container work. An e2e smoke test runs the whole pipeline against a tmp fixture git repo with a fake `claude` binary — zero tokens, zero docker in CI.

**Tech Stack:** TypeScript (ESM/NodeNext style, Node 20), Vitest, `js-yaml` (already a dep), `ignore` (already a dep, gitignore-style matching for `test_patterns`), `gh` CLI, Docker, bash.

**Conventions that are non-negotiable in this repo:**
- `npm run build` emits sibling `.js` next to each non-test `.ts`. **Every commit must include both the `.ts` and its built `.js`** (CI enforces sync). Test files (under `tests/`) are excluded from build — no `.js` siblings for them.
- Stage specific files; never `git add -A` or `git add .`.
- Work on branch `feat/benchmark-harness`; PR at the end; invoke pr-monitor after pushing.
- No real names from any proprietary codebase in fixtures — generic fixtures only.

---

## File structure

```
benchmark/
├── harness/
│   ├── types.ts              # shared interfaces (Task 2)
│   ├── repo_config.ts        # load+validate repos/<id>.yaml (Task 2)
│   ├── sanitize.ts           # issue-body sanitization (Task 3)
│   ├── bench_checkpoint.ts   # state machine + atomic persistence + pair scheduler (Task 4)
│   ├── exec.ts               # Runner interface + realRunner (Task 5)
│   ├── mine_filters.ts       # pure eligibility logic (Task 6)
│   ├── mine_tickets.ts       # gh-driven mining CLI (Task 7)
│   ├── docker_args.ts        # pure docker-argv builders (Task 8)
│   ├── session.ts            # prompt template + session-result classifier (Task 9)
│   ├── run_ticket.ts         # paired-arm batch orchestrator CLI (Task 10)
│   ├── grade.ts              # grade decision + calibrate/grade drivers (Task 11)
│   ├── report.ts             # RESULTS.md renderer CLI (Task 12)
│   ├── build_wiki.ts         # one-time atlas-build CLI (Task 13)
│   ├── cli.ts                # subcommand dispatcher (Task 14)
│   ├── docker/
│   │   ├── Dockerfile        # (Task 8)
│   │   ├── entrypoint.sh     # session+grade dispatch (Task 8)
│   │   ├── grade.sh          # checkout/apply/overlay/test (Task 11)
│   │   └── init-firewall.sh  # Anthropic-only egress (Task 8)
│   └── tests/                # *.test.ts per module + fixtures/ (excluded from build)
├── repos/vitest.yaml         # pilot repo config (Task 2)
├── tickets/                  # mined ticket sets (committed; .gitkeep)
├── runs/                     # raw run artifacts (gitignored; .gitkeep)
├── wiki-cache/               # built wiki overlays (gitignored)
├── README.md                 # how to run (Task 14)
├── METHODOLOGY.md            # skeleton now, numbers later (Task 15)
└── RESULTS.md                # generated by report (Task 12)
```

Run-artifact layout (written by Tasks 10–11, read by 12): `benchmark/runs/<repo>/<issue>/<arm>/{prompt.txt, result.json, stderr.log, exit_code, diff.patch, transcript/, grade.json}`. Checkpoint: `benchmark/runs/<repo>/state.json`.

One spec refinement made here: the run state machine gains an intermediate `ran` status (artifacts captured, awaiting grade) so a crash between session and grading resumes as grade-only instead of burning another session. `pending → running → ran → passed|failed`, with `error`/`rate-limited` side-exits that revert to `pending` on resume.

---

### Task 1: Scaffolding + build/test wiring

**Files:**
- Modify: `tsconfig.json` (include), `tsconfig.build.json` (include), `vitest.config.ts` (test include), `.gitignore`, `package.json` (script)
- Create: `benchmark/tickets/.gitkeep`, `benchmark/runs/.gitkeep`

- [ ] **Step 1: Wire benchmark/ into the toolchain.** In `tsconfig.json` change the include line to:

```json
"include": ["agents/**/*.ts", "skills/**/*.ts", "launch/**/*.ts", "benchmark/**/*.ts", "vitest.config.ts"],
```

In `tsconfig.build.json`:

```json
"include": ["agents/**/*.ts", "skills/**/*.ts", "benchmark/**/*.ts"],
```

In `vitest.config.ts` test.include:

```ts
include: ["agents/**/*.test.ts", "skills/**/*.test.ts", "benchmark/**/*.test.ts"],
```

In `package.json` scripts add:

```json
"benchmark": "node benchmark/harness/cli.js",
```

(`cli.js` arrives in Task 14; the script is inert until then.) Append to `.gitignore`:

```
benchmark/runs/
benchmark/wiki-cache/
```

- [ ] **Step 2: Create the directory skeleton.**

```bash
mkdir -p benchmark/harness/tests/fixtures benchmark/harness/docker benchmark/repos benchmark/tickets benchmark/runs
touch benchmark/tickets/.gitkeep benchmark/runs/.gitkeep
```

- [ ] **Step 3: Verify nothing broke.** Run: `npm run typecheck && npm test`. Expected: both green (vitest `passWithNoTests` covers the empty dir).

- [ ] **Step 4: Commit.**

```bash
git checkout -b feat/benchmark-harness
git add tsconfig.json tsconfig.build.json vitest.config.ts .gitignore package.json benchmark/tickets/.gitkeep benchmark/runs/.gitkeep
git commit -m "chore(benchmark): scaffold benchmark/ and wire into build/test"
```

---

### Task 1.6: V1 supersession sweep

**Context (added 2026-06-10 after Task 1):** a V1 benchmark already exists in `benchmark/` (committed 2026-06-04): hand-curated 25-issue manifest (`repos.yaml`, SHAs verified in `results/curation-report.md`), a 1,266-line harness (`harness/run.ts`, `score.ts`, `validate.ts`, `verify-curation.ts`), strategy docs (`PLAN.md`, `ANALYSIS.md`, `PUBLISH.md`, `dispatch.md`, `README.md`), and published results (`results/RESULTS.md`, `results/raw.csv`) showing baseline 94.4% vs with-doc-wiki 100%. **User decision:** V2 supersedes V1; the V1 results are withdrawn with an explanatory commit (methodology gaps: sessions had unrestricted network access, several curated ticket bodies contain root-cause analysis, no training-data contamination controls, and the README's model attribution contradicts raw.csv). The curated manifest is salvaged for V2's django/cal.com/mastodon phase.

**Files:**
- Delete: `benchmark/harness/run.ts`, `benchmark/harness/score.ts`, `benchmark/harness/validate.ts`, `benchmark/harness/verify-curation.ts`, `benchmark/results/RESULTS.md`, `benchmark/results/raw.csv`
- Keep untouched: `benchmark/repos.yaml`, `benchmark/results/curation-report.md`
- Modify (prepend supersession banner): `benchmark/PLAN.md`, `benchmark/ANALYSIS.md`, `benchmark/PUBLISH.md`, `benchmark/dispatch.md`, `benchmark/README.md`
- Modify: `README.md` (§Reproducible benchmark rewrite + line ~379 table row), `launch/cold-email-alex-albert.md` (one checklist line), `.gitignore` + `benchmark/.gitignore` (consolidation)

- [ ] **Step 1: Delete the V1 harness + withdrawn results.**

```bash
git rm benchmark/harness/run.ts benchmark/harness/score.ts benchmark/harness/validate.ts benchmark/harness/verify-curation.ts benchmark/results/RESULTS.md benchmark/results/raw.csv
```

- [ ] **Step 2: Prepend this banner** (exact text, as the first lines after the H1) to each of `benchmark/PLAN.md`, `benchmark/ANALYSIS.md`, `benchmark/PUBLISH.md`, `benchmark/dispatch.md`, `benchmark/README.md`:

```markdown
> **Superseded (2026-06-10).** The V1 harness and its published runs were withdrawn: sessions ran with unrestricted network access, several curated ticket bodies contained root-cause analysis, and there were no training-data contamination controls. The V2 harness (container isolation, Anthropic-only egress firewall, sanitized tickets, pre-registered calibration) replaces it — see [`docs/superpowers/specs/2026-06-10-benchmark-harness-design.md`](../docs/superpowers/specs/2026-06-10-benchmark-harness-design.md). The curated 25-issue manifest in [`repos.yaml`](repos.yaml) remains valid input and will be re-used (re-sanitized + calibrated) for the V2 django/cal.com/mastodon phase.
```

- [ ] **Step 3: Rewrite root `README.md` §"Reproducible benchmark"** — replace the entire section body (from the `## Reproducible benchmark` heading down to, but not including, `## Apache 2.0 — forever`) with exactly:

```markdown
## Reproducible benchmark

A hardened SWE-bench-style harness is being rebuilt in [`benchmark/`](benchmark/) — container-isolated runs with an Anthropic-only egress firewall, sanitized ticket bodies, training-data contamination floors, and pre-registered test calibration. Design: [`docs/superpowers/specs/2026-06-10-benchmark-harness-design.md`](docs/superpowers/specs/2026-06-10-benchmark-harness-design.md).

An earlier (V1) run of this benchmark was **withdrawn** on 2026-06-10: its sessions had unrestricted network access, several curated ticket bodies carried root-cause analysis, and there were no training-data contamination controls — together these inflate both arms, so the numbers were not defensible in either direction. The V1 post-mortem stays in [`benchmark/ANALYSIS.md`](benchmark/ANALYSIS.md). New numbers will be published here when the V2 pilot (vitest-dev/vitest) completes.

The "~10% → ~80%" headline on this README is the author's measurement on a **private 500k-LOC enterprise codebase**, not an OSS benchmark — explicitly anecdotal.
```

Also update the reference-table row at ~line 379 from `benchmark/PLAN.md | Reproducible benchmark methodology` to `docs/superpowers/specs/2026-06-10-benchmark-harness-design.md | Reproducible benchmark methodology (V2)` (keep the table's markdown-link formatting). Leave the README's other `benchmark/` links (hero lines ~7/19/24) untouched.

- [ ] **Step 4: Fix the stale launch-doc reference.** In `launch/cold-email-alex-albert.md` change the checklist line `- [ ] benchmark/results/RESULTS.md populated with real numbers` to `- [ ] benchmark/RESULTS.md populated with real V2 numbers`.

- [ ] **Step 5: Consolidate gitignore.** Append `wiki-cache/` to `benchmark/.gitignore`; remove the two lines Task 1 added to the root `.gitignore` (`benchmark/runs/`, `benchmark/wiki-cache/`) along with their comment line.

- [ ] **Step 6: Verify.** `npm run typecheck && npm run build && git status --short` — typecheck/build clean, and **no stray `benchmark/harness/*.js` files** appear (the V1 `.ts` files are gone, so the Task 1 build wiring no longer produces orphan `.js`). `npx vitest run benchmark/` still exits 0.

- [ ] **Step 7: Commit** (deletions + edits in one commit; message must carry the explanation):

```bash
git add -u benchmark/ .gitignore README.md launch/cold-email-alex-albert.md
git add benchmark/.gitignore
git commit -m "feat(benchmark)!: withdraw V1 results, supersede V1 harness with V2

V1's published runs were not defensible: sessions had unrestricted
network access (could look up the real fix), several curated ticket
bodies contained root-cause analysis, there were no training-data
contamination controls, and the README's model attribution
contradicted raw.csv. V2 (container isolation, egress firewall,
sanitized tickets, pre-registered calibration) replaces the harness.
The hand-curated 25-issue manifest (repos.yaml) and its verification
report are kept as input for the V2 django/cal.com/mastodon phase."
```

---

### Task 2: Shared types, repo config loader, pilot config

**Files:**
- Create: `benchmark/harness/types.ts`, `benchmark/harness/repo_config.ts`, `benchmark/repos/vitest.yaml`
- Test: `benchmark/harness/tests/repo_config.test.ts`

- [ ] **Step 1: Write `types.ts`** (the contract every later task uses — copy exactly):

```ts
/** Shared types for the benchmark harness. See docs/superpowers/specs/2026-06-10-benchmark-harness-design.md. */

export type Arm = "baseline" | "wiki";

export type RunStatus =
  | "pending"
  | "running"
  | "ran" // session artifacts captured, grade not yet recorded
  | "passed"
  | "failed"
  | "error"
  | "rate-limited";

export interface RepoConfig {
  id: string;
  github: string; // "owner/name"
  clone_url: string;
  language: string;
  ticket_source: "github" | "trac-commits";
  install: string[];
  test_command: string; // must contain "{test_files}"
  test_patterns: string[];
  test_retries: number;
  ticket_after: string; // ISO date floor (training-data contamination control)
  wiki_commit: string; // "" until the wiki is built
  toolchain: string[];
  services: string[];
}

export interface TicketCalibration {
  paths_stable: boolean;
  tests_fail_on_base: boolean;
  tests_pass_on_fix: boolean;
}

export interface TicketRecord {
  issue: number;
  issue_url: string;
  title: string;
  body: string; // verbatim, for audit
  body_sanitized: string; // what sessions receive
  fix_pr: number;
  fix_pr_url: string;
  base_commit: string; // fix PR parent — what the agent gets
  fix_commit: string; // fix PR merge commit
  test_files: string[];
  src_files: string[];
  changed_lines: number;
  merged_at: string; // ISO timestamp, published for contamination audit
  calibration?: TicketCalibration;
  excluded?: string; // exclusion reason; excluded tickets never run
}

export interface TicketsFile {
  schema_version: 1;
  repo: string;
  mined_at: string;
  tickets: TicketRecord[];
}

export interface RunRecord {
  status: RunStatus;
  started_at?: string;
  finished_at?: string;
  cost_usd?: number;
  session_id?: string;
  detail?: string; // error text / rate-limit reset hint / grade detail
}

export interface BenchState {
  schema_version: 1;
  repo: string;
  runs: Record<string, RunRecord>; // key: `${issue}:${arm}`
}

export interface SessionResult {
  kind: "ok" | "rate-limited" | "error";
  costUsd?: number;
  sessionId?: string;
  detail?: string;
}

export type GradeOutcome = "passed" | "failed";

export interface GradeRecord {
  outcome: GradeOutcome;
  detail: string; // "apply-failed" | "tests-failed" | "tests-passed" | "tests-passed-on-retry"
  graded_at: string;
}
```

- [ ] **Step 2: Write the failing test** `benchmark/harness/tests/repo_config.test.ts`:

```ts
import { mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { loadRepoConfig } from "../repo_config.js";

const VALID = `
id: demo
github: acme/demo
clone_url: https://github.com/acme/demo.git
language: typescript
ticket_source: github
install: ["npm ci"]
test_command: "npx vitest run {test_files}"
test_patterns: ["test/**", "**/*.test.ts"]
ticket_after: 2025-06-01
toolchain: ["node:22"]
`;

function writeCfg(yaml: string): string {
  const dir = mkdtempSync(join(tmpdir(), "benchcfg-"));
  const p = join(dir, "demo.yaml");
  writeFileSync(p, yaml);
  return p;
}

describe("loadRepoConfig", () => {
  it("parses a valid config and applies defaults", () => {
    const cfg = loadRepoConfig(writeCfg(VALID));
    expect(cfg.id).toBe("demo");
    expect(cfg.test_retries).toBe(0);
    expect(cfg.services).toEqual([]);
    expect(cfg.wiki_commit).toBe("");
    // js-yaml parses unquoted ISO dates as Date — loader must normalize back
    expect(cfg.ticket_after).toBe("2025-06-01");
  });

  it("rejects a test_command without {test_files}", () => {
    expect(() => loadRepoConfig(writeCfg(VALID.replace("{test_files}", "")))).toThrow(/test_command/);
  });

  it("rejects missing test_patterns", () => {
    expect(() => loadRepoConfig(writeCfg(VALID.replace(/test_patterns:.*\n/, "")))).toThrow(/test_patterns/);
  });

  it("rejects unknown ticket_source", () => {
    expect(() => loadRepoConfig(writeCfg(VALID.replace("github\n", "linear\n")))).toThrow(/ticket_source/);
  });
});
```

- [ ] **Step 3: Run it to verify it fails.** Run: `npx vitest run benchmark/harness/tests/repo_config.test.ts`. Expected: FAIL (cannot resolve `../repo_config.js`).

- [ ] **Step 4: Write `repo_config.ts`:**

```ts
import { readFileSync } from "node:fs";
import { load } from "js-yaml";
import type { RepoConfig } from "./types.js";

function isoDate(v: unknown): string {
  // js-yaml's default schema parses unquoted YAML timestamps into Date objects.
  if (v instanceof Date) return v.toISOString().slice(0, 10);
  return String(v);
}

export function loadRepoConfig(path: string): RepoConfig {
  const raw = load(readFileSync(path, "utf8"));
  if (typeof raw !== "object" || raw === null) throw new Error(`${path}: not a YAML mapping`);
  const cfg = raw as Record<string, unknown>;

  const required = [
    "id", "github", "clone_url", "language", "ticket_source",
    "install", "test_command", "test_patterns", "ticket_after", "toolchain",
  ];
  for (const k of required) {
    if (cfg[k] === undefined || cfg[k] === null) throw new Error(`${path}: missing required key "${k}"`);
  }
  if (cfg.ticket_source !== "github" && cfg.ticket_source !== "trac-commits") {
    throw new Error(`${path}: ticket_source must be "github" or "trac-commits"`);
  }
  if (typeof cfg.test_command !== "string" || !cfg.test_command.includes("{test_files}")) {
    throw new Error(`${path}: test_command must contain the "{test_files}" placeholder`);
  }
  if (!Array.isArray(cfg.test_patterns) || cfg.test_patterns.length === 0) {
    throw new Error(`${path}: test_patterns must be a non-empty list`);
  }
  if (!Array.isArray(cfg.install)) throw new Error(`${path}: install must be a list`);
  if (!Array.isArray(cfg.toolchain)) throw new Error(`${path}: toolchain must be a list`);

  return {
    id: String(cfg.id),
    github: String(cfg.github),
    clone_url: String(cfg.clone_url),
    language: String(cfg.language),
    ticket_source: cfg.ticket_source,
    install: cfg.install.map(String),
    test_command: cfg.test_command,
    test_patterns: cfg.test_patterns.map(String),
    test_retries: cfg.test_retries === undefined ? 0 : Number(cfg.test_retries),
    ticket_after: isoDate(cfg.ticket_after),
    wiki_commit: cfg.wiki_commit === undefined ? "" : String(cfg.wiki_commit),
    toolchain: cfg.toolchain.map(String),
    services: Array.isArray(cfg.services) ? cfg.services.map(String) : [],
  };
}
```

- [ ] **Step 5: Run tests to verify they pass.** Run: `npx vitest run benchmark/harness/tests/repo_config.test.ts`. Expected: 4 PASS.

- [ ] **Step 6: Write the pilot config** `benchmark/repos/vitest.yaml` (values from the screening record in the spec):

```yaml
id: vitest
github: vitest-dev/vitest
clone_url: https://github.com/vitest-dev/vitest.git
language: typescript
ticket_source: github
install:
  - "corepack enable"
  - "pnpm install --frozen-lockfile"
  - "pnpm run build"
test_command: "pnpm vitest run {test_files}"
test_patterns:
  - "test/**"
  - "**/*.test.ts"
  - "**/__tests__/**"
test_retries: 1
ticket_after: "2025-06-01"
wiki_commit: ""
toolchain:
  - "node:22"
services: []
```

- [ ] **Step 7: Build + commit.**

```bash
npm run build
git add benchmark/harness/types.ts benchmark/harness/types.js benchmark/harness/repo_config.ts benchmark/harness/repo_config.js benchmark/harness/tests/repo_config.test.ts benchmark/repos/vitest.yaml
git commit -m "feat(benchmark): shared types + repo config loader + vitest pilot config"
```

---

### Task 3: Issue-body sanitization

**Files:**
- Create: `benchmark/harness/sanitize.ts`
- Test: `benchmark/harness/tests/sanitize.test.ts`

- [ ] **Step 1: Write the failing test:**

```ts
import { describe, expect, it } from "vitest";
import { sanitizeIssueBody } from "../sanitize.js";

describe("sanitizeIssueBody", () => {
  it("strips cross-references >= the issue number, keeps earlier ones", () => {
    const r = sanitizeIssueBody("Same as #100. Fixed properly in #205 maybe.", 200);
    expect(r.text).toContain("#100");
    expect(r.text).not.toContain("#205");
    expect(r.redactions).toContain("#205");
  });

  it("strips github pull/commit URLs", () => {
    const r = sanitizeIssueBody("See https://github.com/acme/demo/pull/123 for the fix", 50);
    expect(r.text).not.toContain("pull/123");
  });

  it("strips bare commit SHAs", () => {
    const r = sanitizeIssueBody("broken since deadbeefcafe1234", 50);
    expect(r.text).not.toContain("deadbeefcafe1234");
  });

  it("strips whole lines saying fixed-by/closed-by", () => {
    const r = sanitizeIssueBody("Repro steps here.\nFixed by the patch in the linked PR.\nMore context.", 50);
    expect(r.text).toContain("Repro steps");
    expect(r.text).toContain("More context");
    expect(r.text).not.toMatch(/Fixed by/i);
  });

  it("returns empty redactions for a clean body", () => {
    const r = sanitizeIssueBody("Just a plain bug report.", 10);
    expect(r.redactions).toEqual([]);
    expect(r.text).toBe("Just a plain bug report.");
  });
});
```

- [ ] **Step 2: Run to verify FAIL** (`npx vitest run benchmark/harness/tests/sanitize.test.ts` — module not found).

- [ ] **Step 3: Implement `sanitize.ts`:**

```ts
export interface SanitizeResult {
  text: string;
  redactions: string[];
}

/**
 * Strip references that could leak the real fix to the agent:
 * - `#N` cross-references where N >= the issue's own number (the fix PR is
 *   always opened after the issue);
 * - github pull/issues/commit URLs;
 * - bare commit SHAs (7-40 hex chars);
 * - whole lines containing "fixed/closed/resolved by/in/via".
 * Every removal is logged so the sanitization is auditable.
 */
export function sanitizeIssueBody(body: string, issueNumber: number): SanitizeResult {
  const redactions: string[] = [];
  let text = body;

  text = text.replace(/https:\/\/github\.com\/\S+\/(?:pull|issues|commit)\/\S+/g, (m) => {
    redactions.push(m);
    return "[link-removed]";
  });

  text = text.replace(/(^|[^\w&])#(\d{1,7})\b/g, (m, pre: string, num: string) => {
    if (Number(num) >= issueNumber) {
      redactions.push(`#${num}`);
      return `${pre}#[ref-removed]`;
    }
    return m;
  });

  text = text.replace(/\b[0-9a-f]{7,40}\b/g, (m) => {
    redactions.push(m);
    return "[sha-removed]";
  });

  text = text
    .split("\n")
    .map((line) => {
      if (/\b(?:fixed|closed|resolved)\s+(?:by|in|via)\b/i.test(line)) {
        redactions.push(line.trim());
        return "[line-removed]";
      }
      return line;
    })
    .join("\n");

  return { text, redactions };
}
```

- [ ] **Step 4: Run tests to verify 5 PASS.**

- [ ] **Step 5: Build + commit.**

```bash
npm run build
git add benchmark/harness/sanitize.ts benchmark/harness/sanitize.js benchmark/harness/tests/sanitize.test.ts
git commit -m "feat(benchmark): auditable issue-body sanitization"
```

---

### Task 4: Checkpoint state machine + pair scheduler

**Files:**
- Create: `benchmark/harness/bench_checkpoint.ts`
- Test: `benchmark/harness/tests/bench_checkpoint.test.ts`

- [ ] **Step 1: Write the failing test:**

```ts
import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { loadState, nextPairs, runKey, saveState, setRun } from "../bench_checkpoint.js";
import type { BenchState } from "../types.js";

const fresh = (): BenchState => ({ schema_version: 1, repo: "demo", runs: {} });

describe("checkpoint state machine", () => {
  it("round-trips through disk and reverts transient states to pending", () => {
    const dir = mkdtempSync(join(tmpdir(), "benchst-"));
    const file = join(dir, "state.json");
    const s = fresh();
    setRun(s, 1, "baseline", { status: "running" });
    setRun(s, 1, "wiki", { status: "error", detail: "container crashed" });
    setRun(s, 2, "baseline", { status: "rate-limited" });
    setRun(s, 2, "wiki", { status: "passed" });
    saveState(file, s);
    const loaded = loadState(file, "demo");
    expect(loaded.runs[runKey(1, "baseline")]?.status).toBe("pending");
    expect(loaded.runs[runKey(1, "wiki")]?.status).toBe("pending");
    expect(loaded.runs[runKey(2, "baseline")]?.status).toBe("pending");
    expect(loaded.runs[runKey(2, "wiki")]?.status).toBe("passed"); // terminal survives
  });

  it("refuses to load a checkpoint for a different repo", () => {
    const dir = mkdtempSync(join(tmpdir(), "benchst-"));
    const file = join(dir, "state.json");
    saveState(file, fresh());
    expect(() => loadState(file, "other")).toThrow(/other/);
  });

  it("never overwrites a terminal run", () => {
    const s = fresh();
    setRun(s, 1, "wiki", { status: "passed" });
    expect(() => setRun(s, 1, "wiki", { status: "running" })).toThrow(/terminal/);
  });

  it("schedules partial pairs before fresh pairs, and batch counts fresh pairs only", () => {
    const s = fresh();
    // ticket 5: baseline done, wiki missing -> partial
    setRun(s, 5, "baseline", { status: "passed" });
    const work = nextPairs(s, [3, 5, 7, 9], 2);
    expect(work[0]).toEqual({ issue: 5, arms: ["wiki"] });
    // batch=2 fresh pairs follow
    expect(work.slice(1)).toEqual([
      { issue: 3, arms: ["baseline", "wiki"] },
      { issue: 7, arms: ["baseline", "wiki"] },
    ]);
  });

  it("skips fully-terminal tickets", () => {
    const s = fresh();
    setRun(s, 3, "baseline", { status: "passed" });
    setRun(s, 3, "wiki", { status: "failed" });
    expect(nextPairs(s, [3], 5)).toEqual([]);
  });
});
```

- [ ] **Step 2: Run to verify FAIL.**

- [ ] **Step 3: Implement `bench_checkpoint.ts`:**

```ts
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
import { dirname } from "node:path";
import type { Arm, BenchState, RunRecord, RunStatus } from "./types.js";

const TERMINAL: ReadonlySet<RunStatus> = new Set(["passed", "failed"]);
const TRANSIENT: ReadonlySet<RunStatus> = new Set(["running", "error", "rate-limited"]);
const ARMS: readonly Arm[] = ["baseline", "wiki"];

export const runKey = (issue: number, arm: Arm): string => `${issue}:${arm}`;

/** Load (or initialize) state; transient statuses revert to pending so resume re-queues them. `ran` survives — it resumes as grade-only. */
export function loadState(file: string, repo: string): BenchState {
  if (!existsSync(file)) return { schema_version: 1, repo, runs: {} };
  const state = JSON.parse(readFileSync(file, "utf8")) as BenchState;
  if (state.repo !== repo) {
    throw new Error(`checkpoint ${file} belongs to repo "${state.repo}", not "${repo}"`);
  }
  for (const rec of Object.values(state.runs)) {
    if (TRANSIENT.has(rec.status)) rec.status = "pending";
  }
  return state;
}

/** Atomic write: tmp file + rename. */
export function saveState(file: string, state: BenchState): void {
  mkdirSync(dirname(file), { recursive: true });
  const tmp = `${file}.tmp`;
  writeFileSync(tmp, `${JSON.stringify(state, null, 2)}\n`);
  renameSync(tmp, file);
}

export function setRun(state: BenchState, issue: number, arm: Arm, rec: RunRecord): void {
  const key = runKey(issue, arm);
  const prev = state.runs[key];
  if (prev !== undefined && TERMINAL.has(prev.status)) {
    throw new Error(`refusing to overwrite terminal run ${key} (${prev.status})`);
  }
  state.runs[key] = rec;
}

export interface WorkItem {
  issue: number;
  arms: Arm[];
}

/**
 * Pair scheduler. Partial pairs (one arm terminal, the other not) come first;
 * then up to `batch` fresh pairs. Terminal runs are never rescheduled.
 */
export function nextPairs(state: BenchState, issues: readonly number[], batch: number): WorkItem[] {
  const partial: WorkItem[] = [];
  const freshPairs: WorkItem[] = [];
  for (const issue of issues) {
    const missing = ARMS.filter((a) => {
      const r = state.runs[runKey(issue, a)];
      return r === undefined || !TERMINAL.has(r.status);
    });
    if (missing.length === 0) continue;
    if (missing.length === ARMS.length) freshPairs.push({ issue, arms: [...missing] });
    else partial.push({ issue, arms: [...missing] });
  }
  return [...partial, ...freshPairs.slice(0, batch)];
}
```

- [ ] **Step 4: Run tests to verify 5 PASS.**

- [ ] **Step 5: Build + commit.**

```bash
npm run build
git add benchmark/harness/bench_checkpoint.ts benchmark/harness/bench_checkpoint.js benchmark/harness/tests/bench_checkpoint.test.ts
git commit -m "feat(benchmark): checkpoint state machine + partial-pair-first scheduler"
```

---

### Task 5: Runner (injectable exec boundary)

**Files:**
- Create: `benchmark/harness/exec.ts`
- Test: `benchmark/harness/tests/exec.test.ts`

- [ ] **Step 1: Write the failing test:**

```ts
import { describe, expect, it } from "vitest";
import { realRunner } from "../exec.js";

describe("realRunner", () => {
  it("captures stdout and zero exit", async () => {
    const r = await realRunner("node", ["-e", "process.stdout.write('hi')"]);
    expect(r).toMatchObject({ code: 0, stdout: "hi" });
  });

  it("resolves (not rejects) on nonzero exit, with stderr", async () => {
    const r = await realRunner("node", ["-e", "console.error('boom'); process.exit(3)"]);
    expect(r.code).toBe(3);
    expect(r.stderr).toContain("boom");
  });
});
```

- [ ] **Step 2: Run to verify FAIL.**

- [ ] **Step 3: Implement `exec.ts`:**

```ts
import { execFile } from "node:child_process";

export interface ExecResult {
  code: number;
  stdout: string;
  stderr: string;
}

export interface ExecOpts {
  timeoutMs?: number;
  env?: NodeJS.ProcessEnv;
  cwd?: string;
}

/** Single injectable boundary for every external process (gh, docker, git, bash). Never rejects on nonzero exit. */
export type Runner = (cmd: string, args: readonly string[], opts?: ExecOpts) => Promise<ExecResult>;

export const realRunner: Runner = (cmd, args, opts = {}) =>
  new Promise((resolve) => {
    execFile(
      cmd,
      [...args],
      {
        timeout: opts.timeoutMs ?? 0,
        env: opts.env ?? process.env,
        cwd: opts.cwd,
        maxBuffer: 64 * 1024 * 1024,
      },
      (err, stdout, stderr) => {
        let code = 0;
        if (err !== null) {
          const c = (err as NodeJS.ErrnoException & { code?: unknown }).code;
          code = typeof c === "number" ? c : 1;
        }
        resolve({ code, stdout: String(stdout), stderr: String(stderr) });
      },
    );
  });
```

- [ ] **Step 4: Run tests to verify 2 PASS.**

- [ ] **Step 5: Build + commit.**

```bash
npm run build
git add benchmark/harness/exec.ts benchmark/harness/exec.js benchmark/harness/tests/exec.test.ts
git commit -m "feat(benchmark): injectable Runner exec boundary"
```

---

### Task 6: Eligibility filters (pure)

**Files:**
- Create: `benchmark/harness/mine_filters.ts`
- Test: `benchmark/harness/tests/mine_filters.test.ts`

- [ ] **Step 1: Write the failing test:**

```ts
import { describe, expect, it } from "vitest";
import { checkEligibility, splitFiles } from "../mine_filters.js";

const PATTERNS = ["test/**", "**/*.test.ts"];

describe("splitFiles", () => {
  it("classifies by the repo's test_patterns", () => {
    const r = splitFiles(
      [
        { path: "test/core/run.spec.ts", additions: 10, deletions: 2 },
        { path: "packages/vitest/src/run.ts", additions: 20, deletions: 5 },
        { path: "packages/ui/render.test.ts", additions: 3, deletions: 0 },
      ],
      PATTERNS,
    );
    expect(r.test).toEqual(["test/core/run.spec.ts", "packages/ui/render.test.ts"]);
    expect(r.src).toEqual(["packages/vitest/src/run.ts"]);
  });
});

const BASE = {
  files: [
    { path: "test/x.test.ts", additions: 10, deletions: 0 },
    { path: "src/x.ts", additions: 30, deletions: 10 },
  ],
  authorIsBot: false,
  bodyLength: 400,
  mergedAt: "2025-09-01T00:00:00Z",
};
const CFG = { test_patterns: PATTERNS, ticket_after: "2025-06-01" };

describe("checkEligibility", () => {
  it("accepts a qualifying PR", () => {
    const r = checkEligibility(BASE, CFG);
    expect(r.ok).toBe(true);
    if (r.ok) expect(r.changed_lines).toBe(50);
  });
  it("rejects test-only PRs", () => {
    const r = checkEligibility({ ...BASE, files: [BASE.files[0]!] }, CFG);
    expect(r).toEqual({ ok: false, reason: "no-source-changes" });
  });
  it("rejects source-only PRs", () => {
    const r = checkEligibility({ ...BASE, files: [BASE.files[1]!] }, CFG);
    expect(r).toEqual({ ok: false, reason: "no-test-changes" });
  });
  it("rejects PRs over 400 changed lines", () => {
    const big = { ...BASE, files: [BASE.files[0]!, { path: "src/y.ts", additions: 500, deletions: 0 }] };
    expect(checkEligibility(big, CFG)).toEqual({ ok: false, reason: "too-large" });
  });
  it("rejects bots, thin bodies, and pre-floor merges", () => {
    expect(checkEligibility({ ...BASE, authorIsBot: true }, CFG)).toEqual({ ok: false, reason: "bot-author" });
    expect(checkEligibility({ ...BASE, bodyLength: 50 }, CFG)).toEqual({ ok: false, reason: "thin-body" });
    expect(checkEligibility({ ...BASE, mergedAt: "2025-01-01T00:00:00Z" }, CFG)).toEqual({ ok: false, reason: "before-ticket-after" });
  });
});
```

- [ ] **Step 2: Run to verify FAIL.**

- [ ] **Step 3: Implement `mine_filters.ts`:**

```ts
import ignore from "ignore";

export interface PrFile {
  path: string;
  additions: number;
  deletions: number;
}

export interface SplitResult {
  test: string[];
  src: string[];
}

/** Classify changed files using the repo's gitignore-style test_patterns — the single definition of "test file". */
export function splitFiles(files: readonly PrFile[], testPatterns: readonly string[]): SplitResult {
  const matcher = ignore().add([...testPatterns]);
  const test: string[] = [];
  const src: string[] = [];
  for (const f of files) {
    (matcher.ignores(f.path) ? test : src).push(f.path);
  }
  return { test, src };
}

export const MAX_CHANGED_LINES = 400;
export const MIN_BODY_LENGTH = 200;

export interface EligibilityInput {
  files: readonly PrFile[];
  authorIsBot: boolean;
  bodyLength: number;
  mergedAt: string; // ISO
}

export interface EligibilityConfig {
  test_patterns: readonly string[];
  ticket_after: string; // ISO date
}

export type Eligibility =
  | { ok: true; test_files: string[]; src_files: string[]; changed_lines: number }
  | { ok: false; reason: string };

export function checkEligibility(input: EligibilityInput, cfg: EligibilityConfig): Eligibility {
  if (input.authorIsBot) return { ok: false, reason: "bot-author" };
  if (input.bodyLength < MIN_BODY_LENGTH) return { ok: false, reason: "thin-body" };
  if (Date.parse(input.mergedAt) < Date.parse(cfg.ticket_after)) {
    return { ok: false, reason: "before-ticket-after" };
  }
  const changed = input.files.reduce((n, f) => n + f.additions + f.deletions, 0);
  if (changed >= MAX_CHANGED_LINES) return { ok: false, reason: "too-large" };
  const { test, src } = splitFiles(input.files, cfg.test_patterns);
  if (test.length === 0) return { ok: false, reason: "no-test-changes" };
  if (src.length === 0) return { ok: false, reason: "no-source-changes" };
  return { ok: true, test_files: test, src_files: src, changed_lines: changed };
}
```

- [ ] **Step 4: Run tests to verify 7 PASS.**

- [ ] **Step 5: Build + commit.**

```bash
npm run build
git add benchmark/harness/mine_filters.ts benchmark/harness/mine_filters.js benchmark/harness/tests/mine_filters.test.ts
git commit -m "feat(benchmark): pure ticket-eligibility filters"
```

---

### Task 7: Mining CLI (github adapter)

**Files:**
- Create: `benchmark/harness/mine_tickets.ts`
- Test: `benchmark/harness/tests/mine_tickets.test.ts` + `benchmark/harness/tests/fixtures/gh_pr_list.json`, `gh_pr_view_42.json`, `gh_issue_17.json`, `gh_commit.json`

The github adapter mines via `gh` (always through the injected `Runner`):
1. `gh pr list --repo <github> --state merged --search "merged:>=<ticket_after>" --limit 200 --json number,url,mergedAt,author` — candidate PRs.
2. Per PR: `gh pr view <n> --repo <github> --json number,url,title,files,closingIssuesReferences,mergeCommit,mergedAt,author`.
3. Skip PRs with zero `closingIssuesReferences`; take the first referenced issue.
4. `gh api repos/<github>/issues/<issue>` — title + body (skip if it's actually a PR: response has `pull_request` key).
5. `gh api repos/<github>/commits/<mergeCommit.oid>` — `parents[0].sha` is `base_commit`.
6. Run `checkEligibility`; eligible records get `sanitizeIssueBody` and are appended.
7. Stop when `--target` (default 30) eligible tickets are collected; write `benchmark/tickets/<repo>.json` (`TicketsFile`, with every excluded candidate logged to stderr).

- [ ] **Step 1: Create the four fixtures** (realistic but generic; abbreviated bodies ≥200 chars). `gh_pr_list.json`:

```json
[{ "number": 42, "url": "https://github.com/acme/demo/pull/42", "mergedAt": "2025-09-01T12:00:00Z", "author": { "login": "alice", "is_bot": false } }]
```

`gh_pr_view_42.json`:

```json
{
  "number": 42,
  "url": "https://github.com/acme/demo/pull/42",
  "title": "fix: handle empty config",
  "files": [
    { "path": "test/config.test.ts", "additions": 12, "deletions": 0 },
    { "path": "src/config.ts", "additions": 8, "deletions": 2 }
  ],
  "closingIssuesReferences": [{ "number": 17 }],
  "mergeCommit": { "oid": "aaaa111122223333444455556666777788889999" },
  "mergedAt": "2025-09-01T12:00:00Z",
  "author": { "login": "alice", "is_bot": false }
}
```

`gh_issue_17.json`:

```json
{
  "number": 17,
  "html_url": "https://github.com/acme/demo/issues/17",
  "title": "Crash when config file is empty",
  "body": "When the config file exists but is empty, the loader crashes with a TypeError instead of falling back to defaults. Steps to reproduce: create an empty demo.config.ts, run the CLI, observe the stack trace. Expected: defaults are used and a warning is printed. This worked in 1.2."
}
```

`gh_commit.json`:

```json
{ "sha": "aaaa111122223333444455556666777788889999", "parents": [{ "sha": "bbbb111122223333444455556666777788880000" }] }
```

- [ ] **Step 2: Write the failing test:**

```ts
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import type { Runner } from "../exec.js";
import { mineGithub } from "../mine_tickets.js";

const fx = (name: string): string =>
  readFileSync(join(import.meta.dirname, "fixtures", name), "utf8");

const fakeGh: Runner = async (cmd, args) => {
  expect(cmd).toBe("gh");
  const joined = args.join(" ");
  if (joined.includes("pr list")) return { code: 0, stdout: fx("gh_pr_list.json"), stderr: "" };
  if (joined.includes("pr view 42")) return { code: 0, stdout: fx("gh_pr_view_42.json"), stderr: "" };
  if (joined.includes("issues/17")) return { code: 0, stdout: fx("gh_issue_17.json"), stderr: "" };
  if (joined.includes("commits/aaaa")) return { code: 0, stdout: fx("gh_commit.json"), stderr: "" };
  return { code: 1, stdout: "", stderr: `unexpected: ${joined}` };
};

const CFG = {
  id: "demo", github: "acme/demo", clone_url: "x", language: "ts",
  ticket_source: "github" as const, install: [], test_command: "t {test_files}",
  test_patterns: ["test/**"], test_retries: 0, ticket_after: "2025-06-01",
  wiki_commit: "", toolchain: [], services: [],
};

describe("mineGithub", () => {
  it("produces a sanitized, eligible ticket record", async () => {
    const out = await mineGithub(CFG, { target: 10, limit: 200, runner: fakeGh });
    expect(out.tickets).toHaveLength(1);
    const t = out.tickets[0]!;
    expect(t).toMatchObject({
      issue: 17,
      fix_pr: 42,
      base_commit: "bbbb111122223333444455556666777788880000",
      fix_commit: "aaaa111122223333444455556666777788889999",
      test_files: ["test/config.test.ts"],
      src_files: ["src/config.ts"],
      changed_lines: 22,
    });
    expect(t.body_sanitized.length).toBeGreaterThan(0);
    expect(out.schema_version).toBe(1);
  });
});
```

- [ ] **Step 3: Run to verify FAIL.**

- [ ] **Step 4: Implement `mine_tickets.ts`:**

```ts
import { writeFileSync } from "node:fs";
import { join } from "node:path";
import { parseFlags } from "../../skills/doc-wiki/scripts/_cli_args.js";
import type { Runner } from "./exec.js";
import { realRunner } from "./exec.js";
import { checkEligibility } from "./mine_filters.js";
import { loadRepoConfig } from "./repo_config.js";
import { sanitizeIssueBody } from "./sanitize.js";
import type { RepoConfig, TicketRecord, TicketsFile } from "./types.js";

interface MineOpts {
  target: number;
  limit: number;
  runner: Runner;
}

async function ghJson<T>(runner: Runner, args: string[]): Promise<T | undefined> {
  const r = await runner("gh", args);
  if (r.code !== 0) {
    process.stderr.write(`gh ${args.join(" ")} failed: ${r.stderr}\n`);
    return undefined;
  }
  return JSON.parse(r.stdout) as T;
}

interface PrListEntry { number: number; url: string; mergedAt: string; author?: { login?: string; is_bot?: boolean } }
interface PrView {
  number: number; url: string; title: string;
  files: Array<{ path: string; additions: number; deletions: number }>;
  closingIssuesReferences: Array<{ number: number }>;
  mergeCommit: { oid: string } | null;
  mergedAt: string;
  author?: { login?: string; is_bot?: boolean };
}
interface IssueView { number: number; html_url: string; title: string; body: string | null; pull_request?: unknown }
interface CommitView { sha: string; parents: Array<{ sha: string }> }

export async function mineGithub(cfg: RepoConfig, opts: MineOpts): Promise<TicketsFile> {
  const tickets: TicketRecord[] = [];
  const seenIssues = new Set<number>();

  const prs = await ghJson<PrListEntry[]>(opts.runner, [
    "pr", "list", "--repo", cfg.github, "--state", "merged",
    "--search", `merged:>=${cfg.ticket_after}`,
    "--limit", String(opts.limit), "--json", "number,url,mergedAt,author",
  ]);
  if (prs === undefined) throw new Error("gh pr list failed");

  for (const entry of prs) {
    if (tickets.length >= opts.target) break;

    const pr = await ghJson<PrView>(opts.runner, [
      "pr", "view", String(entry.number), "--repo", cfg.github,
      "--json", "number,url,title,files,closingIssuesReferences,mergeCommit,mergedAt,author",
    ]);
    if (pr === undefined || pr.mergeCommit === null) continue;
    const issueRef = pr.closingIssuesReferences[0];
    if (issueRef === undefined) { note(pr.number, "no-linked-issue"); continue; }
    if (seenIssues.has(issueRef.number)) { note(pr.number, "duplicate-issue"); continue; }

    const issue = await ghJson<IssueView>(opts.runner, ["api", `repos/${cfg.github}/issues/${issueRef.number}`]);
    if (issue === undefined || issue.pull_request !== undefined) { note(pr.number, "ref-not-an-issue"); continue; }
    const body = issue.body ?? "";

    const verdict = checkEligibility(
      { files: pr.files, authorIsBot: pr.author?.is_bot === true, bodyLength: body.length, mergedAt: pr.mergedAt },
      cfg,
    );
    if (!verdict.ok) { note(pr.number, verdict.reason); continue; }

    const commit = await ghJson<CommitView>(opts.runner, ["api", `repos/${cfg.github}/commits/${pr.mergeCommit.oid}`]);
    const parent = commit?.parents[0];
    if (parent === undefined) { note(pr.number, "no-parent-commit"); continue; }

    const sanitized = sanitizeIssueBody(body, issue.number);
    seenIssues.add(issue.number);
    tickets.push({
      issue: issue.number,
      issue_url: issue.html_url,
      title: issue.title,
      body,
      body_sanitized: sanitized.text,
      fix_pr: pr.number,
      fix_pr_url: pr.url,
      base_commit: parent.sha,
      fix_commit: pr.mergeCommit.oid,
      test_files: verdict.test_files,
      src_files: verdict.src_files,
      changed_lines: verdict.changed_lines,
      merged_at: pr.mergedAt,
    });
    if (sanitized.redactions.length > 0) {
      process.stderr.write(`issue #${issue.number}: redacted ${sanitized.redactions.join(", ")}\n`);
    }
  }
  return { schema_version: 1, repo: cfg.id, mined_at: new Date().toISOString(), tickets };
}

function note(pr: number, reason: string): void {
  process.stderr.write(`PR #${pr}: skipped (${reason})\n`);
}

export async function main(argv: readonly string[]): Promise<number> {
  const { help, values } = parseFlags(argv, {
    "--repo": "repo", "--target": "target", "--limit": "limit", "--out-dir": "outDir",
  });
  if (help || values.repo === undefined) {
    process.stderr.write("usage: benchmark mine --repo <id> [--target 30] [--limit 200] [--out-dir benchmark/tickets]\n");
    return help ? 0 : 2;
  }
  const cfg = loadRepoConfig(join("benchmark", "repos", `${String(values.repo)}.yaml`));
  if (cfg.ticket_source !== "github") {
    process.stderr.write(`ticket_source "${cfg.ticket_source}" not implemented yet (github only)\n`);
    return 2;
  }
  const out = await mineGithub(cfg, {
    target: values.target === undefined ? 30 : Number(values.target),
    limit: values.limit === undefined ? 200 : Number(values.limit),
    runner: realRunner,
  });
  const outPath = join(String(values.outDir ?? "benchmark/tickets"), `${cfg.id}.json`);
  writeFileSync(outPath, `${JSON.stringify(out, null, 2)}\n`);
  process.stderr.write(`${out.tickets.length} eligible tickets -> ${outPath}\n`);
  return 0;
}
```

- [ ] **Step 5: Run tests to verify PASS** (`npx vitest run benchmark/harness/tests/mine_tickets.test.ts`).

- [ ] **Step 6: Typecheck** (`npm run typecheck`) — the cross-directory import of `_cli_args.js` must resolve; if it fails, the fix is to import via the relative path shown (the repo builds in-place, so the `.js` sibling exists after build).

- [ ] **Step 7: Build + commit.**

```bash
npm run build
git add benchmark/harness/mine_tickets.ts benchmark/harness/mine_tickets.js benchmark/harness/tests/mine_tickets.test.ts benchmark/harness/tests/fixtures/gh_pr_list.json benchmark/harness/tests/fixtures/gh_pr_view_42.json benchmark/harness/tests/fixtures/gh_issue_17.json benchmark/harness/tests/fixtures/gh_commit.json
git commit -m "feat(benchmark): github ticket-mining adapter + CLI"
```

---

### Task 8: Docker assets + argv builders

**Files:**
- Create: `benchmark/harness/docker/Dockerfile`, `benchmark/harness/docker/entrypoint.sh`, `benchmark/harness/docker/init-firewall.sh`, `benchmark/harness/docker_args.ts`
- Test: `benchmark/harness/tests/docker_args.test.ts`

- [ ] **Step 1: Write `Dockerfile`:**

```dockerfile
# Benchmark session/grade container. Build: docker build -t docwiki-bench-<repo> --build-arg TOOLCHAIN=node:22 benchmark/harness/docker/
ARG TOOLCHAIN=node:22
FROM ${TOOLCHAIN}
ARG CLAUDE_CODE_VERSION=latest
ENV DISABLE_AUTOUPDATER=1 \
    CLAUDE_CONFIG_DIR=/claude-cfg
RUN apt-get update && apt-get install -y --no-install-recommends git iptables dnsutils ca-certificates bash \
    && rm -rf /var/lib/apt/lists/* \
    && npm install -g @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION} \
    && mkdir -p /claude-cfg /out /work
COPY entrypoint.sh grade.sh init-firewall.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/entrypoint.sh /usr/local/bin/grade.sh /usr/local/bin/init-firewall.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
```

(`grade.sh` is created in Task 11; create an empty executable placeholder now so the image builds: `printf '#!/usr/bin/env bash\nexit 64\n' > benchmark/harness/docker/grade.sh` — Task 11 replaces it.)

- [ ] **Step 2: Write `entrypoint.sh`:**

```bash
#!/usr/bin/env bash
# Modes: entrypoint.sh session | entrypoint.sh grade
# session env: BENCH_BASE_COMMIT, BENCH_MODEL, BENCH_MAX_TURNS, BENCH_INSTALL, CLAUDE_CODE_OAUTH_TOKEN
# mounts:     /bare (ro bare clone), /out (rw artifacts), /wiki (ro overlay, wiki arm only)
set -uo pipefail

mode="${1:?usage: entrypoint.sh session|grade}"
if [ "$mode" = "grade" ]; then
  exec /usr/local/bin/grade.sh
fi

set -e
git clone --no-hardlinks /bare /work
cd /work
git checkout -q "$BENCH_BASE_COMMIT"
git config user.email bench@localhost && git config user.name bench

# Install with normal egress (package registries), BEFORE the firewall comes up.
bash -ec "$BENCH_INSTALL"

# Wiki arm: overlay the pre-built wiki + CLAUDE.md pointer at the repo root.
if [ -d /wiki ]; then
  cp -R /wiki/. /work/
fi

# From here on: Anthropic-only egress. The session cannot look up the real fix.
/usr/local/bin/init-firewall.sh

set +e
claude -p "$(cat /out/prompt.txt)" \
  --model "$BENCH_MODEL" \
  --max-turns "$BENCH_MAX_TURNS" \
  --output-format json \
  --dangerously-skip-permissions \
  >/out/result.json 2>/out/stderr.log
echo "$?" >/out/exit_code
set -e

# Capture the agent's full working-tree delta (incl. new files), minus any wiki overlay noise.
git add -A
git diff --cached --binary >/out/diff.patch

# Publish the transcript for auditability.
mkdir -p /out/transcript
cp -R "$CLAUDE_CONFIG_DIR"/. /out/transcript/ 2>/dev/null || true
```

- [ ] **Step 3: Write `init-firewall.sh`** (Anthropic-only egress; same approach as Anthropic's reference devcontainer):

```bash
#!/usr/bin/env bash
# Lock egress to Anthropic endpoints only. Requires --cap-add=NET_ADMIN.
set -euo pipefail

ALLOWED_DOMAINS=(api.anthropic.com claude.ai statsig.anthropic.com sentry.io)

iptables -F OUTPUT
# Loopback + established flows + DNS stay open.
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -p udp --dport 53 -j ACCEPT
iptables -A OUTPUT -p tcp --dport 53 -j ACCEPT

for domain in "${ALLOWED_DOMAINS[@]}"; do
  for ip in $(dig +short A "$domain" | grep -E '^[0-9.]+$'); do
    iptables -A OUTPUT -d "$ip" -p tcp --dport 443 -j ACCEPT
  done
done

iptables -A OUTPUT -j REJECT
echo "egress locked to: ${ALLOWED_DOMAINS[*]}" >&2
```

- [ ] **Step 4: Lint the shell scripts.** Run: `bash -n benchmark/harness/docker/entrypoint.sh && bash -n benchmark/harness/docker/init-firewall.sh`. Expected: silence (exit 0).

- [ ] **Step 5: Write the failing test** `benchmark/harness/tests/docker_args.test.ts`:

```ts
import { describe, expect, it } from "vitest";
import { buildImageArgs, gradeRunArgs, sessionRunArgs } from "../docker_args.js";

const SPEC = {
  image: "docwiki-bench-vitest",
  outDir: "/abs/runs/vitest/17/wiki",
  bareDir: "/abs/cache/vitest.git",
  wikiDir: "/abs/wiki-cache/vitest/overlay",
  baseCommit: "bbbb0000",
  model: "claude-sonnet-4-6",
  maxTurns: 80,
  install: ["corepack enable", "pnpm install"],
  timeoutSec: 1800,
};

describe("docker argv builders", () => {
  it("session args mount bare ro, out rw, wiki ro; pass token by name only", () => {
    const args = sessionRunArgs(SPEC);
    expect(args[0]).toBe("run");
    expect(args).toContain("--cap-add=NET_ADMIN");
    expect(args).toContain("/abs/cache/vitest.git:/bare:ro");
    expect(args).toContain("/abs/runs/vitest/17/wiki:/out");
    expect(args).toContain("/abs/wiki-cache/vitest/overlay:/wiki:ro");
    expect(args).toContain("CLAUDE_CODE_OAUTH_TOKEN"); // name-only -e: value comes from harness env
    expect(args.join(" ")).not.toMatch(/sk-|oauth/i); // no secret material in argv
    expect(args[args.length - 2]).toBe(SPEC.image);
    expect(args[args.length - 1]).toBe("session");
  });

  it("baseline session has no /wiki mount", () => {
    const args = sessionRunArgs({ ...SPEC, wikiDir: undefined });
    expect(args.join(" ")).not.toContain("/wiki");
  });

  it("grade args end with the grade mode", () => {
    const args = gradeRunArgs({
      image: SPEC.image, outDir: SPEC.outDir, bareDir: SPEC.bareDir,
      baseCommit: "bbbb0000", fixCommit: "aaaa9999",
      testFiles: ["test/a.test.ts"], testCommand: "npx vitest run {test_files}", retries: 1,
    });
    expect(args[args.length - 1]).toBe("grade");
    expect(args).toContain("BENCH_TEST_FILES=test/a.test.ts");
  });

  it("build args pin the toolchain", () => {
    const args = buildImageArgs("docwiki-bench-vitest", "node:22", "benchmark/harness/docker");
    expect(args).toEqual([
      "build", "-t", "docwiki-bench-vitest", "--build-arg", "TOOLCHAIN=node:22", "benchmark/harness/docker",
    ]);
  });
});
```

- [ ] **Step 6: Run to verify FAIL, then implement `docker_args.ts`:**

```ts
export interface SessionSpec {
  image: string;
  outDir: string; // absolute host path for /out
  bareDir: string; // absolute host path of the cached bare clone
  wikiDir?: string; // absolute host path of the wiki overlay (wiki arm only)
  baseCommit: string;
  model: string;
  maxTurns: number;
  install: string[];
  timeoutSec: number;
}

/** docker run argv for one agent session. The OAuth token is passed by NAME only (-e VAR) so it never appears in argv/process listings. */
export function sessionRunArgs(s: SessionSpec): string[] {
  const args = [
    "run", "--rm",
    "--cap-add=NET_ADMIN",
    "--stop-timeout", "10",
    "-v", `${s.bareDir}:/bare:ro`,
    "-v", `${s.outDir}:/out`,
    "-e", "CLAUDE_CODE_OAUTH_TOKEN",
    "-e", `BENCH_BASE_COMMIT=${s.baseCommit}`,
    "-e", `BENCH_MODEL=${s.model}`,
    "-e", `BENCH_MAX_TURNS=${s.maxTurns}`,
    "-e", `BENCH_INSTALL=${s.install.join(" && ")}`,
  ];
  if (s.wikiDir !== undefined) args.push("-v", `${s.wikiDir}:/wiki:ro`);
  args.push(s.image, "session");
  return args;
}

export interface GradeSpec {
  image: string;
  outDir: string;
  bareDir: string;
  baseCommit: string;
  fixCommit: string;
  testFiles: string[];
  testCommand: string;
  retries: number;
}

export function gradeRunArgs(g: GradeSpec): string[] {
  return [
    "run", "--rm",
    "-v", `${g.bareDir}:/bare:ro`,
    "-v", `${g.outDir}:/out`,
    "-e", `BENCH_BASE_COMMIT=${g.baseCommit}`,
    "-e", `BENCH_FIX_COMMIT=${g.fixCommit}`,
    "-e", `BENCH_TEST_FILES=${g.testFiles.join(" ")}`,
    "-e", `BENCH_TEST_COMMAND=${g.testCommand}`,
    "-e", `BENCH_RETRIES=${g.retries}`,
    g.image, "grade",
  ];
}

export function buildImageArgs(tag: string, toolchain: string, contextDir: string): string[] {
  return ["build", "-t", tag, "--build-arg", `TOOLCHAIN=${toolchain}`, contextDir];
}
```

- [ ] **Step 7: Run tests to verify 4 PASS.**

- [ ] **Step 8: Build + commit.**

```bash
npm run build
git add benchmark/harness/docker/Dockerfile benchmark/harness/docker/entrypoint.sh benchmark/harness/docker/init-firewall.sh benchmark/harness/docker/grade.sh benchmark/harness/docker_args.ts benchmark/harness/docker_args.js benchmark/harness/tests/docker_args.test.ts
git commit -m "feat(benchmark): docker image assets + pure argv builders"
```

---

### Task 9: Prompt template + session-result classifier

**Files:**
- Create: `benchmark/harness/session.ts`
- Test: `benchmark/harness/tests/session.test.ts`

- [ ] **Step 1: Write the failing test:**

```ts
import { describe, expect, it } from "vitest";
import { buildPrompt, classifySession } from "../session.js";
import type { TicketRecord } from "../types.js";

const ticket = {
  title: "Crash when config file is empty",
  body_sanitized: "When the config file exists but is empty, the loader crashes.",
} as TicketRecord;

describe("buildPrompt", () => {
  it("is title + sanitized body + the fixed instruction, nothing else", () => {
    expect(buildPrompt(ticket)).toBe(
      "Crash when config file is empty\n\n" +
        "When the config file exists but is empty, the loader crashes.\n\n" +
        "Investigate and fix this issue in this repository. Run the relevant tests to check your fix.",
    );
  });
});

describe("classifySession", () => {
  it("ok: parses cost + session id from the json envelope", () => {
    const r = classifySession(JSON.stringify({ result: "done", total_cost_usd: 1.23, session_id: "s1" }), 0, "");
    expect(r).toEqual({ kind: "ok", costUsd: 1.23, sessionId: "s1", detail: undefined });
  });

  it("rate-limited: detected in result text", () => {
    const r = classifySession(
      JSON.stringify({ result: "You've hit your session limit · resets 3:45pm", session_id: "s2" }),
      0, "",
    );
    expect(r.kind).toBe("rate-limited");
    expect(r.detail).toContain("resets 3:45pm");
  });

  it("rate-limited: detected on stderr even with bad envelope", () => {
    const r = classifySession("", 1, "You've hit your weekly limit · resets Mon 12:00am");
    expect(r.kind).toBe("rate-limited");
  });

  it("error: nonzero exit or unparseable envelope", () => {
    expect(classifySession("not json", 0, "").kind).toBe("error");
    expect(classifySession(JSON.stringify({ result: "x" }), 9, "boom").kind).toBe("error");
  });
});
```

- [ ] **Step 2: Run to verify FAIL, then implement `session.ts`:**

```ts
import type { SessionResult, TicketRecord } from "./types.js";

/** The exact, fixed prompt both arms receive. The wiki is never mentioned — the wiki arm must discover it via CLAUDE.md like a real agent. */
export function buildPrompt(t: TicketRecord): string {
  return `${t.title}\n\n${t.body_sanitized}\n\nInvestigate and fix this issue in this repository. Run the relevant tests to check your fix.`;
}

const RATE_LIMIT = /You've hit your .{0,40}limit[^\n"]*/i;

/**
 * Classify a finished session from its JSON envelope + exit code + stderr.
 * Rate-limit detection is best-effort: anything unrecognized is "error"
 * (re-queued on resume), so a changed message can never corrupt results.
 */
export function classifySession(resultJson: string, exitCode: number, stderr: string): SessionResult {
  const limitHit = RATE_LIMIT.exec(resultJson) ?? RATE_LIMIT.exec(stderr);
  if (limitHit !== null) return { kind: "rate-limited", detail: limitHit[0] };

  let envelope: { result?: unknown; total_cost_usd?: unknown; session_id?: unknown };
  try {
    envelope = JSON.parse(resultJson) as typeof envelope;
  } catch {
    return { kind: "error", detail: `unparseable result envelope (exit ${exitCode})` };
  }
  if (exitCode !== 0) return { kind: "error", detail: `claude exited ${exitCode}: ${stderr.slice(0, 200)}` };
  return {
    kind: "ok",
    costUsd: typeof envelope.total_cost_usd === "number" ? envelope.total_cost_usd : undefined,
    sessionId: typeof envelope.session_id === "string" ? envelope.session_id : undefined,
    detail: undefined,
  };
}
```

- [ ] **Step 3: Run tests to verify 5 PASS.**

- [ ] **Step 4: Build + commit.**

```bash
npm run build
git add benchmark/harness/session.ts benchmark/harness/session.js benchmark/harness/tests/session.test.ts
git commit -m "feat(benchmark): fixed prompt template + session-result classifier"
```

---

### Task 10: Run orchestrator

**Files:**
- Create: `benchmark/harness/run_ticket.ts`
- Test: `benchmark/harness/tests/run_ticket.test.ts`

The orchestrator: load config + tickets (skipping `excluded` and failed-calibration tickets) + state → `nextPairs` → for each work item, for each missing arm (baseline first): mark `running`, write `prompt.txt` into the run dir, `docker run` (session) via Runner, read back `result.json`/`stderr.log`/`exit_code`, classify; `ok` → status `ran`; `rate-limited` → mark + **stop the whole batch**; `error` → mark + continue with next ticket. Checkpoint is saved after every transition. Grading is a separate pass (Task 11) — `run` leaves runs at `ran`.

- [ ] **Step 1: Write the failing test:**

```ts
import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import type { Runner } from "../exec.js";
import { runBatch } from "../run_ticket.js";
import { loadState, runKey } from "../bench_checkpoint.js";
import type { RepoConfig, TicketsFile } from "../types.js";

const CFG: RepoConfig = {
  id: "demo", github: "acme/demo", clone_url: "x", language: "ts",
  ticket_source: "github", install: ["true"], test_command: "t {test_files}",
  test_patterns: ["test/**"], test_retries: 0, ticket_after: "2025-06-01",
  wiki_commit: "cccc", toolchain: ["node:22"], services: [],
};

const ticket = (issue: number) => ({
  issue, issue_url: "u", title: `t${issue}`, body: "b", body_sanitized: "b",
  fix_pr: issue + 1, fix_pr_url: "u", base_commit: "bbbb", fix_commit: "aaaa",
  test_files: ["test/a.test.ts"], src_files: ["src/a.ts"], changed_lines: 10,
  merged_at: "2025-09-01T00:00:00Z",
  calibration: { paths_stable: true, tests_fail_on_base: true, tests_pass_on_fix: true },
});

function setup(tickets: TicketsFile["tickets"]): { root: string; ticketsPath: string } {
  const root = mkdtempSync(join(tmpdir(), "benchrun-"));
  mkdirSync(join(root, "tickets"), { recursive: true });
  const ticketsPath = join(root, "tickets", "demo.json");
  writeFileSync(ticketsPath, JSON.stringify({ schema_version: 1, repo: "demo", mined_at: "x", tickets }));
  return { root, ticketsPath };
}

/** Fake docker: succeeds, writing a plausible /out result envelope. */
const okDocker = (costs: number[]): Runner => async (cmd, args) => {
  expect(cmd).toBe("docker");
  const outDir = String(args.find((a) => a.includes(":/out"))).split(":")[0];
  writeFileSync(join(String(outDir), "result.json"), JSON.stringify({ result: "ok", total_cost_usd: costs.shift() ?? 0.5, session_id: "s" }));
  writeFileSync(join(String(outDir), "stderr.log"), "");
  writeFileSync(join(String(outDir), "exit_code"), "0");
  writeFileSync(join(String(outDir), "diff.patch"), "diff --git a/x b/x\n");
  return { code: 0, stdout: "", stderr: "" };
};

describe("runBatch", () => {
  it("runs both arms of each pair and records ran + cost", async () => {
    const { root, ticketsPath } = setup([ticket(1)]);
    const summary = await runBatch({
      cfg: CFG, ticketsPath, runsRoot: join(root, "runs"), bareDir: "/bare", wikiDir: "/wiki",
      image: "img", model: "claude-sonnet-4-6", maxTurns: 80, batch: 5, timeoutSec: 60,
      runner: okDocker([0.5, 0.7]),
    });
    expect(summary).toMatchObject({ ran: 2, rateLimited: 0, errors: 0 });
    const state = loadState(join(root, "runs", "demo", "state.json"), "demo");
    expect(state.runs[runKey(1, "baseline")]).toMatchObject({ status: "ran", cost_usd: 0.5 });
    expect(state.runs[runKey(1, "wiki")]).toMatchObject({ status: "ran", cost_usd: 0.7 });
  });

  it("stops the batch on rate-limit and persists the reset hint", async () => {
    const { root, ticketsPath } = setup([ticket(1), ticket(2)]);
    let calls = 0;
    const limited: Runner = async (_cmd, args) => {
      const outDir = String(args.find((a) => a.includes(":/out"))).split(":")[0];
      calls += 1;
      const envelope = calls === 1
        ? { result: "ok", total_cost_usd: 0.5, session_id: "s" }
        : { result: "You've hit your session limit · resets 3:45pm", session_id: "s" };
      writeFileSync(join(String(outDir), "result.json"), JSON.stringify(envelope));
      writeFileSync(join(String(outDir), "stderr.log"), "");
      writeFileSync(join(String(outDir), "exit_code"), "0");
      writeFileSync(join(String(outDir), "diff.patch"), "");
      return { code: 0, stdout: "", stderr: "" };
    };
    const summary = await runBatch({
      cfg: CFG, ticketsPath, runsRoot: join(root, "runs"), bareDir: "/b", wikiDir: "/w",
      image: "img", model: "m", maxTurns: 80, batch: 5, timeoutSec: 60, runner: limited,
    });
    expect(summary).toMatchObject({ ran: 1, rateLimited: 1 });
    expect(calls).toBe(2); // ticket 2 never started
    const state = loadState(join(root, "runs", "demo", "state.json"), "demo");
    // rate-limited reverts to pending on load; the persisted detail carried the hint
    const raw = JSON.parse(readFileSync(join(root, "runs", "demo", "state.json"), "utf8"));
    expect(raw.runs["1:wiki"].detail).toContain("resets 3:45pm");
    expect(state.runs["2:baseline"]).toBeUndefined();
  });

  it("skips excluded tickets entirely", async () => {
    const { root, ticketsPath } = setup([{ ...ticket(3), excluded: "calibration-failed" }]);
    const summary = await runBatch({
      cfg: CFG, ticketsPath, runsRoot: join(root, "runs"), bareDir: "/b", wikiDir: "/w",
      image: "img", model: "m", maxTurns: 80, batch: 5, timeoutSec: 60,
      runner: async () => { throw new Error("must not be called"); },
    });
    expect(summary).toMatchObject({ ran: 0 });
  });
});
```

- [ ] **Step 2: Run to verify FAIL, then implement `run_ticket.ts`:**

```ts
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { join, resolve } from "node:path";
import { parseFlags } from "../../skills/doc-wiki/scripts/_cli_args.js";
import { loadState, nextPairs, runKey, saveState, setRun } from "./bench_checkpoint.js";
import { sessionRunArgs } from "./docker_args.js";
import type { Runner } from "./exec.js";
import { realRunner } from "./exec.js";
import { loadRepoConfig } from "./repo_config.js";
import { buildPrompt, classifySession } from "./session.js";
import type { Arm, RepoConfig, TicketsFile } from "./types.js";

export interface RunBatchOpts {
  cfg: RepoConfig;
  ticketsPath: string;
  runsRoot: string;
  bareDir: string;
  wikiDir: string; // overlay dir; only mounted for the wiki arm
  image: string;
  model: string;
  maxTurns: number;
  batch: number;
  timeoutSec: number;
  runner: Runner;
}

export interface RunBatchSummary {
  ran: number;
  rateLimited: number;
  errors: number;
}

function readOut(outDir: string, name: string): string {
  const p = join(outDir, name);
  return existsSync(p) ? readFileSync(p, "utf8") : "";
}

export async function runBatch(opts: RunBatchOpts): Promise<RunBatchSummary> {
  const ticketsFile = JSON.parse(readFileSync(opts.ticketsPath, "utf8")) as TicketsFile;
  const runnable = ticketsFile.tickets.filter(
    (t) => t.excluded === undefined &&
      (t.calibration === undefined ||
        (t.calibration.paths_stable && t.calibration.tests_fail_on_base && t.calibration.tests_pass_on_fix)),
  );
  const byIssue = new Map(runnable.map((t) => [t.issue, t]));
  const stateFile = join(opts.runsRoot, opts.cfg.id, "state.json");
  const state = loadState(stateFile, opts.cfg.id);

  const summary: RunBatchSummary = { ran: 0, rateLimited: 0, errors: 0 };
  const work = nextPairs(state, runnable.map((t) => t.issue), opts.batch);

  for (const item of work) {
    const ticket = byIssue.get(item.issue);
    if (ticket === undefined) continue;
    for (const arm of item.arms) {
      const outDir = resolve(opts.runsRoot, opts.cfg.id, String(item.issue), arm);
      mkdirSync(outDir, { recursive: true });
      writeFileSync(join(outDir, "prompt.txt"), buildPrompt(ticket));

      setRun(state, item.issue, arm, { status: "running", started_at: new Date().toISOString() });
      saveState(stateFile, state);

      const dockerArgs = sessionRunArgs({
        image: opts.image,
        outDir,
        bareDir: opts.bareDir,
        wikiDir: arm === "wiki" ? opts.wikiDir : undefined,
        baseCommit: ticket.base_commit,
        model: opts.model,
        maxTurns: opts.maxTurns,
        install: opts.cfg.install,
        timeoutSec: opts.timeoutSec,
      });
      const exec = await opts.runner("docker", dockerArgs, { timeoutMs: opts.timeoutSec * 1000 });

      const result = classifySession(
        readOut(outDir, "result.json"),
        exec.code !== 0 ? exec.code : Number(readOut(outDir, "exit_code").trim() || "0"),
        readOut(outDir, "stderr.log") + exec.stderr,
      );
      const finished = new Date().toISOString();

      if (result.kind === "ok") {
        setRun(state, item.issue, arm, {
          status: "ran", started_at: state.runs[runKey(item.issue, arm)]?.started_at,
          finished_at: finished, cost_usd: result.costUsd, session_id: result.sessionId,
        });
        summary.ran += 1;
      } else if (result.kind === "rate-limited") {
        setRun(state, item.issue, arm, { status: "rate-limited", finished_at: finished, detail: result.detail });
        saveState(stateFile, state);
        summary.rateLimited += 1;
        process.stderr.write(`rate limit hit (${result.detail ?? ""}) — stopping batch; resume with the same command\n`);
        return summary;
      } else {
        setRun(state, item.issue, arm, { status: "error", finished_at: finished, detail: result.detail });
        summary.errors += 1;
      }
      saveState(stateFile, state);
    }
  }
  return summary;
}

export async function main(argv: readonly string[]): Promise<number> {
  const { help, values } = parseFlags(argv, {
    "--repo": "repo", "--batch": "batch", "--max-turns": "maxTurns",
    "--timeout-sec": "timeoutSec", "--model": "model",
    "--bare-dir": "bareDir", "--wiki-dir": "wikiDir", "--image": "image",
  });
  if (help || values.repo === undefined) {
    process.stderr.write(
      "usage: benchmark run --repo <id> [--batch 10] [--max-turns 80] [--timeout-sec 1800] [--model claude-sonnet-4-6] [--bare-dir d] [--wiki-dir d] [--image i]\n",
    );
    return help ? 0 : 2;
  }
  if (process.env.CLAUDE_CODE_OAUTH_TOKEN === undefined) {
    process.stderr.write("CLAUDE_CODE_OAUTH_TOKEN is not set (run: claude setup-token)\n");
    return 2;
  }
  const repo = String(values.repo);
  const cfg = loadRepoConfig(join("benchmark", "repos", `${repo}.yaml`));
  const summary = await runBatch({
    cfg,
    ticketsPath: join("benchmark", "tickets", `${repo}.json`),
    runsRoot: join("benchmark", "runs"),
    bareDir: String(values.bareDir ?? resolve("benchmark", "wiki-cache", `${repo}.git`)),
    wikiDir: String(values.wikiDir ?? resolve("benchmark", "wiki-cache", repo, "overlay")),
    image: String(values.image ?? `docwiki-bench-${repo}`),
    model: String(values.model ?? "claude-sonnet-4-6"),
    maxTurns: values.maxTurns === undefined ? 80 : Number(values.maxTurns),
    batch: values.batch === undefined ? 10 : Number(values.batch),
    timeoutSec: values.timeoutSec === undefined ? 1800 : Number(values.timeoutSec),
    runner: realRunner,
  });
  process.stderr.write(`ran=${summary.ran} rate-limited=${summary.rateLimited} errors=${summary.errors}\n`);
  return summary.errors > 0 ? 1 : 0;
}
```

- [ ] **Step 3: Run tests to verify 3 PASS.**

- [ ] **Step 4: Build + commit.**

```bash
npm run build
git add benchmark/harness/run_ticket.ts benchmark/harness/run_ticket.js benchmark/harness/tests/run_ticket.test.ts
git commit -m "feat(benchmark): paired-arm batch orchestrator with rate-limit stop"
```

---

### Task 11: Grading + calibration

**Files:**
- Create: `benchmark/harness/grade.ts`; Replace placeholder: `benchmark/harness/docker/grade.sh`
- Test: `benchmark/harness/tests/grade.test.ts`

`grade.sh` does the in-container work; exit codes are the contract: `0` tests-passed, `10` apply-failed, `20` tests-failed, `64` config error. `grade.ts` has the pure decision (`decideGrade`) plus two drivers: `gradeRun` (docker or `--local` for tests/e2e — local runs `grade.sh` directly with env vars) and `calibrateAll`.

- [ ] **Step 1: Write `grade.sh`** (replaces the Task 8 placeholder):

```bash
#!/usr/bin/env bash
# Grade one run. env: BENCH_BASE_COMMIT, BENCH_FIX_COMMIT, BENCH_TEST_FILES, BENCH_TEST_COMMAND, BENCH_RETRIES
# Local mode (tests/e2e): BENCH_BARE_DIR + BENCH_OUT_DIR override /bare and /out.
# Modes via BENCH_GRADE_MODE: grade (default) | calibrate-base | calibrate-fix
# exit: 0 tests-passed | 10 apply-failed | 20 tests-failed | 64 setup error
set -uo pipefail
BARE="${BENCH_BARE_DIR:-/bare}"
OUT="${BENCH_OUT_DIR:-/out}"
MODE="${BENCH_GRADE_MODE:-grade}"

work="$(mktemp -d)"
git clone --no-hardlinks -q "$BARE" "$work" || exit 64
cd "$work" || exit 64

run_tests() {
  local cmd="${BENCH_TEST_COMMAND//\{test_files\}/$BENCH_TEST_FILES}"
  bash -ec "$cmd"
}

case "$MODE" in
  calibrate-base)
    git checkout -q "$BENCH_BASE_COMMIT" || exit 64
    git checkout -q "$BENCH_FIX_COMMIT" -- $BENCH_TEST_FILES || exit 64
    run_tests && exit 20 || exit 0   # tests MUST fail on base: failing = calibration ok (0)
    ;;
  calibrate-fix)
    git checkout -q "$BENCH_FIX_COMMIT" || exit 64
    run_tests && exit 0 || exit 20   # tests MUST pass on fix
    ;;
  grade)
    git checkout -q "$BENCH_BASE_COMMIT" || exit 64
    git apply --index --binary --whitespace=nowarn "$OUT/diff.patch" || exit 10
    git checkout -q "$BENCH_FIX_COMMIT" -- $BENCH_TEST_FILES || exit 64
    if run_tests; then exit 0; fi
    if [ "${BENCH_RETRIES:-0}" -ge 1 ]; then
      run_tests && exit 0
    fi
    exit 20
    ;;
  *) exit 64 ;;
esac
```

Note: `$BENCH_TEST_FILES` is intentionally unquoted in the `git checkout -- ` lines (space-separated list). Calibration installs are the caller's job (`install` runs once into the image/bare cache during `build-image`; vitest's `pnpm install && pnpm run build` happens inside `run_tests` via the repo's `test_command` if the repo config includes it — for the pilot, grading containers reuse the session image which has toolchain but each grade clone needs install. To keep grade self-contained, prepend install: the grade driver passes `BENCH_TEST_COMMAND` as `"<install joined by &&> && <test_command>"`).

- [ ] **Step 2: Run `bash -n benchmark/harness/docker/grade.sh`** — exit 0.

- [ ] **Step 3: Write the failing test** — uses **local mode** with a real tmp git fixture repo (no docker, no network):

```ts
import { execFileSync } from "node:child_process";
import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { decideGrade, gradeRunLocal } from "../grade.js";

/** Build a tiny repo: base commit has a bug + no test; fix commit fixes it and adds a test. Returns { bare, base, fix }. */
function fixtureRepo(): { bare: string; base: string; fix: string } {
  const src = mkdtempSync(join(tmpdir(), "benchfix-src-"));
  const git = (...a: string[]): string => execFileSync("git", a, { cwd: src, encoding: "utf8" }).trim();
  git("init", "-q", "-b", "main");
  git("config", "user.email", "t@t"); git("config", "user.name", "t");
  writeFileSync(join(src, "add.js"), "module.exports = (a, b) => a - b; // bug\n");
  git("add", "add.js"); git("commit", "-qm", "base");
  const base = git("rev-parse", "HEAD");
  writeFileSync(join(src, "add.js"), "module.exports = (a, b) => a + b;\n");
  mkdirSync(join(src, "test"), { recursive: true });
  writeFileSync(join(src, "test", "add.test.js"),
    "const add = require('../add.js');\nif (add(2, 3) !== 5) { console.error('FAIL'); process.exit(1); }\n");
  git("add", "-A"); git("commit", "-qm", "fix");
  const fix = git("rev-parse", "HEAD");
  const bare = mkdtempSync(join(tmpdir(), "benchfix-bare-")) + "/repo.git";
  execFileSync("git", ["clone", "-q", "--bare", src, bare]);
  return { bare, base, fix };
}

const TEST_CMD = "node {test_files}";
const TEST_FILES = ["test/add.test.js"];

describe("decideGrade", () => {
  it("maps grade.sh exit codes", () => {
    expect(decideGrade(0)).toEqual({ outcome: "passed", detail: "tests-passed" });
    expect(decideGrade(10)).toEqual({ outcome: "failed", detail: "apply-failed" });
    expect(decideGrade(20)).toEqual({ outcome: "failed", detail: "tests-failed" });
    expect(() => decideGrade(64)).toThrow(/setup/);
  });
});

describe("gradeRunLocal (real git, no docker)", () => {
  it("passes a correct agent diff and fails an empty one", async () => {
    const { bare, base, fix } = fixtureRepo();

    // "Agent diff": the real fix to add.js (source only, no tests).
    const outDir = mkdtempSync(join(tmpdir(), "benchout-"));
    const goodDiff = execFileSync("git", ["diff", "--binary", base, fix, "--", "add.js"], { cwd: bare, encoding: "utf8" });
    writeFileSync(join(outDir, "diff.patch"), goodDiff);
    const good = await gradeRunLocal({ bareDir: bare, outDir, baseCommit: base, fixCommit: fix, testFiles: TEST_FILES, testCommand: TEST_CMD, retries: 0 });
    expect(good).toEqual({ outcome: "passed", detail: "tests-passed" });

    const outDir2 = mkdtempSync(join(tmpdir(), "benchout-"));
    writeFileSync(join(outDir2, "diff.patch"), "");
    const empty = await gradeRunLocal({ bareDir: bare, outDir: outDir2, baseCommit: base, fixCommit: fix, testFiles: TEST_FILES, testCommand: TEST_CMD, retries: 0 });
    expect(empty).toEqual({ outcome: "failed", detail: "tests-failed" }); // empty diff applies cleanly, then calibrated tests fail
  });
});
```

- [ ] **Step 4: Run to verify FAIL, then implement `grade.ts`:**

```ts
import { readFileSync, writeFileSync } from "node:fs";
import { join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { parseFlags } from "../../skills/doc-wiki/scripts/_cli_args.js";
import { loadState, runKey, saveState } from "./bench_checkpoint.js";
import { gradeRunArgs } from "./docker_args.js";
import type { Runner } from "./exec.js";
import { realRunner } from "./exec.js";
import { loadRepoConfig } from "./repo_config.js";
import type { GradeRecord, RepoConfig, TicketsFile } from "./types.js";

const GRADE_SH = fileURLToPath(new URL("./docker/grade.sh", import.meta.url));

export function decideGrade(exitCode: number): Omit<GradeRecord, "graded_at"> {
  if (exitCode === 0) return { outcome: "passed", detail: "tests-passed" };
  if (exitCode === 10) return { outcome: "failed", detail: "apply-failed" };
  if (exitCode === 20) return { outcome: "failed", detail: "tests-failed" };
  throw new Error(`grade.sh setup error (exit ${exitCode})`);
}

export interface GradeLocalSpec {
  bareDir: string;
  outDir: string;
  baseCommit: string;
  fixCommit: string;
  testFiles: string[];
  testCommand: string;
  retries: number;
  mode?: "grade" | "calibrate-base" | "calibrate-fix";
  runner?: Runner;
}

/** Run grade.sh directly on the host (tests/e2e) — same contract as the docker path. */
export async function gradeRunLocal(spec: GradeLocalSpec): Promise<Omit<GradeRecord, "graded_at">> {
  const runner = spec.runner ?? realRunner;
  const r = await runner("bash", [GRADE_SH], {
    env: {
      ...process.env,
      BENCH_BARE_DIR: spec.bareDir,
      BENCH_OUT_DIR: spec.outDir,
      BENCH_BASE_COMMIT: spec.baseCommit,
      BENCH_FIX_COMMIT: spec.fixCommit,
      BENCH_TEST_FILES: spec.testFiles.join(" "),
      BENCH_TEST_COMMAND: spec.testCommand,
      BENCH_RETRIES: String(spec.retries),
      BENCH_GRADE_MODE: spec.mode ?? "grade",
    },
  });
  return decideGrade(r.code);
}

/** Calibrate every un-calibrated ticket; failures get `excluded` set. Mutates + rewrites the tickets file. */
export async function calibrateAll(cfg: RepoConfig, ticketsPath: string, bareDir: string, opts: { local: boolean; image: string; runner: Runner }): Promise<void> {
  const file = JSON.parse(readFileSync(ticketsPath, "utf8")) as TicketsFile;
  const installPrefix = cfg.install.length > 0 ? `${cfg.install.join(" && ")} && ` : "";
  for (const t of file.tickets) {
    if (t.calibration !== undefined || t.excluded !== undefined) continue;
    const common = {
      bareDir, outDir: bareDir, baseCommit: t.base_commit, fixCommit: t.fix_commit,
      testFiles: t.test_files, testCommand: installPrefix + cfg.test_command, retries: 0, runner: opts.runner,
    };
    try {
      // paths must exist at both commits (rename = ill-defined overlay)
      const stable = await pathsStable(opts.runner, bareDir, t.base_commit, t.fix_commit, t.test_files);
      const failsOnBase = stable && (await gradeRunLocal({ ...common, mode: "calibrate-base" })).outcome === "passed";
      const passesOnFix = stable && failsOnBase && (await gradeRunLocal({ ...common, mode: "calibrate-fix" })).outcome === "passed";
      t.calibration = { paths_stable: stable, tests_fail_on_base: failsOnBase, tests_pass_on_fix: passesOnFix };
      if (!stable || !failsOnBase || !passesOnFix) {
        t.excluded = `calibration-failed (stable=${stable} failsOnBase=${failsOnBase} passesOnFix=${passesOnFix})`;
      }
    } catch (err) {
      // setup error (grade.sh exit 64: bad checkout, clone failure) — exclude, don't crash the sweep
      t.calibration = { paths_stable: false, tests_fail_on_base: false, tests_pass_on_fix: false };
      t.excluded = `calibration-error (${err instanceof Error ? err.message : String(err)})`;
    }
    if (t.excluded !== undefined) process.stderr.write(`issue #${t.issue}: ${t.excluded}\n`);
  }
  writeFileSync(ticketsPath, `${JSON.stringify(file, null, 2)}\n`);
}

async function pathsStable(runner: Runner, bareDir: string, base: string, fix: string, files: string[]): Promise<boolean> {
  for (const f of files) {
    for (const commit of [base, fix]) {
      const r = await runner("git", ["-C", bareDir, "cat-file", "-e", `${commit}:${f}`]);
      if (r.code !== 0) return false;
    }
  }
  return true;
}

/** Grade every `ran` run that has no grade yet. */
export async function gradeAll(cfg: RepoConfig, ticketsPath: string, runsRoot: string, bareDir: string, opts: { local: boolean; image: string; runner: Runner }): Promise<void> {
  const file = JSON.parse(readFileSync(ticketsPath, "utf8")) as TicketsFile;
  const byIssue = new Map(file.tickets.map((t) => [t.issue, t]));
  const stateFile = join(runsRoot, cfg.id, "state.json");
  const state = loadState(stateFile, cfg.id);
  const installPrefix = cfg.install.length > 0 ? `${cfg.install.join(" && ")} && ` : "";

  for (const [key, rec] of Object.entries(state.runs)) {
    if (rec.status !== "ran") continue;
    const [issueStr, arm] = key.split(":");
    const t = byIssue.get(Number(issueStr));
    if (t === undefined) continue;
    const outDir = resolve(runsRoot, cfg.id, String(t.issue), String(arm));

    let exitCode: number;
    if (opts.local) {
      const g = await gradeRunLocal({
        bareDir, outDir, baseCommit: t.base_commit, fixCommit: t.fix_commit,
        testFiles: t.test_files, testCommand: installPrefix + cfg.test_command,
        retries: cfg.test_retries, runner: opts.runner,
      });
      exitCode = g.outcome === "passed" ? 0 : g.detail === "apply-failed" ? 10 : 20;
    } else {
      const r = await opts.runner("docker", gradeRunArgs({
        image: opts.image, outDir, bareDir, baseCommit: t.base_commit, fixCommit: t.fix_commit,
        testFiles: t.test_files, testCommand: installPrefix + cfg.test_command, retries: cfg.test_retries,
      }));
      exitCode = r.code;
    }
    const grade: GradeRecord = { ...decideGrade(exitCode), graded_at: new Date().toISOString() };
    writeFileSync(join(outDir, "grade.json"), `${JSON.stringify(grade, null, 2)}\n`);
    rec.status = grade.outcome;
    rec.detail = grade.detail;
    saveState(stateFile, state);
    process.stderr.write(`${key}: ${grade.outcome} (${grade.detail})\n`);
  }
}

export async function main(argv: readonly string[]): Promise<number> {
  const sub = argv[0];
  const { help, values } = parseFlags(argv.slice(1), {
    "--repo": "repo", "--local": "local", "--image": "image", "--bare-dir": "bareDir",
  });
  if (help || values.repo === undefined || (sub !== "calibrate" && sub !== "grade")) {
    process.stderr.write("usage: benchmark <calibrate|grade> --repo <id> [--local] [--image i] [--bare-dir d]\n");
    return help ? 0 : 2;
  }
  const repo = String(values.repo);
  const cfg = loadRepoConfig(join("benchmark", "repos", `${repo}.yaml`));
  const ticketsPath = join("benchmark", "tickets", `${repo}.json`);
  const shared = {
    local: values.local === true,
    image: String(values.image ?? `docwiki-bench-${repo}`),
    runner: realRunner,
  };
  const bareDir = String(values.bareDir ?? resolve("benchmark", "wiki-cache", `${repo}.git`));
  if (sub === "calibrate") await calibrateAll(cfg, ticketsPath, bareDir, shared);
  else await gradeAll(cfg, ticketsPath, join("benchmark", "runs"), bareDir, shared);
  return 0;
}
```

- [ ] **Step 5: Run tests to verify PASS** (`npx vitest run benchmark/harness/tests/grade.test.ts`).

- [ ] **Step 6: Build + commit.**

```bash
npm run build
git add benchmark/harness/grade.ts benchmark/harness/grade.js benchmark/harness/docker/grade.sh benchmark/harness/tests/grade.test.ts
git commit -m "feat(benchmark): calibration + grading with docker/local parity"
```

---

### Task 12: Report renderer

**Files:**
- Create: `benchmark/harness/report.ts`
- Test: `benchmark/harness/tests/report.test.ts`

- [ ] **Step 1: Write the failing test:**

```ts
import { describe, expect, it } from "vitest";
import { renderResults } from "../report.js";
import type { BenchState, TicketRecord } from "../types.js";

const tickets = [
  { issue: 1, title: "bug one", merged_at: "2025-09-01T00:00:00Z" },
  { issue: 2, title: "bug two", merged_at: "2025-10-01T00:00:00Z" },
] as TicketRecord[];

const state: BenchState = {
  schema_version: 1,
  repo: "demo",
  runs: {
    "1:baseline": { status: "failed", cost_usd: 0.4, detail: "tests-failed" },
    "1:wiki": { status: "passed", cost_usd: 0.6, detail: "tests-passed" },
    "2:baseline": { status: "failed", cost_usd: 0.5, detail: "apply-failed" },
    "2:wiki": { status: "pending" },
  },
};

describe("renderResults", () => {
  it("renders pass rates over graded runs and totals cost", () => {
    const md = renderResults("demo", tickets, state);
    expect(md).toContain("| baseline | 0/2 (0%) |");
    expect(md).toContain("| wiki | 1/1 (100%) |");
    expect(md).toContain("$1.50"); // 0.4+0.6+0.5
    expect(md).toContain("bug one");
    expect(md).toContain("⏳"); // pending wiki run for ticket 2
  });
});
```

- [ ] **Step 2: Run to verify FAIL, then implement `report.ts`:**

```ts
import { readFileSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { parseFlags } from "../../skills/doc-wiki/scripts/_cli_args.js";
import { loadState, runKey } from "./bench_checkpoint.js";
import type { Arm, BenchState, RunRecord, TicketRecord, TicketsFile } from "./types.js";

const ARMS: readonly Arm[] = ["baseline", "wiki"];

const cell = (r: RunRecord | undefined): string => {
  if (r === undefined || r.status === "pending") return "⏳";
  if (r.status === "passed") return "✅";
  if (r.status === "failed") return `❌ ${r.detail ?? ""}`.trim();
  return r.status;
};

export function renderResults(repo: string, tickets: readonly TicketRecord[], state: BenchState): string {
  const lines: string[] = [`## ${repo}`, ""];
  let cost = 0;

  lines.push("| arm | passed/graded (rate) |", "|---|---|");
  for (const arm of ARMS) {
    let passed = 0;
    let graded = 0;
    for (const t of tickets) {
      const r = state.runs[runKey(t.issue, arm)];
      if (r?.cost_usd !== undefined) cost += r.cost_usd;
      if (r?.status === "passed" || r?.status === "failed") {
        graded += 1;
        if (r.status === "passed") passed += 1;
      }
    }
    const rate = graded === 0 ? 0 : Math.round((passed / graded) * 100);
    lines.push(`| ${arm} | ${passed}/${graded} (${rate}%) |`);
  }
  lines.push("", `Total session cost: $${cost.toFixed(2)}`, "");

  lines.push("| ticket | merged | baseline | wiki |", "|---|---|---|---|");
  for (const t of tickets) {
    lines.push(
      `| #${t.issue} ${t.title} | ${t.merged_at.slice(0, 10)} | ${cell(state.runs[runKey(t.issue, "baseline")])} | ${cell(state.runs[runKey(t.issue, "wiki")])} |`,
    );
  }
  lines.push("");
  return lines.join("\n");
}

export async function main(argv: readonly string[]): Promise<number> {
  const { help, values } = parseFlags(argv, { "--repo": "repo", "--out": "out" });
  if (help || values.repo === undefined) {
    process.stderr.write("usage: benchmark report --repo <id> [--out benchmark/RESULTS.md]\n");
    return help ? 0 : 2;
  }
  const repo = String(values.repo);
  const ticketsFile = JSON.parse(readFileSync(join("benchmark", "tickets", `${repo}.json`), "utf8")) as TicketsFile;
  const active = ticketsFile.tickets.filter((t) => t.excluded === undefined);
  const state = loadState(join("benchmark", "runs", repo, "state.json"), repo);
  const md = `# Benchmark Results\n\n> Generated by \`npm run benchmark -- report\`. Methodology: [METHODOLOGY.md](METHODOLOGY.md).\n\n${renderResults(repo, active, state)}`;
  writeFileSync(String(values.out ?? join("benchmark", "RESULTS.md")), md);
  return 0;
}
```

- [ ] **Step 3: Run tests to verify PASS.**

- [ ] **Step 4: Build + commit.**

```bash
npm run build
git add benchmark/harness/report.ts benchmark/harness/report.js benchmark/harness/tests/report.test.ts
git commit -m "feat(benchmark): RESULTS.md renderer"
```

---

### Task 13: Wiki build CLI

**Files:**
- Create: `benchmark/harness/build_wiki.ts`
- Test: `benchmark/harness/tests/build_wiki.test.ts`

This prepares the wiki arm's overlay **once per repo**: clone bare cache → verify `wiki_commit` (must be set in the repo yaml and be an ancestor of every ticket's `base_commit`) → run a container session with the doc-wiki plugin mounted (`--plugin-dir /plugin`) and prompt `/doc-wiki:init` + `/doc-wiki:atlas --cross-service --yes` at that commit, normal egress (atlas needs no firewall — it never sees ticket content) → copy `CLAUDE.md` + the generated wiki dir out to `benchmark/wiki-cache/<repo>/overlay/`. Only the argv/validation logic is unit-tested; the live path is exercised manually in the pilot.

- [ ] **Step 1: Write the failing test:**

```ts
import { describe, expect, it } from "vitest";
import { validateWikiCommit, wikiSessionArgs } from "../build_wiki.js";

describe("validateWikiCommit", () => {
  it("requires wiki_commit to be set and older than every base_commit", () => {
    expect(() => validateWikiCommit("", [])).toThrow(/wiki_commit/);
    // ancestor check is delegated to git merge-base --is-ancestor; here we test the empty guard only
  });
});

describe("wikiSessionArgs", () => {
  it("mounts the plugin and overlay dirs and disables the firewall stage", () => {
    const args = wikiSessionArgs({
      image: "docwiki-bench-vitest", bareDir: "/b", overlayDir: "/o", pluginDir: "/p",
      wikiCommit: "cccc", model: "claude-sonnet-4-6",
    });
    expect(args).toContain("/p:/plugin:ro");
    expect(args).toContain("BENCH_SKIP_FIREWALL=1");
    expect(args[args.length - 1]).toBe("wiki-build");
  });
});
```

- [ ] **Step 2: Run to verify FAIL, then implement `build_wiki.ts`:**

```ts
import { resolve } from "node:path";
import { parseFlags } from "../../skills/doc-wiki/scripts/_cli_args.js";
import type { Runner } from "./exec.js";
import { realRunner } from "./exec.js";
import { loadRepoConfig } from "./repo_config.js";

export function validateWikiCommit(wikiCommit: string, baseCommits: readonly string[]): void {
  if (wikiCommit === "") {
    throw new Error(
      "wiki_commit is not set in the repo yaml. Rule: parent of the oldest eligible ticket's base_commit (see spec).",
    );
  }
  void baseCommits; // ancestor verification happens via git at run time (checkWikiIsAncestor)
}

export async function checkWikiIsAncestor(runner: Runner, bareDir: string, wikiCommit: string, baseCommits: readonly string[]): Promise<void> {
  for (const base of baseCommits) {
    const r = await runner("git", ["-C", bareDir, "merge-base", "--is-ancestor", wikiCommit, base]);
    if (r.code !== 0) {
      throw new Error(`contamination guard: wiki_commit ${wikiCommit} is not an ancestor of base_commit ${base}`);
    }
  }
}

export interface WikiBuildSpec {
  image: string;
  bareDir: string;
  overlayDir: string;
  pluginDir: string;
  wikiCommit: string;
  model: string;
}

export function wikiSessionArgs(s: WikiBuildSpec): string[] {
  return [
    "run", "--rm",
    "-v", `${s.bareDir}:/bare:ro`,
    "-v", `${s.overlayDir}:/out`,
    "-v", `${s.pluginDir}:/plugin:ro`,
    "-e", "CLAUDE_CODE_OAUTH_TOKEN",
    "-e", `BENCH_BASE_COMMIT=${s.wikiCommit}`,
    "-e", `BENCH_MODEL=${s.model}`,
    "-e", "BENCH_SKIP_FIREWALL=1",
    s.image, "wiki-build",
  ];
}

export async function main(argv: readonly string[]): Promise<number> {
  const { help, values } = parseFlags(argv, {
    "--repo": "repo", "--plugin-dir": "pluginDir", "--image": "image", "--bare-dir": "bareDir", "--model": "model",
  });
  if (help || values.repo === undefined) {
    process.stderr.write("usage: benchmark build-wiki --repo <id> [--plugin-dir .] [--image i] [--bare-dir d] [--model m]\n");
    return help ? 0 : 2;
  }
  const repo = String(values.repo);
  const cfg = loadRepoConfig(`benchmark/repos/${repo}.yaml`);
  validateWikiCommit(cfg.wiki_commit, []);
  const args = wikiSessionArgs({
    image: String(values.image ?? `docwiki-bench-${repo}`),
    bareDir: String(values.bareDir ?? resolve("benchmark", "wiki-cache", `${repo}.git`)),
    overlayDir: resolve("benchmark", "wiki-cache", repo, "overlay"),
    pluginDir: resolve(String(values.pluginDir ?? ".")),
    wikiCommit: cfg.wiki_commit,
    model: String(values.model ?? "claude-sonnet-4-6"),
  });
  const r = await realRunner("docker", args, { timeoutMs: 4 * 60 * 60 * 1000 });
  process.stderr.write(r.stderr);
  return r.code;
}
```

- [ ] **Step 3: Extend `entrypoint.sh`** — add a `wiki-build` mode branch right after the `grade` branch:

```bash
if [ "$mode" = "wiki-build" ]; then
  set -e
  git clone --no-hardlinks /bare /work && cd /work
  git checkout -q "$BENCH_BASE_COMMIT"
  claude -p "/doc-wiki:init --yes" --plugin-dir /plugin --model "$BENCH_MODEL" --output-format json --dangerously-skip-permissions >/out/init.json 2>&1
  claude -p "/doc-wiki:atlas --cross-service --yes" --plugin-dir /plugin --model "$BENCH_MODEL" --output-format json --dangerously-skip-permissions >/out/atlas.json 2>&1
  mkdir -p /out/overlay
  cp CLAUDE.md /out/overlay/CLAUDE.md 2>/dev/null || true
  cp -R docs /out/overlay/docs
  exit 0
fi
```

Also guard the firewall line in session mode: `[ "${BENCH_SKIP_FIREWALL:-0}" = "1" ] || /usr/local/bin/init-firewall.sh`.

- [ ] **Step 4: Run `bash -n` on entrypoint.sh, run tests to verify PASS, typecheck.**

- [ ] **Step 5: Build + commit.**

```bash
npm run build
git add benchmark/harness/build_wiki.ts benchmark/harness/build_wiki.js benchmark/harness/docker/entrypoint.sh benchmark/harness/tests/build_wiki.test.ts
git commit -m "feat(benchmark): one-time wiki-overlay build with ancestor contamination guard"
```

---### Task 14: CLI dispatcher + README

**Files:**
- Create: `benchmark/harness/cli.ts`, `benchmark/README.md`
- Test: `benchmark/harness/tests/cli.test.ts`

- [ ] **Step 1: Write the failing test:**

```ts
import { describe, expect, it } from "vitest";
import { dispatch } from "../cli.js";

describe("cli dispatch", () => {
  it("routes known subcommands and rejects unknown ones", async () => {
    expect(await dispatch(["definitely-not-a-command"])).toBe(2);
    expect(await dispatch([])).toBe(2);
    expect(await dispatch(["mine", "--help"])).toBe(0);
    expect(await dispatch(["run", "--help"])).toBe(0);
    expect(await dispatch(["grade", "--help"])).toBe(0);
    expect(await dispatch(["calibrate", "--help"])).toBe(0);
    expect(await dispatch(["report", "--help"])).toBe(0);
    expect(await dispatch(["build-wiki", "--help"])).toBe(0);
  });
});
```

- [ ] **Step 2: Run to verify FAIL, then implement `cli.ts`:**

```ts
import { main as buildWiki } from "./build_wiki.js";
import { main as grade } from "./grade.js";
import { main as mine } from "./mine_tickets.js";
import { main as report } from "./report.js";
import { main as run } from "./run_ticket.js";

const USAGE = "usage: npm run benchmark -- <mine|build-wiki|calibrate|run|grade|report> [flags]\n";

export async function dispatch(argv: readonly string[]): Promise<number> {
  const [sub, ...rest] = argv;
  switch (sub) {
    case "mine": return mine(rest);
    case "build-wiki": return buildWiki(rest);
    case "calibrate": return grade(["calibrate", ...rest]);
    case "grade": return grade(["grade", ...rest]);
    case "run": return run(rest);
    case "report": return report(rest);
    default:
      process.stderr.write(USAGE);
      return 2;
  }
}

const isMain = process.argv[1] !== undefined && import.meta.url.endsWith(process.argv[1].split("/").pop() ?? "");
if (isMain) {
  dispatch(process.argv.slice(2)).then((code) => process.exit(code));
}
```

- [ ] **Step 3: Run tests to verify PASS.**

- [ ] **Step 4: Write `benchmark/README.md`** — NOTE: this file already exists (the bannered V1 readme from Task 1.6); Read it, then fully REPLACE its contents with the operator runbook below (the V1 history note lives in ANALYSIS.md/PLAN.md banners, not here):

```markdown
# doc-wiki benchmark

Reproducible measurement of Claude Code's ticket-fix pass rate, baseline vs with a doc-wiki wiki.
Design: [`docs/superpowers/specs/2026-06-10-benchmark-harness-design.md`](../docs/superpowers/specs/2026-06-10-benchmark-harness-design.md).
Methodology and caveats: [`METHODOLOGY.md`](METHODOLOGY.md). Numbers: [`RESULTS.md`](RESULTS.md).

## One-time setup

1. `claude setup-token` → export the printed token as `CLAUDE_CODE_OAUTH_TOKEN` (draws on your Claude subscription; never commit it).
2. Build the image: `docker build -t docwiki-bench-vitest --build-arg TOOLCHAIN=node:22 benchmark/harness/docker/`
3. Cache a bare clone: `git clone --bare https://github.com/vitest-dev/vitest.git benchmark/wiki-cache/vitest.git`

## Pipeline (pilot: vitest)

| step | command |
|---|---|
| mine tickets | `npm run benchmark -- mine --repo vitest --target 30` |
| set `wiki_commit` | parent of the oldest `base_commit` in `benchmark/tickets/vitest.json` → `benchmark/repos/vitest.yaml` |
| build wiki overlay | `npm run benchmark -- build-wiki --repo vitest --plugin-dir .` |
| calibrate | `npm run benchmark -- calibrate --repo vitest` |
| run a batch (both arms) | `npm run benchmark -- run --repo vitest --batch 10` |
| grade | `npm run benchmark -- grade --repo vitest` |
| report | `npm run benchmark -- report --repo vitest` |

Hitting your subscription's rate limit mid-batch is expected: the run stops, prints the reset time, and the same `run` command resumes exactly where it left off (completing half-finished pairs first).

Artifacts land in `benchmark/runs/<repo>/<issue>/<arm>/` (gitignored): `prompt.txt`, `result.json`, `diff.patch`, `transcript/`, `grade.json`.
```

- [ ] **Step 5: Build + commit.**

```bash
npm run build
git add benchmark/harness/cli.ts benchmark/harness/cli.js benchmark/harness/tests/cli.test.ts benchmark/README.md
git commit -m "feat(benchmark): CLI dispatcher + operator runbook"
```

---

### Task 15: End-to-end smoke test + METHODOLOGY skeleton

**Files:**
- Create: `benchmark/harness/tests/e2e_smoke.test.ts`, `benchmark/METHODOLOGY.md`

The smoke test exercises mine→(skip)→run→grade→report **without docker or tokens**: a fixture tickets file, a fake docker Runner whose "session" writes a correct diff (computed from the fixture repo's real fix), local-mode grading with real git, and the report renderer. It proves the pipeline's joints — file layouts, state transitions, grade integration — hold together.

- [ ] **Step 1: Write the test:**

```ts
import { execFileSync } from "node:child_process";
import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import type { Runner } from "../exec.js";
import { gradeAll } from "../grade.js";
import { renderResults } from "../report.js";
import { runBatch } from "../run_ticket.js";
import { loadState } from "../bench_checkpoint.js";
import type { RepoConfig, TicketsFile } from "../types.js";

function fixtureRepo(): { bare: string; base: string; fix: string; goodDiff: string } {
  const src = mkdtempSync(join(tmpdir(), "e2e-src-"));
  const git = (...a: string[]): string => execFileSync("git", a, { cwd: src, encoding: "utf8" }).trim();
  git("init", "-q", "-b", "main");
  git("config", "user.email", "t@t"); git("config", "user.name", "t");
  writeFileSync(join(src, "calc.js"), "module.exports = (a, b) => a - b;\n");
  git("add", "-A"); git("commit", "-qm", "base");
  const base = git("rev-parse", "HEAD");
  writeFileSync(join(src, "calc.js"), "module.exports = (a, b) => a + b;\n");
  mkdirSync(join(src, "test"));
  writeFileSync(join(src, "test", "calc.test.js"),
    "const c = require('../calc.js');\nif (c(1, 2) !== 3) process.exit(1);\n");
  git("add", "-A"); git("commit", "-qm", "fix");
  const fix = git("rev-parse", "HEAD");
  const goodDiff = git("diff", "--binary", base, fix, "--", "calc.js");
  const bare = `${mkdtempSync(join(tmpdir(), "e2e-bare-"))}/repo.git`;
  execFileSync("git", ["clone", "-q", "--bare", src, bare]);
  return { bare, base, fix, goodDiff };
}

describe("e2e smoke: run -> grade -> report (fake claude, real git)", () => {
  it("baseline fails, wiki passes, report shows the delta", async () => {
    const { bare, base, fix, goodDiff } = fixtureRepo();
    const root = mkdtempSync(join(tmpdir(), "e2e-root-"));
    const cfg: RepoConfig = {
      id: "demo", github: "acme/demo", clone_url: "x", language: "js",
      ticket_source: "github", install: [], test_command: "node {test_files}",
      test_patterns: ["test/**"], test_retries: 0, ticket_after: "2025-01-01",
      wiki_commit: base, toolchain: ["node:22"], services: [],
    };
    const tickets: TicketsFile = {
      schema_version: 1, repo: "demo", mined_at: "x",
      tickets: [{
        issue: 7, issue_url: "u", title: "subtraction instead of addition", body: "b", body_sanitized: "b",
        fix_pr: 8, fix_pr_url: "u", base_commit: base, fix_commit: fix,
        test_files: ["test/calc.test.js"], src_files: ["calc.js"], changed_lines: 2,
        merged_at: "2025-09-01T00:00:00Z",
        calibration: { paths_stable: true, tests_fail_on_base: true, tests_pass_on_fix: true },
      }],
    };
    mkdirSync(join(root, "tickets"), { recursive: true });
    const ticketsPath = join(root, "tickets", "demo.json");
    writeFileSync(ticketsPath, JSON.stringify(tickets));

    // Fake docker: the "wiki" arm's agent writes the correct fix; "baseline" writes nothing.
    const fakeDocker: Runner = async (_cmd, args) => {
      const outDir = String(args.find((a) => a.includes(":/out"))).split(":")[0];
      const isWiki = args.some((a) => a.includes(":/wiki"));
      writeFileSync(join(String(outDir), "result.json"),
        JSON.stringify({ result: "done", total_cost_usd: 0.3, session_id: "s" }));
      writeFileSync(join(String(outDir), "stderr.log"), "");
      writeFileSync(join(String(outDir), "exit_code"), "0");
      writeFileSync(join(String(outDir), "diff.patch"), isWiki ? goodDiff : "");
      return { code: 0, stdout: "", stderr: "" };
    };

    const runsRoot = join(root, "runs");
    const summary = await runBatch({
      cfg, ticketsPath, runsRoot, bareDir: bare, wikiDir: join(root, "overlay"),
      image: "img", model: "m", maxTurns: 10, batch: 5, timeoutSec: 60, runner: fakeDocker,
    });
    expect(summary.ran).toBe(2);

    await gradeAll(cfg, ticketsPath, runsRoot, bare, { local: true, image: "img", runner: (await import("../exec.js")).realRunner });

    const state = loadState(join(runsRoot, "demo", "state.json"), "demo");
    expect(state.runs["7:baseline"]?.status).toBe("failed");
    expect(state.runs["7:wiki"]?.status).toBe("passed");

    const md = renderResults("demo", tickets.tickets, state);
    expect(md).toContain("| baseline | 0/1 (0%) |");
    expect(md).toContain("| wiki | 1/1 (100%) |");
  });
});
```

- [ ] **Step 2: Run it.** `npx vitest run benchmark/harness/tests/e2e_smoke.test.ts`. Expected: PASS (this test drives already-implemented code; if it fails, the joints between Tasks 10–12 are wrong — fix there, not in the test).

- [ ] **Step 3: Write `benchmark/METHODOLOGY.md`** (skeleton; pilot fills numbers):

```markdown
# Methodology

**Claim under test:** a doc-wiki generated wiki in the repo improves Claude Code's autonomous ticket-fix pass rate on real closed issues.

**Design:** paired two-arm runs per ticket (baseline / wiki) — identical container, model (`claude-sonnet-4-6`, pinned full ID), prompt, and flags; the only delta is the presence of the pre-built wiki + `CLAUDE.md` in the checkout. Grading: the real fix PR's tests, overlaid onto the agent's diff (SWE-bench style). Pass = all overlaid tests pass.

**Ticket eligibility:** closed issue with a merged linked fix PR touching both test and non-test source, <400 changed lines, natural-language body ≥200 chars, human author, merged after the repo's `ticket_after` floor. The committed `tickets/<repo>.json` is the exact set, including every exclusion and its reason.

**Contamination controls:**
1. *Fix leak:* the wiki is built at `wiki_commit`, verified (`git merge-base --is-ancestor`) to predate every ticket's base commit.
2. *Issue-body leak:* bodies are sanitized (forward references, SHAs, "fixed by" lines stripped); every redaction is logged in the ticket record.
3. *Online-lookup leak:* agent sessions run behind an egress firewall allowing only Anthropic endpoints.
4. *Training-data leak:* tickets postdate `ticket_after` (set from the pinned model's training cutoff); merge dates are published per ticket.

**Calibration (pre-registered):** before any agent runs, each ticket's fix-PR tests must fail on the clean base commit and pass on the fix commit, with stable test-file paths. Failures are excluded up front, with reasons logged.

**Known caveats:**
- Single run per (ticket, arm): no variance estimate per ticket; treat per-repo aggregates, not per-ticket outcomes, as the signal.
- OSS repos ≠ enterprise codebases. The author's enterprise-codebase experience is an anecdote, not this benchmark's claim; the benchmark's claim is whatever RESULTS.md says.
- Ticket discovery uses GitHub's `closingIssuesReferences` (keyword-linked issues only) — PRs that reference an issue solely in free-text prose are not mined, so the candidate pool understates true fix volume. Selection bias is toward well-linked, process-followed fixes.
- Rebase-merged PRs can make `base_commit` (merge-commit parent) partially contain the fix; the calibration gate excludes them. `merge_parents` on each ticket record flags true merge commits (=2); rebase merges have a single parent and are detectable only via calibration.
- `--max-turns` and container timeout: <set during pilot calibration>.

**Reproduction:** see [README.md](README.md). Total cost and wall-clock for the published runs: <filled from RESULTS.md>.
```

- [ ] **Step 4: Full suite + commit.** Run: `npm test` (everything green), `npm run typecheck`, `npm run build`.

```bash
git add benchmark/harness/tests/e2e_smoke.test.ts benchmark/METHODOLOGY.md
git commit -m "test(benchmark): zero-token e2e smoke + methodology skeleton"
```

---

### Task 16: PR

- [ ] **Step 1: Final verification.** Run: `npm run typecheck && npm run build && npm test && git status --short`. Expected: green; no unstaged build drift (if `.js` files appear modified, a commit missed its sibling — amend the right commit or add a fix commit).

- [ ] **Step 2: Push + PR.**

```bash
git push -u origin feat/benchmark-harness
gh pr create --title "feat(benchmark): reproducible ticket-fix benchmark harness (launch Anchor 1)" --body "Implements docs/superpowers/specs/2026-06-10-benchmark-harness-design.md: deterministic TS harness around headless Claude Code sessions in Docker — mining (github adapter), sanitization, checkpointed paired-arm runs, egress-firewalled sessions, real-fix-test grading with pre-registered calibration, RESULTS/METHODOLOGY rendering. Pilot target: vitest-dev/vitest. Zero-token CI (fake claude + real git e2e smoke)."
```

- [ ] **Step 3: Invoke pr-monitor** (mandatory, same response as the push).

---

## Post-merge pilot runbook (NOT part of this plan's code — operator steps)

1. `claude setup-token` → env var. 2. Build image + bare clone (README). 3. `mine --repo vitest --target 30`. 4. Set `wiki_commit` per the rule, commit the yaml + tickets file. 5. `build-wiki` (one atlas run ≈ one evening's quota). 6. `calibrate`. 7. `run --batch 10` per evening until done. 8. `grade`, `report`, eyeball transcripts. 9. Tune `--max-turns`/timeout in METHODOLOGY.md. 10. Scale: add `repos/django.yaml` (needs the `trac-commits` adapter — separate follow-up plan) and `repos/calcom.yaml`.

## Self-review notes (run after drafting — resolved)

- **Spec coverage:** mining criteria 1–6 → Tasks 6–7; contamination controls 1–4 → Tasks 8 (firewall), 3 (sanitize), 13 (ancestor guard), 2+6 (`ticket_after`); state machine + pairing + batch → Task 4; rate-limit stop/resume → Tasks 9–10; grading + calibration incl. path stability + retries → Task 11; reporting → Tasks 12, 15; fake-claude e2e → 15; docker + token handling → 8; runbook → 14. Trac adapter and cal.com integration-mode services are explicitly deferred (spec's full-mix phase, noted in Task 16/runbook).
- **Type consistency:** `RunStatus` includes `ran` everywhere it's branched on (Tasks 4, 10, 11, 12); `GradeRecord.detail` strings match `decideGrade` and the report's `cell()`; `Runner` signature identical across Tasks 5–13.
- **Placeholder scan:** grade.sh placeholder in Task 8 is explicitly replaced in Task 11; METHODOLOGY has two `<filled during pilot>` slots by design (they're data, not code).
```