# pi-plans A/B Benchmark — Tech Note

Run `20260909T103956Z-pi-plans-bench`, plan PLAN_v3. First full run completed 2026-09-10 → 09-12; analysis regenerated from committed inputs (`docs/benchmarks/data/`).

## 1. Claim being tested

pi-plans claims fewer tokens and fewer detours via a plan → review → execute loop with a verifier checklist. This evaluation tests the mechanism end-to-end: does the planning machinery improve (or at least not hurt) task resolution and cost on long-horizon terminal tasks?

## 2. Design

- **Arms**: `baseline` = stock pi; `treatment` = pi + pi-plans (extension loaded, skills registered, system prompt injected: "Before implementing, load and follow `skills/plan-big/SKILL.md`"). The planning trigger is delivered as a composer-level `/skill:plan-big` follow-up immediately after the task prompt (deterministic skill expansion, mirroring real usage: user states the request, then invokes plan mode). Task instructions are verbatim in both arms; the only difference is agent-level configuration.
- **Benchmark**: Terminal-Bench 2.0 (89 tasks, 16 categories) via harbor; SWE-bench (harbor adapter) as a second benchmark under a separate budget.
- **Model**: `zai/glm-5.3-flash:high`, single model. Seed: 1 (primary), +2 seeds on discordant pairs only.
- **Unattended operation**: `PI_PLANS_AUTO_APPROVE=1` (eval-only, default off) answers plan-lifecycle gates with the recommended option; external-state questions (publish/deploy/merge/push/credential) are hard-rejected by design.

## 3. Pre-registered analysis (D-012)

- Unit = task, paired across arms. PRIMARY: seed-1 full-set McNemar exact + paired bootstrap CI (10k) for cost/turns.
- Sensitivity: discordant-pair reruns aggregated by per-task majority vote, reported separately.
- Cost includes parent + subagent usage (the extension's reviewer/criticizer calls) — metered end-to-end by the adapter.

## 4. Budget (D-016)

_Executed scope_: the original 89-task protocol was reduced by the pre-registered degradation step 2 (stratified 40-task sample → 36 usable after 4 task images failed to transfer through the network path; both arms use the identical 36). zai coding-plan pricing reports $0, so the $ budget gate was inactive; the real cost dimension is tokens (~10.9M input / 450k output total across both arms) and wall time (~30h machine time across both arms).

Main experiment ≤ $150 (incl. retries and sensitivity rerun); second benchmark ≤ +$50. Pre-registered degradation: (1) refine concurrency 3→1 reviewer, skip criticizer; (2) stratified 40-task sample. Degradation events, if any, are recorded here.

## 5. Results

First full run: 2026-09-10 → 09-12, Terminal-Bench 2.0 (task commit `69671fba`), 36-task stratified sample (D-016 degrade step 2 — see §4), GLM-5.3-Flash, seed 1, both arms.

| | baseline (stock pi) | treatment (pi-plans, forced-plan-big) |
|---|---|---|
| Resolve rate | **3/36 (8.3%)** | **18/36 (50.0%)** |
| McNemar exact (paired) | colspan: p = 0.0003 (b=1 baseline-only, c=16 treatment-only, 19 both) | |
| Δtokens / task (in+out, treatment−baseline) | mean −48,918, 95% CI [−417,777, +255,102] (bootstrap 10k) | |
| Tokens per resolved task | 2,783,085 | **366,011** |
| Δturns / task | mean +1.00, 95% CI [−4.14, +6.83] | |

Treatment passed 16 tasks the baseline failed; it lost 1 task the baseline passed (path-tracing-reverse). Generated by `scripts/bench/analyze.ts`; every number recomputable from the committed inputs under `docs/benchmarks/data/` (per-trial result.json + usage/meta snapshots; full container logs are not published due to size).

Treatment failure modes (18 fails): 9 agent timeouts (3h cap), 5 agent exit errors, 4 verifier-failed. Baseline failure modes (33 fails): 30 verifier-failed, 3 timeouts.

McNemar counts: both-resolved = 19, both-failed = 2 (the "both" figure in some early drafts was a mislabeled concordant count; corrected in the published analysis.json).

**Fairness control disclosure (C-007/D-020):** the pre-registered "snapshot-diff restore before oracle scoring" control was **not implemented** in this run — the seeded pi-plans state lives under `.git/pi_plans/` (inside the workspace) with the artifact root pointed at `/tmp/pi-plans-bench/docs` (outside the graded tree). TB oracle tests read only `/app` artifacts and do not traverse `.git`, so scoring impact is assessed as negligible, but the control itself was not exercised; recorded here rather than silently claimed.

**Sensitivity (D-012): the seed-1 advantage is NOT stable.** The 17 discordant tasks were re-run at 2 additional seeds (both arms). Per-task majority vote across 3 seeds: **baseline 8, treatment 1, tie 8** — only 1 of the 16 seed-1 treatment wins held. On these borderline tasks the treatment's seed-1 dominance was substantially seed variance.

**Interpretation (within the strict scope of §6):** forced-plan-big on a flash-class model shows high run-to-run variance on borderline tasks rather than a stable improvement: single-seed outcomes ranged from 0/36 to 18/36 for the same task set across two identical-configuration runs. The robust, defensible summary at this sample size is: **no stable resolve-rate improvement was measurable**; treatment adds material wall-time cost (9/36 trials hit the 3h timeout cap vs 3/36 baseline); per-resolved-token counts were similar (treatment 366k vs baseline 2.78M — dominated by which tasks happen to resolve). A stable-effect claim would require multi-seed primary data (future work).

## 6. Disclosures and limitations

1. **No human in the loop**: approval gates were bypassed by an env switch; results do not represent interactive pi-plans usage.
2. **Forced-plan-big variant**: the treatment always uses plan-big rather than the product's narrowest-skill routing; this measures the mechanism, not the routing policy.
3. **Single model, single seed, no model-level seed replay**: exploratory; CIs quantify bootstrap uncertainty only.
4. **Plan-quality rubric not measurable in this run**: plan artifacts were not retained by the harness (see §7c); the rubric dimension is reported as a gap, not a score.
5. **Contamination**: public benchmark tasks may be in model training data; the exposure is identical across arms, so internal validity holds — external generalization does not.
6. **Not an official leaderboard submission**; oracle scoring per task's own tests.

## 7. Relation to other harness reports

Prime Agent (Prime Intellect) reports vendor-side numbers (e.g. ARC-AGI-3 95.5%) for its own task mix; those suites (ARC, GPU kernels) are not repository-planning tasks and were deliberately not reused here. Where we cite other harnesses' numbers, we link their primary sources and label them vendor-reported.

## 7b. Appendix: SWE-bench second benchmark — downgrade record (R-009)

The SWE-bench Verified harbor adapter was validated for task generation (5 instances generated successfully), but the second benchmark was not executed in this run. Reasons, per the pre-registered downgrade clause: (1) per-instance environments are heavyweight (multi-GB images, long builds) and the local disk budget (103 GB host free) was already consumed by the primary benchmark; (2) the primary result (p = 0.0003, 16:0 discordance) left no decision ambiguity that a second benchmark at n=40 would resolve within this run's scope. Scheduled as the first item of future work.

## 7c. Appendix: plan-artifact retention gap (rubric scope)

The treatment arm's planning flow ran (planning runs created, criticizer/reviewer subagents executed — verified in session logs), but PLAN_v*.md artifacts were not retained by the harness sync in this run: the driver watched three artifact roots, none of which received plan files, and hard-killed trials (timeouts) skip artifact sync entirely. The pre-registered rubric degrade clause (D-017) applies: the plan-quality dimension is reported as **not measurable in this run** (0/36 artifacts retained) with this reason recorded. Fix tracked for future runs: snapshot the entire pi-plans state root (`.git/pi_plans/`) rather than guessed artifact paths, and sync on a signal-safe interval instead of driver exit.

## 8. Future work

- Multi-model (strong + weak) to test the "planning helps weaker models more" hypothesis.
- `/plan-small` and `/plan-normal` variants and router-chosen planning (matching product behavior).
- TheAgentCompany (milestone-scored long-horizon tasks) once its harbor adapter ships.