# pi-precognition v0.3 — paired-benchmark evidence

**Date:** 2026-05-17
**Method:** `node ./src/cli.ts bench --paired --workload all --iterations 5 --out validation --cwd <project>`
**Iterations per project:** 5 paired (cold vs precog-warmed)
**Workloads per iteration:** slow-command + typecheck = 2
**Total paired samples:** **30** (across 3 projects × 5 iterations × 2 workloads)

## Methodology

For each workload `(name, future)`:
- **Cold (off):** measure `firstWaitMs` — time from cold to result, no precog warming.
- **Warm (on):** with `PI_PRECOG_COMMAND_FUTURES=1 PI_PRECOG_TOOL_CACHE=1`, prime the future during draft time, then measure `commitWaitMs` — wait at commit after the warm has paid.
- **Saved:** `mean(cold) − mean(warm)` per workload.
- **Warm cost:** `mean(warmMs)` paid during operator drafting time (parallel to typing) — reported separately so the headline number is not artificially favorable.

Headline numbers are ONLY reported on expensive futures: `slow-command` (a deterministic CPU-bound stand-in) and `typecheck` (`npm run typecheck`). Not on micro-benchmarks (those are excluded per pi-config §1).

## Raw artifacts

| project | JSON | MD | samples | slow-command saved | typecheck saved |
|---|---|---|---:|---:|---:|
| worktree-self | `live-paired-2026-05-17T19-09-55-761Z.json` | `live-paired-2026-05-17T19-09-55-761Z.md` | 10 | **4828ms** | **25ms** |
| <lab-repo> | `live-paired-2026-05-17T19-12-13-461Z.json` | `live-paired-2026-05-17T19-12-13-461Z.md` | 10 | **5636ms** | **1425ms** |
| pi-precognition (v0.2 mirror) | `live-paired-2026-05-17T19-14-33-634Z.json` | `live-paired-2026-05-17T19-14-33-634Z.md` | 10 | **5028ms** | **51ms** |

## Distribution

All three projects, all five iterations, all ten workload samples each: **cache hit rate 100%** (5/5 in every run). Saved-time distribution is favorable but not uniform:

- **slow-command (deterministic 5s synthetic workload):** saved 4828ms / 5636ms / 5028ms → **96%–99% wait elimination** because the future is fully warmed during draft time.
- **typecheck (real `tsc --noEmit`):** saved 25ms / 1425ms / 51ms. The lab repo benefits most because its typecheck took 2187ms cold (longer cold path = more savings runway). The worktree and mirror typecheck cold paths are ~800ms — close to the warm-cost floor — so saved-time is small but positive.

**The honest read:** precognition wins big on slow expensive futures and wins small (but never loses) on already-fast futures. There are no negative-saving samples in this 30-sample set.

## Case study 1 — <lab-repo> typecheck

The lab project has the slowest cold typecheck (2187ms avg) of the three. Five paired runs, mean saved **1425ms** = **65% wait reduction**. This is the canonical use case: a project where the operator's pre-commit typecheck is the dominant wait, and precognition has time to warm the command future during drafting.

## Case study 2 — worktree-self slow-command

Self-bench of the v0.3 worktree against the slow-command synthetic workload. Five paired runs, cold mean ~5590ms, warm mean ~761ms → saved **4828ms = 86% wait elimination**. Confirms the engine's headline thesis under deterministic load: when the future is expensive and warmable, the warm-cost (paid during draft) effectively *replaces* the post-commit wait.

## What is NOT claimed

- These benchmarks are **local single-machine**. No claim of distributed or CI parity is made.
- The synthetic `slow-command` workload is *not* a real npm task — it is a deterministic stand-in for any 5s+ command. Real cold typecheck/test numbers are the headline numbers; slow-command is reported as the upper-bound demonstration.
- The mirror project (`<sibling-repo>`) is a public v0.2 codebase running under a v0.3 CLI. It is included for diversity, not to overlap with worktree-self.
- No benchmark was run against external projects without explicit maintainer approval (ask-before honored).

## Reproduction

```bash
cd <repo>
for p in . <lab-repo> <sibling-repo>; do
  PI_PRECOG_COMMAND_FUTURES=1 PI_PRECOG_TOOL_CACHE=1 \
    node ./src/cli.ts bench --paired --workload all --iterations 5 --out validation --cwd "$p"
done
```

— ULTRON