# poke-crucible

Run masses of headless **Gen 3 OU** battles to rank Teams by how often they win.
Both sides of every Battle are driven by one fixed, competent heuristic **Pilot**,
so a result measures the **Team**, not the player. Three capabilities, each a
single seam built on the last:

- **`evaluate`** (Slice 1) — rank Teams by **Strength** against a fixed Gauntlet, or round-robin.
- **`critique`** (Slice 2) — an LLM **Critic** explains one Team's weaknesses and matchup holes.
- **`design`** (Slice 3) — the **Designer** evolves stronger Teams, with the LLM proposing mutations and naming the Archetypes it discovers.

Vocabulary lives in [`CONTEXT.md`](CONTEXT.md); the specs are the
[PRDs under `docs/`](docs); decisions are the [ADRs](docs/adr).

## Install & build

```bash
npm install
npm run build      # compiles TypeScript to dist/
```

Requires Node ≥ 20. The stack is in-process TypeScript on the `@pkmn/*` toolkit
(`@pkmn/sim`, `@pkmn/sets`, `@pkmn/data`) — no server, no network, no Python
(ADR 0001).

## CLI

```bash
# Absolute Strength vs the fixed Gauntlet
node dist/cli.js eval --mode gauntlet team.txt

# Relative Strength: rank several of your own Teams round-robin
node dist/cli.js eval --mode round-robin teamA.txt teamB.txt teamC.txt

# Pipe a single Team paste in on stdin
cat team.txt | node dist/cli.js eval
```

Options: `--mode <gauntlet|round-robin>` (default `gauntlet`), `--battles <N>`
(default 100), `--seed <string>`, `--gauntlet-version <v>`, `--out <file>`
(default `poke-crucible-results.json`). Each team file is standard Showdown
export/paste text; multiple Teams in one file are separated by
`=== [gen3ou] Name ===` header lines.

The CLI prints a ranked Strength table (with a 95% confidence interval and a
per-matchup breakdown) and writes the full structured result to `--out`.

## Programmatic API

The base seam is `evaluate` — the Critic and Designer build on it, not on the CLI:

```ts
import { evaluate } from 'poke-crucible';

const result = await evaluate({
  teams: showdownPasteText,              // string or array of pastes/{name,paste}
  opposition: { mode: 'gauntlet' },      // or { mode: 'round-robin' }
  battlesPerPairing: 100,
  seed: 'my-seed',
});
// result: ranked Strength rows (per-matchup win-rates tagged with Archetype),
// the seed, the Gauntlet version, and any illegal-Team validation errors.
```

Illegal Teams never throw past the seam — they come back as `result.errors`,
named, so a bad Team can't silently poison a run.

## How it works

- **Pilot** (ADR 0002): one shared Gen 3 heuristic, identical on both sides,
  built in competence layers — secure a KO → pivot out of a losing matchup →
  recover → lay Spikes / spin → spread Toxic/burn/paralysis → set up → max
  damage. Every Strength number is relative to this Pilot.
- **Determinism**: a master seed threads through `@pkmn/sim`'s PRNG; each
  pairing's Battles derive sub-seeds deterministically and sides are swapped
  50/50, so a fixed seed reproduces identical results.
- **Gauntlet** (ADR 0003): a fixed, **versioned** roster of Archetype-labeled
  Gen 3 OU Teams. Every result records the version, so `62% Strength` means the
  same thing next week. Two versions ship: **`g1`** (default; hand-authored) and
  **`g2`** (real [Smogon ADV OU sample teams](https://www.smogon.com/forums/threads/adv-ou-sample-teams.3687813/),
  fetched verbatim, `--gauntlet-version g2`). `g1` stays the default baseline;
  swapping it is a deliberate rebaseline.
- **Parallelism**: `--concurrency <N>` (or `POKE_CONCURRENCY`) runs Battles across
  `N` worker threads. All Battles of a run are dispatched together so the pool
  stays saturated; results are **byte-identical** to serial (each Battle keeps
  its own seed). Default is 1 (inline). It pays off on big, expensive runs — the
  Designer, large evaluations — where it gives roughly a 2× speedup on 4 workers;
  for small/fast runs the thread overhead can make it a wash, hence the default.

### Gauntlet provenance (a deliberate deviation)

The design envisioned sourcing the Gauntlet from Smogon's sample teams + usage
stats via fetch-and-parse. That is deferred: the `g1` roster is **hand-authored**
from canonical Smogon Gen 3 OU sets (see
[`src/gauntlet/teams-g1.ts`](src/gauntlet/teams-g1.ts)). Because the Gauntlet is
versioned and every result records its version, re-sourcing later is a `g2`
bump, not a rewrite — nothing silently drifts.

## Critic (LLM analysis — Slice 2)

Turn an evaluation into a readable, grounded critique of one Team:

```bash
node dist/cli.js critique team.txt
node dist/cli.js critique --model claude-sonnet-5 --effort low --battles 100 team.txt
```

The Critic runs the evaluation, then asks Claude (via the **Claude Agent SDK**)
to explain the Team's weaknesses, the Archetypes that exploit them, the
mechanism, and concrete legal fixes — writing a structured `Critique` JSON. It
is told the Pilot under-rates stall (ADR 0004), so it caveats defense-dependent
verdicts rather than presenting them as ground truth.

**Auth (individual use):** the Critic runs on your **Claude subscription**, not
a pay-per-token key. Log in with Claude Code (or set `CLAUDE_CODE_OAUTH_TOKEN`),
and make sure **`ANTHROPIC_API_KEY` is unset** — it shadows the subscription
OAuth token and reverts to per-token billing. The model is called with all tools
disabled and low reasoning effort; nothing runs on your filesystem.

Programmatically the seam is `critique(input, model)` / `critiqueTeam(paste,
model, opts)`; the model is any `CritiqueModel` (use `StubCritiqueModel` in tests,
`AgentSdkCritiqueModel` live). See [`docs/prd-slice-2-critic.md`](docs/prd-slice-2-critic.md).

## Designer (discovery — Slice 3)

Evolve stronger Teams automatically. A population of legal Teams is scored each
generation by a **composite fitness** (Gauntlet Strength + peer round-robin + a
worst-matchup robustness term), bred by mutation and crossover, with the LLM
proposing variants — driven by the Critic's findings — and naming the Archetypes
it discovers.

```bash
# Evolve from a starting Team (omit the file to search from scratch)
node dist/cli.js design --population 8 --generations 4 team.txt

# Random operators only — no LLM, no subscription call
node dist/cli.js design --no-llm --population 8 --generations 4
```

Options: `--population`, `--generations`, `--battles` (per fitness pairing),
`--model` / `--effort` (Designer LLM), `--no-llm`, `--seed`, `--gauntlet-version`,
`--out`. Every candidate is validated legal Gen 3 OU; the run is deterministic
under its seed and reports each Team's lineage, Archetype name, fitness
components, plus the budget used and total Battles run. To bound compute it
caches unchanged Teams, screens cheaply before spending full battles on
contenders, and honours a hard battle cap.

Programmatically the seam is `design(input, model, options)`; the model is any
`DesignerModel` (`StubDesignerModel` for tests, `AgentSdkDesignerModel` live),
and an optional `critic` lets the Critic drive the mutation proposals. Auth is
the same subscription path as the Critic. See
[`docs/prd-slice-3-designer.md`](docs/prd-slice-3-designer.md).

**Fitness is deliberately not raw win-rate (ADR 0004)** — it rewards well-rounded
Teams over glass cannons and dampens the Pilot's offense lean (see the limitation
below for how far that goes).

## Testing (the trust gate)

```bash
npm run typecheck
npm test
```

The suite asserts external behavior only, through the `evaluate()` seam (never
the Pilot's individual move choices): determinism, sanity ordering (a strong
Team out-Strengths a weak one), **archetype non-bias** (a solid stall Team stays
competitive in a head-to-head ladder of the eight archetypes and can beat
offense — the live check that the Pilot isn't biased against defense), legality,
and stable win-rates. Every Gauntlet Team is asserted legal Gen 3 OU.

### Known limitation: the Pilot's offense lean

The shared greedy Pilot systematically **under-rates stall** — offense plays
near-optimally under greedy heuristics while stall needs long-term planning. In
an archetype-vs-archetype ladder the best stall Team lands mid-lower (≈38%
Strength, 6th of 8), not mid-pack.

This was probed exhaustively ([ADR 0004](docs/adr/0004-greedy-pilot-underrates-stall-search-deferred.md)):
three Pilot improvements (anticipatory switching, tempo tweaks, a positional-eval
Pilot) all failed to lift stall; real-sim look-ahead is infeasible (`@pkmn/sim`
state can't be resumed after deserialization); and the Designer's composite
fitness only *partly* corrects the lean — stall's worst matchups are genuinely
~0% under the Pilot, so a robustness term can't recover them.

The lean is therefore a **known, bounded, documented property of the greedy-Pilot
design**, not a bug — every Strength number is explicitly relative to this Pilot
(ADR 0002). Closing it needs a stronger Pilot (a shallow search or ML policy —
deferred) or a non-Pilot signal of a Team's merit; both are future work. The
Critic caveats defense-dependent verdicts and the Designer's fitness avoids
rewarding glass cannons, so the slices degrade gracefully rather than silently
trusting the lean.