# ⚔️ pi-arena

[![npm](https://img.shields.io/npm/v/pi-arena)](https://www.npmjs.com/package/pi-arena)
[![license: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
[![node ≥ 23](https://img.shields.io/badge/node-%E2%89%A5%2023-brightgreen)](#development)

Run the same task across multiple models **in parallel** and compare their answers side by side — with an optional referee judge that ranks the anonymized answers.

![pi-arena demo: type /arena, watch the models race, browse the ranked answers](demo/arena-demo.gif)

Works with **every model your pi installation knows about**, including models released after this package: contestants are discovered at runtime from pi's model registry, nothing is hardcoded. New model out? `pi update`, and it's in the arena.

## Install

```bash
pi install npm:pi-arena
```

## Quick start

```
/arena-models                 # pick 2+ contestants from your configured providers
/arena Explain the tradeoffs of SQLite vs Postgres for a small CMS
/arena-history                # browse past runs of this session anytime
```

That's it. On your very first `/arena` the contestant picker opens automatically, so you can also just start with a prompt.

## Commands

| Command | What it does |
|---|---|
| `/arena <prompt>` | Run the prompt against all selected models, side by side |
| `/arena-models` | Pick which models compete — live multi-select with filtering |
| `/arena-judge [spec]` | Pick the referee: interactive picker, `current`, `off`, or `provider/id` |
| `/arena-history` | Reopen any past run of this session in the interactive browser |

`/arena-models` opens a live multi-select: **↑/↓** to move, **space** to toggle, type to filter, **enter** to save.

## The live race

The moment you submit a prompt, an animated status panel appears above the editor: every contestant gets its own spinner, token/turn counters tick up as answers stream in, and the progress slots settle into solid blocks as models cross the finish line.

![Animated live widget: four models racing, two finished, spinners still going for the rest](demo/live-race.png)

Each contestant runs in its own isolated `pi` subprocess (read-only tools by default), so answers never touch your files. Contestants that hang are killed on timeout — and no contestant process ever outlives its arena run, even if the run itself fails.

## The scoreboard

When all runs finish, a compact scoreboard lands in your chat history: medals, judge scores as bars, and time / tokens / cost per model. Press **ctrl+o** on the message to expand it into the full answers and the referee's notes.

![Compact scoreboard: ranked models with score bars, per-model stats, and the crowned winner](demo/scoreboard.png)

The report is injected into your session as a normal message, so you can keep chatting about the results with your main model.

## The results browser

Right after the scoreboard (and any time later via `/arena-history`), an interactive browser opens:

![Results browser: scoreboard on top, the selected answer rendered below, keyboard hints in the footer](demo/browser.png)

| Key | Action |
|---|---|
| **↑ / ↓** | Switch between contestants (and the referee's verdict) |
| **enter** | Toggle a full-height, distraction-free reading view |
| **pgup / pgdn** | Scroll long answers |
| **esc** | Leave the reading view, then close |

## The referee

If a judge is configured (default: your current session model), the arena runs one extra round after the contestants finish: the answers are **anonymized and shuffled** — the judge sees "Answer 1..N" in random order, never model names — so it can't develop position or brand bias. It scores each answer out of 10 with a one-line reason, then the winner is revealed.

```
/arena-judge                  # interactive picker
/arena-judge current          # judge with the session's model (default)
/arena-judge openai/gpt-5.2   # or any explicit model
/arena-judge off              # skip judging entirely
```

## Configuration

Settings live in `~/.pi/agent/arena.json` (managed by the commands above, hand-editable):

```json
{
	"models": ["anthropic/claude-opus-4-8:low", "anthropic/claude-opus-4-8:high"],
	"judge": "current",
	"tools": ["read"],
	"timeoutSec": 300
}
```

| Field | Meaning |
|---|---|
| `models` | Contestant specs as `provider/id`, with an optional `:<thinking>` suffix |
| `judge` | `"current"` (session model), `"off"`, or an explicit `provider/id` |
| `tools` | Tool allowlist given to contestants (`[]` = no tools) |
| `timeoutSec` | Per-contestant timeout in seconds |

Model specs pass through pi's `provider/id:<thinking>` syntax, so you can race the *same* model against itself at different thinking levels — the config above pits Opus-low against Opus-high.

> [!WARNING]
> Beware of granting `write` / `edit` / `bash`: parallel contestants share your working directory.

## Headless / CI benchmarking

```bash
PI_ARENA_TEST="Refactor suggestion for src/foo.ts?" PI_ARENA_OUT=report.md \
  pi --mode json -p --no-session "arena"
```

Runs the arena without a TUI and writes the markdown report to `PI_ARENA_OUT` (stderr if unset). Handy for nightly "which model handles our codebase best" checks.

## Development

```
extensions/arena.ts     entry point — command registration and wiring
extensions/arena/       types, config, contestant (process management),
                        judge, report, run (orchestration), widget (live
                        animation), render, browser, picker (TUI)
```

`npm run check` type-checks, `npm test` runs the process-reaping and TUI test suites (Node ≥ 23). Both suites carry watchdogs: a hung test kills its own process tree and aborts instead of lingering.

## License

MIT