# ARC Pi extension evals

These suites exercise every extension registered by `package.json` under a
small deterministic Pi replay harness. They follow the Anthropic-style
prompt/action/assertion shape used by
[`arc-skill-eval`](https://github.com/andysolomon/arc-skill-eval), but do not
call a model or an external service: the harness invokes compiled extension
factories against a fake Pi API and isolated temporary workspaces.

## Run

```sh
npm run build
npm run evals:check
npm run evals
```

Use `node scripts/extension-evals.mjs --list` to list cases, or filter a run
with `--extension <name>` and `--case <id>`. Each run writes `summary.json` and
per-case replay, output, trace, grading, and timing artifacts under
`evals-runs/<run-id>/`. Generated runs are ignored by Git.

## Suite contract

`manifest.json` maps each package extension to one suite. A suite declares its
extension entrypoint and cases. Cases may set up fixture files and environment
variables, invoke registered tools or commands, emit lifecycle events, sleep,
or shut down the session. Assertions cover registration, returned values,
expected errors, notifications, and bounded filesystem artifacts.

The manifest check compares the suite list with `package.json`'s
`pi.extensions` list, so a newly registered extension must add a suite before
CI can pass. Extension entrypoints are loaded from `dist/`; build first when
running the replay.