Compare arbitrary agent profiles against the same real Pi Coding Agent tasks.
## Summary `pi-coding-agent-eval` runs repeatable evaluations through real Pi sessions. It balances agent profile order, prepares isolated workspaces, validates outcomes, records agent telemetry, and calculates per-agent-profile and pairwise statistics. The package defines only one agent profile: `vanilla`, with no additional extensions and all registered tools. Consumers define every other agent profile and explicitly select the profiles to run. Executable behavior contracts and runnable comparisons live together under [`./test/examples/`](./test/examples/). ## Installation ```bash npm install pi-coding-agent-eval ``` The install includes the TypeScript API, the `pi-eval` CLI, and all npm dependencies. Eval uses [`pi-coding-agent-test`](https://github.com/alexshpunt/pi-coding-agent-test) to run the `pi` executable available on the system `PATH`. From a checkout, install the repository dependencies instead: ```bash npm install ``` ## Optional Pi skills The package includes three skills for creating and inspecting evaluations: - `using-pi-coding-agent-eval` — orient before changing a benchmark; - `write-pi-coding-agent-eval` — write suites, presets, and profile comparisons; - `inspect-pi-coding-agent-eval` — inspect artifacts, reports, and pairwise results. Install the published package as a Pi package: ```bash pi install npm:pi-coding-agent-eval ``` From a checkout, install the local package with the helper script: ```bash npm run skills:install npm run skills:install -- --local ``` The first command updates user settings. The second writes a project-local Pi setting. ## Prerequisites - Node.js 22.19 or newer; - Pi installed with the `pi` executable available on `PATH`; - model credentials configured for paid model runs. ## Run an evaluation The package does not provide a built-in benchmark suite. A suite is supplied by the caller through a config file. The repository includes a complete deterministic config at [`./test/examples/tool-profile-comparison/eval.config.mjs`](./test/examples/tool-profile-comparison/eval.config.mjs). It registers the real `tool-comparison` suite and two scripted profiles, so it makes no model request. Run it from the repository root: ```bash npm run example:cli ``` The equivalent direct CLI command is: ```bash npm run build node ./dist/cli.mjs run tool-comparison easy \ --config ./test/examples/tool-profile-comparison/eval.config.mjs \ --agent-profiles filesystem-tools,bash-only \ --run-id cli-example ``` To exercise the global model, thinking, skill, and prompt flags without making a model request: ```bash node ./dist/cli.mjs run tool-comparison easy \ --config ./test/examples/tool-profile-comparison/eval.config.mjs \ --agent-profiles filesystem-tools,bash-only \ --model scripted/scripted-model \ --thinking off \ --skill "$(pwd)/skills/write-pi-coding-agent-eval" \ --system-prompt "Use only the tools exposed by your profile." \ --append-system-prompt "Complete the task and stop when it is done." \ --run-id cli-settings-example ``` The run writes its artifacts to `./.tmp/tool-profile-comparison/cli/results/cli-example`. Rebuild that report without rerunning Pi: ```bash node ./dist/cli.mjs report ./.tmp/tool-profile-comparison/cli/results/cli-example ``` The CLI accepts the global `--model`, `--thinking`, `--skill`, `--system-prompt`, and `--append-system-prompt` settings. Repeat `--skill` and `--append-system-prompt` when you need more than one value. A profile can override these settings in the config file. The evaluator calculates every supplied agent profile pair as `right - left`. ## Thinking off enforcement When a profile's effective thinking level is `off`, the evaluator automatically loads its built-in `disable-thinking` extension. The extension checks the selected model's metadata and applies the provider's real hard-off request control only when `off` is supported. It never substitutes `minimal` or `low` reasoning. The control covers Pi's OpenAI, Anthropic, DeepSeek, Qwen, Google, Vertex, Together, Baseten, Mistral, Bedrock, and compatible chat-template request formats. Each trial records control evidence in `agent/provider-control.jsonl`; reported reasoning tokens or thinking blocks make an `off` trial fail. The extension path is also exported for direct Pi use: ```js import { disableThinkingExtension } from "pi-coding-agent-eval"; ``` The extension's package subpath is `pi-coding-agent-eval/disable-thinking`. ## TypeScript API The complete direct API example is [`./test/examples/tool-profile-comparison/run-api.mjs`](./test/examples/tool-profile-comparison/run-api.mjs). It uses the exported `runEvaluation` function with the real `tool-comparison` suite and scripted profiles, so it makes no model request. Run the exact example: ```bash npm run example:api ``` The example uses a global model, thinking level, system prompt, and appended system prompt. It overrides `thinking` for `bash-only`, calculates a custom `exact-files` metric, writes a custom `api-report.json`, and checks that both profiles pass. This is the canonical runnable example for the direct API. The evaluator keeps these responsibilities separate: - a suite owns task selection, workspace preparation, prompts, validation, and cleanup; - a benchmark preset names the task set, seed, and default attempts; - an agent profile owns the model, thinking level, skills, extensions, tools, prompt settings, and optional scripted conversation; - run-level model, thinking, skills, and prompt settings are inherited by profiles unless a profile overrides them. The public API is documented with TSDoc comments next to its source declarations. The published package exposes the same contract through its `dist/*.d.ts` files. There is no generated documentation site or manually maintained API copy. ## Runnable examples From a checkout, run all free deterministic checks, including the integration suite and the executable API/CLI examples: ```bash npm test ``` To run only the integration suite: ```bash npm run test:integration ``` The exact CLI and direct API examples use the same suite without model requests: ```bash npm run example:cli npm run example:api ``` The CLI example writes `./.tmp/tool-profile-comparison/cli/results/cli-example`; the API example writes `./.tmp/tool-profile-comparison/api/results/api-example` and adds `api-report.json`. | Example | Covers | | --- | --- | | [`tool-profile-comparison`](./test/examples/tool-profile-comparison/) | Filesystem tools versus Bash, scripted and real-model live modes | | [`multi-profile-comparison`](./test/examples/multi-profile-comparison/) | Filesystem tools, Bash, unrestricted `vanilla`, and every pair among three agent profiles | Watch the exact scripted tests in Pi's native TUI: ```bash npm run example:scripted-live npm run example:multi-scripted-live ``` Run the three-profile comparison with a real model: ```bash npm run example:multi-live -- \ --model opencode-go/deepseek-v4-flash \ --thinking high \ --attempts 3 ``` ## License [MIT](./LICENSE)