---
name: using-pi-coding-agent-eval
description: >-
    Orient an agent before writing or changing a pi-coding-agent-eval benchmark. Use this skill whenever a task
    involves an eval suite, benchmark preset, agent profile comparison, scripted Pi evaluation, eval metrics,
    reports, or evaluation artifacts. First inspect the package examples and the closest existing suite, then use
    the focused eval-writing or eval-inspection skill. Do not use it for unrelated Pi tests or generic TypeScript work.
---

# Use pi-coding-agent-eval

This is the orientation layer for the eval skills. The goal is to understand the repository's contract before adding
another benchmark shape.

## First-use rule

Before writing or debugging an eval, check whether this orientation was already completed in the current session. If
not, do this once:

1. Find the package root. In a checkout it is the repository root. In an installed package, resolve
   `pi-coding-agent-eval/package.json` with Node and use its containing directory.
2. Read the closest executable example under `test/examples/`:
   - `tool-profile-comparison` for a small suite and two profiles;
   - `multi-profile-comparison` for three profiles and all pairwise comparisons.
3. Read the closest integration test in the target repository. Treat the example and test as the usage contract.
4. Read the public declarations in `dist/core/types.d.ts` and the runner declaration when the API shape is unclear.
5. Record the behavior the new eval must protect, the observable result, and the smallest existing pattern to adapt.

Do not read every file when one example answers the question. Add a new executable example only when the behavior is
reusable or a regression would otherwise be easy to miss.

## Choose the focused workflow

- Writing or changing a benchmark: use `write-pi-coding-agent-eval`.
- Inspecting a completed run, report, or live comparison: use `inspect-pi-coding-agent-eval`.
- Testing the underlying real Pi process: also use the `using-pi-coding-agent-test` skill from `pi-coding-agent-test`
  when that package is installed.

## Keep the boundaries clear

An eval suite owns task selection, workspace preparation, prompts, validation, and cleanup. A benchmark preset names
suite-owned tasks and scheduling defaults. An agent profile describes the Pi setup being compared.

Do not move task selection into profiles. Do not make a profile-specific prompt when the task itself is supposed to be
identical for every profile. Use profile settings only for the agent configuration that is intentionally being compared.