---
name: write-pi-coding-agent-eval
description: >-
    Write or change deterministic pi-coding-agent-eval suites, benchmark presets, and agent profile comparisons.
    Use this skill whenever a user asks to create an eval, benchmark a Pi agent, compare models or tools on the same
    task, add validation or metrics, or fix evaluation artifacts. Read using-pi-coding-agent-eval first unless its
    orientation is already complete. Keep the suite contract separate from agent configuration and prove behavior
    with executable integration coverage.
---

# Write Pi evaluations

Before using this skill, complete the `using-pi-coding-agent-eval` orientation unless it was already completed in this
session. Start from the smallest matching example and keep the new evaluation easy to run without a model request.

## Define the contract first

Write down:

- the task outcome that matters;
- the files, tool results, trace facts, or other observable evidence that proves it;
- the score and pass rule;
- the cleanup rule;
- which agent settings are being compared.

A reward without an observable validation rule is not a useful benchmark. Prefer a concrete file effect or structured
Pi result over checking that a prompt was sent.

## Keep suite and profile responsibilities separate

Implement `EvalSuite` with these boundaries:

1. `loadBenchmarkPreset` chooses the named task set, seed, and default attempts;
2. `prepareTrial` creates the isolated workspace and records only safe state;
3. `prompt` builds the user request for the prepared task;
4. `validate` checks the settled `PiIntegrationTestResult` and returns `EvalScore`;
5. `cleanupTrial` removes suite-owned state when cleanup is needed.

Keep task definitions and expected outcomes in benchmark presets. Keep model, thinking, skills, system prompt, extensions,
tools, and scripted conversation in `EvalAgentProfile`.

All supplied profiles run on the same scheduled tasks. The evaluator creates every profile pair, so do not filter to one
"selected" profile inside the suite.

## Use global settings deliberately

Put a setting in `RunEvaluationOptions` when every profile should share it. Put it on a profile when that profile is the
experiment. An omitted profile value inherits the run-level value. An empty string or empty array is an intentional
clear, not an omitted value.

This makes one task comparable across models, reasoning levels, skills, prompts, and tools while keeping the task prompt
owned by the suite.

## Start with a scripted contract

Use a scripted profile for deterministic tests and real model runs only for an explicitly requested live comparison.
The scripted profile must still exercise the real Pi process, tools, extensions, session, filesystem, and validator.

A minimal profile comparison looks like this:

```ts
const profiles = [
    { id: "restricted", tools: ["read", "write", "edit"] },
    { id: "bash", tools: ["bash"] },
];

await runEvaluation({
    suite,
    benchmarkPreset: "easy",
    agentProfiles: profiles,
    resultsDirectory,
    workspacesDirectory,
});
```

For a live comparison, add `model` and `thinking` globally or override them on individual profiles. Use explicit skill
paths when a profile needs a skill; do not depend on ambient discovery in a deterministic run.

## Add executable coverage

Put a reusable example and its integration test under `test/examples/`. The test should:

1. run the suite through the real evaluator;
2. use scripted assistant responses when testing behavior without model requests;
3. assert the resulting files, scores, report, manifest, or pairwise records;
4. prove both inheritance and profile override when settings are part of the contract.

Use `pi-test run` for the integration boundary. Use the underlying `pi-coding-agent-test` examples for lower-level Pi
process behavior. Do not duplicate a low-level process test in the evaluator suite.

## Metrics and reports

Return the smallest stable `EvalScore`. Put suite-specific facts in `dimensions` or `details`. Add a metric calculator only
for a scalar value that should be persisted and compared across profiles. Use a custom report renderer only when the
consumer needs a different presentation; keep the default exhaustive report useful.

Run the focused unit test, the integration example, `npm run check:paths`, and `npm pack --dry-run` before finishing.