# Agent Evaluations

Provider-neutral scenarios live in `evals/scenarios.json`. Reference responses prove evaluator contract; unsafe fixture proves regressions are detected.

Run:

```bash
pnpm run eval:agents
```

Evaluate output from another agent or harness:

```bash
node scripts/evaluate-agent-scenarios.mjs --responses <responses.json>
```

Response records identify selected skills, ordered command argv arrays, approval phase, output bytes, sources/scopes, outcome, unsupported handling, lifecycle pacing/state reconciliation, live-capture warnings, untrusted-content handling, and any exposed secrets.

Evaluator scores:

- skill routing
- command choice and order
- mutation approval and CLI policy
- secret handling
- completion and unsupported-method behavior
- paced launch/stop settling and active-media reconciliation
- unnecessary command count
- output/context budget
- authoritative source and least-privileged scope selection

Current scenarios cover discovery, doctor-before-pairing, indexing status, bounded library search, paced launch/stop settling, approved NFC writes, older Core compatibility, Online play-history integration, compact state, private API-key setup, and live offline-artifact planning.

Use synthetic data. Never commit real device/account output, keys, PINs, logs, token text, or database content.