--- title: Evaluating agent runs description: How to run an Agency agent against a test suite, score it with graders, judge its output by hand, and keep everything in one run directory. --- # Evaluating agent runs `agency eval` runs, grades and compares agent runs. Everything it produces lives in a **run directory**: a folder with the runs' statelog and an append-only file of annotations (grades, checklist answers), plus a free-form `notes.md` you write yourself. The main commands are: ``` agency eval run ([:] | --agent-cmd '') (--suite | --input ) [-n ] [--trials ] agency eval grade [--suite ] [--goal ] [-n ] agency eval logs [-f] agency eval optimize [:] [--suite ] [--goal ] [--graders ] agency label --checklist [--annotator ] agency runs add [--statelog ]... [--code ] [--workdir ] agency runs list ``` `agency label` and `agency runs` also work without the `eval` prefix, as written above. ## Running a test suite `agency eval run` runs an Agency agent against every test in a suite and writes a run directory: ```bash agency eval run agent.agency:evalMain --suite suite.json --out runs/smoke ``` A suite is a JSON file with `{ "inputs": [...] }` or a directory with one `.json` file per test. A test looks like this: ```json { "id": "fizzbuzz-write", "input": "Write fizzbuzz in Agency", "goal": "Should produce a typechecking fizzbuzz program.", "files": "./fixtures/empty-project" } ``` A test says nothing about the agent. Tests describe the work; whoever runs the eval names the agent as the positional argument, `file.agency:node`. `input` is what the agent is told: a string, or a JSON object for agents that take structured data. The runner passes it as the entry node's single parameter, whatever it is named. An agent that takes no input runs a test with none — then the entry node takes no parameter. Within one suite, either every test has an input or none does; the runner checks the parameter count before anything runs. `goal` is the success criterion. The agent never sees it. The default LLM judge needs it; a test with its own graders makes it optional (see [Custom graders](#custom-graders)). `id` defaults to a generated id and must be filesystem-safe when you supply one. `expected` is an optional gold output that match graders read. `files` names the test's fixture directory (see [Test files and suites](#test-files-and-suites)). `timeoutSec` overrides the suite's wall clock for one test. `graders` names the test's own grading module, resolved relative to the test; in the test-directory form a `graders.ts` beside `test.json` is picked up automatically. Graders are code the harness runs, so pulling a remote suite means trusting it. For a single ad-hoc run, use `--input` instead of `--suite` (grade it afterwards with `eval grade --goal`): ```bash agency eval run agent.agency --input "Summarize the README in one sentence" ``` Options: - `[:]` (positional): the agent. A directory resolves to `main.agency` inside it. The node defaults to `main`. - `--suite `: the tests. A JSON file, a directory, or a git source (`URL[//subdir][?ref=...]`). Mutually exclusive with `--input`. - `--input `: one inline test whose input is this text, no suite file needed. Mutually exclusive with `--suite`. - `--test `, `--tags `: run a subset of the suite (a glob over test ids, or tests carrying every listed tag). Preview the selection with `agency eval ls`. - `-o, --out `: the run directory. It must not exist yet. Defaults to `runs/-` (or under `eval.runsDir` from `agency.json`), so run directories list in creation order. - `-n, --parallel `: run up to this many tests at once. Above 1, per-agent output is replaced by a live status board on stderr with each test's name, state, elapsed time and cost so far. Drill into a live run with `agency eval logs -f`. - `--trials `: run every test this many times, each repetition in its own run directory at `//`. Agent configuration (strict types, tool-loop caps) comes from the `agency.json` beside the agent, not from eval flags. Every test runs whatever the others did: a test whose agent errored is a `run` row that grades 0, not a reason to stop. Running never grades. When the run finishes it prints the run directory and the grade command, and exits 2 if any test errored (the run directory is complete either way; 1 means it could not run at all): ``` Run completed: 3/3 tests ok total LLM cost: $0.42 runs written under runs/smoke grade it with: agency eval grade runs/smoke ``` Each run's agent process gets a wall-clock limit of 60 seconds unless `eval.limits.wallClockSec` in `agency.json` raises it: ```json { "eval": { "limits": { "wallClockSec": 900, "maxCostUsd": 50, "maxBatchCostUsd": 100 } } } ``` `maxCostUsd` is a per-run LLM spend ceiling, $50 by default. The harness watches each run's cost as it happens and kills the run when it passes the cap. Enforcement lags by one LLM call, because a call's cost is known only when it returns, so treat it as an accident stopper rather than an exact budget. `maxBatchCostUsd` caps one `eval run` invocation as a whole — every test and every trial — and has no default. Spend is summed as runs finish; once it crosses the cap no further test starts, the ones already running finish under their own `maxCostUsd`, and the run reports `batch cost cap exceeded`. So the worst case is `maxBatchCostUsd` plus `--parallel` times `maxCostUsd`. The tests that never started have no run directory, so the batch's trial grid is uneven: `eval grade` still grades every run that exists and reports the batch as having no statistics, and statelog shows it as incomplete. ## Command agents `--agent-cmd` runs a CLI as the agent instead of compiling an `.agency` file — the way to benchmark `agency agent` itself: ```bash agency eval run \ --agent-cmd 'agency agent --agent code --policy approve-all --max-tool-call-rounds 100 --verbose -p -- {input}' \ --suite evals/terminal-bench-mini ``` The command string must contain `{input}`; every occurrence is replaced with each test's input. The string is tokenized by a minimal quote-aware splitter and **never passes through a shell**: no expansion, no operators — and substitution happens after tokenization, per token, so a hostile task is inert (`; rm -rf /` inside a task is bytes in one argv entry, not a command). An object task substitutes as its JSON serialization. The command runs in each input's isolated workdir. The workdir is seeded from the input's `files` plus the invoking directory's `agency.json`/`.env` (so two machines running the same benchmark see the same config); there is no agent closure and nothing is compiled. **Agency CLIs only.** The harness hands the command's process the statelog path via `AGENCY_CONFIG_OVERRIDES` and a shared trace id via `AGENCY_TRACE_ID` — every compiled Agency process honors both, so the agent's own execution record (tool calls, cost, interrupts, its whole process tree) becomes the eval record, and grading/judging work unchanged. A non-Agency command writes no statelog and the run fails saying so. Rules that follow from the mechanism: - **The command must run headless and one-shot** (`agency agent --policy approve-all -p -- {input}`). There is no IPC channel, so eval's interrupt auto-approval cannot reach a command agent — an interactive command just waits for input that never comes until the wall clock kills it. - **Do not pass `--log` inside the command** — it overrides the statelog path the harness set, and the run fails with an error naming this. `--trace` is safe (merged, both survive). - **Compile-time eval flags don't apply.** `--max-tool-call-rounds`, `--max-tool-result-chars`, and `--strict` are baked into compiled file agents; a command target compiles nothing, so combining them with `--agent-cmd` is an error — put the agent's own flags inside the command, budget flags (`--max-cost`) included. - **Keep credentials in the environment, never in the command.** The command string is recorded verbatim in the run directory's annotations; the child inherits your environment, so API keys need no argv. - **Limits:** the wall clock applies (enforced by the harness); memory is capped via `NODE_OPTIONS --max-old-space-size` (V8 heap of Node processes — weaker than the file-target sandbox); output is drained and capped for display but never fails the run. - `eval optimize` does not accept `--agent-cmd`: the optimizer mutates agent files, which a command target does not expose. ## Test files and suites A workdir is seeded from two ingredients: the test's declared files and the agent's own code. The agent's code means the entry `.agency` file, everything it imports, and any local TypeScript those imports use. It is computed, so you never list agent files by hand, and the same suite can grade any agent. A test declares its fixture directory with `files`. The contents land at the workdir root: ```jsonc { "id": "summarize", "input": "Summarize report.txt into summary.md", "goal": "summary.md captures the report's findings", "files": "./fixtures/summarize" } ``` For file-heavy tests there is a directory form: a directory of test directories, each holding `test.json` and an optional `files/` directory. The test's `id` defaults to the directory name. Point `--suite` at the parent. A directory holds one suite shape, never a mix: a single suite file with an `inputs` array, loose one-test `.json` files, or test directories. Suites and fixtures can come from git. Anywhere a directory is accepted, a git source works too: ```bash agency eval run a.agency --suite 'github.com/you/evals//tests?ref=v1.2' ``` `//subdir` names a directory inside the repo. `?ref=` takes a branch, tag, or commit sha. Whatever you wrote, the run records the resolved sha, so any past run is pinnable by copying its sha into `?ref=`. Clones cache under `~/.agency/cache/git/`; branch refs re-fetch per run, shas never do. An agent that reads a project file that was never seeded gets a file-not-found error inside the workdir, and the run's error names what was seeded. ## Creating a test with a form `agency eval new ` asks for the parts of a test and writes the folder, so you never have to remember the layout or the keys of `test.json`. Point it at a suite directory in the test-directory form, existing or empty: ```bash agency eval new bar-evals ``` ```text NEW TEST bar-evals folder name says-its-name description The script is told its name and asked for it. input What's your name? files a directory of files to seed into the working directory, relative to the suite (optional) goal The reply says its name is 'the amazing Agency language'. It may add a greeting, but it must not give any other name.█ ↑↓ move Enter save Ctrl+S save Ctrl+C quit ``` The form opens with the cursor in the first field, so you can start typing at once. Whatever you type is kept as you type it. Enter moves to the next field, and on the last field it offers to save; the arrow keys move too. `input` and `goal` are often several sentences: when `$EDITOR` is set, Ctrl+E opens them in your editor and the text comes back when you close it (`code --wait` works, and a quoted path with a space in it does not). Without `$EDITOR` they are typed on one line like the others. The folder name and `goal` are required; the rest is optional, and an empty field says so in its help. An empty `input` means the agent takes none; a filled one is a plain string, which is what the agent is given: the entry node's one parameter, or `{input}` in a command agent. Enter on the last row, or Ctrl+S from anywhere, asks `write this test? y / n`. `y` writes `bar-evals/says-its-name/test.json`, copies the files directory to `files/` when you named one, closes the form, and prints the run and grade commands to copy. Ctrl+C quits; if you typed anything, it asks first. Nothing else is written. A test with a `goal` and no graders is scored by the bundled goal judge, so this is all a plain test needs. Below the fields is a `graders` section. Ctrl+N adds a grader. Its first row picks what to check with the arrow keys, and the questions for that kind appear below it as you answer; text is typed into the row, and a yes/no or a list is picked with the arrow keys too: ```text graders 1. what to check count things in the trace count tool calls which tool fetchMarkdown only repeats yes at most 0 name no-repeated-fetches must pass yes ``` Ctrl+X on a grader's first row removes it, after a question. Save then writes `graders.json` beside `test.json`; the five kinds and their fields are described in `docs/dev/evals/graders-json.md`. Once a test has graders, its `goal` is not judged (the goal row says so); add a goal grader to judge it. The form does not take structured inputs, tags or timeouts; write those by hand, and the form keeps them when it opens the test again. ## Editing a test `agency eval edit ` opens the same form over an existing test folder, loaded from its `test.json` and `graders.json`: ```bash agency eval edit bar-evals/says-its-name ``` Two rows cannot be changed. The folder name is the test's id, so renaming a test is moving the folder. The files row shows `files/ (12 files)` or `none`; replacing seeded files would mean deleting the old copy, which the form does not do. A structured `input` (a JSON object) shows `structured input: edit the file`. A test that has a `graders.ts` opens with the graders section reading `graders.ts: edit the file`; the rest of the form works. Save writes `test.json` and `graders.json` back in place and leaves everything else in the folder alone. Keys the form does not show, such as `tags`, `timeoutSec`, `expected` and `metadata`, are kept. Removing every grader deletes `graders.json`. Quitting an unchanged test asks nothing. A folder whose `test.json` names an `id` that differs from the folder name is refused; rename one of them. ## The run directory Each run writes one directory: ```text runs// statelog.jsonl # every test's trace, one file annotations.jsonl # what anyone concluded: run rows, grades, labels notes.md # your free-form notes about the run (optional) code// # the agent code that ran, keyed by its content hash workdir// # the isolated directory each test ran in checklists/ # your labeling checklists (see below) ``` The statelog is the record of what happened. Everything else is derived from it or appended beside it, so a directory with only a `statelog.jsonl` is already a valid run directory. `agency runs list ` shows one line per trace: ``` TRACE STARTED ENDED TIME COST LLM TOOLS SCORE NOTES LABELED INPUT rpGXbww2 2026-08-18 09:14 ok 41s $0.12 5 12 0.71 yes Write fizzbuzz… ``` You can also build one by hand. `agency runs add --statelog run.jsonl` merges a statelog in by trace, `--code agent.agency` stores the agent's code, and `--workdir ./project --trace ` snapshots a directory for one trace. `agency run agent.agency --capture-workdir ` does all three for a plain run. ## Scoring a run `agency eval grade ` scores a finished run and prints an objective between 0 and 1: ```bash agency eval grade runs/smoke # 3/3 tests ok # objective 0.71 # goal 0.71 ``` `-n, --parallel ` grades up to that many run directories at once — judge calls are LLM calls, so a big group grades much faster in parallel. A progress line per graded run (`graded 3/15 — objective 0.933 ($0.02, 12s)`) goes to stderr, and when the pass's judges cost money the report ends with `grading cost: $x.xx`. Each grader's verdict is appended to `annotations.jsonl` as a **score**. Grading again appends another pass rather than rewriting anything, so every grading pass survives and the latest complete pass is the one listings show. Re-grading costs nothing for `ExactMatch`, `Contains`, `Similarity` and function graders that do not call `judge`; an `LlmJudge` still makes a live LLM call each time. Each test grades itself with the `graders` module it carries; a test without one is scored by the bundled goal judge against its `goal`. `eval run` stores a copy of every test's graders in the run directory, and `eval grade` uses that copy, so a run scores the same days later wherever it is read. When you improve a grader and want old runs re-scored without re-running the agent, pass the suite: `eval grade runs/smoke --suite evals/smoke` grades each run with its test's *current* graders, matched by test id. A run whose test is missing from the suite is an error. `agency eval logs ` opens the run in the interactive viewer, one trace per test, with each trace's grades, notes and labels summarised on its row. Press `t` for the timeline views (see the observability guide). ### Custom graders A grading module default-exports one grader or a list of them: ```ts // graders.ts import { grader, ExactMatch } from "agency-lang/eval"; import { existsSync } from "fs"; import { join } from "path"; export default [ // the return value grader(({ output }) => String(output).length < 500, { name: "concise" }), // a file the agent wrote grader(({ workdir }) => existsSync(join(workdir, "analyze.py")), { name: "wrote-script" }), // what it did along the way grader(({ record }) => record.metrics.costUsdTotal < 0.05, { name: "cheap" }), // compare against the test's `expected` field new ExactMatch({ mustPass: true }), ]; ``` A test names its module with `"graders"` in its spec, or, in the test-directory form, by a `graders.ts` beside its `test.json`. Graders are test-side, like `goal` and `expected`: there is no suite-wide module. A grader function receives `{ output, test, workdir, record, judge }` and returns a number from 0 to 1, a boolean, or a full `Grade`. A `Grade` pairs a score with the reasoning that is stored next to it, and its score is tagged with its kind: ```ts grader(({ output }) => ({ score: { kind: "binary", pass: String(output).includes("done") }, feedback: "looked for the word done", }), { name: "says-done" }); // scalar form: { score: { kind: "scalar", value: 0.75 }, feedback: "..." } ``` `judge(...)` already returns a `Grade`, so a metric can return its result directly. Options control how it counts: `mustPass` makes it a gate, `weight` sets its share of the objective, `threshold` sets the passing bar for scalar scores, and `samples` runs it k times. When a grading module is supplied, `goal` becomes optional on your tests. ### Pass and fail A `mustPass` grader is the assertion. If one fails, that test scores 0, the run reports `gatesPassed: false`, and `agency eval grade` exits 2, which makes it usable as a CI check. Every other grader is a measurement you track over time. A test whose agent run errored, or which the harness killed, scores 0 and fails every gate. It is counted, not skipped: one bad test out of fifty pulls the mean down by its share rather than zeroing the whole run. A test that produced no output still grades, with `output` set to `null`, because for file-writing agents the deliverable is the workdir. ## Judging output by hand Some questions have no automatic grader. Is this summary really about today? Do these citations point at pages that exist? Is this explanation worth reading? You find out by looking. `agency label` is the tool for those questions. You write a checklist of yes-or-no questions, then answer them one trace at a time. Your answers are appended to the run directory's `annotations.jsonl`, next to the graders' scores, so you can turn them into training data later or measure how well an LLM judge agrees with you. ```bash agency label runs/smoke --checklist news.json ``` Every trace in the directory is one thing to judge. The screen shows the trace's input and its output. When a trace recorded no output, it shows the agent's last message instead and says so. ### Notes For a one-off remark, skip the checklist and write a `notes.md` file in the run directory with any editor: ```bash echo "too slow, and it never checked the date" > runs/smoke/notes.md ``` Nothing else writes that file. When it has text, `agency runs list` shows `yes` in the `NOTES` column, the log viewer says `notes` on the trace's row, and the optimizer's reflection prompt includes it (see [Optimizing agents](optimize.md)). ### Writing a checklist A checklist is a JSON file listing your questions: ```json { "name": "news-quality", "questions": [ { "text": "Is every story actually from today?" }, { "text": "Does each claim have a source?" }, { "text": "Would I want to read this?", "weight": 2 } ] } ``` `weight` is optional and defaults to 1. A trace's score is the weighted share of questions you answered yes. Do not agonize over the list. You add questions while labeling, and discovering them halfway through is the normal way this goes, not a planning failure. ### One rule for questions **Every question must be answerable from the trace's input and output alone.** Those two things and your answer are the whole training example. An agent learning from your labels sees nothing else. A question that depends on anything outside them produces a label about information the agent never receives. "Is every story from today?" breaks that rule when the output never says which day it was written. The fix is to make the output say so, not to reword the question. Nothing checks this for you; the tool cannot know that "today" is unanchored. ### Labelling `agency label runs/smoke --checklist news.json` opens the screen. One trace fills the left, the checklist fills the right. | Key | What it does | |---|---| | `space` | answer the current question yes or no | | `↑` `↓` | move between questions | | `←` `→` | move between traces | | `enter` | sign off on this trace and go to the next | | `a` | add a question | | `d` | remove a question, or bring one back | | `m` | write a note | | `q` | quit | Signing off is deliberate. Your answers are not recorded until you press `enter`, so moving through a trace to read it changes nothing. Quitting halfway is safe. The session is saved as you go, and reopening it puts you back where you stopped. ### Where your answers go Each sign-off appends one `checklist` row to `annotations.jsonl`, naming the trace, the checklist version and your answers per question. Every version of a checklist is kept under `checklists/`. Editing a question does not rewrite history, so a label made against an older version still says what it meant. Two people labeling the same directory keep separate answers, keyed by `--annotator` (your user name by default). Only one labeling session can have a run directory open at a time. ## Optimizing marked declarations `agency eval optimize` (also `agency optimize`) rewrites the declarations you mark with the `optimize` modifier, grading candidates against your suite and keeping the best one. It has its own page: see **[Optimizing agents](optimize.md)** for marking targets, custom graders, validation sets, configuration, and run artifacts. ## The eval record Graders and judges do not read raw statelog events. They read an **eval record**: a small, normalized view of one trace, computed from the statelog whenever it is needed and never written to disk. It is what a grader's `record` argument holds and what `agency eval judge` compares. ## How to annotate a run Import `std::statelog` and call `evalValue` / `evalOutput` where values cross the user-facing boundary: ```ts import { evalValue, evalOutput } from "std::statelog" node main(prompt: string): string { evalValue(prompt) const reply = doWork(prompt) evalOutput(reply) return reply } ``` `evalOutputs` is **the entry node's return value** — the same thing `agency run` prints and a TypeScript caller receives. An agent that returns its answer needs no annotation at all. There is no guessing here. The last LLM completion is never used as a stand-in for a return value, because a plausible-looking chat reply is worse than an honest gap: it makes an agent that produced nothing look like it produced something. If the node returns nothing, or returns `null`, and `evalOutput()` was never called, the record has no output and says so in `warnings`. Returning `null` counts as no output rather than as the value `null`. An unmatched `match` or a missing key normalizes to `null` in Agency, so grading the string `"null"` would silently score a program that produced nothing. If your node genuinely has no return value, call `evalOutput(value)` to say what should be graded. `evalValues` is different: it is inferred from the last user-role message of the first top-level `promptCompletion`, and warns when it does so. That one is a genuine guess — annotate with `evalValue(prompt)` if you care about it. ## Record shape (overview) ```jsonc { "traceId": "...", "recordVersion": 2, "formatVersion": 1, "durationMs": 12345, "source": "/path/to/runs/smoke/statelog.jsonl", "evalValues": [{ "value": "what the user asked", "threadId": "0", "tMs": 120 }], "evalOutputs": [{ "value": "what the agent replied", "threadId": "0", "tMs": 420 }], "threads": [{ "threadId": "0", "label": "main", "parentThreadId": null, ... }], "events": [{ "kind": "llm", "threadId": "0", "model": "gpt-5", ... }, ...], "interrupts": [...], "errors": [...], "incomplete": [...], "metrics": { "llmCalls": 5, "toolEnds": 12, "toolCounts": { "grep": 8, ... }, ... }, "warnings": [] } ``` Every entry in `events` is one of three discriminated shapes: - `{ kind: "llm" }` — one per `promptCompletion`. Carries model, tools, duration, cost, token counts. - `{ kind: "tool_start" }` — one per `toolCallStart`. Carries `argsPreview`. - `{ kind: "tool_end" }` — one per `toolCall`. Carries `outputPreview` and duration. All three carry `threadId`, `spanId`, `parentSpanId`, and `tMs` (milliseconds from the start of the run). Every entry in `evalValues` and `evalOutputs` has this shape: ```jsonc { "value": unknown, "threadId": "0", "tMs": 420, "truncated": true } ``` - `value` is the JSON-serializable value passed to `evalValue` / `evalOutput`, or a heuristic fallback value when annotations are missing. - `threadId` identifies the active thread that recorded the value, or `null` when unavailable. - `tMs` is milliseconds from the trace start, derived from the statelog envelope timestamp. - `truncated` is present only when the serialized value exceeded `STATELOG_EVAL_MAX_VALUE_BYTES`. The default cap is 100KB; set that environment variable to override it. Oversized string values are kept as readable string prefixes; oversized non-string values are converted to JSON-preview strings. Consumers that need one response typically read `record.evalOutputs.at(-1)?.value`. A pairwise judge compares the last element of `evalOutputs`; without annotations that value is the entry node's return value, and the array is empty when the node returned nothing. ## Behavioral-flag recipe Common "did the agent do X?" questions are consumer recipes, not built-ins. Two examples: ```typescript import type { EvalRecord } from "agency-lang/lib/eval/types.js"; function consultedOracle(rec: EvalRecord): boolean { return rec.threads.some(t => t.label === "oracle"); } function grepBeforeWrite(rec: EvalRecord): boolean { const firstWrite = rec.events.findIndex(e => e.kind === "tool_end" && (e.tool === "write" || e.tool === "edit")); if (firstWrite === -1) return true; // no write happened return rec.events.slice(0, firstWrite).some(e => e.kind === "tool_end" && e.tool === "grep"); } ``` If a convention emerges (a set of rules every project wants), it can be promoted to a built-in `agency eval check --rules ` later. ## Downstream chain `evalValues` and `evalOutputs` are hoisted to the top level because graders and pairwise judges need the user-facing inputs and outputs without digging through raw `promptCompletion` events. `threads[*].label` is what behavioral queries grep on. Next: use [`agency eval judge`](./eval-judge.md) to compare two runs against a plain-English goal. To hand one trace to someone else, `agency logs extract run.jsonl --trace -o trace.jsonl` copies it out of a statelog. ## Legacy traces Statelog traces captured before the relevant runtime fields landed (thread labels/sessions, `toolCallStart`, interrupt summaries, per-event `threadId`) still produce a record. Fields that are not present in the source come through as `null` and a single warning is emitted in `record.warnings`. Don't rely on those fields when grading legacy traces; recapture if you can.