# The `benchmark` tool and pipeline CLI

Benchmarks are one Edge MCP tool, not a separate harness. The
connector on the user's machine and Edge's own nightly pipeline run the exact
same engine (`packages/edge-eval`), the same graders and the same leaderboard
math.

## The `benchmark` tool

One tool, `benchmark`, takes one or several skills plus tasks. With one skill it
compares skill vs no skill; with several it ranks them all against a shared
no-skill baseline on one leaderboard.

- `skills`: 1 to 6 entries, each `none`, an Edge catalog reference
  (`owner/repo@skill` or `edge:owner/repo@skill`), or a local path (skill
  directory or SKILL.md). A no-skill baseline condition is added automatically
  if not already present.
- Tasks: `tasks` (inline), `task_file` (JSONL), or `category` — Edge then
  suggests one task from the benchmark specs that ship in the package
  (`evals/specs/`) and shows it for approval. If the user wants three tasks,
  pass `task_count: 3` with `category`. Inline tasks and task files keep their
  explicitly supplied task counts.
- `runs` (default 1 for a single skill, 2 when comparing several), `agent`
  (claude or codex, default: whichever is installed), `model`, `timeout_ms`
  (default 600000 = 10 min).

With one skill the result reads as a plain skill-vs-no-skill comparison; with
several it's a ranked leaderboard: mean, 95% bootstrap CI, paired lift vs
no-skill with its own CI, runs, tokens, mean time.

**The tool never runs immediately.** The first call returns the plan —
executions = conditions × tasks × runs, the per-task time limit and the
worst-case agent-hours — and a confirmation token for exactly that plan. The
agent must show it to the user and only call again, with the same arguments and
`confirmation_token`, after an explicit yes. `confirm: true` alone never runs
anything; a token is single use, lasts 30 minutes, and is refused if the
arguments changed since the plan.

## Execution

The connector spawns the user's own headless agent (`claude -p` or
`codex exec`) with the user's subscription login. API-key environment overrides
(`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, …) are removed so a run never silently
becomes a billable API call. Every execution gets an isolated temporary
workdir, removed afterwards; in agent mode the task workspace is copied into
it, so the agent's tools can never modify the source materials. Every condition
gets the same budget (same timeout, model and runs). Grading is deterministic
(exact, JSON fields, or a trusted local grader script) or a blind LLM judge
that sees the output and reference but never which condition produced the
output.

## Results and sharing

The full report (JSON + HTML, with outputs and transcripts) is written to
`~/.config/edge/evals/<id>/` and never overwritten. The tool returns a short
summary.

Sharing is on by default and minimal by construction: category, skill ids,
scores, run counts, agent and model, plus the install id / key id the connector
already sends. The backend route `POST /api/mcp/eval_result` validates exactly
this shape (`src/eval-result.ts`) — there is no field for prompts, outputs or
local skill contents, so they cannot leave the machine by accident. Edge's own
pipeline runs are flagged `internal`. Sharing failures never affect the local
report; `EDGE_EVAL_SHARE=off` disables sharing.

## The pipeline CLI

```sh
npx @getedge/mcp benchmark SPEC.json [--yes] [--out DIR] [--leaderboard FILE] \
  [--agent claude] [--model MODEL] [--runs N] [--concurrency N] [--timeout MS]
```

One spec skill compares against the baseline; several rank on a leaderboard,
same as the MCP tool. A spec file (see `evals/specs/security-review/spec.json`
for a working example):

```json
{
  "name": "security-review",
  "category": "security-review",
  "tasks": "cases.jsonl",
  "skills": ["edge:getedgehq/skills@security-audit-checklist", "skills/local-dir"],
  "includeNoSkill": true,
  "runs": 2, "agent": "claude", "timeout": 600000, "concurrency": 4
}
```

Skill paths resolve relative to the spec file. Without `--yes` the CLI shows
the cost plan and asks on the terminal (non-interactive shells must pass
`--yes`). `--internal` marks the shared result as an Edge run; `--no-share`
skips sharing. `--leaderboard leaderboard.json` merges the result into the
file the nightly job publishes for the Benchmarks and SEO pages. Exit codes: 0
completed, 1 invalid invocation, 2 completed with execution errors.

## SPEC-v1 skill benchmarks (blind pointwise judge)

For open-ended work (web pages, visual artifacts, documents) there is no exact
answer to grade against. These benchmarks use the protocol the Edge posts were
scored with. It is ported unchanged from the `edge-fast-evals-visual`
prototype: the prompt, rubric wording and critical-failure list are the same,
and golden tests check them byte for byte.

```sh
npx @getedge/mcp benchmark screen  --suite evals/suites/edge-posts --skill PATH --out DIR
npx @getedge/mcp benchmark heldout --suite FRESH_SUITE --skill PATH --screen DIR --out DIR2
npx @getedge/mcp benchmark check   --suite DIR --attempt TASK_ID=ATTEMPT_DIR [--attempt ...] --out DIR
npx @getedge/mcp benchmark judge   RUN_DIR... --suite DIR [--pairwise] [--force]
npx @getedge/mcp benchmark aggregate RUN_DIR [--instrument-check FILE]
npx @getedge/mcp benchmark router  --suite DIR --out DIR --exclude a,b
```

- **Grader.** A blind pointwise judge. Each artifact is judged alone, in a
  fresh context, and never sees the arm label or another arm's output. It
  scores four dimensions from 0 to 3: brief fidelity, hierarchy/readability,
  usability and craft. Q is their mean. A predeclared critical failure (blank
  or not loading, required content missing, …) sets Q = 0.
- **Judge input.** For visual and web tasks the judge sees what a user would
  see: renderer v3 (Playwright and Chrome) produces reveal, keyboard, stacked,
  scroll-frame, page-tile, SVG, PDF/office, video contact-sheet or image
  renders. Every judgment records `total_cost_usd`.
- **Aggregation.** Runs are averaged within a task, and tasks are weighted
  equally. The primary number is **dQ = Q_skill − Q_base**. The pairwise
  preference, judged in both orders, is secondary, and orders that disagree
  count as a conflict, never a tie. It is computed only with `--pairwise`.
- **Stages.**
  - `screen`: 6 shared tasks × 1 run per arm.
  - `heldout`: 6 fresh tasks × 2 runs per arm.
  - Arms are interleaved in a seeded random order recorded in `stage.json`.
  - A held-out stage refuses a skill whose hash differs from the screen's, a
    changed protocol, prompt or renderer, and any task already screened (by
    id or content hash).
  - A proven infrastructure failure gets one logged retry; agent failures are
    outcomes.
- **Agents.** Each attempt runs `claude -p` in a fresh copy of the task input,
  with a fresh `CLAUDE_CONFIG_DIR`. In the skill arm the skill is installed
  there for the agent to discover. It is not pasted into the prompt, and
  whether it was read is recorded as `skill_read`.
- **Instrument check.** `benchmark check` must pass before a score is
  published. It re-judges original artifacts and requires:
  - byte-identical copies within 0.25 Q of the original (repeatability);
  - a blank page scoring Q = 0;
  - the page with its styles and scripts stripped scoring lower than the
    original.

  `aggregate --instrument-check FILE` accepts only a passing check made with
  the same judge model, prompt version and render version. Without one, the
  summary reads `NOT PUBLISHABLE`.
- **Label.** Published results carry "Scored blind by an AI judge (Claude Opus
  5), 0-3 rubric". The name comes from the judge model id.
- **Billing.** Agents and the judge run through the logged-in subscription:
  `claude -p` with the subscription OAuth token, and API keys removed.

**Task suites** use the `tasks/edge-posts` format: `<suite>/<task-id>/task.json`
plus `input/`. `evals/suites/edge-posts` is the six-task suite behind the Edge
posts. `render.kind` in task.json selects the renderer, and `check.py` is an
optional deterministic pre-check.

**Router benchmark.** This mode asks whether the way an agent finds skills
changes the result. It runs three arms on the same tasks:
- `base`: no way to find a skill.
- `find-skills`: vercel-labs `find-skills` plus keyword rewriting, vendored in
  `evals/router/`.
- `edge`: the Edge connector.

Skills the suite was written around (`--exclude`) are removed from retrieval in
both search arms. A local proxy filters Edge's `find_skill` and refuses
`use_skill` for them. An `npx skills` shim does the same for find-skills.
Every search, load and refusal is logged per attempt in `retrieval.jsonl`.
Edge traffic is tagged `edge-cli/router-benchmark`, so it counts as internal.
The result is Q per arm and the difference from `base`.

## Task files and graders

One JSON object per line; see `packages/edge-eval/README.md` for the full
contract. Graders: `exact`, `json-fields`, `llm-judge`, and `script` — a trusted
local module exporting `grade(expected, output)` returning `{pass, fields?}`,
used by the security-review benchmark for its exact planted-issue checks.
`mode: "agent"` gives the agent tools and a copied `workspace` directory;
the default `document` mode is the no-tools extraction harness.

## Tests

The connector suite (`npm test`) covers the route, validation, resolution and
leaderboard. The engine suite lives in the package:
`cd packages/edge-eval && npm test`.