--- title: Evals description: blume eval gives your docs a test suite — an AI agent answers your users' questions using only the documentation, a judge grades the answers, and CI fails when the docs can't answer. --- `blume audit` tells you whether crawlers can find your docs. `blume eval` tells you whether anyone can actually _use_ them: an AI agent reads your documentation the way a stranger would and tries to answer real user questions from it. When the docs don't state the answer, the run fails and names the page that should. ```bash blume eval ``` ``` blume eval 3 question(s) · Claude Code ✔ install-node-version pass 1.00 14.2s $0.14 ✖ deploy-vercel fail 0.40 38.9s $0.31 missing: the adapter is auto-detected ⊘ search-providers skipped fix: content/docs/deployment.mdx Docs could not answer: "How do I deploy to Vercel?" — missing: the adapter is auto-detected 2 passed · 1 failed · 1 skipped · 1m 42s · $0.45 ``` ## How it works Each question runs through two agent sessions, using an agent CLI you already have installed — [Claude Code](https://claude.com/claude-code) by default, or [Codex](https://developers.openai.com/codex/cli) with `--agent codex`. Blume holds no API keys and calls no model itself. 1. **The reader** answers the question using _only_ your documentation. It runs in an empty directory with its file, shell, and web tools disabled, connected to a private [MCP server](/docs/configuration/ai) that serves your docs — the same `search_docs`/`get_page` tools a real agent uses against your deployed site. It cannot read your repo, so it experiences the docs exactly like a fresh user: what isn't written doesn't exist. 2. **The judge** grades the answer against the facts you listed, with no tools at all. Paraphrase passes; a missing or contradicted fact fails — and so does "the documentation doesn't say." The MCP snapshot is built from your content sources directly, so there is no need to run `blume build` first, and nothing is deployed or uploaded anywhere. An answer the docs _can't_ support fails even when the agent's prior knowledge happens to be right — that's the point. Your docs are the only source that ships. ## Writing evals Questions live in `evals.yaml` at the project root. To have an agent draft a starter file from your existing docs: ```bash blume eval init ``` Or write it by hand: ```yaml questions: - id: install-node-version question: What is the minimum Node.js version required? expected: - Node 22.12 or newer routes: /docs/quickstart - id: deploy-vercel question: How do I deploy to Vercel? expected: - run blume build - the output directory is dist routes: - /docs/deployment - id: search-providers question: Which search providers are supported? expected: - pagefind is the default severity: warning # a miss warns instead of failing CI skip: true # temporarily excluded, reported as skipped ``` - `expected` lists the facts a correct answer must state, in substance — the judge accepts paraphrase and rejects contradiction. - `routes` names the page(s) that should answer the question. A failure is then anchored to that page's source file in the report; a hint that no longer matches a page is warned about rather than silently dropped. - `severity: warning` keeps a question in the report without failing CI; `skip: true` sits a question out entirely. Write questions your users actually ask — the ones from support threads, GitHub issues, and onboarding calls. The best evals encode a promise your docs make ("zero-config deploys") as a question that breaks when a PR breaks the promise. ## Failing CI The exit code is the contract: any failed question exits non-zero. `--threshold` relaxes the gate to a passing fraction when you're digging out of a backlog: ```bash blume eval # every question must pass blume eval --threshold 0.8 # at least 80% must pass blume eval --json # machine-readable report on stdout ``` The JSON report carries the same `diagnostics` + `summary` shape as `blume validate --json` and `blume audit --json`, with the per-question results (answer, score, missing facts, cost) alongside. Because each question is two model sessions, an eval run costs real money and minutes — the per-question spend is printed as it runs. A sensible CI setup runs `blume eval` on docs changes rather than every push. ## Fixing the findings Each failure names the missing facts and the page that should state them. To hand the whole report to the agent instead: ```bash blume eval --fix ``` This writes the full JSON report to a file and opens the agent interactively with a prompt that walks it through each failed question: read the named page, add the missing facts in the page's voice, and rerun `blume eval` until everything passes. The session is interactive by design — you review the edits through the agent's own permission flow — and the agent is told never to delete questions or weaken expected facts to get to green. ## Flags - `--agent claude|codex` — which agent CLI runs the reader and judge. Defaults to `claude`. - `--file ` — the evals file. Defaults to `evals.yaml`. - `--threshold <0..1>` — minimum passing fraction before the run exits non-zero. Defaults to `1`. - `--timeout ` — reader time limit per question. Defaults to `180`. - `--json` — emit the report as JSON on stdout. - `--fix` — after a failing run, hand the report to the agent to fix the docs interactively. - `--verbose` — include the reader's full answer under each failure.