# Edge eval An independent, private Node 20+ package. No install or MCP connector build is required. ```sh cd packages/edge-eval node src/cli.js eval --skill demo/skills/line-items-careful \ --vs demo/skills/line-items-basic --cases demo/cases.jsonl \ --agent claude --model sonnet --runs 3 --concurrency 4 npm test ``` The package exposes the `edge` executable when linked or installed locally. Direct execution above avoids global changes. `npm run demo` produces the Comstruct comparison under `demo/results`. Every invocation writes timestamped JSON and a self-contained HTML report, never overwriting an earlier run. The default destination is `eval-results//.json` and `.html`. ## Privacy and execution The package makes no HTTP calls and sends nothing to Edge, except the `benchmark router` Edge arm, which proxies the connector's own Edge requests. The selected local Claude or Codex CLI sends the prompt to its provider under your existing authentication. This is local orchestration, not offline model inference. Claude uses safe mode, disables tools and MCP, and avoids session persistence. Codex ignores user configuration, uses an ephemeral read-only sandbox, and disables shell and web search. Each call gets an empty temporary working directory under the current directory, removed afterward. API key environment overrides are removed to prefer subscription authentication. Verify your CLI login before running. Provider cost estimates are recorded as estimates, not charges; this package cannot verify subscription billing. Skills must be trusted local instruction files. A directory loads its SKILL.md; a file path loads that file. Supporting scripts, linked resources, and tool workflows are not executed. Expected answers are never included in extraction prompts. Only the optional judging step receives the reference. Model rationales are short stated explanations, not private chain-of-thought or verified causal diagnoses. Reports contain expected answers and model outputs, so store them with the same care as source documents. ## Cases and graders One JSON object per line: ```json {"id":"row-1","input":"2 bags at EUR 10 each","expected":{"total":20},"grader":"json-fields","fields":["total"]} ``` `input` accepts text or a UTF-8 file path relative to the JSONL file. An `@` prefix requires file loading; missing files fail early. No PDF/image parsing is included. IDs must be unique nonempty strings. `synthetic: true` marks synthetic cases in the report only when every case has that marker. - `exact`: strict JSON equality for objects or exact text equality, with no whitespace normalization or numeric coercion. - `json-fields`: exact equality at selected dot paths, including array indexes such as `line_items.0.quantity`. Selected arrays compare in order and in full. Missing fields fail. The reference must contain every selected field. - `llm-judge`: asks the same local CLI to compare actual and expected using an optional case `rubric`, requires a boolean verdict and short explanation, and records judge usage separately. This is model-based judgment and can vary. `--runs` repeats every case for both skills. The report includes each run's pass rate, mean and minimum. Field accuracy uses attempts; paired comparisons average repetitions per unique case before voting so repetitions do not inflate sample size. The two-sided exact sign test ignores ties, is exploratory, and does not correct for multiple fields. Fewer than 30 cases or 10 discordant pairs is explicitly flagged as a small sample. `--timeout` sets the per-agent deadline in milliseconds (default 120000). `--concurrency` defaults to 1, maximum 8. Nonzero agent exits, timeouts, invalid output, and judge failures become visible failed attempts, never silently dropped. Exit code 0 means evaluation completed, even if some cases failed; 1 means invalid input/invocation; 2 means one or more execution errors. Stderr is not persisted to avoid copying arbitrary agent diagnostics into reports. ## SPEC-v1 skill benchmarks `node src/cli.js benchmark screen|heldout|check|judge|aggregate|router` runs the blind pointwise protocol for open-ended and visual work. It uses a 0-3 rubric on four dimensions, renderer v3, the instrument check and dQ as the primary number. See `docs/BENCHMARK.md` at the repo root. Modules: - `pointwise.js`, `pairwise.js`, `judge-input.js`: prompts and judge input. - `render.js`: renderer v3, with Playwright loaded only when needed. - `claude-judge.js`: the subscription `claude -p` judge. - `aggregate.js`, `instrument.js`, `stage.js`, `runner.js`, `router.js`, `suite.js`. `test/pointwise.test.js` holds golden tests against the prototype's own `build()` output. `scripts/regress-prototype.mjs` re-judges the prototype's artifacts and compares dQ. ## Import accountant corrections ```sh node src/cli.js eval import --csv demo/corrections.csv \ --map input=doc_text,expected=corrected_json,id=case_id \ --out imported-cases.jsonl --fields document_type,line_items,vat_rate,currency ``` Quoted commas, doubled quotes, CRLF, multiline cells and UTF-8 BOMs are supported. Default grader is `json-fields`, with all top-level expected keys when `--fields` is omitted. `--grader exact` preserves expected text literally. JSON is required for other imported graders. Output is created exclusively and will not overwrite an existing file. Imported paths resolve relative to the resulting JSONL file. ## Reproduce the demo ```sh node demo/generate.mjs npm run demo node demo/public-acceptance.mjs ``` The 30 documents contain 27 invoices, two delivery notes and one rental invoice, all explicitly synthetic. Ground truth is authored independently of model outputs. The basic skill deliberately uses fixed invoice defaults, ignores discounts and mistakes rental units. The careful skill preserves stated totals, document types and absent prices. All failure records are retained, with the first three expanded for presentation. Do not imply that synthetic accuracy measures production performance. The additional public-skill acceptance uses frozen copies of this public repository's `skills/edge/SKILL.md` and `skills/edge-kpis/SKILL.md` from commit `59a97803340e7b94194faef43cc8ff905e53341d`, with 20 synthetic policy questions each. Source: `getedgehq/edge-mcp`. It measures application of their instructions without executing setup, reading telemetry or contacting MCP. These are separate acceptance runs, not invoice extraction competitors. Copies preserve the original text; relative references are not loaded. The report reuses landing-page header/footer markup from the supplied local getedge-site checkout, uses monochrome styling for this lane and embeds its existing Inter font. External links open only when clicked; rendering the report requires no network resources. Use `python3 -m http.server 8766 --bind 127.0.0.1` from the lane root, then open a report under `/packages/edge-eval/demo/results/`.