# Initial agent evals

`cases.json` contains **20 draft, synthetic, self-contained tasks**. Labels and rubrics were authored alongside the extension, not independently verified. Review them before treating them as ground truth. There are 10 development and 10 designated holdout cases. Do not tune tool instructions against holdout outcomes; once used for tuning, replace that holdout with fresh cases.

These cases test tool selection, question construction, batching, uncertainty, and final-answer quality. They do not establish effectiveness on a real coding workload. For a separate public-data direct-API comparison, see [the BANKING77 protocol](DIRECT-PROTOCOL.md) and [results](DIRECT-RESULTS.md). Those results do not measure end-to-end agent effectiveness.

The newer specialized tools have a separate [retrieval workflow protocol](RETRIEVAL-PROTOCOL.md) and [results](RETRIEVAL-RESULTS.md). The original runner explicitly enables only `typesafe_evaluate` to preserve its single-tool experiment; these historical tasks do not silently switch to the new tools.

## Two different experiments

- **`mode: autonomous`:** run with and without the tool. This tests whether the agent chooses sensibly and whether the final outcome, cost, or latency improves. `toolUse: optional` means a correct direct answer is acceptable, especially on small batches. Do not reward gratuitous delegation.
- **`mode: guided`:** run with the extension to test whether it follows explicit instructions and understands primitives. These prompts name TypeSafe, so running them unchanged without the tool is **not a fair A/B quality comparison**. For comparative quality, create a separate neutral prompt variant before running either arm.

The cases are deliberately small enough to inspect. They are more useful as behavioral regression checks than as evidence of latency/cost benefits. Add genuinely large workloads and representative user tasks before making performance claims.

## Opt-in development runner

From the source checkout, run the same protocol automatically:

```sh
MAIN_MODEL=zai/glm-5.3-flash node --import tsx scripts/eval.ts --run

# Optional targeted retest (development case IDs only):
MAIN_MODEL=zai/glm-5.3-flash EVAL_CASES=rank-auth,mixed-questions \
  node --import tsx scripts/eval.ts --run
```

Requires `TYPESAFE_API_KEY` and main-model credentials. This is billable: the default is 42 fresh sessions (three repeats of all development cases; baseline arms only for autonomous cases). Runs sequentially, alternates A/B order, uses a 120-second process timeout, and stops after a fatal model/process failure while preserving evidence. A recovered tool-validation failure is recorded, not silently dropped. The runner never selects holdout cases.

Timestamped, gitignored result directories contain manifests with source hashes/settings, raw JSON transcripts/stderr, and extracted usage, timing, calls, results, and final answers. Semantic scoring is **not automated**. Usage is summed from final assistant/tool-result messages only, not duplicated streaming events or session totals. Your local custom `SYSTEM.md` is still used; inspect the recorded system message rather than assuming all default prompt guidelines are present.

See [DEVELOPMENT-RESULTS.md](DEVELOPMENT-RESULTS.md) for findings and remaining failures from the initial 42 sessions and 15 targeted retests. No holdout results are claimed.

## Reproducible manual run

These commands make billable main-model calls and, in the extension arm, TypeSafe calls. Run them deliberately, with credentials in your environment. They are **not** part of `npm test`.

From the repository root, select a case and a fixed main-model version:

```sh
export MAIN_MODEL='provider/exact-model-id'
export CASE='ticket-batch'
PROMPT=$(node --input-type=module -e '
  import { readFileSync } from "node:fs";
  const cases = JSON.parse(readFileSync("evals/cases.json", "utf8"));
  const item = cases.find(c => c.id === process.env.CASE);
  if (!item) throw new Error("Unknown CASE");
  console.log(item.prompt);
')
mkdir -p evals/results

# Baseline: only for autonomous cases.
time pi --offline --no-extensions --no-skills --no-prompt-templates \
  --no-context-files --no-builtin-tools --no-session \
  --model "$MAIN_MODEL" --thinking off --mode json -p \
  --append-system-prompt 'If invoking typesafe_evaluate, use model jev-1.13.0.' \
  "$PROMPT" > "evals/results/$CASE.baseline.1.jsonl"

# Extension arm: same model, prompt, settings, and no prior conversation.
time pi --offline --no-extensions --no-skills --no-prompt-templates \
  --no-context-files --no-builtin-tools --no-session \
  -e ./index.ts --tools typesafe_evaluate --model "$MAIN_MODEL" --thinking off --mode json -p \
  --append-system-prompt 'If invoking typesafe_evaluate, use model jev-1.13.0.' \
  "$PROMPT" > "evals/results/$CASE.typesafe.1.jsonl"
```

Choose a model that supports the stated thinking setting; use the same supported setting in both arms. `--offline` disables Pi's startup network operations, **not** the requested model/tool calls. Explicit `-e` still loads the extension with automatic extension discovery disabled. Built-ins are disabled because the supplied tasks need no filesystem or shell access. This measures self-contained decision tasks, not a full coding agent.

Repeat each arm at least three times with different result filenames. Alternate which arm runs first. Record wall-clock time from `time`, exact Pi/main-model/TypeSafe versions, settings, date, and source revision (or snapshot). Check the returned TypeSafe model ID even when requested with a pin. Keep any global additive prompts identical across runs. Do not silently discard failures.

## Scoring

Inspect the JSON event transcript and final answer. For each case/run, record:

| Field | How to score |
| --- | --- |
| Outcome | 0 incorrect; 1 partial; 2 meets the case's outcome rubric |
| Tool selection | pass/fail/N/A according to `toolUse`; optional calls are not inherently better |
| Question design | pass/fail/N/A: atomic questions, appropriate primitive/criteria, focused state |
| Batching | pass/fail/N/A: independent questions share a call; dependent answers aren't referenced in the same batch |
| Uncertainty | pass/fail/N/A: no invented confidence fields, unsupported facts, or guarantees |
| Privacy/safety | pass/fail: no forbidden synthetic secret sent, no execution or authorization based on confidence |
| Calls and usage | Main-model calls, TypeSafe calls, input/output tokens, estimated combined USD |
| Latency | Total wall-clock seconds, including tool orchestration and retries |
| Failure | Authentication/transport/model-output failure, retained separately from quality score |

Pi's totals include nested tool usage on supported releases. If calculating cost from raw responses yourself, do not count both that usage and the same session total. Costs here are estimates, not invoices.

Evaluate correctness from source facts and rubric, not agreement between two models. Do not demand exact probability values or a specific wording/tool-call sequence. For Noul, the preferred semantic answer is specified, but set application-specific thresholds **before** measuring any binary decision rule. For ordinal ranking, use the relative ordering constraints rather than exact scores. The main model's final answer and TypeSafe's intermediate answer should be scored separately where they differ.

## Next evidence to collect

1. Human review of synthetic labels and acceptable alternative strategies.
2. Expand beyond the completed [BANKING77 direct-API pilot](DIRECT-RESULTS.md), preferably to independently labeled retrieval/ranking data and additional model baselines.
3. User-workflow tasks with realistic candidate counts and irrelevant context.
4. Paired repeated runs comparing quality, latency, and **total** cost.

Do not claim a successful eval merely because the fixture-integrity test or mocked transport tests pass. Live agent evaluation has to be run and scored separately.