# Eval Procedures

## Step 1: Prerequisites Check

1. Call MCP `execute` with `{ command: "stores" }` to list indexed stores
2. If no stores indexed, show error and abort
3. Record available store names — pass to BK Full agent
4. Build `STORE_PATHS` mapping from store response: `- **<name>**: \`<path>\`` per store

## Step 2: Resolve Queries

### Predefined mode (`--predefined`)

1. Read: `$CLAUDE_PLUGIN_ROOT/evals/agent-quality/queries/predefined.yaml`
2. For each query, check if ANY `store_hint` values match available stores
3. Split into runnable and skipped lists
4. If `--predefined N`, select only query at index N

### Arbitrary mode

1. Use raw query as question
2. Set `expected_topics` and `anti_patterns` to empty
3. Set `id` to "arbitrary", `category` to "general", `difficulty` to "unknown"

## Step 3: Load Templates

From `$CLAUDE_PLUGIN_ROOT/evals/agent-quality/templates/`:
1. `without-bk-agent.md` — baseline agent
2. `bk-grep-agent.md` — grep-only agent
3. `with-bk-agent.md` — full BK agent
4. `judge.md` — grading rubric

## Step 4: Run Eval

Spawn ALL THREE agents in parallel per query using the Task tool:

**Without-BK**: Replace `{{QUESTION}}` in `without-bk-agent.md`
**BK Grep**: Replace `{{QUESTION}}` and `{{STORE_PATHS}}` in `bk-grep-agent.md`
**BK Full**: Replace `{{QUESTION}}` and `{{STORES}}` in `with-bk-agent.md`

### Capture Token Usage

From each Task response, extract `total_tokens` and `duration_ms` from `<usage>` block.

### Judge

Score all 4 criteria (1-5) per answer:
- **Factual Accuracy**: Are claims correct?
- **Specificity**: Does it cite specific files, functions, code?
- **Completeness**: Does it cover the full answer?
- **Source Grounding**: Are claims backed by evidence?

Check `expected_topics` coverage and `anti_patterns` violations.

## Notes

- Each query spawns 3 subagents. Process one query at a time.
- Without-BK agent may use WebSearch — intentional baseline.
- BK Grep agent may NOT use WebSearch — isolates vector search value.
- Scoring is subjective; value is in relative comparison (deltas).
- Token Efficiency reveals cost-effectiveness across modes.