# Code-Eval Batcher

Evaluating and ranking **multiple code projects** — candidate take-homes, submissions, a folder of repos — against the production-readiness + product-quality rubric. The main agent performs this directly rather than delegating it: the rollup needs every scorecard in one context, so the session that ranks must be the one that fans out. A single project does **not** need this rule — invoke the `evaluator` skill directly. Scoring how a person *steered a session* is `interview-eval`, not this rule.

## Division of labor

Per-project scoring — the rubric, gates, absolute banding, the 8 sub-criteria, every recipe — belongs to the **`evaluator` skill** (`.claude/skills/evaluator/`); never re-derive it here. This rule owns only what the skill deliberately doesn't: **enumeration, parallel dispatch, weighting, and the cross-project rollup.** **Never re-band a project against the others** — cohort-relative banding is the anti-pattern this design removed; the only cross-project levers live in the weights block.

## Procedure

| # | Step |
| --- | --- |
| 0 | **Preflight once:** `bash .claude/skills/evaluator/preflight.sh`. Halt on non-zero and report the missing tools — never score with degraded signals. |
| 1 | **Enumerate** the projects → local paths, one slug each (its dir name). Choose an `out_dir` (default `./code-eval-out`). |
| 2 | **Dispatch in parallel — batches of 4.** Per project: an `Agent` call with `subagent_type="general-purpose"` instructing it to invoke the `evaluator` skill on that one path, write the workspace to `<out_dir>/<slug>/`, validate it (`python3 .claude/skills/evaluator/validate_scorecard.py <out_dir>/<slug>/scorecard.yaml`), and return the workspace path + the scorecard's `normalized`. **Issue all four calls in one block** — that is what makes them concurrent; start the next batch when the block returns. |
| 3 | **Collect** every `<out_dir>/*/scorecard.yaml`. Re-dispatch a validation failure once; on the second failure mark the project `VALIDATION_FAILED` and continue. |
| 4 | **Weight + normalize** every project (math below) using the weights block. |
| 5 | **Rollup** → `<out_dir>/results/{scorecard.csv, ranking.md, summary.md}` (contents below), sorted by rank score desc. Leave everything uncommitted. |

## Weighting math

For each project `p`, read each sub-criterion `s`'s raw 0–5 `score` (the evaluator's absolute band) from its `scorecard.yaml`:

1. **Weighted (0–100)** over sub-criteria with a non-null `score` and `weight > 0`:
   `weighted_p = 100 × Σ_s (score_{p,s} / 5 × weight_s) / Σ_s weight_s`
2. **Gate penalty:** `adjusted_p = max(0, weighted_p − Σ_{g ∈ gates_triggered_p} gate_penalties[g])`.
3. **Normalize** across the set `{adjusted_p}` per `normalization`:
   - `percent_of_max` (default) → `adjusted_p` unchanged — absolute, comparable across runs.
   - `min_max` → `100 × (adjusted_p − min) / (max − min)`; `max == min` ⇒ all 100. Spreads a tight set; not comparable across runs.
   - `z_score` → `clamp(0, 100, 50 + 10 × (adjusted_p − mean) / sd)`; `sd == 0` or n < 2 ⇒ fall back to `adjusted_p`.

`VALIDATION_FAILED` projects are excluded from min/max/mean/sd and listed separately. A null sub-criterion drops out of that project's weighted average — never impute a 0. Compute with a short throwaway script over the scorecards; round reported scores to 2 decimals.

| Output | Contents |
| --- | --- |
| `scorecard.csv` | `rank, project, <8 scores>, gates, short_circuited, evaluator_total, weighted, penalty, rank_score` |
| `ranking.md` | ranked table headed with the normalization method — the sort key |
| `summary.md` | n scored, the exact weights / penalties / normalization used, top-5 / bottom-5, gate frequency, any `VALIDATION_FAILED` |

## Weights — edit this block to change the ranking

The math above consumes these values verbatim. Edits here are overwritten on a harness re-install — to persist a custom weighting, drop a `code-eval-batcher.weights.json` with the same shape in the `out_dir` and read that instead.

```json
{
  "weights": {
    "code_quality": 12.5,
    "testing": 12.5,
    "deployment_readiness": 12.5,
    "maintainability": 12.5,
    "problem_substantiveness": 12.5,
    "ux_mechanics": 12.5,
    "value_articulation": 12.5,
    "mechanical_ambition": 12.5
  },
  "gate_penalties": {
    "DEMO": 0, "BROKEN": 0, "UNTESTED": 0, "TESTS_TRIVIAL": 0,
    "SECRETS": 0, "VULN": 0, "AUTH": 0, "OBSERVABILITY": 0
  },
  "normalization": "percent_of_max"
}
```

- **`weights`** — relative importance; any non-negative numbers, normalized to sum 100 over the present, weighted sub-criteria. `0` drops a sub-criterion. The all-12.5 default reproduces the evaluator's own equal-weight total.
- **`gate_penalties`** — flat 0–100-scale points docked per triggered gate, on top of the band caps the evaluator already applied. Default 0 = rely on the evaluator's caps alone. E.g. `SECRETS: 15` docks 15 from any project with a committed secret.

**No silent fallback.** A malformed weights block, a failed preflight, or a scorecard that won't validate is reported loudly — never worked around.