# Evaluator — Preflight

> **Run before scoring a project.** If a tool is missing, **install it** — never fall back to grep or an LLM-only read. A silent fallback corrupts the scorecard twice: the absolute banding tables are calibrated against real tool output (`lizard` CCN, `semgrep` finding density), so an estimate lands in the wrong band; and `gitleaks` catches entropic tokens `grep` cannot while `pip-audit` / `trivy` catch CVEs no heuristic will, so a missed `SECRETS` or `VULN` hit becomes a dishonest pass.

The evaluator installs its own tools via `brew` / `pipx` / `npm -g` / `go install`. Installing is cheap (~1 minute total) and idempotent.

## Invocation

Run `bash .claude/skills/evaluator/preflight.sh` — idempotent, so re-running no-ops when every tool is present. `preflight.sh` is the executable contract and holds the canonical install commands; edit there, then bring the table below to match.

## Required tools

| Tool | Purpose | Recipes that use it | Install (macOS, primary) |
| --- | --- | --- | --- |
| `gh` | GitHub API (CR scrape, GHA runs, PR enumeration) | 1.1, 1.2, 2.4 | `brew install gh` |
| `gitleaks` | committed-secret scan — fires `SECRETS` gate | 1.4 | `brew install gitleaks` |
| `lizard` | polyglot complexity / god-function detection (per-language CCN: Py 10 / JS-TS 10 / Java 12 / Go 15) | 1.1, 1.4 | `pipx install lizard` — **never** `brew install lizard`, which is an unrelated LZ5 compressor of the same name |
| `scc` | LoC + COCOMO estimate | 1.4, 2.4 | `brew install scc` |
| `semgrep` | polyglot static analysis (quality + resilience + multi-protocol auth) | 1.1, 1.2, 2.4 | `brew install semgrep` |
| `hadolint` | Dockerfile lint | 1.3 | `brew install hadolint` |
| `ruff` | Python lint + formatter | 1.1 (Python repos), 2.4 (Tier-A) | `brew install ruff` |
| `shellcheck` | shell-script lint | 2.2 (CLI path) | `brew install shellcheck` |
| `tree` | folder-tree read for LLM context | 1.4, 2.1, 2.2, 2.4 | `brew install tree` |
| `golangci-lint` | Go aggregate linter | 1.1 (Go repos) | `brew install golangci-lint` |
| `pnpm` | installs and audits pnpm-locked JS repos | 1.4 | `brew install pnpm` |
| `yarn` | installs and audits yarn-locked JS repos | 1.4 | `brew install yarn` |
| `trivy` | filesystem dep-vuln scan (OS + language packages) — feeds `VULN` gate | 1.4 | `brew install trivy` |
| `biome` | TS/JS lint (preferred over eslint for speed) | 1.1 (TS/JS repos) | `npm i -g @biomejs/biome` |
| `knip` | TS/JS dead-code + unused-deps | 1.4 | `npm i -g knip` |
| `pipx` | isolated install for the Python CLIs below | — | `brew install pipx && pipx ensurepath` |
| `deptry` | Python unused-deps | 1.4 | `pipx install deptry` |
| `pip-audit` | Python dep-vuln scan — feeds `VULN` gate | 1.4 (Python repos) | `pipx install pip-audit` |
| `govulncheck` | Go dep-vuln scan — feeds `VULN` gate | 1.4 (Go repos) | `go install golang.org/x/vuln/cmd/govulncheck@latest` — only when `go` is on PATH; otherwise skipped and Go vuln scans record `_NOT_RUN: true` with reason `"govulncheck unavailable (go not installed)"` |

Host prerequisites — the script never installs these and exits 2 if any is missing:

- `brew`
- `jq`
- `node` + `npm` (≥ 18)
- `python3` (≥ 3.10) — `pipx` layers on top

Each install runs under a 180 s cap from `timeout` or `gtimeout`; on a host with neither, the script warns and installs uncapped (`brew install coreutils` restores the cap). Preflight ends by checking that the `lizard` on PATH is the cyclomatic-complexity analyzer, and fails if brew's compressor shadows it.

## Required behaviour for scoring agents

1. **Run preflight first** — before extracting any signal, on every invocation; the first run installs, later runs no-op.
2. **If preflight exits non-zero, stop.** Never score with missing tools; return a failure message to the caller naming the tools that failed to install.
3. **No silent fallback.** A signal that needs a tool either runs it or emits `{signal}_NOT_RUN: true` with a `reason` naming the cause in that sub-criterion's `data` block in `scorecard.yaml` — a loud failure, never a graceful substitute.

## When a tool does not apply

A language-specific tool (`ruff` on Python, `golangci-lint` on Go, `knip` on TS/JS) does not apply to a repo without that language:

- Emit the signal as `{signal}: null, reason: "language_not_present"` — **not** as `_NOT_RUN`, which means the tool was missing from the host (a failure), where `null` means not applicable here (expected).
- Preflight still installs every tool; language filtering happens in the recipes, not in preflight.