# Maintainability — Extraction Plan - **Sub-criterion:** 1.4 (AI-only) - **Weight:** 12.5 pts (4 Code AI × 12.5 = 50 pts half) - **Contenders:** gitleaks + lizard (**per-language CCN**: Py 10 / JS-TS 10 / Java 12 / Go 15) + knip/deptry + **dep-vuln scan** (`pip-audit` / `npm audit --json` / `govulncheck` / Trivy) + LLM folder/README read (layered) - **Source quote:** _"Folder structure and clarity that makes it easy for another engineer to pick up and extend."_ Bands are **absolute** — per-signal thresholds in the _Banding_ section. Gates this recipe owns: **`SECRETS`** (trigger: `gitleaks_findings ≥ 1`, caps `maintainability ≤ 2`) and **`VULN`** (fires when dep-vuln scan reports ≥ 1 CVE at CRITICAL OR ≥ 3 at HIGH across any dep, direct or transitive; caps `maintainability ≤ 2`). Dep-vuln tool selection by ecosystem: Python → `pip-audit`, JS/TS → `npm audit --json` (reads package-lock.json), Go → `govulncheck`, built container → `trivy image`. Skipped if no recognized package manifest. Data fields: `dep_vuln_critical` (int), `dep_vuln_high` (int), `dep_vuln_tool` (string), `dep_vuln_by_ecosystem` (map). Raw dumps: `raw/maintainability-dep-vuln-pip-audit.json` / `raw/maintainability-dep-vuln-npm-audit.json` / `raw/maintainability-dep-vuln-govulncheck.json` / `raw/maintainability-dep-vuln-trivy.json` (one file per tool that ran). lizard `--ccn` threshold uses a per-language lookup (see Signal 2 below). --- ## Extraction recipe ### Signal 1 — Committed secrets (SECRETS gate trigger) ```bash # Scan full git history. Single Go binary, JSON out. gitleaks detect --source --report-format json \ --report-path /tmp/gitleaks.json \ --no-banner --exit-code 0 ``` Parse: ```bash jq '[.[] | {rule: .RuleID, file: .File, commit: .Commit, date: .Date}]' /tmp/gitleaks.json ``` Capture: - `gitleaks_findings` (count) - `gitleaks_by_rule` (map rule → count) - `gitleaks_any_still_in_head` — bool; true if any finding's `File` is present in current HEAD (not just history) ### Signal 2 — Polyglot complexity (lizard, per-language CCN) ```bash # 16+ langs. lizard auto-detects languages from file extensions — do NOT pass # `--languages`; it was removed in lizard 1.21+ and the wrapper errors with # `unrecognized arguments` (or, depending on the version, silently no-ops). lizard -o /tmp/lizard.csv \ -x '*/node_modules/*' -x '*/.venv/*' -x '*/dist/*' -x '*/build/*' ``` **Per-language CCN thresholds** — a function is "god" when its CCN exceeds the threshold for its language. Python/JS/TS are idiom-dense and deserve the tightest cap; Go's for/if-heavy control flow tolerates higher CCN before it becomes hard to read. | Language (lizard CSV column 7 / filename extension) | God-function CCN threshold | | --- | --- | | Python (`.py`) | > 10 | | JavaScript / TypeScript (`.js`, `.jsx`, `.ts`, `.tsx`) | > 10 | | Java / Kotlin (`.java`, `.kt`) | > 12 | | Go (`.go`) | > 15 | | Everything else | > 15 (fallback) | Parse using the per-language lookup. The lizard CSV has columns: `NLOC,CCN,token,param,length,location,file,name`. Key on file extension: ```bash # God-functions counted per-language threshold awk -F',' ' NR == 1 { next } { file = $7 ccn = $2 # last path segment → extension n = split(file, parts, ".") ext = (n > 1) ? parts[n] : "" thresh = 15 # default if (ext == "py") thresh = 10 else if (ext == "js" || ext == "jsx" || \ ext == "ts" || ext == "tsx") thresh = 10 else if (ext == "java" || ext == "kt") thresh = 12 else if (ext == "go") thresh = 15 if (ccn + 0 > thresh) n_god++ n_total++ sum_ccn += ccn + 0 } END { printf "total=%d god=%d mean_ccn=%.2f\n", n_total+0, n_god+0, (n_total?sum_ccn/n_total:0) }' /tmp/lizard.csv ``` Capture: - `lizard_total_functions` - `lizard_god_functions` (CCN > per-language threshold) - `lizard_god_by_language` — map `{py: N, js: N, ts: N, go: N, java: N, other: N}` - `lizard_god_ratio` = `lizard_god_functions ÷ lizard_total_functions` - `lizard_god_per_kloc` - `lizard_mean_ccn` - `lizard_thresholds_applied` — `{py: 10, js: 10, ts: 10, java: 12, go: 15, default: 15}` (constant, for audit) ### Signal 3 — Dependency hygiene Run per ecosystem present in the repo. Detect via package manifest: ```bash # JS/TS — install project deps BEFORE running knip. knip walks imports and # needs the repo's own node_modules to resolve them; a bare `npx knip` on a # freshly-cloned repo with no deps installed silently produces `_NOT_RUN` / # empty output (observed when a sub-app reported `Cannot find module 'vite'` # and knip never ran). # # This branch is MANDATORY when package.json exists. The deps-install step is # not optional — do not skip it for time/budget reasons. If the install # genuinely fails (registry down, lockfile broken), knip still runs and # records the failure in /tmp/knip.err; the resulting `knip_NOT_RUN: true` in # the scorecard MUST be accompanied by a `knip_reason` quoting the install-step # error from /tmp/jsdeps.log. A reason of "deps-install step not executed this # pass" is non-conformant. if [ -f /package.json ]; then cd # Detect lock file → pick the matching package manager. If no lock file # exists, assume pnpm (preflight installs pnpm/yarn/npm up front). if [ -f pnpm-lock.yaml ]; then install_cmd="pnpm install --frozen-lockfile --prefer-offline" elif [ -f yarn.lock ]; then install_cmd="yarn install --frozen-lockfile --prefer-offline" elif [ -f package-lock.json ]; then install_cmd="npm ci --prefer-offline" elif [ -f bun.lockb ]; then install_cmd="bun install --frozen-lockfile" else install_cmd="pnpm install --prefer-offline" # no lock → assume pnpm per convention fi # 300s cap — slow registries shouldn't hang the run. Install errors are # non-fatal here; knip still runs and records the failure in its stderr. timeout 300 bash -c "$install_cmd" > /tmp/jsdeps.log 2>&1 || true # knip proper — preserve stderr for audit (a NOT_RUN outcome needs the # install log + knip stderr to explain itself). npx --yes knip --reporter json > /tmp/knip.json 2> /tmp/knip.err || true fi # Python — detect src-layout / Poetry packages declarations so deptry doesn't # false-positive on first-party imports. One observed project reported 292 # bogus DEP001s because deptry treated `app.*` imports (declared via Poetry's # `packages = [{ include = "app", from = "src" }]`) as third-party. # # Strategy: pull every `tool.poetry.packages[].include` and every # `tool.setuptools.packages.find` package name out of pyproject.toml, plus # any top-level package directory under src/ (setuptools-src layout # convention). Pass them as a comma-separated list to --known-first-party. # Falls back gracefully (empty list → flag omitted) if no pyproject.toml or # no parseable declarations. if [ -f /pyproject.toml ] || [ -f /requirements.txt ]; then cd kfp="" # known-first-party packages, comma-separated if [ -f pyproject.toml ]; then # Poetry packages = [{ include = "X", from = "..." }] kfp_poetry=$(python3 -c " import sys, tomllib try: d = tomllib.loads(open('pyproject.toml').read()) pkgs = d.get('tool', {}).get('poetry', {}).get('packages', []) print(','.join(p.get('include', '') for p in pkgs if p.get('include'))) except Exception: pass " 2>/dev/null) [ -n "$kfp_poetry" ] && kfp="$kfp_poetry" fi # Top-level dirs under src/ are first-party by setuptools-src convention if [ -d src ]; then kfp_src=$(find src -maxdepth 1 -mindepth 1 -type d -exec basename {} \; | tr '\n' ',' | sed 's/,$//') [ -n "$kfp_src" ] && kfp="${kfp:+$kfp,}$kfp_src" fi if [ -n "$kfp" ]; then deptry . --known-first-party "$kfp" -o /tmp/deptry.json 2>/dev/null else deptry . -o /tmp/deptry.json 2>/dev/null fi fi # Go if [ -f /go.mod ]; then cd && go mod tidy -diff > /tmp/gomod-diff.txt 2>/dev/null fi ``` Parse (examples): ```bash # knip: unused deps + dead files jq '{unused_deps: (.dependencies // []) | length, unlisted_deps: (.unlisted // []) | length, unused_files: (.files // []) | length, unused_exports: (.exports // []) | length}' /tmp/knip.json # deptry: DEP001-004 counts jq 'group_by(.error.code) | map({code: .[0].error.code, count: length})' /tmp/deptry.json ``` Capture: - `knip_unused_deps`, `knip_unlisted_deps`, `knip_unused_files`, `knip_unused_exports` - `deptry_findings_by_code` (map DEP001-004 → count) - `gomod_tidy_diff_lines` (number of diff lines; 0 = clean) - `dep_hygiene_overall_clean` (bool; true if all above are 0 / empty / no issues) ### Signal 4 — File-size & tech-debt density ```bash # God-file detection — file LOC distribution find -type f \ \( -name '*.py' -o -name '*.ts' -o -name '*.tsx' \ -o -name '*.js' -o -name '*.jsx' -o -name '*.go' \ -o -name '*.java' -o -name '*.kt' -o -name '*.rs' \) \ -not -path '*/node_modules/*' -not -path '*/.venv/*' \ -not -path '*/dist/*' -not -path '*/build/*' \ -exec wc -l {} + 2>/dev/null | sort -rn | head -20 # Tech-debt marker density grep -rnE '\b(TODO|FIXME|HACK|XXX)\b' \ --include='*.py' --include='*.ts' --include='*.tsx' \ --include='*.js' --include='*.jsx' --include='*.go' \ --include='*.java' --include='*.kt' --include='*.rs' \ --exclude-dir=node_modules --exclude-dir=.venv \ --exclude-dir=dist --exclude-dir=build 2>/dev/null \ | wc -l ``` Capture: - `files_over_500_loc` (count — god-file candidates) - `largest_file_loc` - `tech_debt_markers_total` (TODO/FIXME/HACK/XXX) - `tech_debt_per_kloc` ### Signal 5 — Folder structure + README (LLM interpretive) Assemble context for LLM: 1. `tree -L 3 ` (top 3 levels, no `node_modules`/`.venv`) 2. `README.md` (full contents) 3. `CONTRIBUTING.md` if present 4. Three sampled source files (largest + 2 random across distinct dirs) Score on: - Is the top-level folder structure self-explanatory to a new engineer? - Does README explain what the project is and how modules relate? - Are module boundaries clear, or is everything in one soup? - Are naming conventions consistent across files? - Does the commit history (`git log --pretty='%s' | head -100`) show disciplined messages? Capture: - `llm_structure_score` (0–5) - `llm_readme_score` (0–5) - `llm_commit_quality_score` (0–5) - `llm_pickup_score` (0–5) — the integrated "can another engineer pick this up?" judgment - `llm_notes` (one sentence) ### Signal 6 — Dependency vulnerabilities (VULN gate trigger) Scan committed dependency manifests for known CVEs. One tool per ecosystem present. **Skipped when no recognized manifest is present** — emit `dep_vuln_NOT_RUN: true, reason: "no recognized manifest"` in that case. ```bash # Python — pip-audit against requirements.txt / pyproject.toml (reads both). # 120s cap: pip-audit fetches advisory DB on first run; subsequent calls are fast. if [ -f /requirements.txt ] || [ -f /pyproject.toml ]; then cd timeout 120 pip-audit --format json --progress-spinner off \ > /tmp/pipaudit.json 2> /tmp/pipaudit.err || true fi # JS/TS — npm audit --json reads package-lock.json. Requires a lockfile; yarn/pnpm # lockfiles need their native audit (`yarn npm audit --json`, `pnpm audit --json`). # All three output the same vulnerability envelope. if [ -f /package-lock.json ]; then cd timeout 120 npm audit --json > /tmp/npmaudit.json 2>/dev/null || true elif [ -f /pnpm-lock.yaml ]; then cd timeout 120 pnpm audit --json > /tmp/npmaudit.json 2>/dev/null || true elif [ -f /yarn.lock ]; then cd timeout 120 yarn npm audit --json > /tmp/npmaudit.json 2>/dev/null || true fi # Go — govulncheck walks the module graph. Skipped when go is not on PATH # (preflight.sh documents this; sub-agent emits _NOT_RUN when appropriate). if [ -f /go.mod ] && command -v govulncheck >/dev/null 2>&1; then cd timeout 180 govulncheck -json ./... > /tmp/govulncheck.json 2>/dev/null || true fi # Container — Trivy scans a built image OR the filesystem directly. FS scan is # cheaper (no docker build) and covers the same OS+lang package surface. timeout 180 trivy fs --format json --scanners vuln --quiet \ > /tmp/trivy.json 2>/dev/null || true ``` Parse per tool — fields normalize to `{critical, high, medium, low}` counts: ```bash # pip-audit → count by aliased severity (pip-audit uses OSV severity strings) jq '[.dependencies[]?.vulns[]? | .aliases[]?] as $_ | [.dependencies[]?.vulns[]?] | group_by(.severity // "UNKNOWN") | map({sev: .[0].severity, count: length})' /tmp/pipaudit.json # npm audit → advisory severity bucket jq '.metadata.vulnerabilities // {}' /tmp/npmaudit.json # govulncheck → CVSS severity from OSV DB; count only reachable findings jq '[.osv?, .finding?] | map(select(.)) | length' /tmp/govulncheck.json # trivy → JSON envelope, Severity field on each Vulnerability jq '[.Results[]?.Vulnerabilities[]? | .Severity] | group_by(.) | map({sev: .[0], count: length})' /tmp/trivy.json ``` Capture (aggregated across every tool that ran): - `dep_vuln_critical` (int, sum across ecosystems) — `CRITICAL` severity - `dep_vuln_high` (int) — `HIGH` severity - `dep_vuln_medium` (int) - `dep_vuln_low` (int) - `dep_vuln_tool` — comma-joined list of tools that ran on this repo (e.g. `"pip-audit,trivy"`) - `dep_vuln_by_ecosystem` — map `{python: {critical, high, ...}, npm: {...}, go: {...}, container: {...}}` - `dep_vuln_NOT_RUN` — bool; true when no recognized manifest was present - `dep_vuln_reason` — string; populated when `_NOT_RUN` or when a tool's stderr pointed at a genuine failure (lockfile corrupt, registry down) **VULN gate trigger:** `dep_vuln_critical ≥ 1` OR `dep_vuln_high ≥ 3`. Caps `maintainability ≤ 2`. Skipped (gate does not fire) when `dep_vuln_NOT_RUN == true` — no manifest, no signal, no gate. --- ## Raw dumps (flat — files under `raw/` named `maintainability-*`) Each signal writes its raw output to the flat `raw/` directory at the workspace root. Absent files = not extracted / not applicable. Aggregates go to `scorecard.yaml` under `data.`. | File | Source signal | Shape / note | | --- | --- | --- | | `raw/maintainability-gitleaks.json` | Signal 1 | Raw `gitleaks detect --report-format json` output | | `raw/maintainability-lizard.csv` | Signal 2 | Per-function: `{name, file, ccn, nloc, params}` for every function in scope. Shared with 1.1. | | `raw/maintainability-knip.json` | Signal 3 | Raw `npx knip --reporter json` (absent when no `package.json`) | | `raw/maintainability-deptry.json` | Signal 3 | Raw `deptry` output (absent when no Python manifest) | | `raw/maintainability-gomod-diff.txt` | Signal 3 | `go mod tidy -diff` output (absent when no `go.mod`; empty content = clean) | | `raw/maintainability-file-sizes.txt` | Signal 4 | `wc -l` for source files, sorted descending. Shared with 1.1. | | `raw/maintainability-tech-debt.txt` | Signal 4 | `grep -rnE 'TODO\|FIXME\|HACK\|XXX'` output with `file:line` (absent when count = 0) | | `raw/maintainability-llm-read.md` | Signal 5 | LLM's folder-tree + README + commit-quality read: `{structure_score, readme_score, commit_quality_score, pickup_score, one_sentence}` + short rationale | | `raw/maintainability-dep-vuln-pip-audit.json` | Signal 6 | Raw `pip-audit --format json` output (absent when no Python manifest) | | `raw/maintainability-dep-vuln-npm-audit.json` | Signal 6 | Raw `npm/pnpm/yarn audit --json` output (absent when no JS lockfile) | | `raw/maintainability-dep-vuln-govulncheck.json` | Signal 6 | Raw `govulncheck -json ./...` output (absent when no `go.mod` or `go` not on PATH) | | `raw/maintainability-dep-vuln-trivy.json` | Signal 6 | Raw `trivy fs --format json` output (always attempted — covers OS + lang packages) | --- ## Aggregation → 0–5 band Scorer integrates the 5 signals. Rough directionality: | Signal | Pulls score UP | Pulls score DOWN | | --- | --- | --- | | `gitleaks_findings` | 0 | ≥ 1 triggers `SECRETS` gate | | `lizard_god_per_kloc` | < 2 | > 10 | | `lizard_god_ratio` | < 0.05 | > 0.20 | | `dep_hygiene_overall_clean` | true | false with many findings | | `dep_vuln_critical` + `dep_vuln_high` | 0 critical, 0 high | ≥ 1 critical OR ≥ 3 high triggers `VULN` gate | | `tech_debt_per_kloc` | < 1 | > 5 | | `files_over_500_loc` | 0–1 | > 3 | | `llm_pickup_score` | 4–5 | 0–2 | **Absolute bands (per-signal thresholds):** - **5** — `gitleaks_findings == 0`; `dep_vuln_critical == 0` AND `dep_vuln_high == 0`; lizard god-ratio < 0.05; dep hygiene clean; tech-debt < 1/kLoC; clear `src/` layout; LLM pickup ≥ 4 - **4** — Strong but one weaker signal (e.g. 1 high-severity CVE, or god-ratio 0.05–0.10, or tech-debt 1–3/kLoC) - **3** — Reasonable structure; some dep drift or moderate god-functions (ratio 0.10–0.15); tech-debt 3–5/kLoC; README workable - **2** — Flat dump with god-files; god-ratio > 0.15; messy deps; README absent or stale; **`SECRETS` or `VULN` gate fires here (cap at 2)** - **1** — Actively hostile to reading: no README, files > 1000 LOC common, tech-debt > 10/kLoC - **0** — Reserved for extreme cases (would be rare) --- ## Gates | Gate | Trigger | Cap | | --- | --- | --- | | `SECRETS` | `gitleaks_findings ≥ 1` (any secret in git history — even if rotated) | `maintainability ≤ 2` | | `VULN` | `dep_vuln_critical ≥ 1` OR `dep_vuln_high ≥ 3` across any ecosystem tool that ran (`pip-audit` / `npm audit` / `govulncheck` / `trivy fs`). Skipped when `dep_vuln_NOT_RUN == true` (no recognized manifest). | `maintainability ≤ 2` | The `SECRETS` gate fires even if the secret has been rotated out of HEAD — leaked-and-revoked still blocks "pick up and extend" because the leak is permanent in git history. `gitleaks_any_still_in_head` lets the scorer comment on severity but doesn't change the gate. The `VULN` gate fires on committed dependency manifests only; it does not attempt to resolve what version is actually running in production (runtime verification is out of scope — a separate maintainer follow-up). A repo that pins vulnerable versions but ships a build that avoids the vulnerable code path still takes the gate — the signal is "ships committed dependency hygiene", not "is exploitable at runtime". --- ## Fallback — signals missing - Repo has only Go / Rust (no JS/Python) → drop knip/deptry; rely on `go mod tidy -diff` or `cargo tree` + other signals. - lizard doesn't support a language in the repo → drop Signal 2 for that language; rely on file-size distribution + LLM. - Zero-commit or zero-LoC repo → score 0, flag `NO_SIGNAL`, skip. - README missing → LLM Signal 5 degrades; use `llm_confidence: low`. Annotate evidence with `"Signals used: [...]"`. --- ## Output (what the scorer emits) ```yaml sub_criterion: maintainability score: 3 evidence: "Clean src/ layout; god-per-kLoC 1.8; knip clean; tech-debt per kLoC 3.2; no secrets in history; README explains modules but not setup. LLM pickup: 3 — middleware layer is a god-file (src/app.ts = 680 LOC)." data: gitleaks_findings: 0 gitleaks_by_rule: {} gitleaks_any_still_in_head: false lizard_total_functions: 114 lizard_god_functions: 8 lizard_god_ratio: 0.07 lizard_god_per_kloc: 1.8 lizard_mean_ccn: 4.2 knip_unused_deps: 0 knip_unlisted_deps: 0 knip_unused_files: 0 knip_unused_exports: 2 deptry_findings_by_code: {} gomod_tidy_diff_lines: 0 dep_hygiene_overall_clean: true files_over_500_loc: 1 largest_file_loc: 680 tech_debt_markers_total: 14 tech_debt_per_kloc: 3.2 llm_structure_score: 3 llm_readme_score: 3 llm_commit_quality_score: 4 llm_pickup_score: 3 dep_vuln_critical: 0 dep_vuln_high: 1 dep_vuln_medium: 4 dep_vuln_low: 12 dep_vuln_tool: "pip-audit,trivy" dep_vuln_by_ecosystem: python: { critical: 0, high: 1, medium: 2, low: 5 } container: { critical: 0, high: 0, medium: 2, low: 7 } dep_vuln_NOT_RUN: false signals_used: [gitleaks, lizard, knip, filesystem, llm, pip-audit, trivy] gates_triggered: [] ``` --- ## Caveats - **lizard CCN thresholds are language-idiomatic.** Python's 10 ≠ Kotlin's 12 ≠ Go's 15. The per-language lookup in Signal 2 sets the threshold per file extension; absolute and fixed. - **knip / deptry only cover their ecosystems.** Mixed repos run both and aggregate; there's no polyglot dep-hygiene tool. - **LLM "pickup" judgment is brittle at sample size = 3 sampled files.** Evidence should name the files sampled. - **`SECRETS` gate is conservative.** Fires even on rotated secrets — "pick up and extend" assumes trust in git history, and rotation doesn't erase the leak. Scorer should mention severity (`still_in_head` vs. history-only) in evidence. - **`VULN` gate is committed-manifest only.** Does not attempt runtime resolution or version-pinning analysis. A repo that pins a vulnerable version but ships a build that avoids the vulnerable code path still takes the gate — by design, it's a cleanliness signal, not an exploitability signal. - **`CONTRIBUTING.md` / ADR / RFC presence** is a weak signal — absence is common even in healthy repos. Don't penalize for missing; reward when present. - **Tech-debt markers** may be legitimate scoping comments (`TODO: address in v2`) or abandoned work. LLM adjudicates when count is suspiciously high.