# AI-Native Engineering Maturity **The metric set fastpace surfaces so engineering leaders can answer one question: "How mature is my org at AI-native development?"** Industry conversations about AI-native engineering converge on the same nine competencies. fastpace already implements primitives that touch most of them; this doc maps each competency to the metric we surface and the feature behind it. Where there's a gap, we name it as a Tier-4 work item. The aggregate is the **AI-Native Maturity Score** — a 0–100 number per repo and per org, derived from nine weighted sub-scores. It rolls up to the org dashboard and powers a single sentence in stand-up: *"Acme is at 72/100 — held back by Review Maturity (38) and Test Parity (54)."* ## The nine competencies, mapped | # | Competency | What it measures | fastpace coverage today | Metric we surface | F-ID | |---|---|---|---|---|---| | 1 | **Project Setup** | Repo is structured for AI tools (agents, hooks, CLAUDE.md, decisions, patterns, glossary, sanctioned runtime) | `fastpace inventory` lists what's installed | **AI-Readiness Score (0–100)** — checklist of 9 install-time artifacts | F4.1 | | 2 | **Spec-Driven Development** | User stories → PRD → ERD → exec plan before code | PRD/ERD/exec-plan generators (`/fp-write-prd`, etc.); provenance trailers tie commits to specs | **Spec Coverage %** — share of PRs (last 30d) linked to PRD/ERD/exec-plan | F4.2 | | 3 | **Context Management** | Quality of CLAUDE.md, decisions, patterns, glossary, learnings | `fastpace context-score` (6 dimensions, already shipped) | **Context Score (0–100)** + 30-day trend, already in CLI; needs UI tile + org rollup | F4.3 | | 4 | **Testing & Validation** | Tests written before / alongside AI-generated code | SAST/lint required hook (F2.13); correction detector knows which files AI touched | **AI Test Parity %** — share of AI-edited files in a PR with same-PR test changes | F4.4 | | 5 | **Harness Engineering** | Guardrails are tight enough that multiple agents + devs can work without debt | Agent identity manifests with caps; scope-guard hook | **Guardrail Strictness Score (0–100)** — # agents × % with all 4 caps set × avg path-allowlist tightness | F4.5 | | 6 | **Architectural Guardrails** | AI doesn't quietly violate ADRs or established patterns | Explanations module ties code to ADRs; correction-detector tracks AI edits | **ADR Drift Rate** — # AI commits touching ADR-referenced files without updating the ADR (per 100 commits) | F4.6 | | 7 | **Review Maturity** | AI-attributed PRs receive substantive human review | Provenance trailers identify AI-attributed commits | **AI Review Rate** — % of AI-attributed PRs reviewed; **AI Review Depth** — avg review comments per AI PR | F4.7 | | 8 | **AI SDLC Metrics** | DORA (lead time / deploy freq / MTTR / change-fail) split by attribution | DORA dashboard view exists; provenance attribution exists | **DORA-by-Attribution** — same DORA metrics, but split AI-attributed vs human-attributed | F4.8 | | 9 | **Prompt Engineering** | Prompts produce code that doesn't need rework | Reliability score (F2.1) measures correction rate already | **Prompt Effectiveness Score** — per-developer reliability score weighted by prompt specificity proxy (length, role count, system-prompt presence) | F4.9 | ## The aggregate score `fastpace maturity` returns a single 0–100 weighted score plus the nine sub-scores. Default weights (configurable per-org via `fastpace.config.yaml`): ```yaml maturity: weights: project_setup: 10 # foundation; pass or fail at install spec_driven: 12 # spec-coverage % → directly observable context: 12 # context score is fastpace's signature metric testing: 14 # most sensitive to "AI shipping debt" harness: 10 # config-time, lower variance architectural: 12 # ADR drift catches subtle decay review_maturity: 12 # human-in-the-loop signal sdlc_dora: 10 # outcome metric — slow-moving prompt_engineering: 8 # noisy at low N ``` The score-pause guarantee on Team+ tiers is wired to this: if your maturity score drops by ≥10 points in a billing month, your seat charges pause until it recovers. ## Where the gaps are **Already shipped (5/9):** Project Setup (inventory), Spec-Driven (provenance trailers + PRD/ERD pipelines), Context Management (context-score), Testing (SAST hook), Prompt Engineering (reliability score). Each has the underlying data. None is exposed as the metric described above. Wrapping them in `fastpace maturity` is a few days of work. **Partial (2/9):** - *Harness Engineering* — agent caps exist, but no aggregate strictness score. F4.5 wraps existing data. - *Architectural Guardrails* — explanations module knows which ADRs reference which files, and correction-detector knows what AI touched. F4.6 joins them. **New work (2/9):** - *Review Maturity (F4.7)* — needs an integration with GitHub / GitLab / Bitbucket review-state APIs. Read-only, scoped to public PR metadata. M-L effort. - *DORA-by-Attribution (F4.8)* — DORA already exists, but the AI-vs-human split needs the run-manifest attribution to flow into the DORA aggregator. M effort. ## How fastpace surfaces this 1. **CLI:** `fastpace maturity [--repo ] [--org] [--json]` returns the aggregate + nine sub-scores. Cached locally; refreshed on every `fastpace status`. 2. **Local UI:** new "Maturity" view at the top of the sidebar, above "Dashboard". Shows the 0–100 number + sparkline + the nine sub-bars with click-through to the underlying view. 3. **Org dashboard:** "Org Maturity" tile rolls every repo's score into a single org-level number. Heatmap view shows repos × sub-scores. 4. **Export:** `fastpace maturity export` produces a signed JSON bundle suitable for a board deck or a Drata/Vanta evidence push. 5. **Trust portal:** maturity score (with consent) becomes a buyer-facing signal — *"Acme runs at 72/100 across 14 repos, up from 64 last quarter."* ## How metrics stay fresh as people use fastpace Maturity is only useful if it reflects reality this morning, not last quarter. The freshness story has four layers — each builds on primitives fastpace already ships. ### Layer 1 — Single developer, single repo (real-time) Every hook fire and every CLI command appends a signed entry to the **local hash-chained audit log** (F0.2) and a **run manifest** (F1.1) where applicable. Maturity sub-scores are **derived views**, not stored state — they're recomputed from the logs on every read. So: - The dev runs `/fp-write-prd` → audit entry written → next time they open the local UI, Spec Coverage % reflects it. - The dev commits AI-edited code → correction-detector hook fires → audit entry written → AI Test Parity recalculates on next read. Latency: **real-time** (whatever the next CLI/UI read interval is, usually seconds). No daemon, no scheduler, no cache to invalidate — the source of truth is the log, and recomputation is cheap because logs are append-only and indexable. ### Layer 2 — Single developer, many repos (per-machine rollup) Discovery manifest at `~/.fastpace/manifest.json` (F0.6) lists every repo on the machine. `fastpace me` (F3.5) reads each repo's audit log + run manifests + corrections + spec links and rolls them up. Refresh trigger: any of (a) a CLI command in any tracked repo, (b) the local UI's polling tick (default 30s), (c) a manual `fastpace me --refresh`. Cached in `~/.fastpace/me-cache.json` with a 60-second TTL so back-to-back reads don't reparse every audit log. ### Layer 3 — Many developers, one repo (cross-machine) This is where it gets interesting, because audit logs are **identity-signed** per developer and shouldn't be committed to git (they'd merge-conflict and leak per-dev info). The propagation mechanism is **audit-log replication** (F3.11): ``` dev-1 laptop ──┐ ├──▶ S3 / GCS / on-prem ──▶ org dashboard ingests dev-2 laptop ──┤ bucket │ dev-3 laptop ──┘ │ ▼ maturity rollup (per repo · per team · per org) ``` - **Push side:** every hook event optionally fires a webhook (F3.10) → the developer's local replicator queues a shard → on commit / push / every N minutes, the shard is uploaded to the configured backend. Default cadence: 5 min. Configurable. - **Pull side:** the org dashboard polls (or receives webhook callbacks from) the bucket and ingests new shards. Each shard is signature- verified at ingest using F1.14 (org SAML envelope) — no shard from outside the SCIM-provisioned roster is accepted. - **No merge conflicts:** every shard is independent (different signer), so the dashboard just appends. Maturity sub-scores are recomputed from the union of shards. Latency: **5 min by default; sub-second on webhook push.** A new proposal in F4.10 below makes this configurable per metric — e.g. Review Maturity can poll the GitHub API every 15 minutes, but Spec Coverage updates on every shard push. ### Layer 4 — Many teams, many repos (org dashboard) Org dashboard (F1.13) ingests from Layer 3 and computes: - **Per-repo maturity** (the same nine sub-scores, but rolled up across every developer's contributions to that repo). - **Per-team maturity** (mean, weighted by repo activity). - **Per-org maturity** (the headline number on the trust portal). Refresh trigger: ingest event. The dashboard caches per-repo scores for 60 seconds, per-team for 5 min, per-org for 15 min — only the deepest aggregation is "stale-ish," and even there it's well within the cadence a CTO would refresh a board slide. ### What's already shipped vs. new for this story | Layer | Need | Status | |---|---|---| | 1 | Local audit log, run manifests, signed entries | ✅ F0.2, F1.1 | | 1 | Hooks fire on every tool call | ✅ F0.4, F0.5, F2.1 | | 1 | UI polls + recomputes on read | ✅ already in `packages/dashboard/` | | 2 | Discovery manifest | ✅ F0.6 | | 2 | Per-developer rollup (`fastpace me`) | ✅ F3.5 | | 3 | Audit-log replication | ✅ F3.11 | | 3 | Webhook event bus | ✅ F3.10 + producers wired Sprint 20 | | 3 | Signature-verified ingest | ✅ F1.14 | | 4 | Org dashboard ingest + rollup | ✅ F1.13 (note: standalone server lacks built-in admin UI today — see future-work §3.7) | | 4 | **Per-metric refresh cadence config** | **🟡 new — F4.10** | | 4 | **WebSocket / SSE for live UI updates** | **🟡 new — F4.11** | | all | **Maturity score itself (derived view)** | **🟡 new — F4.x bundle** | **Bottom line:** the data plumbing is already in production. F4.1–F4.9 add the maturity *views* on top. F4.10 + F4.11 make the views update live instead of polling. ## Why this is the right framing The current marketing leans on **trust** (audit, evidence, redaction). That sells to CISOs. Maturity leans on **velocity-with-quality** — it sells to VPs of Engineering and CTOs, who want a number that says "we're getting this right." Same product, different headline metric. The audit primitives stay the foundation; maturity is the dashboard view a CTO would put on a board slide. Same way DORA reframed delivery metrics in 2018 — fastpace can do that for AI-native delivery in 2026. ## Source citation The nine-competency framework matches the consensus in industry write-ups on AI-native engineering competencies (Project Setup, Spec-Driven Development, Context Management, Testing & Validation, Harness Engineering, Architectural Guardrails, Review Maturity, AI SDLC Metrics, Prompt Engineering). fastpace's contribution is making each one **measurable** and **comparable across repos**, not redefining them.