Knowledge Freshness

What this system does

Knowledge entries under content/knowledge/ declare a volatility tier and a list of sources. A daily cron prefilters at most ten entries that are due — by cadence or by a changed source hash — runs a grounded LLM audit against the prefetched source bodies, opens one PR per drifted entry, and gates that PR on five checks. In parallel, downstream agents emit knowledge_gap_signal events when they hit a topic the KB does not cover; Lens I aggregates those signals into P1/P2 audit findings, suppressing any topic an entry already covers.

Two arms, two outcomes:

• The refresh arm chases known sources for drift. It ends in a PR that updates an entry.
• The gap arm surfaces unknown topics. It ends in a PR that creates an entry.

Both terminate in a human-merged PR.

How a gap closes

The full lifecycle, end to end:

1. Downstream agents emit signals; they accumulate in the rolling 90-day window.
2. A topic's signal count and distinct-project count cross the threshold.
3. Lens I emits a P1/P2 finding.
4. An operator adds content/knowledge/<category>/<slug>.md.
5. The next audit's knowledge index covers the slug and Lens I suppresses the bucket — the finding disappears.

Signals are not purged when the entry is added. The window is rolling, so yesterday's signals still aggregate tomorrow; suppression filters the emit step, not the aggregation step (src/observability/checks/lens-i-knowledge-gaps.ts:155). Signals only fade as they age out of the 90-day window naturally.

System map

Three real hooks sit beside the two arms: the phase-audit hook (runs Lens H only, never Lens I), the doc-conformance MMR channel (routes Lens I findings into MMR), and the --fix flow (initial + verifier + postfix audit). They are covered below.

Frontmatter, signals, and resolution

Frontmatter schema

Every knowledge entry's frontmatter is a Zod-validated object with four freshness-relevant fields. The schema is the source of truth and runs as Gate 1 of the PR CI (src/validation/knowledge-frontmatter-validator.ts:42-50); runtime readers tolerate missing optional fields.

Cadence model

Three tiers, three windows — 14 / 60 / 180 days for fast-moving / evolving / stable (src/knowledge-freshness/audit-prefilter.ts:5-7). An entry with no last-reviewed always counts as due. Sources with a changed hash also become candidates regardless of age, but the hash check only runs for entries still inside their cadence window.

Which tier does an entry belong in?

Rule of thumb: if a version bump often breaks downstream guidance, lean fast-moving; if drift is extremely rare, stable; otherwise evolving (the default).

Adding a new entry to the KB

1. Choose a category directory under content/knowledge/<category>/. Many categories exist today (backend, core, cli, research, web-app, web3, …); prefer placing into an existing one. Creating a new category is a separate PR.
2. File name = entry slug + .md. The basename must match the name: field (e.g. retry-with-jitter.md ↔ name: retry-with-jitter). Lens I's suppression match reads name: only, not the filename — a mismatch silently breaks suppression.
3. Required frontmatter: name, description. Add volatility + sources[] if you want the cron to audit it — an entry with no sources[] is skipped by the prefilter (src/knowledge-freshness/audit-prefilter.ts:17).
4. Validate locally: make validate-knowledge.
5. Confirm the prefilter will pick it up. A fresh entry has no last-reviewed, so it should appear at priority 100:

   node dist/index.js knowledge-freshness audit-prefilter --max=10 \
     | jq '.[] | select(.name=="<your-new-slug>")'
   

   The daily ceiling is 10, so a flood of new entries may queue past the first day.

Gap-signal payload

A gap signal is a ledger event validated by src/observability/engine/event-schemas.ts:191-220 (payload allow-list at src/observability/engine/event-schemas.ts:12):

{
  "event_id":    "<uuid>",
  "worktree_id": "<sha>",
  "actor_label": "agent | bot | …",
  "branch":      "<branch>",
  "task_id":     null,
  "type":        "knowledge_gap_signal",
  "ts":          "<ISO-8601>",
  "payload": {
    "topic":         "<kebab-slug>",
    "source":        "agent_search",
    "project_id":    "<sha256-hex>",
    "step_name":     "tech-stack",
    "agent_excerpt": "…"
  }
}

topic is ≤80 chars matching /^[a-z0-9]+(-[a-z0-9]+)*$/; source ∈ {agent_search, lessons, manual}; project_id is 64-char sha256 hex (or the literal lessons when source=lessons); step_name and agent_excerpt (≤200 chars) are optional.

KnowledgeRootResolution shape

The resolver returns a three-field record that threads through the audit run (src/observability/knowledge-index.ts:275-291):

export interface KnowledgeRootResolution {
  /** Validated absolute path to a knowledge directory, or null. */
  root: string | null
  /** Pre-loaded index Set, populated by the validator. Null when root is null.
      Lens I reads this directly — no re-walk. */
  index: Set<string> | null
  /** Audit trail of what was tried. Lens I uses this to compose a precise
      warn-once message when root is null. */
  attempts: KnowledgeRootAttempt[]
}

From candidate to merged PR

The cron is a thin bash loop — the brains live in three CLI subcommands and a meta-prompt that runs a grounded LLM against pre-fetched source bodies.

Prefilter

An entry becomes a candidate when (1) it has at least one source, AND (2) either its last-reviewed is older than the cadence window, OR a source's prefetched hash differs from the stored one. Priority orders highest-score first: unreviewed entries (100), then overdue entries (50 + ageDays, so the oldest rank highest), with in-window hash changes at 75; the top --max win (src/knowledge-freshness/audit-prefilter.ts:14-72):

for (const e of entries) {
  if (e.sources.length === 0) continue            // no sources = no audit
  if (!e.lastReviewed)        { select = true; priority = 100 }
  else if (ageDays > window)  { select = true; priority = 50 + ageDays }
  else {
    // hash check — Promise.all over a small per-entry list (1-3 sources)
    if (anyHashChanged) { select = true; priority = 75 }
  }
}
candidates.sort((a, b) => b.priority - a.priority)
return candidates.slice(0, max)

The hash check is a tiebreaker, not a baseline. Entries already past their window are selected immediately at priority 50 + ageDays — no network cost. The hash check only runs in the else branch (still inside the window), runs Promise.all over the entry's 1–3 sources, and swallows fetch errors so a slow upstream doesn't crash the cron.

Audit verdicts

The meta-prompt at content/tools/knowledge-audit-entry.md instructs the LLM to read pre-fetched source bodies (no web tool available) and emit one of four verdicts. Every verdict opens a PR — the dry-run apply runs first so gates can inspect the proposed diff, then --open-pr creates the branch.

PR generation

Branch: knowledge-freshness/<entry>-<YYYY-MM-DD>. renderPrBody renders a summary, the verdict fields, a findings table, the sources, and any preserve warnings (it does not embed the raw verdict JSON). Each candidate gets its own PR off origin/main — the cron git checkout main between iterations and restores the entry between the dry-run apply (for gates) and the final --open-pr call. PRs do not stack; failures isolate per-candidate.

VERSION bump on merge

A dedicated workflow (.github/workflows/knowledge-freshness-version-bump.yml:16) fires on PR closed (merged-only) when the source branch starts with knowledge-freshness/ or the PR carries the knowledge-freshness label. It computes the next SemVer from the PR title and body, writes content/knowledge/VERSION, commits with the prefix chore(knowledge): (deliberately not knowledge-freshness/*) so the commit doesn't re-trigger itself, then git pull --rebase before pushing. Bump rules (src/knowledge-freshness/bump-version.ts:26-45):

The start-of-line anchor on the BREAKING CHANGE body match (/^BREAKING CHANGE:/m) is deliberate — a freshness PR's body embeds an LLM-generated findings table whose evidence excerpts could otherwise mention "BREAKING CHANGE:" and trigger an accidental major bump.

MMR corroboration (manual)

The cron does not dispatch MMR today — the workflow only runs inline gates. To corroborate a freshness PR locally:

git diff origin/main...HEAD -- 'content/knowledge/**/*.md' \
  | mmr review --diff - --focus knowledge-freshness --sync --format json

A native knowledge-freshness MMR channel is the Phase 5 plan. See the MMR guide for the channel architecture.

The five PR gates

The cron's GITHUB_TOKEN-opened PRs don't fire downstream workflows, so the cron also runs the gate code inline (same CLI surface). Human-opened freshness PRs get gated by the workflow at .github/workflows/knowledge-freshness-gates.yml:17.

Lens I — gap detection + suppression

Lens I runs under --scope=docs and --scope=all (src/observability/checks/lens-i-knowledge-gaps.ts:43). It collects signals from the ledger (rolling 90-day window, src/observability/checks/lens-i-knowledge-gaps.ts:52) plus synthetic signals from tasks/lessons.md, buckets them by normalized topic, applies the threshold matrix, and suppresses buckets whose topic an entry already covers.

Threshold matrix

The rules (src/observability/checks/lens-i-knowledge-gaps.ts:148-149):

Topic normalization

Lens I normalizes the raw topic before bucketing, then validates the result. Two distinct steps: normalizeTopic (src/observability/checks/lens-i-lessons-scanner.ts:32-38) always produces a (possibly empty) string; isValidTopic (src/observability/checks/lens-i-lessons-scanner.ts:114-116) decides whether to accept it. Normalization lowercases, strips apostrophes, replaces every other non-slug run with a single hyphen, collapses repeats, and trims. The validator additionally enforces ≤ 80 chars and /^[a-z0-9]+(-[a-z0-9]+)*$/. So Agent Eval Harnesses! → agent-eval-harnesses (valid), but !!! → `` (rejected).

What the lessons.md scanner sees

Lens I synthesizes signals from tasks/lessons.md at audit time (read inline, no ledger writes) via two passes per non-fenced line — code-fenced blocks are skipped (src/observability/checks/lens-i-lessons-scanner.ts:4):

1. Explicit marker — <!-- gap-topic: <slug> --> (slug must already be kebab-case; the marker regex enforces it).
2. Heuristic phrases (case-insensitive): "would have helped to have a guide on X", "missing knowledge entry for X", "no knowledge entry for X" / "no kb entry for X", "missing knowledge: X".

Captured topics run through the same normalizeTopic / isValidTopic. Synthetic signals carry project_id: "lessons" and are excluded from the distinct-projects count by the aggregator's delete('lessons') rule (decision #6) — they corroborate but don't independently satisfy the threshold.

3-tier --knowledge-root resolution

Lens I must know where the KB lives to skip already-covered topics. The resolver (resolveKnowledgeRoot at src/observability/knowledge-index.ts:326-379) tries three tiers in order:

The sharp asymmetry is intentional: an operator who typed a --knowledge-root gets a hard error on a bad path; yaml and auto-detect soft-fail so suppression degrades gracefully. The most instructive case: yaml invalid + auto-detect found — the trail records the yaml failure and the auto-detect success, root is the auto-detect path, and a one-line stderr note points at the stale yaml. This is what an operator sees when npm update -g @zigrivers/scaffold moved the install out from under a pinned yaml path.

Warning policy

emitOnceForAudit (src/observability/knowledge-index.ts:251-259) reads a caller-provided Set created fresh in each runAudit (src/observability/engine/api.ts:114), so the --fix flow's three internal audits each get their own dedup scope.

What a Lens I finding looks like

A single finding excerpt from the audit sidecar (docs/audits/<id>.json):

{
  "id": "a3f2c1d4...",
  "lens_id": "I-knowledge-gaps",
  "severity": "P2",
  "title": "Knowledge base lacks coverage for \"agent-eval-harnesses\" — 4 signals across 2 projects",
  "source_doc": "",
  "evidence": {
    "kind": "knowledge_gap",
    "topic": "agent-eval-harnesses",
    "signal_count": 4,
    "distinct_project_count": 2,
    "distinct_projects": ["a3f2...", "1c4e..."],
    "first_seen": "2026-04-12T09:00:00Z",
    "last_seen": "2026-05-21T14:30:00Z",
    "example_excerpts": ["No knowledge entry for agent eval harnesses"]
  },
  "confidence": "medium",
  "fix_hint": {
    "kind": "edit_doc",
    "target": "content/knowledge/<category>/agent-eval-harnesses.md",
    "prompt": "Propose a new knowledge entry for \"agent-eval-harnesses\". Evidence: 4 signals from 2 projects in the last 90 days."
  }
}

The allowlist

Out-of-allowlist sources warn but don't block (decision #4). Bare hostnames match subdomains; host/path entries additionally require the URL path to start with the prefix; github_repos is locked to specific owner/repo.

The off-allowlist warning is advisory and is surfaced by the frontmatter-validation path (validateKnowledgeFile), not by a gate. Gate 3 (lint-unsourced) is a separate advisory check that flags nearby links not covered by the entry's declared sources[] domains. Off-allowlist sources still get fetched, hashed, and audited — they just warn. It is not a security boundary: the SSRF guard (src/knowledge-freshness/source-url-validator.ts) runs independently, so a new host never unlocks private-IP fetches. The editorial bar is: "would the maintainers want this URL to be the verbatim grounding for a P0/P1 finding?"

Most-cited hosts

Counted live from every entry's sources[*].url at build time.

The full allowlist

Every host plus its category, and the pinned GitHub repos.

47 allowlisted hosts and 3 GitHub repos. Out-of-list sources warn (they do not block).

GitHub repos: modelcontextprotocol/specification, gastownhall/beads, joelparkerhenderson/architecture-decision-record

KB inventory

Totals over content/knowledge/, broken down per category.

301 entries across 21 categories:

How to expand the allowlist

Adding a host is a one-line PR to docs/knowledge-freshness/authoritative-sources.yaml:

 hosts:
   - owasp.org
+  - developers.cloudflare.com
   - nist.gov

1. Pick the form. Bare hostname for vendor docs whose path layout changes; host/path prefix for shared-tenancy hosts where you only trust a sub-path; owner/repo under github_repos: for specific GitHub repos. Skip www. (bare entries auto-match subdomains).
2. Verify the host is live — curl -sI https://<host>/<path> should return 2xx (or a 3xx that ultimately resolves).
3. Mirror the category in CATEGORY_MAP in scripts/build-freshness-reference.mjs — otherwise the regenerated allowlist table shows the new host as other.
4. Open a normal PR. Allowlist additions are not a separate trust delegation; any maintainer can review.

Providers (cron uses Z.ai → DeepSeek)

The cron runs on Z.ai HTTP with a DeepSeek fallback chained behind it (.github/workflows/knowledge-freshness-audit.yml:83), which removes the local claude CLI dependency from CI entirely. Local audits keep using whichever provider is configured. Precedence is resolved by resolveProvider (src/knowledge-freshness/providers/index.ts:36):

1. --provider <name> — explicit flag, operator override (anthropic | deepseek | zai)
2. KNOWLEDGE_FRESHNESS_PROVIDER env var — same three values
3. A single API key in env — inferred (ANTHROPIC_API_KEY / DEEPSEEK_API_KEY / ZAI_API_KEY)
4. More than one API key present → error (ambiguous)
5. No env, claude on PATH → anthropic (subprocess uses keychain)
6. Nothing → error (no provider configured)

Set KNOWLEDGE_FRESHNESS_FALLBACK_PROVIDER to chain a second provider behind the primary. It must name a different known provider, or construction throws (src/knowledge-freshness/providers/index.ts:149). Unset, there is no fallback: a primary failure is a failure.

The cron wires the chain explicitly (.github/workflows/knowledge-freshness-audit.yml:83):

env:
  ZAI_API_KEY:                            ${{ secrets.ZAI_API_KEY }}
  DEEPSEEK_API_KEY:                       ${{ secrets.DEEPSEEK_API_KEY }}
  KNOWLEDGE_FRESHNESS_PROVIDER:           zai
  KNOWLEDGE_FRESHNESS_FALLBACK_PROVIDER:  deepseek
  GH_TOKEN:                               ${{ secrets.GITHUB_TOKEN }}

A missing API key fails the run loudly at preflight rather than silently exiting 0 with zero PRs.

Every command that touches the system

All commands ship in the published CLI.

Refresh-arm commands

Gap-arm commands

The --fix flow (runFixFlow at src/observability/engine/fix-flow.ts:71) runs a three-audit loop: (1) the initial audit produces a fix plan; (2) for each blocking finding, dispatch a fix agent then re-audit just that finding (the verifier); (3) one postfix audit runs everything for the final report. The --knowledge-root override threads into all three (decision #20) so suppression is consistent throughout.

Gate-side subcommands (also runnable locally for triage)

Operations cheat sheet

An entry's audit failed in the cron

The cron logs audit failed for <name> — moving on and continues; the entry stays in tomorrow's queue. Causes: provider auth (key rotated), source URL now 404s, a fetch/HTTP error or the 5 MiB fetch-and-hash cap, dispatcher error, or LLM timeout. (A source body over the 96 KiB embed cap is truncated and flagged truncated: true — it does not fail the audit.) Reproduce locally:

DEEPSEEK_API_KEY=sk-… node dist/index.js knowledge-freshness \
  audit-run-entry content/knowledge/<cat>/<name>.md
# read stderr to see if it's a URL issue or a provider issue

Lens I keeps surfacing a topic the KB already covers

Suppression didn't match. Either the resolver returned root: null (look for [Lens I] knowledge-root not located in stderr) or the entry's name: doesn't normalize to the same slug as the bucket topic — the match is exact and post-normalize.

scaffold observe audit --lens I-knowledge-gaps --json \
  --knowledge-root /path/to/content/knowledge \
  | jq '.findings[] | select(.lens_id=="I-knowledge-gaps")'
grep -A1 "^---" content/knowledge/<cat>/<slug>.md | grep "^name:"

Downstream auto-detect can't find the KB

findScaffoldKnowledgeRoot walks parents from the CLI install's module location looking for package.json#name === '@zigrivers/scaffold'. Symlinked or repackaged installs may miss. Pin it via the tier-2 yaml:

lenses:
  I-knowledge-gaps:
    knowledge_root: /opt/homebrew/lib/node_modules/@zigrivers/scaffold/content/knowledge

Yaml knowledge_root stops working after an upgrade

The yaml tier soft-fails and records the reason in the attempts trail; Lens I appends it to the warning. Validation requires all four: the path exists, is a directory, contains a <path>/VERSION marker, and loadKnowledgeIndex runs without throwing (an empty index is OK). The usual cause after an upgrade is a moved install path:

find / -name VERSION -path '*content/knowledge*' 2>/dev/null

Then update lenses.I-knowledge-gaps.knowledge_root to the new path.

A source URL fetches in curl but the cron rejects it

The SSRF guard re-resolves the hostname at fetch time and rejects any IP in a non-globally-routable range (RFC1918, link-local, loopback, CGNAT, ULA, IPv4-mapped IPv6, …). Common cause: an internal DNS view returning a private IP for an outwardly-public hostname.

node -e 'require("node:dns").promises.lookup("<host>", { all: true }).then(console.log)'

Fix: move the source to a globally-routable host, or remove it. Allowlisting does not bypass the SSRF guard.

--knowledge-root resolves to a path you didn't expect

Auto-detect may pick a stale npm-global install. The successful-resolution path doesn't log its attempts trail today (only the failure path warns), so pin and compare:

scaffold observe audit --lens I-knowledge-gaps --json \
  --knowledge-root /path/you/expected/content/knowledge \
  | jq '.findings[] | select(.lens_id=="I-knowledge-gaps") | .evidence.topic'
# compare against the unset behavior; if the lists differ, auto-detect picked a different KB

Fix: pin lenses.I-knowledge-gaps.knowledge_root in .scaffold/observability.yaml. A pinned yaml path takes precedence over auto-detect.

Config reference

Everything operator-tunable lives in .scaffold/observability.yaml. Anything outside this list is hardcoded (decision #7 invariant) so an untrusted project can't redirect dispatch commands or LLM URLs.

lenses:
  I-knowledge-gaps:
    knowledge_root: /path/to/content/knowledge   # tier-2 resolver override

disabled_lenses: [I-knowledge-gaps]              # opt-out

phase_audit:
  enabled: true                                  # default
  timeout_s: 60
  detached: false                                # fire-and-forget when true

fix:
  dispatcher_command: "claude -p"                # default
  timeout_s: 300
  per_finding_max_attempts: 3

Roadmap and known divergences

Phase 5 (planned)

• Native MMR knowledge-freshness channel — runs automatically on freshness PRs (today the cron dispatches no MMR).
• Frontier scan — augment cadence/hash triggers with a model-driven check: "has the underlying technology meaningfully changed since last-reviewed?"
• Taxonomy cross-reference — detect when two entries assert contradictory facts and route to a reviewer.

Known divergences

The reference page's own audits surfaced these doc-vs-code mismatches; the code is ground truth:

• MMR-in-cron framing — three docs describe it differently; decision #3 (Phase 5 deferral) is authoritative.
• Gate 4 override — spec says PR-description marker; code reads a PR label.
• Daily ceiling — spec implies a yaml knob; only the --max flag exists.
• operations.md lags — labels the native MMR channel "Phase 4" and describes suppression in future tense; both have shipped.
• www. prefix inconsistency (P3) — mixed www. use in the allowlist; bare entries already auto-match subdomains, so the prefix is redundant.