# Quality Gates **Never ship code without passing all quality gates.** ## The Quality Gates (8 blocking default-on + 3 advisory default-on + 1 opt-in) Every gate below is wired into the orchestration loop (`autonomy/run.sh`). Read the count honestly, split by what a gate actually does: - **8 BLOCKING default-on gates** (the numbered table below): run by default and FAIL the build / block completion when they trip. - **3 ADVISORY default-on gates** (LSP diagnostics, semantic test-authenticity, invariant/property; see the advisory table further down): run by default, SURFACE actionable findings into the next iteration's prompt, and never block by default. Two of them gain a blocking arm only via an explicit `*_BLOCK` opt-in flag; LSP diagnostics has no blocking arm at all. - **1 OPT-IN gate** (coverage, `LOKI_COVERAGE_GATE`, default OFF): does not run unless explicitly enabled. It is the lone opt-in because it doubles test runtime (instrumented re-run); see the coverage section for why. So: 8 blocking gates run by default, plus 3 advisory surfacing gates and 1 opt-in coverage gate. Only the 8 numbered gates block out of the box; the advisory gates surface findings without blocking, and the coverage gate is OFF unless explicitly enabled. Never count the advisory or opt-in gates as blockers. The numbered table below lists the 8 BLOCKING default-on gates: exactly what each detects, what it does NOT detect (so you never over-trust a green gate), its opt-out flag, and its blocking behavior. Transcribe this list verbatim; do not recompute it. | # | Gate | Detects | Does NOT detect | Blocking | Opt-out flag | |---|------|---------|-----------------|----------|--------------| | 1 | Static Analysis | ESLint/Pylint, type-checker findings on the diff | Logic bugs that pass the linters | Yes (severity ladder) | `PHASE_STATIC_ANALYSIS=false` | | 2 | Test Suite (pass/fail) | Whether the project test runner passes or fails (red blocks) | Coverage % (not measured in this release) | Yes (red blocks) | `PHASE_UNIT_TESTS=false` | | 3 | Blind Code Review (3-reviewer council + severity blocking) | Correctness/security/design issues via 3 blind reviewers; Critical/High block, Medium/Low advisory | Issues none of the 3 reviewers surface | Yes (Crit/High block) | `PHASE_CODE_REVIEW=false` | | 4 | Anti-Sycophancy / Devil's Advocate (on unanimous PASS) | Sycophantic unanimous approvals: a Devil's Advocate re-review on a unanimous PASS; its Crit/High findings block | Problems the Devil's Advocate reviewer also misses | Yes (DA Crit/High block) | `LOKI_GATE_DEVILS_ADVOCATE=false` | | 5 | Mock Integrity Detector | Tautological assertions, internal-mock ratio, tests that do not import source (`tests/detect-mock-problems.sh`); HIGH blocks | Semantic correctness of mocks (whether a mock faithfully models the real dependency) | Yes (HIGH blocks) | `LOKI_GATE_MOCK=false` | | 6 | Test Mutation Detector | Assertion-value churn alongside implementation changes (test-fitting), low assertion density (`tests/detect-test-mutations.sh`); HIGH blocks | Logically-correct-but-weak assertions | Yes (HIGH blocks) | `LOKI_GATE_MUTATION=false` | | 7 | Documentation Coverage | README presence, docs freshness within 10 commits, API docs for exported symbols in packages | Whether the docs are accurate or useful | Yes | `LOKI_GATE_DOC_COVERAGE=false` | | 8 | Magic Modules Debate | Spec-vs-implementation debate findings on generated Magic Modules; BLOCK-severity findings block | Issues outside the Magic Modules debate scope | Yes (BLOCK severity) | `LOKI_GATE_MAGIC_DEBATE=false` | The three advisory default-on gates (LSP diagnostics, semantic test-authenticity, invariant/property) are documented in their own table under "Advisory default-on verification gates" below. Semantic test-authenticity used to appear here as "gate 9 (opt-in, default OFF)"; it is now a default-on advisory gate and has moved to that table. See the migration note there. **Severity-based blocking** ties the review gates together: any Critical or High finding blocks completion. Medium, Low, and cosmetic findings are advisory and become TODO comments rather than blockers. It is the block policy inside code review (gate 3), not a separate gate function. **Follow-up (later release):** real coverage measurement (Fix A). Today gate 2 decides purely on the test runner's pass/fail; the coverage percentage is not measured. ### Conditional auditor (not numbered): Backward Compatibility (v6.67.0) This is a healing-mode SPECIALIST reviewer, not one of the 8 loop gates. It fires only when `LOKI_HEAL_MODE=true`, when `loki heal` is active, or when the diff touches files flagged in `.loki/healing/friction-map.json`. Greenfield projects skip it entirely. To suppress on a healing project, set `LOKI_HEAL_MODE=false`. **Purpose:** Prevent accidental removal of institutional logic or behavioral changes to legacy code without explicit documentation. **Checks:** 1. **Friction Safety** - If modified code matches a friction-map entry, verify `safe_to_remove` is true or `classification` is `true_bug` 2. **Characterization Test Coverage** - Modified legacy components must have characterization tests in `.loki/healing/characterization-tests/` 3. **Comment Preservation** - Deleted comments containing business rule keywords (hack, workaround, compliance, per requirement) must be extracted to `institutional-knowledge.md` first 4. **Adapter Verification** - Replaced components must have an adapter layer that preserves the original interface 5. **Behavioral Baseline** - If a baseline exists in `.loki/healing/behavioral-baseline/`, outputs must match or differences must be documented as intentional **Severity:** - Removing friction point classified as `business_rule` or `unknown` without approval = **Critical** (BLOCK) - Missing characterization test for modified legacy component = **High** (BLOCK) - Deleted business rule comment without knowledge extraction = **Medium** (BLOCK) - Missing adapter for replaced component = **High** (BLOCK) - Behavioral baseline mismatch without documentation = **Medium** (BLOCK) --- ## Gate 7: Documentation Coverage (v6.75.0) **Triggers when:** Diff touches public APIs, new files added, library/package releases **Checks:** - Every exported function/class/endpoint has a doc entry in `.loki/docs/` - README.md exists and is non-empty in project root - Documentation SHA is within 10 commits of HEAD - CLAUDE.md (if exists) references current key files **Severity:** - Missing API docs = Medium (BLOCK for npm/pip packages) - Stale docs = Low (TODO) **Skip:** Internal-only changes, test-only changes, config changes **Disabling (not recommended for packages):** ```bash LOKI_GATE_DOC_COVERAGE=false # Disable gate 7 ``` --- ## Gates 5 and 6: Automated Test Integrity Gate 5 (Mock Integrity Detector) and gate 6 (Test Mutation Detector) run during the VERIFY phase and are enabled by default (opt-out, never opt-in). **How they run:** - Gate 5 runs `tests/detect-mock-problems.sh` against all test files in the project - Gate 6 runs `tests/detect-test-mutations.sh` against recent commits (default: last 5, or use `--commit HASH` for targeted checks) - Both produce findings at HIGH/MEDIUM/LOW severity levels - HIGH findings = automatic FAIL (same as other blocking gates); MED/LOW route to the findings-injection file for the next iteration rather than blocking **Disabling:** ```bash LOKI_GATE_MOCK=false # Disable gate 5 (Mock Integrity Detector) LOKI_GATE_MUTATION=false # Disable gate 6 (Test Mutation Detector) ``` --- ## Advisory default-on verification gates (LSP, semantic, invariant) These three gates extend the mock/mutation precedent (gates 5+6: default-on, opt-out) into an ADVISORY-FIRST posture. They run by default, surface actionable findings into the next iteration's prompt, and only BLOCK completion when their opt-in `*_BLOCK` flag is set. LSP diagnostics has no blocking arm at all: it is advisory by construction. Coverage is NOT in this group; it remains opt-in (see the next section). This is the FROZEN knob scheme. All flags accept `true` or `1`. **Route scope (honest disclosure):** Bun-route parity is ACHIEVED for the default-ON behavior. The Bun runner (`loki-ts/src/runner/quality_gates.ts` `readToggles`) now defaults all three advisory gates ON (`LOKI_GATE_SEMANTIC_TESTS`, `LOKI_GATE_INVARIANTS`, `LOKI_GATE_LSP_DIAGNOSTICS` each `flag(X, true)`), matching the bash `loki start` route (`autonomy/run.sh`), with blocking behind the same opt-in `*_BLOCK` flags. Semantic and invariant findings surface into the next iteration's prompt via the `build_prompt` readers (`buildSemanticFindingsBlock` + `buildInvariantFindingsBlock`). The one real remaining gap is a SURFACING ASYMMETRY: LSP runs default-ON on the Bun route, but its advisory findings are NOT yet injected into the Bun prompt (LSP contributes only the grounding instruction, not its diagnostics block). That prompt-injection parity is still pending for LSP on the Bun route. This mirrors the "Reachability note" at the end of this file for the v7.5.0 Phase 1 flags. | Gate | Surfacing (advisory, default-ON, opt-out) | Blocking (opt-in, default-OFF) | |------|--------------------------------------------|--------------------------------| | LSP diagnostics | `LOKI_GATE_LSP_DIAGNOSTICS=true` | advisory-only, no blocking arm | | Semantic test-authenticity | `LOKI_GATE_SEMANTIC_TESTS=true` | `LOKI_GATE_SEMANTIC_TESTS_BLOCK=true` | | Invariant / property | `LOKI_GATE_INVARIANTS=true` | `LOKI_GATE_INVARIANTS_BLOCK=true` | How to read the Surfacing column: each gate is already ON by default. The flag shown is the gate's own toggle, and `true`/`1` is its enabled value (the default). Set the flag to its off value (`false`/`0`) to OPT OUT of surfacing. Setting the `=true` value is a no-op confirmation, not an opt-in: these are not opt-in gates. How to read the Blocking column: by default these gates never block, they only surface. To make a gate also block completion, set its `*_BLOCK` flag (default OFF). The blocking arm fires only on a completion claim, and only on the gate's high-severity findings; lower-severity findings stay advisory either way. **What each gate surfaces:** - **LSP diagnostics** (`LOKI_GATE_LSP_DIAGNOSTICS`, default on): language-server diagnostics (errors/warnings) on the changed files. Advisory only; there is no `_BLOCK` flag and no way to make it block. Inert when no language server is available for the project's languages. - **Semantic test-authenticity** (`LOKI_GATE_SEMANTIC_TESTS`, default on): fake tests that look real but verify nothing (literal-via-variable echo, mock-return echo, deleted assertions) that gates 5+6 miss (`tests/detect-semantic-test-problems.sh`). Surfaces by default; blocks on CRITICAL/HIGH only when `LOKI_GATE_SEMANTIC_TESTS_BLOCK=true`. Does NOT detect deep dataflow, legitimate computed-literal assertions, or non-JS/TS tests (JS/TS only). - **Invariant / property** (`LOKI_GATE_INVARIANTS`, default on): property and metamorphic invariant findings. Surfaces by default; blocks only when `LOKI_GATE_INVARIANTS_BLOCK=true`. **Advisory-first posture and deny-filter:** these gates run by default and feed their findings into the next iteration's prompt so the agent can act on them. They only stop a completion claim when the matching `*_BLOCK` opt-in is set. The gates are deny-filtered: a clean result, an absent toolchain (e.g. no language server, no test files in scope), or a timeout never fires the gate. The gate surfaces or blocks only on real, parseable findings, so a missing toolchain does not produce false surfacing or a false block. ### Migration: semantic and invariant flags were repurposed `LOKI_GATE_SEMANTIC_TESTS` and `LOKI_GATE_INVARIANTS` changed meaning: - **Before:** these flags were default-OFF opt-ins that, when set, made the gate BLOCK completion. Semantic test-authenticity was documented as "gate 9 (opt-in, default OFF)" and blocked on CRITICAL/HIGH when `LOKI_GATE_SEMANTIC_TESTS=true`. - **Now:** these flags are default-ON surfacing toggles (set to `false`/`0` to opt OUT of surfacing). They no longer block. Blocking moved to the new `LOKI_GATE_SEMANTIC_TESTS_BLOCK` and `LOKI_GATE_INVARIANTS_BLOCK` opt-ins (default OFF). **Action required if you relied on the old behavior:** anyone who set `LOKI_GATE_SEMANTIC_TESTS=true` (or `LOKI_GATE_INVARIANTS=true`) specifically to BLOCK the build must now set `LOKI_GATE_SEMANTIC_TESTS_BLOCK=true` (or `LOKI_GATE_INVARIANTS_BLOCK=true`). The old flag set to `true` now only confirms the default-on surfacing and will NOT block. This is the one behavior change in the migration: a flag that used to block now only surfaces. ### Coverage stays opt-in (the exception) The coverage gate is the one verification gate that remains OPT-IN (`LOKI_COVERAGE_GATE`, default OFF). It is the exception because measuring coverage requires an instrumented SECOND test run, which roughly doubles test runtime for every iteration. The advisory gates above add no such cost, so they default on; coverage's cost is why it does not. ```bash LOKI_COVERAGE_GATE=1 # opt in: measure + record coverage. Default OFF because # it doubles test runtime (instrumented re-run). Even # when enabled it measures and warns; it does not block # unless LOKI_ENFORCE_COVERAGE=1 is also set. ``` ## v7.5.0 Phase 1 environment flags These four flags activate the override council and structured-findings pipeline added in v7.5.0. All default off; behavior is byte-identical when unset. ```bash LOKI_INJECT_FINDINGS=1 # inject structured per-finding records into the # next iteration's prompt (instead of just the # comma-separated gate-failure tokens) LOKI_OVERRIDE_COUNCIL=1 # enable the 3-judge override council on BLOCK # when .loki/state/counter-evidence-.json # exists. Requires LOKI_INJECT_FINDINGS=1. LOKI_AUTO_LEARNINGS=1 # auto-write structured learnings to # .loki/state/relevant-learnings.json on every # code_review gate failure LOKI_HANDOFF_MD=1 # write a structured handoff doc to # .loki/escalations/handoff-*.md before PAUSE # (in addition to the bare PAUSE signal) ``` Optional: `LOKI_AUTO_LEARNINGS_EPISODE=1` also writes the learning into the Python episodic memory layer via `memory.engine.save_episode`. ## Default-on verification-integrity gates (v7.41.1) These three gates are default-ON accuracy guards (opt-out, never opt-in). They close "verification theater" gaps where a gate could pass without actually verifying. Set each to its off value only to restore the older half-blind behavior. ```bash LOKI_REVIEW_INCONCLUSIVE_BLOCK=1 # default 1. Treat a code-review round that # produced no parseable VERDICT (NO_OUTPUT # or zero real verdicts) as a BLOCK instead # of a silent pass. Set 0 to disable. LOKI_COMPLETION_TEST_CAPTURE=1 # default 1. The verified-completion gate # captures fresh test evidence when # test-results.json is absent, instead of # passing half-blind. Set 0 to disable. LOKI_AUTO_DOCS=true # default true. Auto-generate the .loki/docs/ # suite in the loop before the documentation # gate scores, instead of nagging the user to # run 'loki docs generate'. Set false to disable. ``` The code-review diff also excludes `.loki/` and `.git/` (top-level and nested `**/.loki/**`) so reviewers score real source changes, not runtime state bloat. This is unconditional, not gated. ## Other opt-in environment flags (Release 3) Two more default-off flags added for hybrid search and parallel concurrency. Both are no-ops when unset (behavior identical to before). ```bash LOKI_DYNAMIC_CONCURRENCY=1 # scale the parallel-session cap DOWN under # CPU/memory pressure (default off). Full # knobs and defaults: skills/parallel-workflows.md # (Dynamic Resource-Aware Session Concurrency) LOKI_CODE_INDEX_AUTOREINDEX=1 # auto incremental re-index of the semantic # code index before a search when stale # (default off = warn-if-stale). Details: # references/mcp-integration.md (Built-in # Hybrid Codebase Search) ``` ## Output-token compressor (caveman, default-on, Claude-only) [caveman](https://github.com/JuliusBrussee/caveman) is a Claude Code skill that instructs the model to compress its OUTPUT tokens only (prose style), keeping all technical substance. Loki ACTIVATES it on free-form generation (the main RARV dev loop) and HARD-SUPPRESSES it on every parsed-output trust-gate subcall (council votes, the code-review `^VERDICT:`, the adversarial probe, the merge-conflict resolver, the USAGE.md regen). The suppression is by construction (one shared helper sets `CAVEMAN_DEFAULT_MODE=off` on every parsed call site), so compression can NEVER flip a verdict or completion decision. The compression level is auto-selected per iteration from the run's RARV tier (no user knob): planning iterations (architecture / design) compress at `lite` to protect nuance; development / fast / unknown iterations stay at the conservative `full`. Inference never picks `ultra` (auto ceiling = `full`); an explicit `LOKI_CAVEMAN_LEVEL` overrides the inference entirely, and the never-raise-a- user's-lower-global-level guard still applies. Claude-provider-only: on Codex / Cline / Aider the run is byte-identical to before. Vendor-less: Loki ships no copy of caveman and bootstraps the pinned version on demand (idempotent, cached under `.loki/`). Savings are real but bounded (output tokens only); there is no price API, so Loki discloses the savings CLASS, never a dollar figure. ```bash LOKI_CAVEMAN=0 # opt out (default on). Disables activation; the # parsed-subcall suppression still runs (it is a # harmless no-op when caveman is absent). LOKI_CAVEMAN_LEVEL=full # explicit override (default: auto-inferred from # the RARV tier). lite | full | ultra | wenyan | # wenyan-lite|full|ultra. Set it to opt out of # inference and pin one level. LOKI_CAVEMAN_VERSION=1.9.0 # pinned caveman version (upgrade by bumping) LOKI_CAVEMAN_AUTO_BOOTSTRAP=0 # disable the on-demand pinned install ``` Note: when `LOKI_LEGACY_COMPLETION_MATCH=true` (the legacy prose-grep completion path), main-loop activation is automatically disabled so compression cannot mangle the prose completion-promise. The default completion path (the `loki_complete_task` MCP tool / completion signal file) is immune to compression and keeps caveman on. ## Verified-completion evidence gate (v7.19.1, default-on) The completion council will not accept a "done" claim without evidence. Before completion is honored (on BOTH the council path AND the default completion-promise route), `council_evidence_gate` requires: - a nonzero git diff vs the run-start SHA (something was actually shipped), AND - green tests (`.loki/quality/test-results.json` shows the runner passed). The diff is the union of committed, staged, unstaged, and untracked changes (`--exclude-standard`, so gitignored artifacts do not count), with `.loki/` runtime state excluded. Inconclusive cases (no git repo, no baseline, no test-results file, `runner=none`) pass through and never false-block a legitimate first run. ```bash LOKI_EVIDENCE_GATE=0 # opt out: completion is honored without the # evidence check (byte-identical to pre-v7.19.1). # Default is on (1). ``` When the gate blocks, it prints the reason and this opt-out to the terminal, writes `.loki/council/evidence-block.json`, and surfaces in the dashboard (`/api/council/gate` -> `evidence`; the Quality Gates panel shows a banner). A persistent block keeps iterating only up to `MAX_ITERATIONS`, then stops cleanly; it cannot hang. Honest limit: this proves something-changed-and-tests- pass, not PRD-semantic correctness (the council vote is the semantic check). The common false-block is a project that was ALREADY red before the run; the one-step opt-out is the escape hatch. **Inconclusive-baseline disclosure (v7.28.0):** when the gate cannot establish a diff baseline (reason `no_git_repo` or `no_run_start_sha`) it still passes through (it never blocks a non-git project), but completion is no longer independently verified. Instead of passing silently, the gate writes `.loki/state/evidence-inconclusive.json` (recording the reason, iteration, and timestamp) and emits an `evidence_inconclusive` trust event. The run summary in `.loki/COMPLETION.txt` then carries one honest line: `Evidence gate: inconclusive () - completion not independently verified`. The record is removed automatically on any later run that resolves a conclusive baseline. This is a diff-baseline-only disclosure: red tests still block completion independently, regardless of the inconclusive state. **Override-judge knobs (v7.5.4+):** ```bash LOKI_OVERRIDE_JUDGES=claude,codex # csv of provider names for the # 3-judge override council. Defaults # to the available installed providers # (claude, codex, cline, aider). LOKI_OVERRIDE_REAL_JUDGE=0 # force the deterministic stub-judge # path (hermetic CI / cost control). # Default: 1 = real provider-backed # judges when their CLIs are present; # falls back to stub on missing CLI # or transient provider failure. ``` Implementation: `loki-ts/src/runner/quality_gates.ts:760` (judge dispatch), `:780` (csv parse), `:987` (real-judge gate). **Reachability note (v7.5.0/v7.5.1)**: these flags activate inside the Bun runtime. Today `loki start ` routes through the bash runner via `bin/loki` shim fall-through, so the flags do not yet trigger on a real `loki start`. They DO activate in any code path that calls `loki-ts/src/runner/runQualityGates` directly (e.g. tests, programmatic integration). End-to-end activation lands when Part A Phase 4 wires the Bun `start` route. See CHANGELOG v7.5.0 NOT-tested section. ### Counter-evidence file format (`.loki/state/counter-evidence-.json`) ```json { "iteration": 7, "evidence": [ { "findingId": "eng-qa::- [Critical] dead code path bug at sdk/python/...", "claim": "this code path is dead duplicate; live code is at sdk/src/gauge/", "proofType": "duplicate-code-path", "artifacts": ["sdk/python/ is excluded by pyproject.toml"] } ] } ``` `findingId` is `canonicalFindingId(finding)` -- `::`. `proofType` MUST be one of: `file-exists`, `test-passes`, `grep-miss`, `reviewer-misread`, `duplicate-code-path`, `out-of-scope`. Entries with any other proofType are silently dropped at load time. The override council uses a stub judge in v7.5.x that approves any of those six trusted proofTypes; real provider-backed judges land in Phase 2 of Part B. **Cross-process gate counter (v7.5.5+)**: the per-iteration gate counter at `.loki/state/gate-counter-.json` is now incremented under a cross-process file lock via `withFileLockSync` in `loki-ts/src/util/atomic.ts`. Concurrent gate runs (parallel worktrees, overlapping `runQualityGates` invocations) no longer race the read-modify-write, so override-council quotas and per-finding counters remain consistent across processes. The lock file lives at `.loki/state/gate-counter-.json.lock` and is released even on crash via the primitive's `finally` cleanup. --- ## Held-out spec evals (v7.28.0, default-on when reserved) Anti-reward-hacking for the checklist. Before the first verification, `checklist_select_heldout` (`autonomy/prd-checklist.sh`) deterministically reserves a slice of checklist items as held-out: `count = clamp(round(0.25 * N), 1, 5)` for checklists with `N >= 4` items (smaller checklists reserve nothing). Selection is reproducible, not random: items are ranked by `sha256(id)` and the first `count` are taken, then written once to `.loki/checklist/held-out.json` (idempotent: never reselected once chosen). Held-out item IDs are EXCLUDED from everything the build loop sees: the checklist summary, the visible counts, and the per-iteration checklist gate all omit them, so the build agent cannot tune to those specific acceptance checks. The completion council evaluates them only at the ship gate via `council_heldout_gate` (`autonomy/completion-council.sh`): a held-out item whose status is `failing` (and not waived) blocks completion exactly like any other critical failure. Each evaluation records a `heldout_eval` trust event with the verdict and pass/fail counts (no event is emitted when nothing is reserved). ```bash LOKI_HELDOUT_GATE=0 # opt out: the held-out gate never blocks completion. # Default is on (1), and the gate is inert anyway when # no held-out items were reserved (N < 4). ``` Honest limit: this protects against the PROMPT FEED, not against filesystem access. The reservation lives on disk at `.loki/checklist/held-out.json`; an adversarial agent with read access to the working tree can open that file and learn which items were held out. The guarantee is that held-out items are kept out of the build loop's own prompt context, not that they are sandboxed. --- ## Uncertainty-gated escalation (v7.19.2, default-on) When Loki is likely stuck or thrashing, it escalates proactively to the human via the existing PAUSE + notification + handoff machinery, rather than silently burning iterations until max-iterations. No new metacognition: the system reuses three proxy signals that already exist and escalates only when at least two of the three co-occur for N consecutive rounds. ### Trigger condition Three proxy signals are evaluated each iteration: - **Proxy 1 (no-change counter):** `consecutive_no_change` in council state.json reaches `LOKI_UNCERTAINTY_NOCHANGE_MIN` (default: `COUNCIL_STAGNATION_LIMIT - 1`, i.e. one below the circuit-breaker limit so escalation fires before the breaker ends the run). - **Proxy 2 (diff-hash oscillation):** the current iteration's combined diff hash matches a hash seen 2+ rounds back in a bounded ring buffer (A -> B -> A pattern). Detects oscillation/revert cycling; does not fire on the trivial immediate-repeat case which proxy 1 already covers. - **Proxy 3 (persistent council split):** the last `LOKI_UNCERTAINTY_SPLIT_ROUNDS` consecutive council verdicts are all REJECTED-with-at-least-one-approver (split verdict). Stale between council votes; fresh exactly when proxy 1 is hot, because proxy 1 hot forces a circuit-breaker vote that refreshes verdicts. Escalation fires when `hot_count >= 2` (at least two proxies hot simultaneously) for `LOKI_UNCERTAINTY_ROUNDS` consecutive rounds AND the episode has not already been escalated (one escalation per stuck-episode, with re-arm when co-occurrence clears). ### Action When the trigger condition is met, the run.sh action block: 1. Prints a loud terminal line with the opt-out env var. 2. Calls `write_structured_handoff "uncertainty_escalation"` (saves `.loki/memory/handoffs/.json` and `.md`). 3. Calls `notify_intervention_needed` with a structured reason string. 4. Writes a `.loki/signals/UNCERTAINTY_ESCALATION` marker file. 5. Touches `.loki/PAUSE`. ### Knobs ```bash LOKI_UNCERTAINTY_ESCALATION=0 # Disable entirely. Byte-identical when off: # zero reads, zero writes, no state file. # Default: 1 (enabled). Toggle value is 0/1, # not false/true. LOKI_UNCERTAINTY_ROUNDS=2 # Consecutive co-occurrence rounds required. # Recommended range 2-3. Default: 2. LOKI_UNCERTAINTY_NOCHANGE_MIN=N # Proxy 1 threshold. Unset = auto-computed as # COUNCIL_STAGNATION_LIMIT - 1 (floored at 1). LOKI_UNCERTAINTY_SPLIT_ROUNDS=2 # Proxy 3 trailing split-round run length. # Default: 2. ``` Configurable via `config.yaml` under `completion.uncertainty.*` (see `autonomy/config.example.yaml`). ### Honest limits - **Perpetual-mode = notify-only by default.** `AUTONOMY_MODE` defaults to `perpetual`. In perpetual mode the existing consumer (`check_human_intervention`) auto-clears PAUSE and continues. Escalation therefore degrades to a notification plus a handoff document; it does NOT halt the run. The terminal prints an explicit warning at the escalation site: "Perpetual mode: PAUSE will be auto-cleared; this is notify-only and will NOT halt the run." - **Proxy 2 is count-blind by origin.** It approximates oscillation with diff-hash recurrence-at-distance; it cannot distinguish a genuine revert from a coincidental identical tree state, and misses oscillation where the hash differs every round. - **Proxy 3 is stale between council votes.** Verdicts are only appended when the council actually votes (every `COUNCIL_CHECK_INTERVAL` or circuit-forced). In practice p3 is always fresh in the regime that matters (proxy 1 hot forces a vote), but it may lag by up to `COUNCIL_CHECK_INTERVAL` iterations otherwise. - **These are heuristics, not true metacognition.** The system does not know it is stuck; it infers stuckness from three correlated symptoms. A legitimately hard refactor that produces no net diff for several rounds while the council remains split can false-fire. Requiring >=2 co-occurring for N rounds reduces but does not eliminate false fires. The cost of a false fire is bounded: one notification + one handoff + one PAUSE (auto-cleared in perpetual), opt-out at the site. --- ## Guardrails Execution Modes - **Blocking**: Guardrail completes before agent starts (use for expensive operations) - **Parallel**: Guardrail runs with agent (use for fast checks, accept token loss risk) **Research:** Blind review + Devil's Advocate reduces false positives by 30% (CONSENSAGENT, 2025) --- ## Chain-of-Verification (CoVe) Protocol **Research:** arXiv 2309.11495 - "Chain-of-Verification Reduces Hallucination in Large Language Models" ### Core Insight Factored, decoupled verification mitigates error propagation. Each verification is computed independently without access to the original response, preventing the model from rationalizing its initial mistakes. ### The 4-Step CoVe Process ``` Step 1: DRAFT Step 2: PLAN Step 3: EXECUTE Step 4: REVISE +-------------+ +---------------+ +-----------------+ +----------------+ | Generate | ---> | Self-generate | --> | Answer each | -> | Incorporate | | initial | | verification | | question | | corrections | | response | | questions | | INDEPENDENTLY | | into final | +-------------+ +---------------+ +-----------------+ +----------------+ "What claims | (factored exec) did I make? | No access to What could be | original response wrong?" ``` ### Step-by-Step Implementation **Step 1: Draft Initial Response** ```yaml draft_phase: action: "Generate initial code/response" model: "sonnet" # Fast drafting output: "baseline_response" ``` **Step 2: Plan Verification Questions** ```yaml verification_planning: prompt: | Review the response above. Generate verification questions: 1. What factual claims did I make? 2. What assumptions did I rely on? 3. What could be incorrect or incomplete? 4. What edge cases did I miss? output: "verification_questions[]" ``` **Step 3: Execute Verifications INDEPENDENTLY (Critical)** ```yaml factored_execution: critical: "Each verification runs in isolation" rule: "Verifier has NO access to original response" # Launch in parallel - each is independent verifications: - question: "Does the function handle null inputs?" context: "Function signature and spec only" # NOT the implementation verifier: "sonnet" - question: "Is the SQL query injection-safe?" context: "Query requirements only" verifier: "sonnet" - question: "Does the API match the documented spec?" context: "API spec only" verifier: "sonnet" ``` **Step 4: Generate Final Verified Response** ```yaml revision_phase: inputs: - original_response - verification_results[] action: "Revise response incorporating all corrections" output: "verified_response" ``` ### Factor+Revise Variant (Longform Code Generation) For complex code generation, use the enhanced Factor+Revise pattern. The key difference from basic Factored execution is an **explicit cross-check step** where the model compares original claims against verification results before revision. ```yaml factor_revise_pattern: step_1_draft: action: "Generate complete implementation" output: "draft_code" step_2_factor: action: "Decompose into verifiable claims" outputs: - "Function X handles error case Y" - "Loop invariant: Z holds at each iteration" - "API call returns type T" - "Memory is freed in all paths" step_3_independent_verify: # CRITICAL: Each runs with ONLY the claim + minimal context # No access to full draft code parallel_tasks: - verify: "Function X handles error case Y" context: "Function signature + error spec" result: "PASS|FAIL + evidence" - verify: "Loop invariant holds" context: "Loop structure only" result: "PASS|FAIL + evidence" step_3b_cross_check: # KEY DIFFERENCE: Explicit consistency check before revision action: "Compare original claims against verification results" prompt: "Identify which facts from the draft are CONSISTENT vs INCONSISTENT with verifications" output: "consistency_report" step_4_revise: inputs: [draft_code, verification_results, consistency_report] action: "Discard inconsistent facts, use consistent facts to regenerate" output: "verified_code" ``` ### Why Factored Execution Matters The paper tested 4 execution variants: - **Joint**: Questions and answers in one prompt (worst - repeats hallucinations) - **2-Step**: Separate prompts for questions vs answers (better) - **Factored**: Each question answered separately (recommended) - **Factor+Revise**: Factored + explicit cross-check step (best for longform) Without factoring (naive verification): ``` Model: "Here's the code" Model: "Let me check my code... looks correct!" # Confirmation bias ``` With factored verification: ``` Model: "Here's the code" Model: "Question: Does function handle nulls?" [New context, no code visible] Model: "Given a function that takes X, null handling requires..." # Independent reasoning ``` **Key principle from the paper:** The verifier cannot see the original response, only the verification question and minimal context. This prevents rationalization of errors and breaks the chain of hallucination propagation. ### CoVe Integration with Blind Review CoVe operates BEFORE blind review as a self-correction step: ``` Developer Code --> CoVe (self-verification) --> Blind Review (3 parallel) | | Catches errors early Catches remaining via factored checking issues independently ``` **Combined workflow:** ```yaml quality_pipeline: phase_1_cove: # Developer runs CoVe on their own code draft: "Initial implementation" verify: "Self-generated questions, factored execution" revise: "Corrected implementation" phase_2_blind_review: # 3 independent reviewers (no access to CoVe results) reviewers: - focus: "correctness" - focus: "security" - focus: "performance" # Reviewers see verified code but don't know what was corrected phase_3_aggregate: if: "unanimous approval" then: "Devil's Advocate review" ``` ### Metrics Track CoVe effectiveness: ``` .loki/metrics/cove/ +-- corrections.json # Issues caught by CoVe before review +-- false_positives.json # CoVe flags that were actually correct +-- review_reduction.json # Reviewer findings before/after CoVe adoption ``` --- ## Velocity-Quality Feedback Loop (CRITICAL) **Research from arXiv 2511.04427v2 - empirical study of 807 repositories.** ### Key Findings | Metric | Finding | Implication | |--------|---------|-------------| | Initial Velocity | +281% lines added | Impressive but TRANSIENT | | Quality Degradation | +30% static warnings, +41% complexity | PERSISTENT problem | | Cancellation Point | 3.28x complexity OR 4.94x warnings | Completely negates velocity gains | ### The Trap to Avoid ``` Initial excitement -> Velocity spike -> Quality degradation accumulates | v Complexity cancels velocity gains | v Frustration -> Abandonment cycle ``` **CRITICAL RULE:** Every velocity gain MUST be accompanied by quality verification. ### Mandatory Quality Checks (Per Task) ```yaml velocity_quality_balance: before_commit: - static_analysis: "Run ESLint/Pylint - warnings must not increase" - complexity_check: "Cyclomatic complexity must not increase >10%" - test_suite: "Tests must pass (coverage % not measured in this release)" thresholds: max_new_warnings: 0 # Zero tolerance for new warnings max_complexity_increase: 10% # Per file, per commit coverage_target: 80% # Target/threshold only; not measured this release (Fix A follow-up) if_threshold_violated: action: "BLOCK commit, fix before proceeding" reason: "Velocity gains without quality are net negative" ``` ### Metrics to Track ``` .loki/metrics/quality/ +-- warnings.json # Static analysis warning count over time +-- complexity.json # Cyclomatic complexity per file +-- velocity.json # Lines added/commits per hour +-- ratio.json # Quality/Velocity ratio (must stay positive) ``` --- ## Specialist Review Pool (v5.30.0) 6 named expert reviewers. Select 3 per review based on change type. **Inspired by:** Compound Engineering Plugin's 14 named review agents -- specialized expertise catches more issues than generic reviewers. | Specialist | Focus Area | Trigger Keywords | |-----------|-----------|-----------------| | **security-sentinel** | OWASP Top 10, injection, auth, secrets, input validation | auth, login, password, token, api, sql, query, cookie, cors, csrf | | **performance-oracle** | N+1 queries, memory leaks, caching, bundle size, lazy loading | database, query, cache, render, loop, fetch, load, index, join, pool | | **architecture-strategist** | SOLID, coupling, cohesion, patterns, abstraction, dependency direction | *(always included -- design quality affects everything)* | | **test-coverage-auditor** | Missing tests, edge cases, error paths, boundary conditions | test, spec, coverage, assert, mock, fixture, expect, describe | | **dependency-analyst** | Outdated packages, CVEs, bloat, unused deps, license issues | package, import, require, dependency, npm, pip, yarn, lock | | **legacy-healing-auditor** | Behavioral preservation, friction safety, institutional knowledge | legacy, heal, migrate, cobol, fortran, refactor, modernize, deprecat | ### Selection Rules 1. **architecture-strategist** is ALWAYS one of the 3 slots 2. Score remaining 4 specialists by counting trigger keyword matches in the diff content and changed file names 3. Top 2 scoring specialists fill the remaining slots 4. **Tie-breaker priority:** security-sentinel > test-coverage-auditor > performance-oracle > dependency-analyst 5. **No triggers match at all:** Default to security-sentinel + test-coverage-auditor ### Dispatch Pattern Launch all 3 in ONE message. Each reviewer sees ONLY the diff -- NOT other reviewers' findings (blind review preserved). ```python # ALWAYS launch all 3 in ONE message (parallel, blind) Task( model="sonnet", description="Review: Architecture Strategist", prompt="""You are Architecture Strategist. Your SOLE focus is design quality. Review ONLY for: SOLID violations, excessive coupling, wrong patterns, missing abstractions, dependency direction issues, god classes/functions. Files changed: {files} Diff: {diff} Output format: VERDICT: PASS or FAIL FINDINGS: - [severity] description (file:line) Severity levels: Critical, High, Medium, Low""" ) Task( model="sonnet", description="Review: Security Sentinel", prompt="""You are Security Sentinel. Your SOLE focus is security vulnerabilities. Review ONLY for: injection (SQL, XSS, command, template), auth bypass, secrets in code, missing input validation, OWASP Top 10, insecure defaults. Files changed: {files} Diff: {diff} Output format: VERDICT: PASS or FAIL FINDINGS: - [severity] description (file:line) Severity levels: Critical, High, Medium, Low""" ) Task( model="sonnet", description="Review: {3rd_selected_specialist}", prompt="""You are {specialist_name}. Your SOLE focus is {focus_area}. Review ONLY for: {specific_checks} Files changed: {files} Diff: {diff} Output format: VERDICT: PASS or FAIL FINDINGS: - [severity] description (file:line) Severity levels: Critical, High, Medium, Low""" ) ``` ### Rules (unchanged from blind review) - ALWAYS use sonnet for reviews (balanced quality/cost) - NEVER aggregate before all 3 complete - ALWAYS re-run ALL 3 after fixes - If unanimous PASS -> run Devil's Advocate (anti-sycophancy check) - Critical/High findings = BLOCK (must fix before merge) - Medium findings = TODO (track but don't block) - Low findings = informational only --- ## Two-Stage Review Protocol **Source:** Superpowers (obra) - 35K+ stars GitHub project **CRITICAL: Never mix spec compliance and code quality review. They are separate stages.** ### Why Separate Stages Matter Mixing stages causes these problems: - **"Technically correct but wrong feature"** - Code is clean, well-tested, maintainable, but doesn't implement what the spec requires - **Spec drift goes undetected** - Quality reviewers approve beautiful code that solves the wrong problem - **False confidence** - "3 reviewers approved" means nothing if none checked spec compliance ### Stage 1: Spec Compliance Review **Question:** "Does this code implement what the spec requires?" ``` Review this implementation against the specification. Specification: {paste_spec_or_requirements} Implementation: {paste_code_or_diff} Check ONLY the following: 1. Does the code implement ALL required features from the spec? 2. Does the code implement ONLY what the spec requires (no scope creep)? 3. Are edge cases from the spec handled? 4. Do the tests verify spec requirements? DO NOT review code quality, style, or maintainability. Output: PASS/FAIL with specific spec violations listed. ``` **Stage 1 must PASS before proceeding to Stage 2.** ### Stage 2: Code Quality Review **Question:** "Is this code well-written, maintainable, secure?" ``` Review this code for quality. Spec compliance has already been verified. Code: {paste_code_or_diff} Check the following: 1. Is the code readable and maintainable? 2. Are there security vulnerabilities? 3. Is error handling appropriate? 4. Are there performance concerns? 5. Does it follow project conventions? DO NOT verify spec compliance (already done). Output: PASS/FAIL with specific issues listed by severity. ``` ### Implementation in Loki Mode ```yaml two_stage_review: stage_1_spec: reviewer_count: 1 # Spec compliance is objective model: "sonnet" must_pass: true blocks: "stage_2" stage_2_quality: reviewer_count: 3 # Quality is subjective, use blind review model: "sonnet" must_pass: true follows: "stage_1" anti_sycophancy: true # Devil's advocate on unanimous on_stage_1_fail: action: "Return to implementation, DO NOT proceed to Stage 2" reason: "Quality review of wrong feature wastes resources" on_stage_2_fail: action: "Fix quality issues, re-run Stage 2 only" reason: "Spec compliance already verified" ``` ### Common Anti-Pattern ``` # WRONG - Mixed review Task(prompt="Review for correctness, security, performance, and spec compliance...") # RIGHT - Separate stages Task(prompt="Stage 1: Check spec compliance ONLY...") # Wait for pass Task(prompt="Stage 2: Check code quality ONLY...") ``` --- ## Severity-Based Blocking | Severity | Action | |----------|--------| | Critical | BLOCK - fix immediately | | High | BLOCK - fix before commit | | Medium | Advisory - TODO comment, fix recommended | | Low | TODO comment, fix later | | Cosmetic | Note, optional fix | See `references/quality-control.md` for complete details. --- ## Scale Considerations > **Source:** [Cursor Scaling Learnings](../references/cursor-learnings.md) - integrators became bottlenecks at high agent counts ### Review Intensity Scaling At high agent counts, full 3-reviewer blind review for every change creates bottlenecks. ```yaml review_scaling: low_scale: # <10 agents all_changes: "Full 3-reviewer blind review" rationale: "Quality critical, throughput acceptable" medium_scale: # 10-50 agents high_risk: "Full 3-reviewer blind review" medium_risk: "2-reviewer review" low_risk: "1 reviewer + automated checks" rationale: "Balance quality and throughput" high_scale: # 50+ agents critical_changes: "Full 3-reviewer blind review" standard_changes: "Automated checks + spot review" trivial_changes: "Automated checks only" rationale: "Trust workers, avoid bottlenecks" risk_classification: high_risk: - Security-related changes - Authentication/authorization - Payment processing - Data migrations - API breaking changes medium_risk: - New features - Business logic changes - Database schema changes low_risk: - Bug fixes with tests - Refactoring with no behavior change - Documentation - Dependency updates (minor) ``` ### Judge Agent Integration Use judge agents to determine when full review is needed: ```yaml judge_review_decision: inputs: - change_type: "feature|bugfix|refactor|docs" - files_changed: 5 - lines_changed: 120 - test_coverage: 85% - static_analysis: "0 new warnings" output: review_level: "full|partial|automated" rationale: "Medium-risk feature with good coverage" ``` ### Cursor's Key Learning > "Dedicated integrator/reviewer roles created more bottlenecks than they solved. Workers were already capable of handling conflicts themselves." **Implication:** At scale, trust automated checks and worker judgment. Reserve full review for high-risk changes only.