# Bundled tools `tools/` holds 32 standalone scripts that ship with loki-mode. They read artifacts a run leaves behind and answer one question each. None of them starts a build or spends money on inference. Two qualifications to that, because "read-only" is not true of all 32: `probe-model-catalog.py` makes network requests to provider documentation pages, and several write files (`gate-init.py`, `cost-history.py record`, `baseline-pin.py set`, `receipt-export.py --out`, `gate-log.py record`). Where a tool asserts read-only in its own help, this page repeats the assertion; `preflight.sh`, `gate-status.py`, `cost-attribute.py` and `run-replay.py` do. Every command and every output on this page was executed against loki-mode 9.8.1 before being written down. Where a tool reports UNKNOWN or refuses to answer, that real output is shown rather than a prettier invented one. ## Running them Most of these files are not marked executable, so invoke them through the interpreter rather than as `./tools/x.py`: ``` python3 tools/.py [args] bash tools/.sh [args] ``` Installed from npm, they live under the package root. `python3 tools/tool-index.py` lists what shipped. **Where you run them from matters.** The receipt and gate tools check drift and the working tree against the **current directory**, and several read or write their state files relative to cwd (`.loki-policy.json`, `.loki/cost-history.jsonl`, `.loki/gate-log.jsonl`, `.loki/baseline.json`). Run those from the workspace the receipt describes, not from the loki-mode package directory. The examples below show this as an explicit `cd`, with the tool referenced through a `$LOKI` variable pointing at the package root: ``` LOKI=/path/to/loki-mode # or the npm package root cd /path/to/your/workspace python3 $LOKI/tools/receipt-attest.py .loki/proofs/good/proof.json ``` Tools that take a workspace as a positional argument and read nothing relative to cwd (`cost-attribute.py`, `run-replay.py`, `receipt-find.py`, `estimate-run.py`, `model-advisor.py`, `preflight.sh`) can be run from anywhere. ## The exit-code convention The tools that compose in CI share one convention. It is enforced by `tests/test_tool_exit_contract.py`, which walks the real directory rather than testing tools one at a time. Enforcement covers exactly **24** of them: that test globs `tools/*.py` (30 files) and excludes six by name. The two shell tools, `preflight.sh` and `verify-demo.sh`, are never globbed by it at all; they follow the convention by observation here, not by enforcement. | Code | Meaning | |---|---| | 0 | Checked, and it passed | | 1 | Checked, and it FAILED | | 2 | Could NOT be checked; no policy configured, or no measurement exists | | 3 | Nothing to check (empty) | | 64 | Usage error | | 66 | Input missing (the path does not exist) | Two properties are worth knowing before you build a gate on these. **2 is not a pass.** A gate that cannot evaluate its axis exits 2, never 0. If cost was never measured, `cost-guard.py` refuses to say the run was within budget, because it does not know. Treat any code `>= 2` as "do not merge on this axis". **Gates and advisors differ deliberately.** A tool named `*-gate`, `*-guard`, or `gate-*`, plus `ci-gate.py`, `receipt-attest.py` and `receipt-bundle.py`, is a gate: its exit code is a verdict a machine branches on, and it may never exit 0 while its own output says it could not evaluate. An advisor such as `model-advisor.py` answering "NO BASIS" has answered the question honestly and exits 0. Forcing advisors non-zero would train operators to ignore them. Six Python tools are excluded from the convention by name, and are described under [Internal and maintenance](#internal-and-maintenance) below. They are benchmark harnesses and repo-maintenance scripts that predate it. One of the six, `hybrid_search.py`, is genuinely user-facing and is documented under [Diagnose a run](#diagnose-a-run) instead. For the exit codes of the `loki` CLI itself (`loki start`, `loki verify`, `loki ci`, `loki doctor`), see [exit-codes.md](./exit-codes.md). That is a separate contract; this page does not restate it. --- ## Verify a receipt An Evidence Receipt (`proof.json`) is what a run leaves behind to describe what it did. These tools re-check one. ### `verify-demo.sh` **Answers:** what does the verification chain actually do, without paying for a build? It creates a scratch git repo in a tempdir, generates a real receipt over it, verifies it, then tampers with the receipt and verifies again so you can see the failure. Synthetic receipts, real verifier: every verdict shown is printed by the actual tool. ``` $ bash tools/verify-demo.sh ... STEP 3 of 3 -- tamper with the receipt, then re-verify edited facts.git.diff.count: 1 -> 999 (claims work that is not there) $ tools/receipt-attest.py .loki/proofs/tampered/proof.json FAILED -- drift, integrity did not pass here; receipt is UNSIGNED, so origin rests on its generator integrity FAILED hash mismatch: recorded 34175ce3..., computed 64eb7c52... -- proof.json was edited after it was written -> exit 1. Caught, with the reason printed above. ``` Pass `--keep` to leave the scratch directory in place. Exits 0 when the whole chain behaves as expected, so it doubles as a smoke test. ### `receipt-attest.py` **Answers:** does this receipt hold up, checked from here? Scores five axes independently (integrity, drift, headline, cost, tree) plus signature state, and never collapses "could not check" into either pass or fail. ``` $ cd /path/to/your/workspace $ python3 $LOKI/tools/receipt-attest.py .loki/proofs/good/proof.json VERIFIED -- every axis checked here passed; receipt is UNSIGNED, so origin rests on its generator receipt proof.json sha256 5854c7a5271d567f915e495b81644551f7418d485422ba1e9dcc3f62f364edc8 checked from /private/var/folders/.../loki-doccheck2-pdtkmezm integrity VERIFIED drift VERIFIED headline VERIFIED cost VERIFIED tree VERIFIED signature UNSIGNED (unsigned) the receipt carries no gpg signature, so its origin rests on the generator that produced it, not on cryptographic proof ``` (the sha256 and path are specific to that receipt; yours will differ) Exit 0 all axes verified, 1 something FAILED, 2 something was UNVERIFIABLE, 64 usage error. **Important:** the drift and tree axes are checked against the **current working directory**, and this tool has no flag to point them elsewhere. Run it from the repository the receipt describes. Run it from anywhere else and you get a truthful but useless FAILED, because the tree genuinely does not match: ``` $ cd /path/to/loki-mode # the WRONG directory for this receipt $ python3 tools/receipt-attest.py /tmp/other-repo/.loki/proofs/good/proof.json FAILED -- drift, tree did not pass here drift FAILED diff drift: the receipt recorded 1 files / +2 / -0, the repository now has 4178 files / +878921 / -0 ``` `receipt-bundle.py` and `receipt-export.py` accept `--repo-dir` for exactly this case; `receipt-attest.py` does not. ### `receipt-bundle.py` **Answers:** do ALL the receipts under this workspace hold up, as one audit trail? A bundle is only as good as its weakest receipt, and failures are counted, never dropped. ``` $ python3 tools/receipt-bundle.py /tmp/ws Receipt bundle: /tmp/ws VERIFIED /tmp/ws/.loki/proofs/good/proof.json total cost UNKNOWN (0 of 1 receipts measured cost) VERIFIED -- all 1 receipts verified. total cost UNKNOWN (0 of 1 receipts measured cost) ``` Takes `--repo-dir` to name the repository the receipts are re-checked against. ### `receipt-export.py` **Answers:** can I hand a third party one self-describing evidence file? Writes every receipt under a workspace into a single record, and includes an explicit "what this export does NOT prove" section rather than letting the reader assume. ``` $ python3 tools/receipt-export.py /tmp/ws --out /tmp/ws/evidence.json ... What this export does NOT prove: - An UNSIGNED receipt proves INTEGRITY, not ORIGIN: the recorded bytes were not edited after they were hashed, but nothing here shows WHICH machine produced them. ``` `--force` overwrites an existing `--out`. Exits 1 when any receipt fails. ### `receipt-diff.py` **Answers:** what actually changed between two runs? Reports UNKNOWN where a value was never measured, rather than printing a misleading zero. ``` $ python3 tools/receipt-diff.py a/proof.json b/proof.json Evidence Receipt diff cost UNKNOWN -> UNKNOWN delta UNKNOWN cache hit ratio UNKNOWN -> UNKNOWN delta UNKNOWN iterations 0 -> 0 delta +0 duration 0 -> 0 delta +0s gates no verdict changes NOT COMPARABLE (reported UNKNOWN, not zero): cost: cost was not measured in a/proof.json and b/proof.json ``` It refuses to diff a receipt that fails integrity, rather than comparing numbers that were edited after the fact: ``` $ python3 tools/receipt-diff.py good/proof.json tampered/proof.json REFUSED: receipt failed integrity verification: tampered/proof.json hash mismatch: recorded 34175ce3..., computed 64eb7c52... -- proof.json was edited after it was written ``` Exit 2, because nothing could be checked. ### `receipt-find.py` **Answers:** which receipts, out of a workspace full of them, does an auditor actually need? Filters by measurable criteria only. ``` $ python3 tools/receipt-find.py /tmp/ws /tmp/ws/.loki/proofs/good/proof.json [-] /tmp/ws/.loki/proofs/tampered/proof.json [-] 2 of 2 receipt(s) matched. Filters applied: none (no filter applied). ``` Filters: `--min-usd`, `--max-usd`, `--failed-only`, `--since YYYY-MM-DD`. ### `signing-status.py` **Answers:** can this machine produce SIGNED receipts? It proves the answer by attempting a sign-and-verify round trip rather than assuming from config. ``` $ python3 tools/signing-status.py Receipt signing: UNSIGNED receipts prove integrity but NOT origin gpg installed: /opt/homebrew/bin/gpg LOKI_PROOF_GPG_KEY: not set sign+verify proof: not proven Why: LOKI_PROOF_GPG_KEY is not set Nothing is broken. Signing is opt-in and off. Turn it on to prove a receipt came from you and not merely that its bytes are intact. Next: gpg --list-secret-keys --keyid-format=long # then: export LOKI_PROOF_GPG_KEY= ``` Exit 2 here: signing is off, so the question could not be answered affirmatively. --- ## Set up a merge gate One exit code a CI job branches on, plus the tools that configure, render, and track it. ### `gate-status.py` **Answers:** is this repo's merge gate actually set up and working? Start here. One screen, read-only. ``` $ python3 tools/gate-status.py /tmp/ws Merge gate status for /tmp/ws (read-only: starts nothing, spends nothing, contacts no provider) COMPONENT STATE DETAIL policy OK /tmp/ws/.loki-policy.json is valid and enforces: --require-receipt baseline PROBLEM NO BASELINE: no baseline pinned. Pin one with: tools/baseline-pin.py set signing UNKNOWN signing is opt-in and off (LOKI_PROOF_GPG_KEY unset) cost_history UNKNOWN no measured cost history; record a run first. would_run OK yes: ci-gate would enforce the loaded policy on the next run GATE: UNKNOWN -- the merge gate cannot be fully verified from here, so it is not known to be working ``` Exit 2: a gate that is not known to be working is not reported as working. ### `gate-init.py` **Answers:** what policy file and CI snippet turn the gate on? Scaffolds them from measured history, and refuses to invent a cost ceiling it has no basis for. ``` $ cd /path/to/your/workspace $ python3 $LOKI/tools/gate-init.py . wrote .loki-policy.json { "require_receipt": true } validated by policy-load.py; ci-gate args: --require-receipt NO CEILING WAS SET: no measured runs in this workspace's cost history. The max_usd key is ABSENT rather than guessed. You must choose a ceiling and add it, for example: "max_usd": 5.00 ``` `--print-workflow` also prints a starting-point CI snippet. `--force` overwrites. ### `ci-gate.py` **Answers:** does this run pass every configured merge policy? This is the one a CI job calls. ``` $ cd /path/to/your/workspace $ python3 $LOKI/tools/ci-gate.py . --require-receipt POLICY STATE DETAIL receipt PASS VERIFIED GATE: PASS -- 1 of 1 policies passed ``` And on a workspace whose receipt does not hold up: ``` $ python3 $LOKI/tools/ci-gate.py . --require-receipt POLICY STATE DETAIL receipt FAIL FAILED GATE: FAIL -- 0 of 1 policies passed ``` Exit 1. `--max-usd N` adds the cost ceiling. `--json` emits the verdict that the four renderers below consume on stdin. The `--require-receipt` policy runs `receipt-attest.py` underneath, so it inherits the cwd dependence described above. Run it from the workspace; run it from elsewhere and it FAILs on drift even for a sound receipt. ### `policy-load.py` **Answers:** what does my version-controlled policy file actually enforce? Keeps the policy reviewable in git instead of buried in a CI flag. ``` $ cd /path/to/your/workspace # where .loki-policy.json lives $ python3 $LOKI/tools/policy-load.py --as-args --require-receipt ``` With no policy file present it is explicit about what that means: ``` $ python3 $LOKI/tools/policy-load.py policy-load: no policy file at .loki-policy.json: a gate with no policy enforces nothing ``` Exit 1. `--file` names a different path; `--json` emits the validated policy. ### `policy-diff.py` **Answers:** does this policy edit tighten the gate or weaken it? A loosening looks like any other line in a diff, so this classifies by safety direction. ``` $ python3 tools/policy-diff.py old-policy.json new-policy.json WEAKENS: max_usd: 5.0 -> 20.0 1 weakening(s), 0 unknown direction, 1 change(s) total ``` `--fail-on-weaken` makes that a blocking review step: ``` $ python3 tools/policy-diff.py old-policy.json new-policy.json --fail-on-weaken WEAKENS: max_usd: 5.0 -> 20.0 policy-diff: 1 weakening(s) require an explicit human ack ``` Exit 1. ### `gate-report.py` **Answers:** how do I show this verdict in CI-native form? Reads a `ci-gate --json` verdict on stdin and re-renders it. It never invents a verdict. ```text $ cd /path/to/your/workspace $ python3 $LOKI/tools/ci-gate.py . --require-receipt --json | python3 $LOKI/tools/gate-report.py --format markdown ## Merge gate: PASS | Policy | State | Detail | | --- | --- | --- | | receipt | PASS | VERIFIED | PASS -- 1 of 1 policies passed ``` (the output is markdown; it is indented here only so it does not render as a heading on this page) `--format github` emits workflow annotations instead. Shown here on a FAILING verdict, since a passing gate has no annotations worth reading: ``` ::error title=gate: receipt (FAIL)::FAILED ::error title=merge gate FAIL::0 of 1 policies passed ``` Formats: `markdown` (step summary), `github` (annotations), `text`. `--file` attaches a path to github annotations, omitted when not given because an annotation on the wrong file is worse than none. ### `gate-explain.py` **Answers:** the gate failed, so what do I run next? Turns a verdict into an actionable command. On a FAILING verdict: ``` $ python3 $LOKI/tools/ci-gate.py . --require-receipt --json | python3 $LOKI/tools/gate-explain.py POLICY: receipt [FAIL] CHECKED: the gate checked this and it failed FOUND: FAILED NEXT: attestation was required and did not hold. Read the attestation's own per-axis states before changing anything: python3 tools/receipt-attest.py /.loki/proofs/*/proof.json --json GATE: FAIL -- 0 of 1 policies passed ``` And on a passing one, it says so rather than manufacturing advice: ``` POLICY: receipt [PASS] CHECKED: the gate checked this and it passed FOUND: VERIFIED NEXT: nothing to do GATE: PASS -- 1 of 1 policies passed ``` ### `gate-badge.py` **Answers:** what is the live gate state, as a README badge? Emits shields.io endpoint JSON without laundering a failure into a green badge. ``` $ python3 $LOKI/tools/ci-gate.py . --require-receipt --json | python3 $LOKI/tools/gate-badge.py {"schemaVersion": 1, "label": "gate", "message": "passing", "color": "brightgreen"} ``` On a failing verdict the same command emits, and the badge exit code follows the verdict rather than always succeeding: ``` {"schemaVersion": 1, "label": "gate", "message": "failing", "color": "red"} ``` `--label` changes the badge label. ### `gate-log.py` **Answers:** one verdict is a fact; what is the pattern across a hundred? A gate that was never recorded is not a gate that never blocked. ``` $ cd /path/to/your/workspace $ python3 $LOKI/tools/ci-gate.py . --require-receipt --json | python3 $LOKI/tools/gate-log.py record gate-log: recorded pass to .loki/gate-log.jsonl $ python3 $LOKI/tools/gate-log.py report gate-log: 1 record(s) pass 1 fail 0 unevaluable (NOT a pass) 0 corrupt 0 most failing policy: none -- no FAIL in 1 readable record(s) trend: UNKNOWN -- fewer than 2 readable records (1) ``` Note that `unevaluable` is counted separately and never folded into `pass`. With no log at all it reports UNKNOWN, not a clean history: ``` $ python3 $LOKI/tools/gate-log.py report gate-log: no log at .loki/gate-log.jsonl -- UNKNOWN, not a clean history. A gate that was never recorded is not a gate that never blocked. ``` Exit 66. The log path is relative to cwd, so `record` and `report` must run from the same directory. --- ## Govern cost Every tool here reports UNKNOWN when cost was not measured. None of them substitutes zero for an absent measurement. ### `preflight.sh` **Answers:** will this run succeed, and what will it cost? Answered BEFORE the run. Read-only. ``` $ bash tools/preflight.sh /tmp/ws Loki preflight -- /tmp/ws Read-only: starts nothing, spends nothing, contacts no provider. Provider OK auto-detection would select: claude Environment OK doctor: 12 checks passed, 1 warning(s) Saved state no resumable run found Projected cost UNKNOWN no measured, priced iteration in this workspace no iteration records found in this workspace: there is NO history to project from, so no cost is estimated (not $0.00) ======================================== VERDICT: READY ``` `--iterations N` projects over N iterations; without it the total reads UNKNOWN rather than being guessed. ### `cost-guard.py` **Answers:** did this run's cost regress past a budget policy? A cost gate for CI. ``` $ python3 tools/cost-guard.py /tmp/ws --max-usd 5 CANNOT EVALUATE: cost is UNMEASURED for /tmp/ws -- no efficiency record carried an observed cost or token count. Unmeasured is not within budget: this gate cannot say whether the run complied, so it reports no verdict rather than a green one. ``` Exit 2. This is the convention's sharpest case: an unmeasured run is not a passing run. `--baseline` plus `--max-increase-pct` gates on relative growth instead of an absolute ceiling. ### `token-guard.py` **Answers:** did token usage regress? A provider-independent work ceiling, for when you want a limit that does not move with pricing. ``` $ python3 tools/token-guard.py /tmp/ws --max-output-tokens 100000 CANNOT EVALUATE: tokens are UNMEASURED for /tmp/ws -- no efficiency record carried an observed token count. Unmeasured is not within budget: this gate cannot say whether the run complied, so it reports no verdict rather than a green one. ``` Exit 2. `--max-output-tokens` bounds output alone (the work signal); `--max-total-tokens` bounds input + output + cache reads, which cache reads dominate. ### `baseline-pin.py` **Answers:** which run is THE cost baseline? Pin one, resolve it later for `cost-guard --baseline`. ``` $ cd /path/to/your/workspace $ python3 $LOKI/tools/baseline-pin.py set . REFUSED TO PIN: receipt ./.loki/proofs/good/proof.json records NO MEASURED COST, so it must not become a baseline: a percentage increase against an unmeasured number is undefined. Unmeasured is not $0.00. Fix the cost instrumentation for that run, then pin it. ``` Exit 1. Subcommands: `set` pins the newest receipt in a workspace, `show` reports what is pinned and whether it is intact, `path` prints the pinned proof path for feeding to `--baseline`. ### `cost-history.py` **Answers:** what is the cost trend across MANY runs? Unmeasured runs are stored as null and excluded from the trend, never counted as zero. ``` $ cd /path/to/your/workspace $ python3 $LOKI/tools/cost-history.py record . recorded /private/var/folders/.../loki-doccheck-cayf1yk6: UNMEASURED (stored as null, excluded from the trend) $ python3 $LOKI/tools/cost-history.py report 1 run(s), 0 measured NO TREND: 1 run(s) recorded but none carried a measured cost; there is nothing to trend. Unmeasured runs are kept as null, not 0. ``` ### `estimate-run.py` **Answers:** what is this run likely to cost, and on what basis? Projects from measured history only. ``` $ python3 tools/estimate-run.py /tmp/ws --iterations 5 Run cost ESTIMATE -- /tmp/ws NO BASIS: no measured, priced iteration to project from. Cost per iteration: UNKNOWN Projected cost: UNKNOWN Records found: 0 measured: 0 priced: 0 Basis model(s): not recorded Model now: not pinned - no iteration records found in this workspace: there is NO history to project from, so no cost is estimated (not $0.00) ``` Exit 0: "no basis" is an honest answer to the question asked, and no gate consumes this exit code. ### `cost-attribute.py` **Answers:** where did a run's cost and time actually GO, stage by stage? ``` $ python3 tools/cost-attribute.py /tmp/ws NO DATA: no events at /tmp/ws/.loki/events.jsonl -- pass the workspace directory of a completed run ``` Exit 66. Needs `.loki/events.jsonl` from a completed run. ### `model-advisor.py` **Answers:** would a cheaper model have done this job, and what would it have saved? An advisor, deliberately not a gate. ``` $ python3 tools/model-advisor.py /tmp/ws Model cost advisor -- /tmp/ws NO BASIS: no measured, priced iteration in this workspace. Model used: UNKNOWN Measured cost: UNKNOWN Recommendation: NONE -- there is no measured basis Projected saving: UNKNOWN Records found: 0 measured: 0 priced: 0 Cited external benchmark (SWE-bench verified) -- NOT a measurement of your workload: MiniMax M2.5 (open weights) score 75.8 cost $36.64 Claude Opus 4.6 score 75.6 cost $275.76 the harness itself was worth about 3.4 points on an identical model equal-or-better score at roughly 7.5x lower cost ON THAT BENCHMARK. It is not a measurement of your workload and does not predict that a cheaper model would complete YOUR task ``` Exit 0 with no basis, by design. `tests/test_tool_exit_contract.py` pins this judgement so a future reader does not "fix" it: forcing an advisor non-zero trains operators to ignore a failing advisor. --- ## Diagnose a run ### `run-replay.py` **Answers:** what did a completed run actually do, iteration by iteration? Reconstructs from artifacts; starts nothing, spends nothing. ``` $ python3 tools/run-replay.py /tmp/ws no events at /tmp/ws/.loki/events.jsonl -- pass the workspace directory of a completed run ``` Exit 66. Point it at the workspace of a run that finished. ### `tool-index.py` **Answers:** what tools shipped, and which can a user reach? ``` $ python3 tools/tool-index.py loki tools baseline-pin.py Pin one run as THE cost baseline, then resolve it later. ci-gate.py One exit code for EVERY merge policy. The gate a CI job actually calls. cost-attribute.py Where did a run's cost and time actually GO, stage by stage. ... ``` `--json` adds a `shipped` flag per tool. `--tools-dir` points at a different directory. ### `hybrid_search.py` **Answers:** where in this codebase is X? Merges lexical (grep/ripgrep) and semantic search under a token budget. ``` $ python3 tools/hybrid_search.py council_should_stop --grep-only --top 3 hybrid search: 'council_should_stop' [grep-only] (ripgrep not found, using grep) budget: 3000 tokens, 3 result(s) [1] tests/test-council-convergence-floor.sh:223 (match: grep, score: 0.016393) # The HARD floor in council_should_stop (ITERATION_COUNT < floor -> not allowed to ``` `--semantic-only` requires the codebase index (see `index-codebase.py` below, which currently does not run on Python 3.14). `--grep-only` needs nothing. --- ## Internal and maintenance Five of the tools below are excluded from the exit-code convention **by name** in `tests/test_tool_exit_contract.py` (the sixth named exclusion, `hybrid_search.py`, is user-facing and appears under [Diagnose a run](#diagnose-a-run)). They are benchmark harnesses and repo maintenance scripts, not CI-composable gates. `verify-demo.sh` is listed here for a different reason: it is user-facing and is documented in full under [Verify a receipt](#verify-a-receipt), but as a shell script it is never globbed by the contract test, so nothing enforces its exit codes. Several of these carry version-stamped internal release notes as their docstrings rather than user-facing descriptions, so what follows is a description of observed behaviour, not a quoted one. ### `bench_memory_retrieval.py` Memory retrieval cold-start benchmark. Seeds N episodes, performs cold retrievals, reports percentiles against a threshold. ``` $ python3 tools/bench_memory_retrieval.py --episodes 50 --runs 5 --json { "episodes_seeded": 50, "runs": 5, "seed_ms": 47.5, "p50_ms": 4.7, "p95_ms": 5.2, "p99_ms": 5.2, "threshold_ms": 500.0, "p95_under_threshold": true, "generated_at": "2026-08-03T02:38:51.517506+00:00" } ``` Defaults are 1000 episodes, 100 runs, 500ms threshold. Its own `--help` notes that 10000 episodes does NOT meet the 500ms bar with file-based storage. ### `bench_cross_project_lift.py` Measures how much retrieval coverage a project gains from sibling projects' memory. The `method` field names its own limitation. ``` $ python3 tools/bench_cross_project_lift.py --json { "goals": 6, "baseline_covered": 0, "cross_covered": 3, "lift_absolute": 3, "lift_pct_points": 50.0, "net_new_from_siblings": 3, "top_k": 5, "method": "retrieval-coverage (keyword-overlap relevance proxy), NOT task-success" } ``` ### `probe-model-catalog.py` Fetches provider documentation pages, extracts model IDs by regex, and compares them against `providers/model_catalog.json`. It reports and never auto-rewrites the catalog. ``` $ python3 tools/probe-model-catalog.py == claude == known in catalog: 4 found in docs: 14 NEW CANDIDATES: claude-haiku-4-5-20251001, claude-opus-4-1, ... To adopt a new model: edit providers/model_catalog.json -> bump latest_ and add to models[]. Then re-run this script to confirm it disappears from new_candidates. ``` `--strict` exits nonzero when new models are found. Note it still probes a `gemini` section; Gemini was removed as a provider, so those candidates are not adoptable. ### `regen-state-machine-refs.py` Verifies that every `file:line (function)` reference in `docs/architecture/STATE-MACHINES.md` still points at that function's definition. `--fix` rewrites stale numbers in place, `--strict` exits nonzero on drift (for CI). ### `index-codebase.py` Builds the semantic index that `hybrid_search.py --semantic-only` consumes. **This does not currently run on this machine.** It imports `chromadb`, which fails on Python 3.14: ``` $ python3 tools/index-codebase.py --help Traceback (most recent call last): File ".../tools/index-codebase.py", line 41, in import chromadb ... UserWarning: Core Pydantic V1 functionality isn't compatible with Python 3.14 or greater. ``` Exit 1 even for `--help`. Use `hybrid_search.py --grep-only` until the dependency supports your interpreter. No usage example is given here because none could be executed. ### `verify-demo.sh` Documented under [Verify a receipt](#verify-a-receipt) above; it is user-facing despite also serving as a smoke test.