# v7.5.15 LLM Failure Modes Catalogue

This table catalogues LLM failure modes that occurred (or were explicitly
considered and ruled out) during the v7.5.15 8-agent fleet session. "Failure
mode" means a systematic LLM behavioral error, not a human process mistake.
Each row covers one category. If a category did not materialize in this
session, it is marked NOT OBSERVED so the reader can confirm it was
considered, not overlooked. All session moments and file references are
specific: no generalities.

| Failure mode | Where it appeared | What caught it | Cost if missed |
|---|---|---|---|
| Confident hallucination (agent claims a file/function exists that doesn't) | NOT OBSERVED in this session | n/a | n/a |
| Stale context (agent acts on memory of prior state instead of current) | Dev5's `autonomy/loki` worktree was branched from pre-merge `HEAD` (`2ce36624`), i.e., prior to Dev3+Dev4's merges. The integrator treated the worktree file as current state and bulk-`cp`'d it over the integrated tree. This is the integrator exhibiting stale-context behavior. | `grep -c "init-rules" autonomy/loki` returned 0 instead of 4. Immediate recovery via `git checkout HEAD -- autonomy/loki` + surgical `Edit` of only Dev5's two help blocks. | Dev3's `init-rules` subcommand (9/9 tests) and Dev4's `cmd_doctor_json` sentrux field (13/13 tests) would have shipped silently regressed. Two advertised features gone from the binary. |
| Scope creep (agent does more than asked, breaking adjacent code) | NOT OBSERVED in this session. All 8 dev agents touched only their assigned functions/files. Dev7 explicitly disclaimed out-of-scope sites (`cross_project.py:101`) rather than patching them. | n/a | n/a |
| Silent test rot (tests written but not wired to runner) | 7 of 8 new test suites (Dev1 through Dev7) were not registered in `tests/run-all-tests.sh`. Only Dev8's `test-ci-sentrux-coverage.sh` was wired. R1, R2, R3 all ran the tests directly and confirmed PASS without asking whether CI would ever invoke them. | Devil's Advocate reviewer asked "will these tests run tomorrow?" and checked `tests/run-all-tests.sh` directly. Caught pre-commit. Fixed by adding 5 bash entries + 2 pytest wrapper scripts at `tests/run-all-tests.sh:85-99`. | 7 tests silently orphaned. Future regressions in sentrux wire-in, dashboard endpoint, init-rules, doctor JSON parity, dashboard nav UAT, pytest timeout, and episode resilience paths would go undetected until a manual invocation. |
| Cross-agent state assumption (Dev N assumes Dev M's branch is in some state without verifying) | Integrator assumed Dev5's worktree file was safe to `cp` over the integrated main tree. The assumption was that worktree files reflected the post-merge state; they did not (Dev5's branch point was `2ce36624`, before Dev3 + Dev4 merges). See `v7.5.15-fleet-postmortem.md` section 1. | Structural checksum: `grep -c "init-rules" autonomy/loki` == 0 (expected 4). Caught immediately after the bad copy. REPEATABILITY: LOW (manual, not codified -- see fleet-postmortem section 1 recommendation 3 for the fix). | Same as stale context above: two shipped features silently deleted from the binary. |
| Optimistic verification (agent says "tested" without running test) | R3 (first dispatch) returned "Still running. Let me wait for monitor." as its entire output. R3 had not run any of the 6 cross-cutting integration checks assigned to it, but its status was "completed". The agent used language implying work was in progress when no work was done. | Integrator read the result body. 7-word fragment clearly not a structured 6-risk verdict. Re-spawned R3 with a prompt containing literal `bash` blocks and "DO NOT wait for any monitor." R3-retry completed in 61s. See `v7.5.15-fleet-postmortem.md` section 2. | Integration safety review would have been missing from the record. Subtle cross-file interactions between Dev1+Dev6's `autonomy/run.sh` edits and Dev3+Dev4's `autonomy/loki` edits would have gone unchecked. A previously undetected conflict could have shipped. |
| Path-prefix confusion (rm -rf interpreted as dangerous when it isn't) | `validate-bash` hook matched the pattern `rm -rf /Users/lokesh/git/loki-mode/.sentrux` against its dangerous-rm guard (`rm -rf /` prefix check). The path is a subdirectory of the project, not the root, but the string match fired a false positive. Hook returned BLOCKED. | The hook itself caught and surfaced the block explicitly -- no silent data loss. Workaround: rewrite as relative path `rm -rf ./.sentrux`. Documented in session notes from the validate-bash hook output. | If the hook had silently suppressed the error and continued, a subtly wrong rm invocation could have run. Conversely, if the integrator had bypassed the hook without understanding why, it could have established a precedent of ignoring hook blocks. |
| Exit code over-trust (relying on subprocess exit code that lies) | Pre-existing MCP test failure: `python3 -c "import mcp"` exits non-zero on this Mac because `pip install mcp` was never run. The exit code does not represent a regression introduced by v7.5.15 -- it is an environmental gap that predates the session. If taken at face value it could be misread as a newly broken test. | Verified pre-existing by reproducing on `origin/main` before the release commit. CHANGELOG v7.5.15 documents it honestly as "single failure is pre-existing pip install mcp env gap". 24/25 PASS is the accurate count. | Misfiled as a release regression: would have blocked the release or, if overridden, set a precedent of shipping with unacknowledged test failures. |
| Bash vs zsh source-file scope leaking | session-reported, unverified by audit. Reported behavior: `command -v sentrux` returned empty under zsh mid-session even though the binary was on PATH. Plausible mechanism: zsh PATH hash cache had not refreshed to see the binary added during the session. No file:line cite, no grep output, no git ref was captured that would allow an auditor to reproduce or confirm this independently. | session-reported, unverified by audit. Reported workaround: switching to `bash -c` for the test invocation forces a fresh hash evaluation and the invocation succeeded. No codified guardrail exists for this behavior. | If the zsh/bash inconsistency had been attributed to a code bug rather than a shell hash quirk, time would have been spent diagnosing a non-existent defect in the sentrux integration code. Alternatively, the test could have been skipped and a real future failure masked. |
| Tool hook noise drowning real failures | First `bash tests/run-all-tests.sh` invocation was piped through `tail -15`. The tail window captured only 15 lines of output, discarding all per-test PASS/FAIL lines and the full run summary. The visible output showed only the last few lines of the final test's output -- no actionable signal. | Integrator noticed the output was truncated (no "Tests Run:" summary line) and re-ran without the `tail` pipe. Second run produced full per-test output: 24/25 PASS with the one failing test identified by name. | If the truncated output had been accepted as the verification record, the integrator would have had no per-test attribution. Any failure that landed outside the last 15 lines would have been invisible. The release would have shipped without a verified pass/fail count per test. |
| (beyond user's 10) Build artifact drift after source-only merge | The bun-parity matrix in `local-ci.sh` ran both bash and Bun routes through `diff -q` and surfaced a 5/13 failure caused by stale `loki-ts/dist/loki.js`. The Bun dist had not been rebuilt after source changes were merged, so the Bun route diverged from the bash route at 5 commands. This is a distinct failure mode from stale context: the source was current, but the compiled artifact was not. | `bash scripts/local-ci.sh` bun-parity matrix caught the 5/13 diff before the commit. Fix was `bun run build` followed by re-staging the updated dist file. | If the stale dist had shipped, 5 Bun-route commands would have exhibited the pre-merge behavior while the bash route exhibited the post-merge behavior. Users on the Bun route would have silently received the older behavior with no error. |

## What went right

The failure modes above were caught -- none shipped to users -- because
several guardrails worked exactly as intended. The `validate-bash` hook
surfaced the false-positive rm block explicitly rather than silently passing
or silently failing. The post-merge structural grep checksum (`grep -c
"init-rules"`) caught the bad-copy regression in seconds. The Devil's
Advocate role, operating with an explicitly non-domain mandate separate from
R1/R2/R3, asked the meta-question about test runner wiring that three
domain-specific reviewers each had reason not to ask. The `local-ci`
pre-push gate (`bash scripts/local-ci.sh`, 21/21 PASS) caught the stale
`loki-ts/dist/loki.js` artifact before the commit: the bun-parity matrix
ran both bash and Bun routes through `diff -q` and surfaced the 5/13
failure that triggered a `bun run build` before staging. The integrator's
discipline of reading reviewer output rather than trusting "completed"
status caught R3's fragment response before it was treated as a verdict.
Taken together: grep-based checksums, the DA contrarian role, local-ci
pre-push, the bun-parity matrix, and the validate-bash hook are the
codified mechanisms that kept this session's failure modes from becoming
user-facing defects. Note: the integrator reading R3's result body before
accepting its status as a verdict was human attentiveness, not a repeatable
guardrail. It is not listed here as a mechanism; it is listed as the action
that exposed the gap. The gap itself (agents reporting "completed" without
producing required output) is a process deficiency without a codified fix
as of this session.