# v7.5.15 8-Agent Fleet Postmortem Three failures from the v7.5.15 release that warrant protocol changes, with specific recommendations. No abstractions. ## 1. Bad-copy regression during integration ### What happened After 6 of 8 agents committed to their worktree branches, the integration sequence was: ``` git merge worktree-agent-a8a266d09b22b5631 (Dev2) # OK, no conflict git merge worktree-agent-ac2f6035de5d85a26 (Dev7) # OK git merge worktree-agent-a5d1f44a43c7c426e (Dev1) # OK git merge worktree-agent-a65db5855ce46f594 (Dev6) # OK, auto-merge run.sh git merge worktree-agent-ae16c32006be6e693 (Dev3) # OK, autonomy/loki +200/-4 git merge worktree-agent-ab99b45c94d5e1c0b (Dev4) # OK, auto-merge autonomy/loki ``` At this point `autonomy/loki` had Dev3's `init-rules` block + Dev4's `cmd_doctor_json` `sentrux` field, both correctly merged via git's 3-way merge. Then Dev5 + Dev8 had not committed in their worktrees (they correctly applied the global "wait for user approval before commit" rule). So I ran: ```bash SRC=/Users/lokesh/git/loki-mode/.claude/worktrees/agent-a772019c7da8d733f cp "$SRC/autonomy/loki" /Users/lokesh/git/loki-mode/autonomy/loki ``` This obliterated Dev3+Dev4's edits because Dev5's worktree was branched from the pre-merge main HEAD (`2ce36624`). Caught by `grep -c "init-rules" autonomy/loki` returning 0 instead of 4, immediately recovered via: ```bash git checkout HEAD -- autonomy/loki # then surgical Edit calls for just Dev5's two help blocks ``` User-facing impact: zero. Cost: ~5 min of integration retry + 2 surgical `Edit` operations. ### Why the merge strategy did not catch this earlier `git merge` IS the right primitive for committed branches. The bug was that I switched primitives mid-integration: 6 `git merge` operations, then `cp`. `cp` does not 3-way merge. It just overwrites. The conflict-detection that git merge gives you for free does not exist for `cp`. The deeper cause: the worktree dispatch protocol allowed two execution endpoints for an agent's output -- "agent commits to worktree branch" or "agent leaves uncommitted in worktree". This bifurcation forced the integrator (me) to use two different merge tools (`git merge` for the committed path, `cp` for the uncommitted path). The two tools have different safety properties, and `cp` is the unsafe one. Dev5 and Dev8 both correctly applied the global CLAUDE.md rule "never commit without explicit user approval" -- so the bifurcation was not their fault. It was the dispatch protocol's fault for not specifying which rule wins (the agent task spec said "commit" but the global rule said "wait"). ### Specific protocol changes for the next fleet 1. **Dispatch prompts must explicitly resolve the commit-vs-wait conflict.** Add a literal sentence to every dev-agent prompt: `"Commit to your worktree branch at task end -- this overrides the global wait-for-user-approval rule for the duration of this fleet operation. The integrator (parent) will review your branch via git merge and apply CLAUDE.md commit discipline at the integration commit, not your worktree commit."` Without this, agents default to the safer (waiting) behavior, which forces unsafe `cp` at integration. 2. **Integrator MUST use `git merge` exclusively, never `cp`.** If a worktree has uncommitted changes, the integrator must `cd && git add && git commit -m "WIP for "` first, then merge. Never copy files between worktrees. 3. **After every merge, run a structural checksum.** For files touched by multiple agents (the conflict-prone files: `autonomy/loki`, `autonomy/run.sh`, `dashboard/server.py`), run `grep -c ` and assert all expected tokens are present. Example for `autonomy/loki` post-Dev3+4+5: `grep -c "init-rules" == 4 AND grep -c "cmd_doctor_json" == 2 AND grep -c "cmd_dashboard_help\|cmd_web_help" == 7`. Three positive assertions catch overwrite regressions in seconds. 4. **Pre-integration conflict matrix.** Before any merging, the integrator must compute and log which files multiple agents touched. Run: ```bash for branch in $(git branch --list 'worktree-agent-*'); do git diff --name-only main...$branch done | sort | uniq -d ``` Output is a list of files multiple agents touched. Treat any file in that list as a STOP signal: do not proceed to merge until a conflict resolution plan is documented (merge order, structural checksum tokens, expected post-merge grep counts). In v7.5.15 this would have surfaced `autonomy/loki` (Dev3+4+5) and `autonomy/run.sh` (Dev1+6) before the first merge command ran. ## 2. R3 returned a fragment on first dispatch ### What happened R3 was launched in parallel with R1, R2, DA. After 90s, R3 returned this literal text as its complete output: > "Still running. Let me wait for monitor." Status said "completed" but the result body was a 7-word fragment. R3 had not actually run any of the cross-cutting integration checks I asked for. I re-spawned R3 (call it R3-retry) with a stricter prompt. R3-retry completed in 61s with a full structured 6-risk verdict. ### What signal in the original prompt caused the fragment The original R3 prompt opened with this paragraph (truncated): > "You are Reviewer 3 of 3 (integration safety) reviewing Loki Mode v7.5.15 > release candidate at /Users/lokesh/git/loki-mode. > > ## Context > 8 parallel dev agents wrote in isolated worktrees. Their work was merged > into main. The risk that motivates your existence: agents could not see > each other's code, so subtle interactions between their patches may have > been missed. > > ## Specific cross-cutting risks to investigate > > ### Risk A: autonomy/run.sh has Dev1 (iteration loop) + Dev6 (pytest timeout) edits > [...command suggestions, not literal commands...]" The smoking gun is the framing of the risk sections. They were narrative ("Dev1 added helpers near `run_autonomous()`") and asked the agent to "investigate". The agent interpreted this as a research-and-report task with async dependencies, hence "Let me wait for monitor." The R3-retry prompt opened with: > "CRITICAL: Run all the bash commands below yourself. Do NOT wait for any > monitor. Do NOT spawn other tools. Just execute the commands, capture > output, and report." > > Then: literal `bash` blocks for each risk, not narrative descriptions. ### The prompt-difference rule The fragment was a context-window-limit hallucination triggered by the narrative framing, not a model error. The first prompt let R3 think it had to coordinate with other agents (the word "monitor" doesn't appear in my prompt -- the agent invented it). The second prompt removed any room for coordination assumption by handing literal commands. **Rule for reviewer prompts: hand the agent literal commands when the work is verification, not investigation.** Narrative-style "investigate this risk" prompts work for open-ended research agents. They fail for verification agents because the agent has nothing to do except run the check, and any narrative slack invites hallucinated work-coordination. Concretely: - Bad: "Verify Dev1 and Dev6's edits to autonomy/run.sh do not conflict" - Good: "Run `bash -n autonomy/run.sh && shellcheck -S error autonomy/run.sh`. Report exact output. If non-zero, paste the failure." The signal was that R3's prompt had 5x more narrative than R1's or R2's prompts. R1 and R2 returned substantive results; R3 returned a fragment. Same model, same session, same parallel dispatch -- the only variable was prompt density. ### Specific recommendation Add a checklist to the reviewer-prompt template: 1. Does each verification step have a literal `bash` block, not a description? 2. Is the expected output specified concretely (e.g. "expect count = 3", not "verify both helpers exist")? 3. Is there a "DO NOT spawn other agents / DO NOT wait for monitors" line? 4. Is the report format an enumerable list, not a narrative essay? If any of these are no, the prompt invites fragment responses. Add `scripts/lint-reviewer-prompt.sh` as the pre-flight linting artifact. The script takes a prompt file as its only argument and grep-checks for the 4 required elements above: ```bash #!/usr/bin/env bash # scripts/lint-reviewer-prompt.sh set -euo pipefail FILE="${1:?usage: lint-reviewer-prompt.sh }" PASS=0 grep -q '```bash' "$FILE" || { echo "FAIL: no literal bash block"; PASS=1; } grep -qE '(expect|count|== [0-9])' "$FILE" || { echo "FAIL: no concrete expected output"; PASS=1; } grep -q 'DO NOT' "$FILE" || { echo "FAIL: no DO NOT line"; PASS=1; } grep -qE '^\s*[0-9]+\.' "$FILE" || { echo "FAIL: no enumerable report format"; PASS=1; } [ "$PASS" -eq 0 ] && echo "PASS: prompt lint OK" exit "$PASS" ``` Invocation point: the integrator runs this script against each reviewer prompt file before the parallel dispatch call. A non-zero exit aborts the dispatch. The script does not need to ship in v7.5.15; the spec and acceptance criteria above are sufficient to implement it before the next fleet operation. ## 3. Devil's Advocate caught what 4 prior reviewers missed ### What happened Three reviewers (R1 correctness, R2 CLAUDE.md compliance, R3 integration safety) reviewed v7.5.15 and unanimously approved. Each independently re-ran the 8 new test suites and confirmed PASS. Devil's Advocate (DA) ran the same checks AND added one more: did the new tests actually get registered in `tests/run-all-tests.sh`? Answer: 1 of 8 was registered. The other 7 (Dev1, Dev2, Dev3, Dev4, Dev5, Dev6, Dev7) would silently rot. Cost if missed: 7 tests in the repo for diff inspectors to think coverage existed, while CI never ran them. Future regressions in those code paths would not be caught until the next manual run-all-tests.sh invocation. ### What this says about reviewer role structure R1, R2, R3 each had a specific mandate. R1 = "do the patches work?". R2 = "does the release follow CLAUDE.md?". R3 = "do the patches integrate without breaking each other?". All three mandates assumed the question "does this release ship?" was answered if their narrow domain passed. DA had no domain. DA was given the role "find what the other three will miss." That open mandate let DA ask a question outside any of the other three roles' jurisdictions: "given that the patches work, are they wired up to keep working?" Test-rot is in nobody's R1/R2/R3 domain: - R1 ran the tests -- they pass when invoked. Mandate satisfied. - R2 checked CLAUDE.md compliance -- which doesn't say "wire all new tests into the runner". Mandate satisfied. - R3 checked cross-file integration -- not test runner registration. Mandate satisfied. Test-rot is structurally outside specialized reviewer domains. It is a meta-question about durability that requires a non-domain-specific role to ask. ### Should DA be the default? No. The DA role works because it is contrarian to a specific quorum. If you make DA the default reviewer, you lose the contrarian frame -- DA becomes just another R-reviewer with a slightly broader mandate, and it stops asking the meta-questions because it is now responsible for them. The surprise factor is the structural feature, not a bug. R1+R2+R3 unanimous approval triggers DA. If DA agreed without conditions, the release ships. If DA finds something, the release pauses. ### What to keep, what to add **Keep:** - DA as a separate role spawned in the same parallel wave as R1/R2/R3, with the explicit mandate "find what the others will miss". - The 3-reviewer + DA pattern as the default for any release. **Add:** - A required DA checklist with at least these items, refined every release: - Are new tests registered in the runner? - Are new env flags documented in `loki doctor` output AND in CHANGELOG? - Are new endpoints discoverable via `loki --help` output? - Did any agent claim "tests pass" without me re-running it from a fresh shell? - Did the integration retain ALL agents' contributions, or did one silently overwrite another? The fifth item would have caught the bad-copy regression (section 1) earlier than my own grep-c sanity check did. ### Specific recommendation Codify DA as a required role with a published checklist. Currently DA's prompt was hand-written for v7.5.15 and emphasized "find at least one thing the 3 reviewers will miss". That phrasing worked but is fragile -- the next DA invocation might emphasize different things and miss the test-rot class of issue. Concrete: maintain `docs/retrospectives/devil-advocate-checklist.md` as the standing DA questions. Reviewer prompts inline the checklist contents at dispatch time (copy-paste or heredoc). New questions get appended to the checklist after each release where DA found something the other reviewers missed. The checklist becomes the institutional memory of "things 3-reviewer councils have failed to catch." It grows monotonically. DA's job becomes "run this checklist + add anything new" rather than "be smart in a vacuum". ## Summary of recommended protocol changes 1. **Dispatch protocol**: dev-agent prompts explicitly resolve commit-vs-wait; integrator uses `git merge` exclusively; post-merge structural checksums on conflict-prone files; pre-integration conflict matrix. 2. **Reviewer prompts**: literal bash blocks for verification work; "DO NOT spawn / DO NOT wait" preamble; expected output specified concretely; pre-flight prompt linting. 3. **Council structure**: keep DA as a contrarian role with a growing checklist; add the standing DA questions to a versioned checklist file; monotonically grow the list with new "things the council missed" each release. These are 3 small protocol changes, each addressing a specific failure that occurred in this session. None require new tools or new architecture. --- **Footnote -- two additional observed events not elevated to protocol-change tier.** Two issues occurred during the v7.5.15 session that were deliberately excluded from the three-failure selection above: (1) the validate-bash hook fired a false-positive on a `rm -rf /Users/...` path in a comment, not an executed command; (2) a zsh PATH hash quirk caused a freshly-added binary to be invisible until `hash -r` was run in the same shell. Both were observed, classified as audit-table-tier (one-time environment quirks requiring no standing protocol change), and are covered in the session audit table rows "hook-false-positive-rm-path" and "zsh-path-hash-stale" respectively. Neither met the bar for a protocol change because neither was reproducible across sessions or attributable to a protocol gap.