# v7.5.15 8-Agent Fleet Postmortem

Three failures from the v7.5.15 release that warrant protocol changes, with
specific recommendations. No abstractions.

## 1. Bad-copy regression during integration

### What happened
After 6 of 8 agents committed to their worktree branches, the integration
sequence was:

```
git merge worktree-agent-a8a266d09b22b5631 (Dev2)  # OK, no conflict
git merge worktree-agent-ac2f6035de5d85a26 (Dev7)  # OK
git merge worktree-agent-a5d1f44a43c7c426e (Dev1)  # OK
git merge worktree-agent-a65db5855ce46f594 (Dev6)  # OK, auto-merge run.sh
git merge worktree-agent-ae16c32006be6e693 (Dev3)  # OK, autonomy/loki +200/-4
git merge worktree-agent-ab99b45c94d5e1c0b (Dev4)  # OK, auto-merge autonomy/loki
```

At this point `autonomy/loki` had Dev3's `init-rules` block + Dev4's
`cmd_doctor_json` `sentrux` field, both correctly merged via git's 3-way merge.

Then Dev5 + Dev8 had not committed in their worktrees (they correctly applied
the global "wait for user approval before commit" rule). So I ran:

```bash
SRC=/Users/lokesh/git/loki-mode/.claude/worktrees/agent-a772019c7da8d733f
cp "$SRC/autonomy/loki" /Users/lokesh/git/loki-mode/autonomy/loki
```

This obliterated Dev3+Dev4's edits because Dev5's worktree was branched from
the pre-merge main HEAD (`2ce36624`). Caught by `grep -c "init-rules" autonomy/loki`
returning 0 instead of 4, immediately recovered via:

```bash
git checkout HEAD -- autonomy/loki
# then surgical Edit calls for just Dev5's two help blocks
```

User-facing impact: zero. Cost: ~5 min of integration retry + 2 surgical
`Edit` operations.

### Why the merge strategy did not catch this earlier

`git merge` IS the right primitive for committed branches. The bug was that
I switched primitives mid-integration: 6 `git merge` operations, then `cp`.
`cp` does not 3-way merge. It just overwrites. The conflict-detection that
git merge gives you for free does not exist for `cp`.

The deeper cause: the worktree dispatch protocol allowed two execution
endpoints for an agent's output -- "agent commits to worktree branch" or
"agent leaves uncommitted in worktree". This bifurcation forced the integrator
(me) to use two different merge tools (`git merge` for the committed path,
`cp` for the uncommitted path). The two tools have different safety
properties, and `cp` is the unsafe one.

Dev5 and Dev8 both correctly applied the global CLAUDE.md rule "never commit
without explicit user approval" -- so the bifurcation was not their fault. It
was the dispatch protocol's fault for not specifying which rule wins (the
agent task spec said "commit" but the global rule said "wait").

### Specific protocol changes for the next fleet

1. **Dispatch prompts must explicitly resolve the commit-vs-wait conflict.**
   Add a literal sentence to every dev-agent prompt: `"Commit to your worktree
   branch at task end -- this overrides the global wait-for-user-approval rule
   for the duration of this fleet operation. The integrator (parent) will
   review your branch via git merge and apply CLAUDE.md commit discipline at
   the integration commit, not your worktree commit."` Without this, agents
   default to the safer (waiting) behavior, which forces unsafe `cp` at
   integration.

2. **Integrator MUST use `git merge` exclusively, never `cp`.** If a worktree
   has uncommitted changes, the integrator must `cd <worktree> && git add
   <files> && git commit -m "WIP for <devN>"` first, then merge. Never copy
   files between worktrees.

3. **After every merge, run a structural checksum.** For files touched by
   multiple agents (the conflict-prone files: `autonomy/loki`, `autonomy/run.sh`,
   `dashboard/server.py`), run `grep -c <each-agent's-distinctive-token>
   <file>` and assert all expected tokens are present. Example for
   `autonomy/loki` post-Dev3+4+5: `grep -c "init-rules" == 4 AND
   grep -c "cmd_doctor_json" == 2 AND grep -c "cmd_dashboard_help\|cmd_web_help"
   == 7`. Three positive assertions catch overwrite regressions in seconds.

4. **Pre-integration conflict matrix.** Before any merging, the integrator
   must compute and log which files multiple agents touched. Run:

   ```bash
   for branch in $(git branch --list 'worktree-agent-*'); do
     git diff --name-only main...$branch
   done | sort | uniq -d
   ```

   Output is a list of files multiple agents touched. Treat any file in
   that list as a STOP signal: do not proceed to merge until a conflict
   resolution plan is documented (merge order, structural checksum tokens,
   expected post-merge grep counts). In v7.5.15 this would have surfaced
   `autonomy/loki` (Dev3+4+5) and `autonomy/run.sh` (Dev1+6) before the
   first merge command ran.

## 2. R3 returned a fragment on first dispatch

### What happened

R3 was launched in parallel with R1, R2, DA. After 90s, R3 returned this
literal text as its complete output:

> "Still running. Let me wait for monitor."

Status said "completed" but the result body was a 7-word fragment. R3 had not
actually run any of the cross-cutting integration checks I asked for.

I re-spawned R3 (call it R3-retry) with a stricter prompt. R3-retry completed
in 61s with a full structured 6-risk verdict.

### What signal in the original prompt caused the fragment

The original R3 prompt opened with this paragraph (truncated):

> "You are Reviewer 3 of 3 (integration safety) reviewing Loki Mode v7.5.15
> release candidate at /Users/lokesh/git/loki-mode.
>
> ## Context
> 8 parallel dev agents wrote in isolated worktrees. Their work was merged
> into main. The risk that motivates your existence: agents could not see
> each other's code, so subtle interactions between their patches may have
> been missed.
>
> ## Specific cross-cutting risks to investigate
>
> ### Risk A: autonomy/run.sh has Dev1 (iteration loop) + Dev6 (pytest timeout) edits
> [...command suggestions, not literal commands...]"

The smoking gun is the framing of the risk sections. They were narrative
("Dev1 added helpers near `run_autonomous()`") and asked the agent to
"investigate". The agent interpreted this as a research-and-report task with
async dependencies, hence "Let me wait for monitor."

The R3-retry prompt opened with:

> "CRITICAL: Run all the bash commands below yourself. Do NOT wait for any
> monitor. Do NOT spawn other tools. Just execute the commands, capture
> output, and report."
>
> Then: literal `bash` blocks for each risk, not narrative descriptions.

### The prompt-difference rule

The fragment was a context-window-limit hallucination triggered by the
narrative framing, not a model error. The first prompt let R3 think it had
to coordinate with other agents (the word "monitor" doesn't appear in my
prompt -- the agent invented it). The second prompt removed any room for
coordination assumption by handing literal commands.

**Rule for reviewer prompts: hand the agent literal commands when the work is
verification, not investigation.** Narrative-style "investigate this risk"
prompts work for open-ended research agents. They fail for verification
agents because the agent has nothing to do except run the check, and any
narrative slack invites hallucinated work-coordination.

Concretely:
- Bad: "Verify Dev1 and Dev6's edits to autonomy/run.sh do not conflict"
- Good: "Run `bash -n autonomy/run.sh && shellcheck -S error autonomy/run.sh`. Report exact output. If non-zero, paste the failure."

The signal was that R3's prompt had 5x more narrative than R1's or R2's
prompts. R1 and R2 returned substantive results; R3 returned a fragment. Same
model, same session, same parallel dispatch -- the only variable was prompt
density.

### Specific recommendation

Add a checklist to the reviewer-prompt template:

1. Does each verification step have a literal `bash` block, not a description?
2. Is the expected output specified concretely (e.g. "expect count = 3", not
   "verify both helpers exist")?
3. Is there a "DO NOT spawn other agents / DO NOT wait for monitors" line?
4. Is the report format an enumerable list, not a narrative essay?

If any of these are no, the prompt invites fragment responses.

Add `scripts/lint-reviewer-prompt.sh` as the pre-flight linting artifact.
The script takes a prompt file as its only argument and grep-checks for the
4 required elements above:

```bash
#!/usr/bin/env bash
# scripts/lint-reviewer-prompt.sh <prompt-file>
set -euo pipefail
FILE="${1:?usage: lint-reviewer-prompt.sh <prompt-file>}"
PASS=0
grep -q '```bash' "$FILE"        || { echo "FAIL: no literal bash block"; PASS=1; }
grep -qE '(expect|count|== [0-9])' "$FILE" || { echo "FAIL: no concrete expected output"; PASS=1; }
grep -q 'DO NOT' "$FILE"         || { echo "FAIL: no DO NOT line"; PASS=1; }
grep -qE '^\s*[0-9]+\.' "$FILE"  || { echo "FAIL: no enumerable report format"; PASS=1; }
[ "$PASS" -eq 0 ] && echo "PASS: prompt lint OK"
exit "$PASS"
```

Invocation point: the integrator runs this script against each reviewer
prompt file before the parallel dispatch call. A non-zero exit aborts the
dispatch. The script does not need to ship in v7.5.15; the spec and
acceptance criteria above are sufficient to implement it before the next
fleet operation.

## 3. Devil's Advocate caught what 4 prior reviewers missed

### What happened

Three reviewers (R1 correctness, R2 CLAUDE.md compliance, R3 integration
safety) reviewed v7.5.15 and unanimously approved. Each independently re-ran
the 8 new test suites and confirmed PASS.

Devil's Advocate (DA) ran the same checks AND added one more: did the new
tests actually get registered in `tests/run-all-tests.sh`? Answer: 1 of 8 was
registered. The other 7 (Dev1, Dev2, Dev3, Dev4, Dev5, Dev6, Dev7) would
silently rot.

Cost if missed: 7 tests in the repo for diff inspectors to think coverage
existed, while CI never ran them. Future regressions in those code paths
would not be caught until the next manual run-all-tests.sh invocation.

### What this says about reviewer role structure

R1, R2, R3 each had a specific mandate. R1 = "do the patches work?". R2 =
"does the release follow CLAUDE.md?". R3 = "do the patches integrate without
breaking each other?". All three mandates assumed the question "does this
release ship?" was answered if their narrow domain passed.

DA had no domain. DA was given the role "find what the other three will
miss." That open mandate let DA ask a question outside any of the other three
roles' jurisdictions: "given that the patches work, are they wired up to
keep working?"

Test-rot is in nobody's R1/R2/R3 domain:
- R1 ran the tests -- they pass when invoked. Mandate satisfied.
- R2 checked CLAUDE.md compliance -- which doesn't say "wire all new tests
  into the runner". Mandate satisfied.
- R3 checked cross-file integration -- not test runner registration.
  Mandate satisfied.

Test-rot is structurally outside specialized reviewer domains. It is a
meta-question about durability that requires a non-domain-specific role to
ask.

### Should DA be the default?

No. The DA role works because it is contrarian to a specific quorum. If you
make DA the default reviewer, you lose the contrarian frame -- DA becomes
just another R-reviewer with a slightly broader mandate, and it stops asking
the meta-questions because it is now responsible for them.

The surprise factor is the structural feature, not a bug. R1+R2+R3 unanimous
approval triggers DA. If DA agreed without conditions, the release ships. If
DA finds something, the release pauses.

### What to keep, what to add

**Keep:**
- DA as a separate role spawned in the same parallel wave as R1/R2/R3, with
  the explicit mandate "find what the others will miss".
- The 3-reviewer + DA pattern as the default for any release.

**Add:**
- A required DA checklist with at least these items, refined every release:
  - Are new tests registered in the runner?
  - Are new env flags documented in `loki doctor` output AND in CHANGELOG?
  - Are new endpoints discoverable via `loki <thing> --help` output?
  - Did any agent claim "tests pass" without me re-running it from a fresh
    shell?
  - Did the integration retain ALL agents' contributions, or did one
    silently overwrite another?

The fifth item would have caught the bad-copy regression (section 1) earlier
than my own grep-c sanity check did.

### Specific recommendation

Codify DA as a required role with a published checklist. Currently DA's
prompt was hand-written for v7.5.15 and emphasized "find at least one thing
the 3 reviewers will miss". That phrasing worked but is fragile -- the next
DA invocation might emphasize different things and miss the test-rot class
of issue.

Concrete: maintain `docs/retrospectives/devil-advocate-checklist.md` as the
standing DA questions. Reviewer prompts inline the checklist contents at
dispatch time (copy-paste or heredoc). New questions get appended to the
checklist after each release where DA found something the other reviewers
missed.

The checklist becomes the institutional memory of "things 3-reviewer councils
have failed to catch." It grows monotonically. DA's job becomes "run this
checklist + add anything new" rather than "be smart in a vacuum".

## Summary of recommended protocol changes

1. **Dispatch protocol**: dev-agent prompts explicitly resolve commit-vs-wait;
   integrator uses `git merge` exclusively; post-merge structural checksums on
   conflict-prone files; pre-integration conflict matrix.
2. **Reviewer prompts**: literal bash blocks for verification work; "DO NOT
   spawn / DO NOT wait" preamble; expected output specified concretely;
   pre-flight prompt linting.
3. **Council structure**: keep DA as a contrarian role with a growing
   checklist; add the standing DA questions to a versioned checklist file;
   monotonically grow the list with new "things the council missed" each
   release.

These are 3 small protocol changes, each addressing a specific failure that
occurred in this session. None require new tools or new architecture.

---

**Footnote -- two additional observed events not elevated to protocol-change tier.**
Two issues occurred during the v7.5.15 session that were deliberately excluded
from the three-failure selection above: (1) the validate-bash hook fired a
false-positive on a `rm -rf /Users/...` path in a comment, not an executed
command; (2) a zsh PATH hash quirk caused a freshly-added binary to be
invisible until `hash -r` was run in the same shell. Both were observed,
classified as audit-table-tier (one-time environment quirks requiring no
standing protocol change), and are covered in the session audit table rows
"hook-false-positive-rm-path" and "zsh-path-hash-stale" respectively. Neither
met the bar for a protocol change because neither was reproducible across
sessions or attributable to a protocol gap.
