# Recovery Guide - What to Do When Things Break

Every failure mode in the pipeline has a deterministic recovery path. This
guide is a single place to look up "X failed - now what?" instead of chasing
the answer across `modes.md`, `operations.md`, and the individual phase specs.

## Fast Triage - Which Path Do I Need?

| Symptom                                                 | Jump to                                         |
| ------------------------------------------------------- | ----------------------------------------------- |
| Pipeline paused/halted mid-run                          | [Resume a paused task](#resume-a-paused-task)   |
| Phase 3 build failed > 3 times                          | [Build retry exhausted](#build-retry-exhausted) |
| Phase 4 triage returned exit 1 (invalid JSON) twice     | [Triage fallback](#triage-fallback)             |
| Phase 4 triage returned exit 2 (over-rejection)         | [Over-rejection](#over-rejection-guard)         |
| Worktree already exists / dirty                         | [Worktree collisions](#worktree-collisions)     |
| `agent-state.json` corrupt or unreadable                | [State corruption](#state-corruption)           |
| Wrong git identity committed                            | [Identity rewind](#identity-rewind)             |
| Commit hook blocked commit (secret scan)                | [Secret scan blocked commit](#secret-scan-blocked-commit) |
| Autopilot ran too long / cost concern                   | [Abort autopilot](#abort-autopilot)             |
| instructionDriven flag set but instruction file missing | [Instruction fallback](#instruction-fallback)   |
| CHANGELOG / version out of sync                         | [Version drift](#version-drift)                 |

---

## Resume a Paused Task

Paused tasks keep their full state in `.worktrees/{jiraId}/agent-state.json`.
Inspect, then resume.

```bash
/multi-agent:status               # Claude Code
multi-agent-status                # Copilot CLI
# → shows #N, jiraId, currentPhase, status

/multi-agent:resume #3
multi-agent-resume 3
```

Resume re-enters the phase whose `status` is `in_progress` or `failed`. If a
phase is `failed`, it restarts from its first step; if `in_progress`, it picks
up at the last logged step.

**Autopilot resume:** the `autopilot` flag on `state.autopilot=true` persists,
so `resume` continues in autopilot.

---

## Build Retry Exhausted

Phase 3 retries the build up to **3 times** on each TDD cycle's green step. On
the 4th consecutive failure the pipeline **pauses** (even in autopilot) and
surfaces the error. Autopilot does not loop infinitely - this is intentional.

Path forward:

1. Read the last 20 lines of `agent-log.md` - the root cause is usually there.
2. If the failure is environmental (missing Xcode sim, wrong JDK), fix the
   environment and `resume` - the 3-retry counter resets.
3. If the failure is logic (compile error in the generated code), edit the
   offending file in the worktree, then `resume` - Phase 3 re-runs build.
4. If the task itself is wrong-shaped (Phase 2 plan is infeasible), `kill` and
   restart with a better-scoped prompt.

---

## Triage Fallback

`validate-triage.mjs` exit codes and what they mean:

| Exit | Meaning                  | Pipeline action                                |
| ---- | ------------------------ | ---------------------------------------------- |
| 0    | Valid & clean            | Proceed to Phase 5                             |
| 1    | Invalid JSON / schema    | Retry triage ONCE. On second exit-1, fallback: treat ALL raw findings as accepted. |
| 2    | Over-rejection (>80%)    | Pause for human confirmation                   |
| 3    | Contradiction corrected  | Use `result.corrected`, continue               |

If exit 1 fires twice (fallback path):

1. The pipeline logs `Phase 4: triage failed twice - fallback, all findings accepted as blocking`.
2. All raw findings loop back into Phase 3 as if they were all real blockers.
3. This is intentionally conservative - we'd rather over-fix than skip
   something real. You can manually mark noise in the Phase 6 PR description.

## Over-Rejection Guard

Exit code 2 means triage rejected >80% of findings, which is suspicious - it
often indicates the triage prompt lost the task context. Pipeline halts and
asks a human to look at the raw findings vs triage output.

**Normal resolution:**

- Open `agent-log.md`, find the triage invocation.
- Compare raw findings (logged) with rejected items.
- If the rejections are correct, run `resume` with `--accept-overrejection` (Phase 4 re-enters validation with the guard soft-failed).
- If not, run `resume` - Phase 4 re-runs triage with the guard still active.

Autopilot behavior: logs the warning, accepts the triage output, continues -
the autopilot contract values throughput over human-in-loop.

---

## Worktree Collisions

Phase 0 creates `.worktrees/{jiraId}`. If that path already exists:

```bash
# Option A - resume the previous run
/multi-agent:resume #N

# Option B - delete the stale worktree
/multi-agent:kill #N        # asks to confirm, removes worktree + branch

# Option C - manual cleanup
git worktree list                        # find stale entries
git worktree remove .worktrees/{jiraId}  # safe if no uncommitted work
git worktree prune                       # drop any dead refs
```

If `git worktree remove` refuses (uncommitted changes), commit the WIP work
inside the worktree first:

```bash
git -C .worktrees/{jiraId} add -A
git -C .worktrees/{jiraId} commit -m "WIP: rescue before remove"
```

---

## State Corruption

`agent-state.json` is validated against `pipeline/schemas/agent-state.schema.json`
at Phase 0 load. If validation fails:

1. Back up the file: `cp agent-state.json agent-state.json.bak`.
2. Try `node pipeline/scripts/validate-schemas.mjs` to identify the violation.
3. Common fixes:
   - **Missing required field** - add the default per the schema description.
   - **Bad enum value** - check `phase.status` and `taskType` enums.
   - **`lastUpdated` not ISO-8601** - regenerate with `date -u +"%Y-%m-%dT%H:%M:%SZ"`.
4. If the file is irrecoverable, `kill` the task and restart. State loss for
   a single task is recoverable; corruption bleeding into the next run is not.

To prevent future corruption, use the atomic writer - never write state with
shell `echo > agent-state.json`:

```bash
echo '{"currentPhase":4}' | node pipeline/scripts/write-state.mjs .worktrees/{id}/agent-state.json
```

---

## Identity Rewind

If a commit landed under the wrong git identity:

```bash
# Check what happened
git log -1 --format='%an <%ae>'

# Amend (only if not pushed)
git -c user.name="Correct Name" -c user.email="correct@example.com" commit --amend --no-edit

# Already pushed? Revert is safer than force-push
git revert HEAD
# then re-do the commit with correct identity
```

Never rewrite published history on `main`/`master`/`develop` without team
consent. Force-push rewrites break downstream clones.

---

## Secret Scan Blocked Commit

The pre-commit hook (`~/.claude/scripts/pre-commit-check.sh`) scans staged
files for hardcoded secrets and blocks the commit if any are found.

If the hit is a **real secret**:

1. `git reset HEAD {file}` - unstage.
2. Move the secret to env / keychain / secret manager.
3. Re-stage the cleaned file and commit.

If the hit is a **false positive** (e.g. an example string in docs):

1. Rename the variable so the pattern doesn't match
   (`EXAMPLE_API_KEY` not `api_key=`).
2. Or, for documentation, break the pattern with a zero-width marker.
3. Never `--no-verify` to bypass - that's how real secrets slip through.

---

## Abort Autopilot

Autopilot runs to completion unless:

- A phase fails 3×+ (hard pause)
- Ctrl-C sent to the running process

Graceful abort without data loss:

```bash
# 1. Interrupt
Ctrl-C

# 2. Verify state
cat .worktrees/{id}/agent-state.json | grep status

# 3. Either resume (continue where it stopped) or kill (wipe and restart)
/multi-agent:resume #N
/multi-agent:kill   #N
```

---

## Instruction Fallback

`state.instructionDriven=true` but the file at `state.instructionFiles.commit`
is missing on disk. Phase 6 logs an error, sets
`state.instructionDrivenFallback=true`, and uses the standard commit path.

This is rarely fatal - usually the instruction file was removed between
sessions or the wrong path was cached. To recover cleanly:

1. Check whether the intended instruction file still exists somewhere:
   `find ~/.claude/skills ~/.copilot/skills -name "{file-basename}"`
2. If found, update `state.instructionFiles.commit` with
   `write-state.mjs` and `resume`.
3. If the skill was removed, either accept the standard commit path (already
   what happened) or update the task definition to not expect instruction
   mode.

---

## Version Drift

`package.json` version does not match the tag the release workflow sees, so
publishing fails with "Tag version does not match package.json version".

Fix:

```bash
# What does the tag expect?
git describe --tags --abbrev=0            # e.g. v3.5.0

# What does package.json say?
node -p "require('./package.json').version"

# Match them and re-tag
npm version {version} --no-git-tag-version
git add package.json package-lock.json
git commit -m "chore: bump version to {version}"
git tag -d v{version}
git tag v{version}
git push origin main
git push origin v{version} --force       # only if tag was never consumed
```

If the tag has already been consumed (release CI ran), bump to the next patch
version and tag fresh - never re-publish a consumed version.

---

## Worktree Corruption (v8.0.0)

`agent-state.json` says the task lives at `.worktrees/<id>/` but the directory
is missing, locked, or `git worktree list` does not list it. Symptoms: `:resume`
exits 1 with `cannot cd to <path>`, or git operations inside the worktree fail
with `fatal: not a git repository`.

Fix:

```bash
# Inspect git's worktree registry - the source of truth
git worktree list --porcelain | grep -A 2 "<task-id>"

# If the path is missing but git still tracks it (common when a stale .git/worktrees/<id>/
# entry survived a manual rm -rf):
git worktree prune --verbose
git worktree list                   # confirm cleanup

# If the worktree is locked:
git worktree unlock .worktrees/<task-id>
git worktree remove --force .worktrees/<task-id>

# Re-create the worktree pointing at the same branch:
git worktree add .worktrees/<task-id> <branch-name>

# Update agent-state.json to point at the new worktree path (only if it changed):
node -e 'const f="$HOME/.claude/logs/multi-agent/<project>/<task-id>/agent-state.json";
         const s=JSON.parse(require("fs").readFileSync(f));
         s.worktreePath="/abs/path/.worktrees/<task-id>";
         require("fs").writeFileSync(f, JSON.stringify(s, null, 2));'
```

If branch and commits look intact but the worktree refuses to recover, the
safest path is `kill` the task (preserves logs + branch) and restart with
`/multi-agent:resume <id>` from a fresh clone.

---

## Tracker State Corruption (v8.0.0)

`phase-tracker.sh render` returns garbage - duplicate phases, missing
elapsed/token fields, JSON parse errors - even though the task is healthy.
Usually caused by a manual edit to `tracker-state.json` or a partial write
mid-update.

Fix:

```bash
TRACKER_STATE="$HOME/.claude/logs/multi-agent/<project>/<task-id>/tracker-state.json"

# 1. Verify it's invalid JSON (otherwise the issue is elsewhere)
jq empty "$TRACKER_STATE"           # exit non-zero == corrupted

# 2. Back it up before fixing
cp "$TRACKER_STATE" "$TRACKER_STATE.bak.$(date +%s)"

# 3. Either restore from a recent valid snapshot in the same dir, or
#    reinitialize from the agent-log timeline:
bash $HOME/.claude/scripts/phase-tracker.sh init "<task-id>"
for p in 0:Init 1:Analysis 2:Planning 3:Dev 4:Review 5:Test 6:Commit 7:Report; do
  bash $HOME/.claude/scripts/phase-tracker.sh add "${p%%:*}" "${p#*:}"
done
# Replay completed phases from agent-log.md headers
grep -E '^## Phase [0-9]+:' "$HOME/.claude/logs/multi-agent/<project>/<task-id>/agent-log.md" \
  | awk -F'[: ]' '{print $3}' | sort -u | while read n; do
    bash $HOME/.claude/scripts/phase-tracker.sh update "$n" completed
done
```

Token counts are not recoverable from the log - those start fresh after a
reinit. A footnote in `agent-log.md` documenting the rebuild is good practice.

---

## Token Rotation Mid-Pipeline (v8.0.0)

A token (Jira / Bitbucket / GitHub / Vercel) was rotated while a long-running
pipeline was active. Phase 6 / Phase 7 then fail with `401 Unauthorized` even
though earlier phases worked.

Fix:

```bash
# 1. Update the keychain entry for the rotated provider. The mapping
#    keys live in prefs.global.keychainMapping (account+label tuple per provider).
ACCOUNT=$(jq -r '.global.keychainMapping.<provider>.account' \
            $HOME/.claude/multi-agent-preferences.json)
LABEL=$(jq -r '.global.keychainMapping.<provider>.label' \
            $HOME/.claude/multi-agent-preferences.json)

# macOS:
security delete-generic-password -s "$LABEL" -a "$ACCOUNT" 2>/dev/null
security add-generic-password   -s "$LABEL" -a "$ACCOUNT" -w "$NEW_TOKEN"

# Linux:
secret-tool clear     account "$ACCOUNT" service "$LABEL"
echo "$NEW_TOKEN" | secret-tool store --label="$LABEL" account "$ACCOUNT" service "$LABEL"

# 2. Sanity-check the doctor reports the new token's prefix
bash pipeline/lib/credential-store.sh doctor

# 3. Resume the pipeline - Phase 6/7 re-resolve the token on each invocation,
#    so no state edit is required.
/multi-agent:resume <task-id>
```

If the rotated token was Vercel, also run `bash pipeline/lib/vercel-deploy.sh
doctor` to confirm the wrapper sees the new value before retrying any deploy.

**Never** copy-paste the new token into a `--token=` argv to test it - see
`pipeline/lib/vercel-deploy.sh` for why this is forbidden.

---

## Author Rewrite + Tag Re-Sync (v8.0.0)

A series of commits (and any tags pointing at them) were authored under the
wrong identity - typically because someone passed `git -c user.email=...` or
the repo-local config drifted. Vercel / npm / GitHub team-membership gates
then reject the push because the author email is not a recognized contributor.

This is the v7.9.1 release-window incident verbatim: 7 commits across
v7.7.0-v7.9.1 plus 4 annotated tags shipped with the wrong author. Recovery
required a rebase + retag dance.

```bash
# 1. Confirm the repo-local identity is now what you actually want.
git config user.email
git config user.name
# Repair if needed:
git config user.email "you@example.com"
git config user.name  "You"

# 2. Find the first wrong commit (the one BEFORE the rebase rewrites everything):
git log --format='%h %ae %s' main..HEAD | grep -i "wrong-email-pattern" | tail -1
# Grab its parent SHA - that's your --onto target.
PARENT_OF_FIRST_WRONG=<sha>

# 3. Rebase with --reset-author to rewrite every commit since that point.
#    --reset-author forces git to take the new author from the repo-local config.
git rebase --reset-author "$PARENT_OF_FIRST_WRONG"

# 4. Verify EVERY rewritten commit's author is now correct:
git log --format='%h %ae' "$PARENT_OF_FIRST_WRONG"..HEAD

# 5. Force-with-lease the branch (NOT plain --force):
git push --force-with-lease origin <branch>

# 6. Now retag every tag whose target SHA changed during the rebase.
#    For each affected tag:
TAG=v7.7.0   # repeat for every tag in the rebased range
OLD_SHA=$(git rev-parse "$TAG")
NEW_SHA=$(git log --format=%H --grep="<commit-subject-or-marker>" -1)

# 6a. Local tag delete + recreate at the rewritten SHA, with annotation:
git tag -d "$TAG"
git tag -a "$TAG" "$NEW_SHA" -m "$(git tag -l --format='%(contents)' "$TAG-old" 2>/dev/null || echo "$TAG release")"

# 6b. Remote tag - DELETE then PUSH (no atomic option for tag content rewrite):
git push origin :refs/tags/"$TAG"
git push origin "$TAG"

# 7. If a CI release pipeline already consumed the OLD tag SHA (npm publish,
#    GitHub Release creation), the published artifact stays attached to the
#    deleted SHA. You CANNOT rewrite published packages - bump the patch
#    version and ship a fresh release instead. Document the orphan in the
#    CHANGELOG with a "yanked" note.
```

**Tag attestation gotcha:** GitHub Releases attached to a deleted tag SHA
remain visible in the UI as "this tag has been deleted". Either delete the
release manually (`gh release delete <tag>`) and recreate it pointing at the
new SHA, or accept the orphan and document it.

**npm publish gotcha:** if `npm publish` ran against the wrong-author tag,
the package is permanently anchored to that SHA on the registry. Bump the
patch version (`npm version patch --no-git-tag-version`), commit, push,
publish - never `npm unpublish` (registry blocks unpublish for stable
versions).

---

## Failed Push + Force-Push Decision (v8.0.0)

`git push` rejects the branch (non-fast-forward, hook rejected, branch
protection mismatch, etc.). The pipeline is paused at Phase 6 Step 3.

Decision tree:

| Condition | Safe action |
|---|---|
| Remote ahead - someone pushed to the same branch | `git pull --rebase origin <branch>`, replay the commit, retry push |
| Hook rejected (lint / signing / size) | Fix locally, amend or new commit, retry push - NEVER `--no-verify` |
| Branch protection (review count) | Push to a new branch with `-u`, open PR - do NOT `--force` |
| Author rewrite needed (wrong identity, see `feedback_git_author_repo_local.md`) | `git rebase --reset-author`, `git push --force-with-lease`, NOT `--force` |

Force-with-lease is the only acceptable force variant: it refuses to push if
someone else committed in the meantime. The plain `--force` flag silently
overwrites their work.

Bitbucket-specific: a force-push wipes default reviewers if the PR description
PUT happens before reviewers re-attach. The `channels.md` adapter's
reviewer-preserving payload handles this - never `gh pr edit --body` on a
Bitbucket PR after a force-push without re-reading the reviewer list first.

---

## When None of These Apply

1. Capture `agent-log.md`, `agent-state.json`, and the last 100 lines of
   whatever command last ran.
2. Open a repo issue with the artifacts attached.
3. For urgent cases, `kill` the task to unblock, then diagnose offline.

Recovery is preferable to reset, but reset is preferable to carrying forward a
broken state.