--- kind: knowledge when-and-why-to-read: When a node is marked wedged or remains active without progress while its inbox accumulates, this reference should be read so recovery restores progress without losing context or killing the wrong process. short-form: a child stuck mid-turn (runaway bash, or a stalled broker with no subprocess) is AUTO-DETECTED by the daemon (⚠ wedged + a doctrine wake) — kill the subprocess if one exists; if none, the daemon SIGTERM-kicks the broker and automatically starts a fresh recovery cycle system-prompt-visibility: none file-read-visibility: preview --- A base node can wedge INDEFINITELY on a single runaway bash command (classic: `grep -rln "..." /` — a recursive grep from filesystem root that scans all of disk and never returns; also any unbounded find/scan over `/` or a network mount). The node's pi turn is blocked waiting on the bash call, so it never `push`es and never finishes. **Detection is automatic (crtrd, issue #110).** The daemon's supervision tick corroborates two signals on every live/busy node: the busy-marker heartbeat (re-touched on every tool start/stream/end and turn end) gone quiet for 20+ minutes, AND the broker's whole process tree (broker pid + every descendant) reading near-zero CPU. Only when BOTH hold does it fire — routine long tool calls that are genuinely still producing output or burning CPU never trip it. On a 'wedged' verdict the daemon records a `daemon→node`/`wedged` Fault (the local canvas graph and browse UI show ⚠ "wedged · needs you", the same surface provider-fault stalls use) and fans a doctrine wake to the node's subscribers — ONE notice per wedge episode, not a repeat every tick. In the local browse UI, the hanging-node recovery action runs `node lifecycle revive --now`; there is no hanging filter on `node inspect list`. **Remediation branches on whether the tree has a descendant (issue #119):** - **A subprocess exists (the common case)** — a runaway bash/find/grep is the likely cause. The daemon does NOT touch the engine here; you kill the subprocess yourself (see below). - **No subprocess exists** — the broker pid stands alone in its own tree (the engine itself stalled, e.g. mid a `model_change`, with nothing a human could kill). The daemon performs the ONLY remaining remediation itself: it SIGTERMs the broker (the same on-demand kick `node lifecycle revive --now` does). The surviving `job/busy` marker proves the process died mid-turn, so its exit-policy job starts a fresh cycle on the saved session tree. That cycle automatically receives the normal roadmap/feed/context kickoff; it does not reopen the dead turn at an idle prompt, and no manual nudge is required. **Why it used to be invisible to the orchestrator:** the wake spine only fires on a child's push/finalize/crash. A child wedged *mid-turn* does none of those — its row stays `status=active` and the dashboard shows it `●` working. Before #110, the parent got no wake at all, so an entire wave could stall for hours looking healthy; cli-verbs-B2 sat wedged ~2h14m on a `grep /` before a safety-deadline inspection caught it. The daemon-side detection above closes that gap; you should no longer need to notice a wedge by elapsed-time vibes alone — but if you do (the daemon hasn't caught up yet, or the wedge is younger than its grace window), the manual inspection below still works. ## Subprocess case: kill the runaway command **How to inspect further / confirm:** `tmux capture-pane -p -t ` shows the live turn (look for `Elapsed NNNNs ⠼ Working...` on one bash command); `pgrep -P ` + `ps -axo pid,ppid,etime,command` reveals the runaway subprocess and how long it's run. **How to apply:** kill the runaway SUBPROCESS, not the node — `kill `. The bash call then returns and the pi turn resumes on its own, no context lost, no respawn. (Killing the bash `-c` wrapper can orphan the grep; kill the actual scanner PID. A pipeline `grep / | head` resumes fastest if you kill the first-stage grep — the pipe closes and bash proceeds.) Then notify the OWNING orchestrator (deferred tier) so it steers the child to completion and knows the lost time — the doctrine wake above already does this automatically for a detected wedge, but do it yourself if you found the wedge before the daemon did. Related: the `revive-stuck-orchestrator` doc covers the orchestrator-level false-finish failure mode. ## No-subprocess case: broker wedged mid-stream (`streaming:true` stuck) The wedge can outlive any subprocess, or exist with none at all. A node (including a top-level **resident orchestrator**, not just a child) can sit stuck with the canvas snapshot showing `streaming: true` and a stale `last_activity` for **days**, while the broker pid is alive but doing nothing (0% CPU) and NO subprocess exists to kill — the engine deadlocked mid-stream (often right after a bash call returned, or after a `model_change`) and never closed the turn. Tells: `job/busy` lockfile present with an old mtime, `job/telemetry.json` `updated_at` frozen, `job/broker.log` stopped growing, inbox piling up (`queued (N)` in the dashboard) because the turn never ends. The `busy` marker alone is harmless (it is always AND-ed with `pidAlive`, so a stale one from a dead pid is ignored) — do not bother deleting it. **This is now the daemon's own remediation (issue #119) — it auto-kicks the broker for you.** If it hasn't caught up yet (younger than the wedge grace, or the daemon was down), do it manually: 1. `kill -TERM ` — the wedged broker. Confirm it dies (`ps -p `) and that no orphan subprocess lingers (`ps -o pid,ppid,command | awk '$2=='`). 2. The **daemon auto-resumes** it within ~20s on the saved session — a fresh broker pid appears (`node inspect show`), `status=active`, `intent=null`, `view.sock` accepts connections. (No manual `node lifecycle revive` needed if the daemon is up; do it manually only if it doesn't come back.) 3. **The daemon starts the replacement as a fresh recovery cycle** — the stale `job/busy` marker distinguishes an interrupted turn from an idle crash, so the daemon selects `resume:false`. The saved session tree remains on disk, while the new cycle re-orients from `context/roadmap.md`, feed, reports, and context automatically. Confirm success by watching `telemetry.json` `updated_at` go current and the node run tools again. A manual `--tier critical` message is now only a fallback if the replacement broker itself fails to boot or provider access is still unhealthy.