# M38: interrupted runs carry on

The first of the durable-inspired pieces of [any agent](any-agent.md): a run that was cut
short is not left for the user to notice; it carries on by itself, a bounded number of
times.

## Before

- A worker that crashed mid-run (non-zero exit) was restarted once.
- A host gone under a running job (killed, out of memory) failed it ("host vanished").
- A run cut by the machine or the service going down failed, waiting for a reply.
- A worker killed by a signal showed as **done**: node-pty reports it as exit code 0 with
  the signal's number, and only the code was read. Found by the new eval (below).

## Now

`autoResume` in the user's `roost.json`:

| | restarts by itself |
|---|---|
| `"interrupted"` (default) | a worker crash or kill mid-run; a host gone under a working run; a run that was working when the machine or the service went down, if its last progress is within 6 hours |
| `"crash"` | worker crashes and kills mid-run only |
| `"off"` | nothing |

A restart sends "Continue where you left off." to the session (a session pi never wrote
runs its first task again). The budget: 3 automatic restarts in a row, after 1, 5 and
15 s (`PI_ROOST_RESUME_DELAYS_MS`); a turn that ends normally once the worker has been up
a minute gives it back. Past it, the row fails with why ("not restarted again after 3
automatic restarts in a row (reply to continue)").

While a restart waits, the row stays **working** (no "finished" notification for a run
that is still meant to go on) and says so ("worker killed by SIGKILL; continuing in 5 s
(automatic restart 2 of 3)"). `resumeAt` is kept in the job's state as a durable write:

- `stop` (or a `/stop` reply) calls it off (the row is stopped); `rm` too. A subagent's
  parent hears of either, as of any end.
- Any start takes its place once it can start: attach or Enter twice carry the run on
  ("Continue where you left off."), a reply goes in instead. It does not count as an
  automatic restart, nor give the budget back. A start that fails (the session locked)
  leaves the waiting restart to run.
- The service going down while it waits: the next start takes it up.
- A restart that cannot start (the session locked by another pi, its directory gone)
  fails with the reason and is not tried again; a parent hears of it.

Exits read as crashes now: a non-zero code or a signal (`killed by SIGKILL`); for
commands and terminal agents too.

pi jobs here; protocol agents and terminal agents with `resume` since [M40](m40-agents-resume.md).

## Checked

- Tests: a crash loop restarted 3 times then failed; `off` restarts nothing; a host
  that died while the service was down resumed with "Continue where you left off."; a host
  killed under a working run restarted (`crash`: failed); stop while a restart waits; a
  restart pending across a service restart; a reply, an attach, a failed start and a
  `/stop` while one waits; signal deaths read as failures. 363 tests. The review found a
  start that failed while a restart waited left the row waiting for ever, attach losing
  the "continue", stop and rm not telling a parent: fixed.
- Eval, real pi on haiku: the worker SIGKILLed 6 s into a 15 s `sleep`; restarted once,
  pi saw the interrupted tool call, ran it again, wrote the file: 3/3.
- Stress: first run of the branch showed 1000 replies in 8 s instead of 5 and hub event
  lag p95 ~1000 ms instead of ~100: every idle report wrote `crashRestarts: 0`, which
  had been made a durable key (an fsync per report). It is written only when there is a
  count, and is not durable (losing a reset to a crash only leaves fewer restarts).
  After: 4.8 to 4.9 s, p95 119 to 142 ms, as main.