# Conditional Thread Continuation — `run_job`

**Status:** Gateway-native since task 139 (2026-08-15). Supersedes the
watcher-with-deadline convention adopted 2026-07-26, which is described at the
end for anyone reading old threads.

## The problem

A thread kicks off something long-running with an uncertain finish time — a
compile, a test suite, a deploy, a download. The thread should continue **the
moment the outcome is known** (success _or_ failure), not after an arbitrary
sleep, and not by burning a scheduled AI turn every N minutes to poll ("did it
finish yet?" cron turns dilute thread history — the exact failure class tasks
088–092 fixed).

## The answer

```
run_job("npm run build", label: "build")
  → { started: true, id: "job_a7f3c2d1", logPath: "..." }
```

Then **end the turn**. When the job exits, the gateway starts a new turn on the
thread carrying the exit code, the working directory, the log path and the tail
of the output.

Companions: `list_jobs()`, `job_log(id, tail?)` (works while it is still
running, so progress can be checked without waiting), `cancel_job(id)`.

### Write the command plainly

No `&`, no `> log 2>&1`, no `nohup`, no `setsid`. The gateway supplies all of
that, and adding your own breaks the log capture that the completion report
reads from. A command ending in `exit N` is fine — the status is captured by an
`EXIT` trap, not by trailing statements.

### For a condition that is not a process exit

Put the wait inside the job. A job _is_ a shell, so the same loop that used to
go in a watcher goes here, and it inherits supervision and reporting for free:

```
run_job("until curl -sf http://localhost:3000/health; do sleep 5; done", label: "wait for health")
```

### When NOT to use it

A command that finishes within a couple of minutes should just run in the
foreground. If it runs past the Bash tool's limit the harness backgrounds it and
notifies the thread, which keeps waiting in-turn — correct for twenty minutes,
wrong for six hours. `run_job` is for the second case, and for when the thread
should be free meanwhile.

## Why this is a tool and not a documented recipe

Cumulus tried the documented-recipe route three times — task 108 (redirect both
streams), 109 (watcher + deadline), 116 (the worked one-liner in the content
store) — and it kept failing in the world, at a standing cost of ~290 prompt
tokens on every turn of every thread. Three measured reasons:

1. **The cure had to be applied at every layer.** A driver script that redirects
   its own output does not save the children it spawns. @ordimor got two of
   three layers right and lost the job anyway.
2. **The watcher could die independently of the job.** It was itself background
   work with its own redirect requirement. Task 109's postmortem found exactly
   that: the watcher never fired.
3. **One killer was not curable from the thread side at all.** systemd's default
   `KillMode=control-group` reaps every process in the unit's cgroup when the
   main process exits, and `systemctl reload` makes the gateway exit (Rule #9).
   Neither `setsid`, `nohup`, nor redirecting escapes a cgroup — a cgroup is not
   a session. Every reload killed every background job on the box.

`run_job` removes 1 and 2 by construction: the daemon spawns the job with a file
descriptor rather than a pipe (so SIGPIPE is unreachable, not cured) and outside
the turn's process tree (so interjecting cannot reach it), and the registry that
reports completion cannot die independently of the daemon that owns the job.

Killer 3 needs `KillMode=process` on the service unit — shipped in the unit
template generated by `cumulus-gateway setup`. **Without it, jobs still work but
do not survive a gateway restart.** They are not lost silently either way: on
startup the registry checks every recorded job, re-adopts the ones still
running, and reports the rest — with their real exit code if the job got far
enough to record one, otherwise honestly as `interrupted`, which explicitly says
the work may or may not have completed.

## `schedule_trigger` is still here, for a different job

`schedule_trigger({id, trigger: "once"|"cron", at|cron, message})` injects a
message into the thread at a **time**. That is deferred reminders, drip
sequences and genuinely periodic work — not condition-waiting.

It is no longer needed as a deadline backstop for background work: that leg
existed because an unsupervised watcher could vanish, and the registry cannot.

### Anti-pattern: cron polling turns

Do **not** use `trigger: "cron"` to poll a condition every N minutes. Each poll
burns a full AI turn and appends "checked, nothing yet" noise to thread history,
degrading retrieval for every future turn.

## Superseded: the watcher-with-deadline convention (2026-07-26 → 2026-08-15)

Threads used to compose two primitives by hand: a detached `curl` to
`POST /api/agents/inject` wrapped around the job, plus a one-shot
`schedule_trigger` as the floor in case the watcher died. Recorded here only so
older thread history reads coherently — do not write new ones.

Its four known limitations are what `run_job` was built to remove:

| Limitation of the convention                                              | Status                                                        |
| ------------------------------------------------------------------------- | ------------------------------------------------------------- |
| No supervision — a bare background process, nothing restarts or tracks it | Fixed: the registry owns the job                              |
| No visibility — no way to enumerate or cancel a pending watcher           | Fixed: `list_jobs` / `job_log` / `cancel_job`                 |
| Needed a gateway API key in the script's command line                     | Fixed: no key involved at all                                 |
| Reboot amnesia — schedules survived restarts, watchers did not            | Fixed: adoption on startup, or an honest `interrupted` report |

The key point was also a real hazard rather than a theoretical one: the recipe
read `apiKeys[0]` — the gateway's **admin** key — and because it was seeded into
every thread's content store as topic-free boilerplate, it surfaced on
unrelated queries and taught an app's visitor-facing model to invent HTTP calls
against the gateway (task 134). `run_job` needs no credential, and the seeded
recipe has been deleted.