# How to run a worker on another machine The pane is local and the pi process is remote. mu never connects and knows no ssh flags: a host profile holds your own command templates, and `mu agent spawn --host` fills them. The full recipe, with collecting and the traps, is in [skills/mu/recipes/remote-workers.md](../../skills/mu/recipes/remote-workers.md). Read it before your first remote spawn. ## Prepare the host 1. On the host, install pi and the mu package: `pi install npm:@mu-crew/mu`. The host needs no mu DB: a remote worker cannot run `mu task` verbs. It ends with a `CLOSE:` block. `mu wait` applies it when it sees the run settle; `mu agent wait` does not. Re-run the evidence first, or close the task yourself. 2. Allow unix-socket forwards in the host's sshd: `AllowStreamLocalForwarding yes`. 3. Have a git repo there for `--worktree` to branch from. One profile serves every repo on the host: `--max-sessions` caps the machine, so never add the same machine twice. Use one orchestrator DB on your machine. Do not run a second mu on the host for the same work: task ownership points into your local `agents` table. ## Add a profile and check it ```bash mu host init dev --ssh # prints a starter `mu host add`; runs nothing mu host add dev --shell '' --exec '' --fetch '' --max-sessions 1 mu host check dev --root '~/repo' # host answers, repo, pi, mu extension ``` ## Spawn the agent ```bash mu agent spawn worker-1 -w auth --host dev --worktree --root '~/repo' --task build --send - <<'EOF' EOF ``` - The pane runs the profile's `--shell` template, filled with the remote identity (`MU_MANAGED_AGENT`, `MU_AGENT_NAME`, `MU_WORKSTREAM`, `MU_CTL_SOCK`, `MU_REMOTE`, never your DB, state or sync dir, which are unset in the pane), your `--command` (default `pi`), the local and remote socket paths and the remote cwd. The template starts the remote pi and forwards its control socket to the local derived path. - `--worktree` makes a detached `git worktree add` in the remote repo `--root` names (default: the profile's `--repo`; with neither the spawn refuses) through its `--exec`, at `~/.local/state/mu-next/remote-worktrees//` on the host. Its path, base commit and the repo's absolute toplevel are the agent's `cwd`, `base` and `repo`; close and recovery use that repo, never the profile's. `--cwd` names an existing remote directory instead; without either, the agent starts in `--root` (default the profile's repo), else the login directory. Without a repo, templates get `.` for `{cwd}`. `--root` is part of a `--task` attempt's identity: the same attempt in another repo conflicts. - `--task` is the same one journaled attempt as for a local agent: claim for the new agent, then the brief, never sent twice. - mu reserves the host's channel before its first remote command and keeps it while the pane lives (`--max-sessions` counts it). A full host refuses the spawn up front and names the holder. A failed spawn removes only the worktree it made (still clean, still at its base) and frees the slot. If the host's answer was lost after it made the worktree (a timeout), spawning the same name again takes that tree over while it is clean, at its base and unused; anything else at the path refuses. - The pane is the transport: closing it, or its death, ends the agent and frees the slot. tmux and Rex; herdr refuses `--host` in one line. - A non-pi `--command` (`--host dev --command 'npm test'`, no `--cli pi`) is a **command gate**, not a worker: its pane is kept after the command ends, and `mu wait agent:auth/gate-1` returns the command's own exit status (the `exit` of the result and of the process; 23 stays 23). Never a held `ssh dev …`. The remote command writes a token-bound completion record that mu reads once the pane has exited; no record, a signal or a lost transport is exit 6, never 0. `mu agent close gate-1` frees the pane and removes that record. No `--worktree`, no `--task`. - If the spawn reports `ctl refused` or `missing`, run `mu host check dev --agent worker-1 -w auth`, then check `AllowStreamLocalForwarding` on the host. ## Drive it like a local agent `mu agent send`, `wait`, `abort` and `check` go through the forwarded socket and behave as they do locally. `mu state` shows remote workers as agent records with `host`, `cwd`, `base` and `collected_head`. ## Collect and close ```bash cd ~/repo # the local repo the commits come home to mu agent collect worker-1 -w auth # runs the profile's --fetch, prints the merge git merge # mu prints the exact sha; it never merges mu agent close worker-1 -w auth # preview, then --if ``` - `collect` runs the `--fetch` template locally, in the current directory's git repo, under one channel. It pins the fetched commit as `refs/mu/collect//` (gc keeps it) and records it as the agent's `collected_head`. A fetch that fails, or succeeds without leaving a commit, records nothing. Collecting the same HEAD again says so. - The pane is the transport, so a live worker holds its host's channel. At `--max-sessions 1`, `collect` is refused with the holder named; raise the cap to collect while the worker runs. - `close` kills the pane first (that frees the channel), then removes the remote worktree **only if** it is clean, its HEAD is the base, the collected HEAD or on a branch (or remote-tracking ref) of the root repo (checked again just before the removal), and no process on the host is inside it. Its `git worktree remove` never uses git's `--force`. A process inside it, or a host with neither /proc nor lsof to check, refuses naming `close --force`, which removes past either with a warning. A newer HEAD refuses with one line: `collect` again, or `close --keep-branch` to keep the commits on the remote branch `mu//-` and remove the worktree. Uncommitted files always refuse (`--discard` and `--keep-branch` do not override that). - A `--cwd` agent has no remote worktree of its own: close never touches the host. When a row is deleted (its pane died and the reaper took it, a workstream teardown, a peer's workstream delete, even a raw SQL delete) the database keeps a machine-local `remote_recovery` record of the host, path and base (`mu sql "select * from remote_recovery"`; `mu rebuild` loses it). A failed spawn whose own rollback removed the tree (or found it gone) leaves none. With no agent row left, `mu agent close -w ` (or `/`) previews and closes THAT worktree under the rule above (no pane, no task; the recorded host profile only), and `mu agent collect ` fetches from it. Several records of one name need `--recovery `; a current agent of the name or path, a newer record of the same path, or a changed `--exec`, refuses (a changed default repo does not). ## Host profiles A host profile stores your own command templates, so you stop retyping them. mu never connects and knows no ssh flags: it fills the templates and runs them with `/bin/sh -c`. `spawn --host` uses them (above); `agent collect` runs the `--fetch` template. ```bash mu host init dev --ssh # prints a starter `mu host add`; runs nothing mu host add dev --shell '' --exec '' --fetch '' --max-sessions 1 mu host check dev --root '~/repo' # host answers, repo, pi, mu extension mu host show dev # templates and the holders of its channels ``` - Placeholders: `{env}` (the five `MU_*` identity variables only), `{cmd}`, `{local_sock}`, `{remote_sock}`, `{cwd}`; a lease profile adds `{job_file}` (its `shell`, exactly once) and `{job}` (its `stop`, required). A name valid in only some fields is refused in the others. Each value is one quoted word and is exactly the text you passed, also inside your own `"..."` or `'...'`. A template that hands command text to a second shell (ssh's remote login shell) says so per placeholder: `{cmd:sh}`, `{cwd:sh}` quote the value for that shell too. Write `{{` and `}}` for literal braces. Anything else in braces is refused. - A command that times out, or whose caller is killed, takes its whole process group with it (TERM, then KILL after 5 s) before its channel is freed, so `--max-sessions` never undercounts a live transport. - An `exec` or `fetch` template that exits 3 is a hand-back (a hardware key, a prompt): mu stops with exit 5, `human needed`, and the template's first line. Never retried; `host check` reports a `handback` step. - A `shell` template must not ride a shared ssh master (keep `-o ControlMaster=no -o ControlPath=none`, as `host init --ssh` prints): a `-L` through a master is bound by the master, outlives the pane, and is never re-bound, so the next spawn of that name gets no socket. `host check` warns when the template lacks either option. On a session-capped host use a lease ([below](#session-capped-hosts)). - `mu host show` and `list` print the transport (`session` or `lease`) and, for a lease, the `stop` template. - `--max-sessions N` counts every channel mu holds to the host. At the cap mu refuses with one line naming the holder. - `host check --agent -w ` also probes that agent's forwarded control socket, and on a lease agent reports its recorded job handle. ## Session-capped hosts Some hosts allow one ssh session per connection (`MaxSessions 1`, the Meta devvm). A `session` worker's pane holds that session for the worker's whole life, so every other `ssh` to the host (the murmur collector, `git fetch`, you) is refused, and it shows as a misleading `Permission denied`. A `lease` profile fixes that: the pane runs a local **carrier** that owns a detached remote job on its own connection, so no session is held. ```bash mu host init dev --mule # prints the starter below; runs nothing mu host add dev --transport lease \ --shell 'mule run --host dev --lease --human --cwd {cwd} --job-file {job_file} --forward-unix {local_sock}:{remote_sock} -- env {env} sh -c {cmd}' \ --exec 'mule exec --host dev -- sh -c {cmd}' \ --stop 'mule rm --host dev {job}' \ --fetch 'git fetch ssh://dev{cwd} HEAD' mu host check dev --root '~/repo' ``` - The `shell` template must contain `{job_file}` once, `{local_sock}` and `{remote_sock}`: the carrier writes the job handle to the file, and mu records it in the name's `carrier_jobs` reservation (machine-local; `mu sql "SELECT * FROM carrier_jobs"`). The reservation is taken before the pane opens. While it exists, every spawn of that name refuses. Once a carrier may have launched, a failed spawn that never got a handle keeps the reservation and the tree for `mu agent close `. `--stop` must contain `{job}`, and must be idempotent: exit 0 for a job that already ended. `{env}` is refused there, `{cwd}` is always `.`. - `local_sock` determines `remote_sock`: both derive from (workstream, agent), so a carrier's forward table never maps one local socket to two remote ones. - A lease pane holds no host channel, so `--max-sessions` counts only what mu opens itself (`exec`, `fetch`, `stop`, `check`), and `collect` works while a lease worker runs. - Why `--fetch` stays plain ssh: a fetch is a bulk transfer, and the carrier's channel is never lent to one. - Why a session template must not ride a shared master: a `-L` asked of a control master is bound by the master, not by the client, and outlives the pane. mu then deletes the socket file but the master keeps the listener; the next spawn of that name asks for the same forward, ssh exits 0 and no file appears (`ctl: missing`, for that name, for good). A lease carrier binds and verifies its own forward; a session template keeps `ControlMaster=no ControlPath=none`. - mule jobs run in a non-login shell: a host that finds `pi` only through login files edits `sh -c` to `zsh -ilc`, or fixes `PATH` in `.bashrc`. `host check` runs `pi` without a login shell on a lease profile for that reason. - An exit 3 from any host template (`stop` included) is a hand-back: stop and ask a human. mu never retries it. - `agent spawn --host` on a lease profile waits for the job handle (`MU_SPAWN_JOB_MS`, default 30 s) even under `--no-ctl`. A spawn that fails after the handle runs `stop` before it removes the worktree. A failed stop keeps both the handle and the worktree, and a respawn of that name refuses until `mu agent close ` stops the job. A lease profile runs no command gates. - `agent close` on a lease agent kills the pane and proves it gone as usual, then runs `stop` for the job, and only then removes the worktree. A stop that fails (the template's first stderr line is named), hands back (exit 3) or finds the `stop` template changed keeps the agent and its handle: fix that, then close again. `--force` never skips the stop. A handle the spawn had not recorded yet is read from the job file; with neither, close refuses (`spawn in progress`). - With no agent row left (the carrier was `kill -9`ed and reconcile reaped it, a teardown, a raw delete), `mu agent close -w ` stops the job first, then closes the recovery worktree, if any. `agent collect` never stops a job. - A host that is gone for good: `mu agent close --abandon-job` drops the handle without a stop and logs the host and handle; the job may still run there. It is the only way past an unreachable host. ### Smoke-test a lease end to end `scripts/smoke-lease.sh` drives a real mule against a loopback `sshd MaxSessions 1` on this machine (no other host), with a ctl-speaking fake pi (`scripts/smoke-fake-pi.mjs`) as the remote job. Vitest never runs it: ```bash MU=~/hacking/crew/mu MULE=~/hacking/crew/mule bash scripts/smoke-lease.sh ``` - It builds both (`cargo build --release`, `npm run build`), then checks spawn (a job handle, no host channel, the forward bound by mule's master), a second master with `ssh -S` during the worker, `collect` at `--max-sessions 1`, send and wait (the settled `lastText` must echo the sent text), and a close that leaves no job, forward, socket, job file, worktree or `carrier_jobs` row. It then `kill -9`s a carrier, lets reconcile reap the row, and closes by name, which must stop the job. - Exit 0 is a pass. A missing prerequisite (sshd, ssh, tmux, node, cargo, lsof, `MULE`, `node_modules`) exits 2. It is never a pass. - Everything is private: its own `/tmp/mu-smoke.*` dir, `HOME`, `TMPDIR`, `XDG_*`, `MU_DB_PATH`, a blank `MU_SYNC_DIR`, a private tmux server and mule `--config`. Your mule config, mule state and mu DB are never read. An exit trap stops every process it started, on failure too. `SMOKE_KEEP=1` keeps the dir for a post-mortem. Spec: [lease transport](../specs/2026-10-28-lease-transport.md).