# boxery backends

`host/boxery-host` is backend-agnostic. It generates each VM's identity (cloud-config with ssh keys, LLM proxy token, pi/opencode config), manages tokens, leases and locks, waits for ssh, and runs the LLM proxy. A backend is a bash file that implements the functions below.

The host (Node) runs one backend function per process: `host/backend.sh FUNCTION ARG…` sources the backend into a small prelude and calls that function, with arguments as argv. The prelude is everything a backend may use from the host: `die`, `die_unchanged` (exit 3, see `backend_reimage`), `log`, `need`, `wait_ssh NAME HOST [PORT]`, `vm_ssh_cmd NAME HOST PORT CMD…` (both check VM NAME's pinned host key), `profile_unit BASE` (the name for a host-wide unit in this profile: `BASE` for the default profile, `BASE-NAME` for profile NAME), and the variables `ROOT`, `CACHE`, `PROXY_PORT`, `PROXY_UNIT` (this profile's proxy unit) and `SSH_OPTS`. Its settings come from the environment and `$BOXERY_CONF/host.env`. Only functions named in this contract can be called. Because every call is a fresh process, a backend can't keep state in shell variables between calls; keep it on disk or in the hypervisor. `boxery-host connect` (the ssh ProxyCommand) also goes through `backend.sh`, without starting Node.

## VMs

| Function | Contract |
|---|---|
| `backend_check` | verify dependencies (fail with `die`) |
| `backend_init` | one-time host setup (storage pools etc.) |
| `backend_proxy_services` | install host-side bridge services for the guest→proxy path; print unit names to enable (may print nothing). Name each with `profile_unit` and order it `After=$PROXY_UNIT.service`: two profiles of one backend run side by side, each on its own proxy port |
| `guest_proxy_upstream` | socat address the guest's `boxery-llm` service forwards 127.0.0.1:8088 to (e.g. `VSOCK-CONNECT:2:8088`, `TCP:192.168.5.2:8088`) |
| `backend_build_base` / `backend_has_base` | build / detect the golden desktop image. The cloud-config to build it from is `$BASE_USERDATA` (`image/base-user-data.yaml` with the [layers](#image-layers-desktop-and-headless-vms) filled in). `backend_has_base` exits 0 if the base exists and 1 if it doesn't; when it can't tell (a missing tool, an unreachable API), it says why on stderr and exits with another status, so `new` doesn't report a missing base |
| `backend_exists NAME` | 0 if the VM exists |
| `backend_new NAME CPUS MEM_MIB DISK_GIB USERDATA [SOURCE]` | create + boot a VM and apply the cloud-config in USERDATA (see *Identity* below). SOURCE: empty = the desktop base; `VM@LABEL` = that image; `stock` = the backend's stock image, no base needed (headless VMs, see [layers](#image-layers-desktop-and-headless-vms)). `stock` has no `@`, so it is never an image name |
| `backend_list` | lines `NAME<TAB>STATE<TAB>SSH<TAB>DISK_BYTES<TAB>DISK_USED_BYTES[<TAB>TUNNEL]` (VMs only, never images; empty field = unknown). SSH: the VM's ssh endpoint as this host reaches it, always `HOST:PORT` (`192.168.122.50:22`, `127.0.0.1:60022`), empty unless the VM is running and reachable. DISK_BYTES: the disk's configured (virtual) size. DISK_USED_BYTES: what the VM's own disk takes on this host (allocated blocks; a shared base or backing image is not counted), empty if the backend can't know. TUNNEL only for backends that reach VMs through a tunnel: `up`, `connecting`, `down`. `boxery-host` turns these into the `ls --json` objects (sizes in GiB) |
| `backend_ssh_endpoint NAME` / `backend_wait_endpoint NAME` | `HOST PORT` where the VM's sshd is reachable from the host (now / after waiting) |
| `backend_endpoint_problem NAME` | optional, local checks only (it runs while `boxery-host` waits for ssh). Print nothing and return 0 when nothing is known to be wrong; else print what's wrong and the fix, and return 1 (may recover: `boxery-host` keeps waiting for ssh and adds the message to its timeout) or 2 (won't recover without the fix: it gives up at once). `boxery-host` asks after each failed ssh attempt in `new`, `start`, `resize`, `revert` and lease resets; the prelude's `wait_ssh` doesn't. exe: the VM's tunnel |
| `backend_start/stop/rm NAME` | lifecycle. `rm` removes the VM, not its images (`boxery-host rm` does that) |
| `backend_size NAME` | `CPUS MEM_MIB DISK_GIB` as configured (not usage). `ls` asks it once for a VM without a size record and records the answer |
| `backend_resize NAME CPUS MEM_MIB DISK_GIB` | optional; empty = keep. Apply the new size. It may stop NAME, but **must leave it running if it was** (also on failure) and stopped if it was. `boxery-host resize` holds the VM lock, refuses to shrink the disk, drops values that don't change anything, and waits for ssh afterwards. libvirt: `setvcpus`/`setmaxmem`/`setmem --config` + offline `vol-resize`; Lima: `limactl edit` on the stopped instance |

### Optional functions

boxery-host asks the backend which functions it defines (`host/backend.sh __functions`) and falls back when one of these is missing:

| Function | Without it |
|---|---|
| `backend_resize` | `resize` fails with "the BACKEND backend cannot resize VMs" |
| `backend_proxy_bind` | the LLM proxy listens on `127.0.0.1` (no backend overrides it today; for one whose guests reach the proxy over a host interface instead of vsock or a tunnel) |
| `backend_endpoint_problem` | waits for ssh report only a timeout, not why the VM can't be reached (remote backends: [`remote_endpoint_problem`](#remote-backends)) |
| `backend_host_key NAME` | the guest's live ed25519 ssh host key (`ssh-ed25519 AAAA…`), read through the hypervisor rather than the VM network. boxery-host pins it for a VM that has no record (made before keys were pinned, or `boxery pin`). libvirt: the qemu guest agent. Without it, only loopback endpoints (Lima, the remote backends' tunnels) are trusted for that, or `pin --from-guest` |
| `backend_platform_host_key NAME` | for platforms that give every machine its own host key and report it (boat): that key. boxery-host then seeds no key of its own and re-pins this one whenever it waits for ssh |

### Persisted settings

`boxery-host init` saves the backend's settings in `$BOXERY_CONF/host.env`, and later commands export the saved values that the environment doesn't set. The keys are listed once, in [host/host-env.list](../host-env.list), which both [host/backend.sh](../backend.sh) and [host/config.mjs](../config.mjs) read; a new backend adds its `BOXERY_<NAME>_*` keys there. Never list secrets: host.env isn't private (boat keeps its API key in its own 0600 file).

Platform support, requirements and capabilities are declared in
[`backends/index.mjs`](../../backends/index.mjs). See the
[queryable support contract](../../docs/backends.md); every new user-facing backend
must add a declaration (one without runs unchecked: no preflight, capabilities unknown). The `fake` backend is test-only.

## Images: one primitive for snapshots, forks and pool resets

An **image** is an immutable copy of a VM's disk, named `VM@LABEL`. boxery-host builds everything on four functions:

| Function | Contract |
|---|---|
| `backend_capture NAME IMAGE` | copy NAME's current disk into a new image. **Must work while NAME is running** (it may stop it briefly, but must leave it in the state it found it). Must not die leaving a partial image |
| `backend_images` | lines `IMAGE<TAB>CREATED<TAB>SIZE` for every image |
| `backend_image_rm IMAGE` | delete an image. VMs made from it must keep working |
| `backend_reimage NAME IMAGE` | replace NAME's disk with a copy of IMAGE, **keeping NAME's identity** (endpoint, MAC, seed). Ends running. When it fails before changing NAME at all (no such image, a copy that failed or was refused), it exits 3 (`die_unchanged`): `revert` then drops its journal entry instead of leaving recovery to finish the revert later, over whatever the VM did since. Any other failure is finished by recovery |

| boxery operation | built from |
|---|---|
| `snap NAME LABEL` | `capture NAME NAME@LABEL` |
| `revert NAME LABEL` | `reimage NAME NAME@LABEL` |
| `new NAME --from IMAGE` | `new … IMAGE` |
| `fork SRC A B…` | `capture SRC SRC@fork-…`, then `new A … SRC@fork-…` (in parallel), then `image_rm` |
| `pool add NAME` / lease reset | `capture NAME NAME@boxery-clean` / `reimage NAME NAME@boxery-clean` |
| `rm NAME` | `rm NAME`, then `image_rm NAME@*` |

Rules every backend must follow:

- **Disk only.** Images never include RAM. Processes restart; runtime secrets that grants push to tmpfs are never captured.
- **Independent.** VMs made from an image, and images of those VMs, never depend on the source VM or image staying around. (libvirt pulls image data into the new overlay; Lima clones are full or copy-on-write copies.) This is what keeps fork-of-a-fork-of-a-fork cheap and lets snapshots be deleted in any order.
- **Cheap.** Captures and VMs from images should cost about as much disk as the data they change, not a full disk copy, where the platform allows it.
- **No locking needed.** boxery-host holds lifecycle resource locks through the complete workflow, including capture/reimage/rm, image readers, guest readiness and key installation. Backends don't need their own. See [lifecycle coordination](../../docs/lifecycle.md).

## Image layers: desktop and headless VMs

What boxery adds to the starting image is in three layers (and one optional one), `image/layers/*.sh` (root shell scripts, idempotent), run in order by `boxery-provision` (`image/layers/provision.sh`):

| Layer | What | Notes |
|---|---|---|
| `plumbing` | the `agent` user + sudoers, `boxery-llm.service` (socat bridge to the host LLM proxy; keys stay on the host), security updates (`apt-get full-upgrade`, kernel included), sshd `MaxSessions`, sysctls (no core dumps, no MPTCP), cloud-init default user, `/etc/profile.d/boxery.sh`, tmux config, `socat`, `qemu-guest-agent` | ~10 s, plus the upgrade: up to a few minutes on an old starting image |
| `devtools` | git (from the git-core PPA on Ubuntu), build tools, python3, jq, ripgrep, fd, node 24, `pi`, `opencode` | keeps what the image ships: a package only when missing (except git, upgraded from the PPA: Ubuntu 24.04's is 2.43), node only when the image's can't run the job helper (needs 22.18+), an agent CLI only when `agent` can't run the image's own (exeuntu's `pi` links into exedev's home, so it's replaced; see #30). Run again, it updates what it installed: its node 24 to the latest 24.x (`/etc/boxery/node-installed`; an older VM's `/usr/local` node unless the image is exeuntu) and the agent CLIs that are npm's global install of their package to their latest release. ~30 s |
| `desktop` | XFCE on `:1`, TigerVNC (password required: `boxery-vnc-password`) + noVNC, Firefox, Chrome (Playwright Chromium on arm64) + its AppArmor profile, `playwright-core`, xdotool, ImageMagick, the agent's desktop user services | ~2 min; the only reason for a built base. Every run installs the latest `playwright-core`; on arm64 it then installs that release's Chromium, points `google-chrome` and the AppArmor profile at it, and removes older builds (stopping a browser still running one). On amd64, Chrome and Firefox update from their apt repositories in `plumbing`'s upgrade |
| `github-runner` (optional) | the GitHub Actions runner, a pinned release checked against its pinned SHA-256, in `/opt/actions-runner` (owned by `agent`) with its `installdependencies.sh`, and `boxery-runner`, which the host's runner manager drives | headless VMs only: `new NAME --headless --layers github-runner`. No credential: each lease gets a single-use config ([GitHub runners](../../docs/github-runners.md)) |

- **Desktop VMs** (the default) clone the base, which `build-base` makes by applying all three layers once.
- **Headless VMs** (`new --headless`) call `backend_new … stock`. boxery-host's USERDATA then also carries every layer file (as `write_files`), `/etc/boxery/provision` (`plumbing devtools`, plus any `--layers`), a `users:` entry for `agent` (the stock image's default user isn't agent) and, as the first `runcmd` step, `boxery-provision`. Seeded backends (libvirt, cloud-hypervisor) get all of that from cloud-init. Backends that replay the cloud-config themselves (Lima, exe) must run `sudo /usr/local/sbin/boxery-provision` after writing the files. boxery-host waits for `/var/log/boxery-provision.ok` and shows `/var/log/boxery-provision.log` if it's missing.
- `boxery-provision` records the applied layers in `/etc/boxery/layers` and writes `/etc/boxery/headless` when `desktop` isn't one of them. Last, it writes `/etc/boxery/build-info` (`lib.sh`, `build_info`): `key=value` lines with `built_at`, `layers`, `layers_hash` (lib, provision and each layer file, hashed as `host/build-info.mjs` does), `kernel` (the newest installed) and, with the desktop, `chrome` and `playwright_core`. Each run starts with fresh package lists. The desktop tools (`vm_screenshot` … `vm_browser`, `boxery desktop`) check that file and refuse with a clear error. The host records each VM's kind and guest OS (`linux`, the only one so far: [guests/](../../guests/)) in its store (`vm_records` in `$CONF/leases.db`; no record = desktop, `linux`) for `ls`.
- Images, snapshots and forks need nothing extra: they copy the disk, so a VM from a headless VM's image is headless and isn't provisioned again. A VM keeps its kind for life; `--headless` with a desktop image is refused.
- **Build records and `patch`** need nothing from backends either. The host reads `/etc/boxery/build-info` (and `/var/run/reboot-required`) over ssh when it reaches a VM anyway, and keeps it in `$CONF/build-info/` per VM and per image (copied at capture, back at a pool reset). `patch` (`host/patch.mjs`) runs over ssh as `agent` with sudo: it writes the `__layer_files` entries into the guest, runs `boxery-provision` with the layers in `/etc/boxery/layers`, and restarts the VM from inside (`systemctl reboot`), waiting for a new boot id through `backend_ssh_endpoint`, so a backend must let a guest reboot itself and come back on its own, at an endpoint `backend_ssh_endpoint` reports (a VM that isn't back within `BOXERY_SSH_WAIT_TRIES` × 2 s fails the patch). For a pool VM it then removes and captures `NAME@boxery-clean` with `backend_image_rm` and `backend_capture`, under the VM's lifecycle lock; a crash in between is recovered by reimaging the VM from its clean image, or marking it broken if there is none (`pool-patch` in `host/recovery.mjs`).

Measured headless `new` (Lima vz on Apple Silicon, November 2026): 51 s from `limactl create` to ready (plumbing 10 s, devtools 30 s), versus ~13 s for a clone of the desktop base, and no `build-base`. Snapshot 10 s, fork 20 s, `new --from` 10 s, as for desktop VMs. That's fast enough that there is no cached headless base; if one is wanted later (CI fleets creating many VMs), it would be `build-base` with only `plumbing devtools`.

## Identity: what a VM made from an image must get

A VM made from an image starts with someone else's disk. boxery-host's cloud-config (`boxery_seed`) and post-boot checks take care of the guest side on every backend:

- new hostname, proxy token, pi/opencode config: cloud-config `write_files`/`runcmd`, as for any new VM
- **a new ssh host key** (`host/hostkeys.mjs`): `write_files` puts it in `/etc/ssh/boxery_host_ed25519_key` with an sshd drop-in (`05-boxery-hostkey.conf`) that makes it sshd's only host key, and boxery-host records the public half. Every ssh to the VM checks it. A backend that replays the cloud-config instead of seeding it must make sshd reload after writing the files (`remote_identity_script` restarts sshd when the user-data carries a key)
- **`authorized_keys` reset**: a per-instance `bootcmd` deletes it before sshd starts, then cloud-init writes this VM's keys. boxery-host then sets them again over ssh. A fork of a leased VM never accepts the lease holder's key
- **fresh machine-id**: per-instance `bootcmd`
- pushed grants (`/run/user/<uid>/boxery`) cleared. It's tmpfs, so it's normally gone already

The backend's job is the hypervisor side:

- a new MAC address, vsock CID, UUID and seed. Never reuse the source's
- a network identity that doesn't depend on the guest's machine-id. libvirt passes a network-config with `dhcp-identifier: mac`, so a fork never gets its source's DHCP lease
- if the backend can't deliver a cloud-init seed (Lima), apply USERDATA itself and regenerate the machine-id

## Crash recovery contract

The supervisor journals intent before side effects and stops an abandoned worker group before replay. Every backend also implements these **idempotent** helpers, including artifacts not visible in its ordinary inventory:

| Function | Contract |
|---|---|
| `backend_cleanup_new NAME` | Remove an incomplete VM and its partial disks/seed/staging instances, even if no domain/instance was fully registered. Does not remove unrelated images |
| `backend_cleanup_image IMAGE` | Remove a complete or partial image, including a clone killed before its image metadata was published; succeed if already absent |
| `backend_recover_capture NAME IMAGE ORIGINAL_STATE` | Cancel outstanding capture jobs, thaw/restore capture-related source state as needed, then remove the incomplete image. Never remove the source VM |

`backend_reimage` must be retryable after an interruption between destructive steps. libvirt tolerates a missing old disk; Lima resumes a staged replacement after the original instance was deleted. Return errors if cleanup cannot complete so the supervisor retains the journal/locks for retry. Inventory commands must distinguish a healthy empty result from an unavailable backend. Do not hide an inventory failure as an empty host.

These hooks complement process-group shutdown: submitted hypervisor jobs can survive the CLI that launched them. Real-hypervisor fault testing is still required in addition to fake-backend tests.

## Implementations

| Backend | Platform | Images |
|---|---|---|
| `libvirt` | Linux + KVM | qcow2 overlays on a root volume: the base, or for headless VMs `boxery-stock.qcow2` (`BOXERY_STOCK_VOL`, the cloud image as downloaded, uploaded on the first headless `new`). An image has its VM's root; `build-base` only refuses while desktop VMs or images use the base. Images are `$PREFIX$IMAGE.qcow2`. Capture = shallow live `blockcopy` + `domfsfreeze` through the guest agent (no pause; a stopped VM is booted `--paused` for the copy). New-from-image = overlay on the image + `blockpull --base`, so every disk is one overlay deep on the base. Disks are attached with `discard=unmap`, so space the guest frees (Ubuntu mounts root with online discard; `fstrim` weekly) leaves the overlay instead of being copied into every image, fork and reimage made from it. Domains made before this get it from `init`, effective on their next cold start (stop + start, revert, lease reset). Guest isolation: every interface on `BOXERY_NET` references the nwfilter `boxery-guest-isolation` (no spoofing, DHCP/DNS only to the host, private ranges dropped; the cloud-hypervisor policy). `init` (re)defines the filter and adds it to existing domains; `new`, `start`, `reimage` and `resize` add it before booting, so older domains get it on their next cold start. `BOXERY_NET_ISOLATION=off` removes it the same way. Measured: capture ~1 s, fork into 2 running VMs ~17 s, 3 VMs with a 3 GiB working set ~23 s |
| `lima` | macOS (vz) or Linux (qemu) | Headless VMs are fresh instances of Lima's `ubuntu-24.04` template, not clones of the base. Images: stopped Lima instances `boxery-img.<hash>`, made with `limactl clone` (copy-on-write on APFS/btrfs). Lima only clones stopped instances, so capture stops a running VM briefly. Works on `vz`, unlike Lima's own snapshots. Lima maps the guest's gateway to the host's loopback, so on macOS every start runs under `sandbox-exec` with a profile that lets the host agent reach only the proxy and the instance's ssh port on `127.0.0.1` (pinned with `--ssh-port`; confined qemu instances use slirp's DNS, `--dns 192.168.5.3`). `BOXERY_LIMA_CONFINE=0` turns it off |
| `cloud-hypervisor` | Linux + KVM | **Experimental.** qcow2 overlays on the base, or for headless VMs on `$BOXERY_CH_DIR/stock.qcow2` (a copy of the cloud image); images are `$BOXERY_CH_DIR/images/IMAGE.qcow2`. Capture = guest `sync`, pause (v54 drains and flushes qcow2 metadata), reflink copy of the overlay, resume. New-from-image and reimage = reflink copy of the image. See [below](#cloud-hypervisor-experimental) |
| `exe` | anywhere with ssh ([exe.dev](https://exe.dev)) | Headless VMs are new exeuntu VMs (`exe.dev new`), not copies of the base. Images: paused exe.dev VMs made with `cp` (a copy boots fresh from disk), named `PREFIXi-HASH`, with the image name in the exe.dev comment. Reimage = `cp` to a new exe.dev name, then `rm` the old one (names are never reused). See [below](#exe) |
| `boat` | anywhere with ssh ([boat.dev](https://boat.dev)) | **Experimental.** No headless VMs yet. Images: boat saved snapshots named `PREFIXi-HASH` (exact, independent disk copies of a running sandbox), the image name kept in `$BOXERY_BOAT_DIR/images/`. New-from-image = a sandbox `from` the snapshot. Reimage = a new sandbox from the image, then delete the old one. See [below](#boat-experimental) |
| `fake` | tests | directories |

### Writing a new backend

The image contract maps onto most VM platforms:

| Platform | capture | new from image | reimage |
|---|---|---|---|
| exe.dev (implemented: `exe`) | `cp NAME IMAGE-vm`, then `pause` it | `cp IMAGE-vm NAME` | `cp IMAGE-vm NAME-NEWGEN && rm NAME-OLDGEN` |
| boat.dev (implemented: `boat`) | `POST /saved-snapshots {sandboxId, name}` | `POST /sandboxes {from: name}` | new sandbox from the image, then `DELETE` the old one |
| Firecracker / Cloud Hypervisor | reflink-copy the rootfs (`cp --reflink`) | boot from a reflink copy | swap the rootfs file |
| ZFS / btrfs-backed anything | `zfs snapshot` + `zfs clone` | `zfs clone` | `zfs rollback` or re-clone |

Run `bash test/shell/host-images.sh` against the fake backend to see the behaviour every backend has to match.

### What stable means for a backend

A backend is **stable** when every line below holds. This is the bar libvirt and Lima were promoted on (offline suites, `e2e.sh` 19/19 on both, their `tested` entries), written down. Lines 6 and 7 matter most for hosted backends, where the platform keeps working after the CLI dies and the host is a separate machine. A backend that misses a line stays experimental, and its section says which lines it misses.

| # | | Requirement |
|---|---|---|
| 1 | Contract | Every function in the [VM](#vms), [image](#images-one-primitive-for-snapshots-forks-and-pool-resets) and [crash recovery](#crash-recovery-contract) tables, following their rules (images disk-only and independent, inventory errors never an empty host, only its own namespace touched). Each capability it doesn't offer is declared unsupported in [`backends/index.mjs`](../../backends/index.mjs) |
| 2 | Offline tests | `test/shell/host-NAME-backend.sh`, against command doubles (no hypervisor, no account), run by `npm run test:shell`: every lifecycle function, capture of a running VM, a failed create or capture leaving nothing behind, every recovery hook run twice, inventory failing when the platform is unreachable, a create killed mid-way recovered by `reconcile`, and the host's or account's other VMs untouched |
| 3 | Declaration | `stability`, `supported`, `requirements` and `capabilities` in `backends/index.mjs`, and a `tested` entry for every host configuration that passed lines 4–7 |
| 4 | Live: e2e | `e2e.sh` passes from a controller on current code, on every host OS the docs say the backend runs on. For a backend with host-side tunnels, that means both supervisors: systemd user units with linger (Linux) and launchd (macOS) |
| 5 | Live: lifecycle | What `e2e.sh` doesn't cover: `build-base`, `new --headless`, `new --from`, `stop`/`start`, pool lease and reset, and every account or plan option the docs offer (or that option is listed as unverified) |
| 6 | Live: interruptions | A create and a capture killed mid-way, then `reconcile`: no VM, image or local state is left, including work the platform finishes after the CLI died |
| 7 | Live: host reboot | After a host reboot with VMs running, each VM is in the state the docs say and is reachable, at most after `start` |
| 8 | Security | A guest reaches credentials only through the boxery proxy, with its per-VM token and grants. No host service or platform feature (integrations, credential stores, metadata services) gets it around them, unless the account owner turned that feature on and the backend warns about it wherever it reaches a VM. Its sshd accepts only boxery's lease-fenced keys, through the pinned endpoint |
| 9 | Security notes | Every remaining limit (what the platform, its admins or the account's key can do) is in the backend's security notes and [docs/security.md](../../docs/security.md) |
| 10 | Docs | `docs/install.md` gives requirements, account or plan prerequisites and setup; the backend's section here gives layout, lifecycle mapping, recovery, security notes and a verification record (date, host OS and arch, platform plan or version, timings). Nothing the docs state is contradicted by the platform, and no "Not verified yet" item touches lines 4–8 |
| 11 | One answer | Every statement of stability agrees: the `docs/install.md` table, the backend's heading here, `ARCHITECTURE.md` (backends row, roadmap), the README's Status, `stability` in `backends/index.mjs`, and the "works with" artwork |

Each live check is recorded where it was run: the backend's section here, and a `tested` entry in `backends/index.mjs`.

### Remote backends

A backend whose VMs live on a hosted platform (exe.dev, boat.dev, and likely Daytona, #61) faces the same problems every time: the guest has no cloud-init, the host may be a laptop behind NAT, and boxery-host wants a stable `HOST PORT` for each VM's sshd. [`lib/remote.sh`](lib/remote.sh) solves them once:

- **One supervised tunnel per VM** (`remote_tunnel_write/start/stop/active/remove`): `ssh -N … -L 127.0.0.1:PORT:127.0.0.1:22 -R 127.0.0.1:GPORT:127.0.0.1:PROXY_PORT` as the platform's admin user, run by a systemd user unit (Linux) or a launchd agent (macOS). `-L` is the VM's endpoint (`remote_ssh_endpoint`, `remote_wait_endpoint`); `-R` carries the guest's `boxery-llm` back to the host's proxy, so `guest_proxy_upstream` is `TCP:127.0.0.1:GPORT`. The backend passes the ssh argv; the lib owns the supervisor.
- **Tunnel state** (`remote_tunnel_state`: `up`, `connecting`, `down`, local checks only): the `TUNNEL` column of `backend_list`, and `remote_endpoint_problem` / `remote_ssh_endpoint` / `remote_wait_endpoint` say why a VM can't be reached (its tunnel's last error, where the log is, the fix) instead of an ssh timeout. `REMOTE_PLATFORM` and `REMOTE_CHECK` fill in the messages.
- **Host ports** (`remote_alloc_slot`, `remote_port`, `remote_clear_stale_slot_lock`): slot S ↔ `REMOTE_PORT_BASE + S`, kept for the VM's life (also across reimages).
- **Identity without cloud-init** (`remote_identity_script USERDATA NAME BOOT_SCRIPT regen|keep`): replays `boxery_seed`'s user-data as one root script, replaces `authorized_keys`, makes a fresh machine-id, records the hostname for a boot unit to restore, and regenerates ssh host keys unless the platform pins its own (`keep`, boat).
- **The base build** (`remote_build_script POST_FN OK_MARKER`, `remote_build_wait`): the agent user, `$BASE_USERDATA` and the backend's post-build script, run detached in a builder VM and polled. `remote_post_agent_session`, `remote_post_boot_unit` and `remote_hostname_restore` are the post-build pieces most hosted images need.

A remote backend sets `REMOTE_DIR`, `REMOTE_SUPERVISOR`, `REMOTE_PORT_BASE`, `REMOTE_MAX_SLOT`, `REMOTE_PLATFORM`, `REMOTE_CHECK` and `REMOTE_TUNNEL_WAIT`, defines `remote_unit NAME`, and then only has to talk to its platform's API: create, copy, stop/start, list, delete. Two rules carry over from exe.dev:

- **Guest sshd lets `agent` in from loopback only.** The tunnel's `-L` end arrives from `127.0.0.1`, so the lease holder's key works through the host and nowhere else (exe.dev: `ListenAddress 127.0.0.1`; boat.dev, whose own sshd must stay public: `Match User agent Address *,!127.0.0.1,!::1` → `DenyUsers agent`).
- **Inventory never reports an unreachable API as an empty account**, and every listing is filtered to the backend's own prefix or labels: the account's other VMs are never touched.

Select a backend with `boxery-host init --backend NAME` (persisted in `$BOXERY_CONF/host.env`) or `BOXERY_BACKEND`. A second backend on the same host is a second profile: `boxery-host --profile NAME init --backend B` (or `boxery host add NAME DEST --backend B` from the controller) gives it `~/.config/boxery-NAME`, a free proxy port and its own services. Init refuses a new profile with the same backend and the same namespace settings as an existing one; a backend's namespace keys are listed in `NAMESPACE` in [host/setup.mjs](../setup.mjs) (add yours there).

## OpenShell (experimental)

`openshell` is a restricted headless backend, not a full implementation of the VM/image contract. See [setup, trust model and smoke checks](../../docs/openshell.md). It uses prebuilt OCI images, OpenShell policy/providers and gateway-authenticated SSH. No root provisioning, Boxery proxy tokens, pushed grants, desktop, disk images, forks, resizing or pools are supported. Inventory is scoped to an explicit gateway, workspace and prefix.

Backends can optionally implement `backend_connect NAME` to relay SSH through their authenticated control plane instead of a TCP endpoint. The host's guest transport must use the same relay and retain strict pinned host-key checks. OpenShell bootstraps its platform-managed key through that authenticated relay. A gateway relay does **not** imply guest `authorized_keys` enforcement: OpenShell pool membership is refused.

## Guest contract

Ubuntu, user `agent` with passwordless sudo, sshd, `~/project`, `boxery-llm.service` reading `/etc/boxery/proxy-upstream`, and cloud-init (for per-instance `bootcmd`; exe and Lima replay it instead). Desktop VMs also have XFCE on `:1` with VNC/noVNC on localhost (VNC needs the password `boxery-vnc-password --new` sets and prints), xdotool + ImageMagick, a `google-chrome` binary and `/opt/boxery/node_modules/playwright-core`. The layers above provide all of it.

## cloud-hypervisor (experimental)

Runs VMs with [Cloud Hypervisor](https://www.cloudhypervisor.org/) directly, without libvirt. Tested end to end with Cloud Hypervisor v54 on arm64 (Ubuntu 24.04 in a nested-virtualization Lima VM). Not yet tested on an x86 host.

**Requirements:**
- `cloud-hypervisor` and `ch-remote` **v54 or newer**. Older versions can't open qcow2 overlays (no `backing_files`), and their `pause()` doesn't flush qcow2 metadata, so a capture could copy a stale disk.
- Firmware: `CLOUDHV.fd` (x86) or `CLOUDHV_EFI.fd` (arm64) from the cloud-hypervisor/edk2 releases, or `hypervisor-fw` from rust-hypervisor-firmware, at `BOXERY_CH_FIRMWARE` (default `~/.local/share/boxery/ch/firmware`).
- `qemu-img`, `cloud-localds`, `iproute2`, `nftables`, `socat`, read/write `/dev/kvm`.
- Passwordless `sudo` for network setup (`BOXERY_CH_SUDO`, default `sudo -n`; `""` when running as root), and `cap_net_admin` on the VMM so it can open its TAPs.

```bash
sudo setcap cap_net_admin+ep "$(command -v cloud-hypervisor)"
./host/boxery-host init --backend cloud-hypervisor
./host/boxery-host build-base
```

| | |
|---|---|
| Storage | `$BOXERY_CH_DIR` (default `~/.local/share/boxery/ch`): `base.qcow2`, `images/IMAGE.qcow2`, and `vms/NAME/{disk.qcow2, seed.iso, launch.sh, slot, serial.log}` |
| Process | `boxery-ch-NAME.service` (systemd user unit) runs `launch.sh`. `stop` presses the ACPI power button; `rm` and `reimage` stop the unit |
| Network | VM slot S: TAP `bxchS`, host `10.77.S.1`, guest `10.77.S.2`, MAC `52:54:00:77:…:S`, all static from the seed's network-config. A VM made from an image gets a new slot. `BOXERY_CH_NET` changes `10.77`; DNS is `BOXERY_CH_DNS` (default `1.1.1.1 9.9.9.9`) |
| Isolation | nftables table `inet boxery_ch`, replaced atomically on every `start`/`new`. Guests can't reach the host, each other, RFC1918, CGNAT/Tailscale, link-local or IPv6, and can't spoof source addresses (strict reverse-path check). Everything else is NATed out. The host can still ssh in |
| Proxy | Guest → `VSOCK-CONNECT:2:8088` → `vms/NAME/vsock.sock_8088` (symlink) → `proxy.sock` served by `boxery-ch-vsock.service` → `127.0.0.1:8088` |
| Recovery | `cleanup_new` stops and removes the unit, deletes the TAP and the VM directory (and a dead holder's slot lock); `cleanup_image` removes an image and its `.part`; `recover_capture` resumes a VM a dead capture left paused. `reimage` stages the new disk as `disk.qcow2.new` and renames it, so a retry after a crash is safe. Inventory fails if the systemd user manager is unreachable rather than reporting every VM stopped |
| Images | An image is a copy of a VM's overlay, so it depends only on the base. Capturing a running VM: guest `sync` over ssh, pause, `cp --reflink=auto` (instant on XFS/btrfs, a full copy of the overlay elsewhere), resume. There's no guest agent to freeze filesystems, so captures are crash-consistent after the sync. Measured on arm64 (4 GiB guests): capture ~1 s, revert or pool reset ~20 s (a reboot), `new --from` ~22 s, fork into 2 VMs ~25 s |

Known gaps:
- VMs don't autostart after a host reboot (`start` recreates TAPs and rules).
- Other firewalls with a FORWARD drop policy block guest traffic: Docker (allow `bxch*` in `DOCKER-USER`), ufw (`DEFAULT_FORWARD_POLICY`), firewalld.
- `net.ipv4.ip_forward=1` is host-wide, so the host also routes between its other interfaces.

## exe

Runs VMs on [exe.dev](https://exe.dev) instead of local KVM, so the host can be any machine with ssh (a Mac works). Everything goes through exe.dev's ssh API (`ssh exe.dev CMD --json`); the host needs an exe.dev account whose key ssh offers first, or `BOXERY_EXE_KEY`. Verified end to end on a Personal-plan account (November 2026): build, new, snapshots, revert, fork, `new --from`, stop/start, resize, rm; and again on 2026-11-04 on both host OSes, below.

```bash
ssh exe.dev whoami                  # must print your account, not "Please complete registration"
./host/boxery-host init --backend exe
./host/boxery-host build-base       # an exeuntu VM + image/base-user-data.yaml (~3 min), kept paused as the base (not needed for --headless)
```

| | |
|---|---|
| Names | exe.dev VM names are global, so every name starts with a per-host prefix (`BOXERY_EXE_PREFIX`, default `bx<6 hex>-`, generated by `init` into `$BOXERY_EXE_DIR/prefix`): `PREFIXvGEN-NAME` VMs, `PREFIXi-HASH` images, `PREFIXbase`, `PREFIXbuild`. GEN (6 hex) is new for every disk a VM gets (`new`, `revert`, pool resets), and the current name is kept in `$BOXERY_EXE_DIR/vms/NAME/vm`: exe.dev's edge keeps routing a deleted VM's name for minutes (`ssh` says "VM not found", `-R` forwards are refused), so VM names are never reused. The account's other VMs are never listed or touched |
| ssh | exe.dev's edge authenticates `ssh NAME.exe.xyz` with the **account's** keys and never uses the guest's sshd (exeuntu masks `ssh.service`). boxery's lease fencing needs the guest's own `authorized_keys`, so the build enables sshd on the guest's **loopback only** (`AllowUsers agent`, keys only), and each VM gets a supervised host-side tunnel: `ssh -N NAME.exe.xyz -L 127.0.0.1:PORT:127.0.0.1:22 -R 127.0.0.1:18088:127.0.0.1:PROXY_PORT`. The VM's endpoint is `127.0.0.1:PORT` (`BOXERY_EXE_PORT_BASE` + slot, default 22401…), as on Lima. Controllers and lease holders only ever reach the guest's sshd through it. The edge (account key → `exedev`, passwordless sudo) is the backend's admin channel for build, identity and `sync`, like host root on libvirt |
| Tunnels | systemd user unit `boxery-exe-PREFIXv-NAME.service` on Linux, launchd agent `dev.boxery-exe-PREFIXv-NAME` on macOS (`BOXERY_EXE_SUPERVISOR`); the label and port stay when the exe.dev name changes. Enabled while the VM runs, so they come back after a host reboot: on Linux at boot (with linger), on macOS when you log in (exe.dev VMs keep running while the host is down). The supervisor restarts a dropped tunnel every 5 s (`ServerAliveInterval` 15 s × 4 notices a dead connection within a minute). `ls` reports each tunnel's state (local checks, no API call): `up` (supervised and its `-L` end accepts connections), `connecting` (supervised, not connected: host offline or just awake, exe.dev unreachable, VM paused outside boxery) or `down` (not supervised); a running VM whose tunnel isn't up shows that state in place of its address. `connect`, `ip`, and the waits in `new`/`start`/`revert`/`resize` (`BOXERY_EXE_TUNNEL_WAIT`, 120 s) fail with that state, the tunnel's last error, where its log is (`$BOXERY_EXE_DIR/vms/NAME/tunnel.log` on launchd, `journalctl --user -u UNIT` on systemd) and the fix, instead of an ssh timeout |
| Host availability | **Run the exe host on an always-on machine** and use laptops as controllers only. Each VM's tunnel carries both its ssh endpoint (`-L`) and its only route to the LLM proxy (`-R`), so while the host sleeps or is offline, agents lose their LLM mid-task and nobody can ssh in; the VMs themselves keep running and the tunnels reconnect once the host is back. On Linux, enable linger so the user units run without a login session. `init` warns when the host has a battery (`pmset` on macOS, `/sys/class/power_supply` on Linux). LLM traffic stays on the host by design ([#42](https://github.com/modem-dev/boxery/issues/42)): exe.dev's HTTP-proxy integration or workload identity would put the trust at exe.dev and give up per-VM tokens, revocation, grants and logging, and a proxy on an exe.dev VM would put the keys where the account key is root |
| Proxy | Guest `127.0.0.1:8088` → socat → guest `127.0.0.1:18088` (the tunnel's `-R` end, `BOXERY_EXE_GUEST_PORT`) → host `127.0.0.1:PROXY_PORT`. Credentials stay on the host. exe.dev's own HTTP-proxy integration can't be used instead: its target must be a public URL |
| Identity | exeuntu has no cloud-init, so like Lima the backend replays the cloud-config over the edge (`host/cloudconfig-sh.mjs` for `write_files`), **replaces** `authorized_keys`, and gives every VM a fresh machine-id, ssh host keys and hostname. exe.dev's init resets the hostname and `/etc/hosts` to the exe.dev name at every boot, so the base's `boxery-exe-boot.service` restores boxery's (from `/etc/boxery/hostname`) and grows `/` to the disk exe.dev gave the VM. A copy boots fresh from disk, so RAM and tmpfs grants never travel. The devtools layer keeps exeuntu's `opencode` and `pi` if the `agent` user can run them and replaces them otherwise (exeuntu's `pi` links into exedev's home); exeuntu's `claude` and `codex` stay |
| Lifecycle | exe.dev has no stop/start. `stop` clears pushed grants (`/run/user/*/boxery`) and `sync`s, then `pause`s the VM (RAM snapshot to disk, frees its CPU and memory); `start` = `resume` (also cold-boots a `stopped` VM). `resize` = exe.dev `resize`, then a `restart` of a running VM so the guest sees it. A paused VM stays paused, and its next `start` is a `restart` instead of a `resume` (a resumed VM keeps its old CPUs, memory and disk). `backend_size` reports the size recorded at create/resize (after capping). `backend_list` reports that recorded disk size and no usage (exe.dev doesn't say what a disk holds). `ls` shows exe.dev's status (`running`, `paused`, `shut off` for `stopped`) |
| Sizes | A VM is capped at the largest the account allows, which `init`, `new`, `resize` and reimages read from exe.dev (`billing plan`: `max_cpus`, `max_memory_gb`, `max_disk_gb`) and keep in `$BOXERY_EXE_DIR/plan.json` for when exe.dev can't be asked (with neither, 2 vCPU / 4 GB, the smallest plan). On Personal that's the account's whole capacity, which all its VMs share: exe.dev doesn't refuse VMs that add up to more (it ran four 1-vCPU VMs on a 2-vCPU plan), they share the CPU. `BOXERY_EXE_MAX_CPUS` / `BOXERY_EXE_MAX_MEM` (MiB) override the limit, e.g. to a Work pool's size. A disk beyond `max_disk_gb` is refused. Disks only grow: `--disk` is passed when it's bigger than the source's (the base is 20 GB). Images and the base are paused, so they hold disk only |
| Images | `capture` = guest `sync` over the edge (a paused source was synced by `stop`), `cp VM PREFIXi-HASH --copy-tags=false` (~0.5 s; the copy boots from the disk as it is, unsynced writes included) (with the VMs' pool and egress policy: the image VM boots the agent's disk until it's paused), `comment` it `boxery-image:IMAGE:DISK_GB`, `pause` it. The source keeps running, untouched. A `cp` that dies before its comment is unlisted but still removed by `cleanup_image` (the name is a hash of the image name) |
| Headless | `new --headless` creates an exeuntu VM (`exe.dev new`), runs the plumbing + devtools layers and the base build's exe.dev specifics (loopback sshd, boot unit) over the edge, then the identity script. The devtools layer keeps exeuntu's node and build tools, upgrades git from the git-core PPA if exeuntu identifies as Ubuntu, and replaces only what `agent` can't run. With `BOXERY_EXE_EGRESS` set, egress is limited (`exe.dev egress VM …`) after the layers are installed, since they download packages. Live-tested (November 2026, Personal plan): headless `new` 42 s (plumbing 15 s, devtools 16 s: exeuntu's node 22 and `opencode` kept, `pi` installed), LLM bridge through the tunnel, snapshot 8 s, fork 19 s, rm. Not yet tested with `BOXERY_EXE_EGRESS` |
| Options | `BOXERY_EXE_EGRESS=integrations` cuts new VMs off the public internet (`cp --egress`); the `-R` tunnel and exe.dev integrations still work. `BOXERY_EXE_POOL` puts VMs in a Work-plan pool. `BOXERY_EXE_ALLOW_INTEGRATIONS` names integrations not to warn about (see the security notes). These, and the other `BOXERY_EXE_*` settings, are saved by `init` |
| Recovery | `cleanup_new` stops and removes the tunnel and deletes every `PREFIXv*-NAME` and the local state; `cleanup_image` deletes the image VM; `recover_capture` has no source state to restore. `reimage` copies to a new generation before deleting anything, then records it and deletes every other generation, so a retry leaves exactly one VM; `ls` ignores a generation that isn't the recorded one. Inventory (`ls --json`) fails if exe.dev is unreachable rather than reporting an empty account. A `cp` still running on exe.dev's side after its CLI was killed can land after cleanup: it shows up in `ls` (or `images`), and `boxery-host rm NAME` removes it |

Security notes:
- Anyone holding the exe.dev account's ssh key has root in every VM through the edge. Treat that key like host root; give boxery its own with `BOXERY_EXE_KEY` if you share the account. The backend's ssh never forwards the host's agent or X11 into a guest, whatever `~/.ssh/config` says.
- exe.dev [integrations](https://exe.dev/docs/integrations) keep credentials off a VM and use them for it: `llm` (the account's LLM credits, or keys stored in it), HTTP proxies, GitHub, S3, token mints, Anthropic and OpenAI workload identity. boxery lets VMs use them (they're the account owner's choice), but one that reaches a boxery VM works outside the proxy and its grants, and new accounts attach `llm` to every VM (`auto:all`). So `init`, `new`, `start` and reimages (revert, pool resets) warn about each integration exe.dev attaches to the VM, except `reflection` and `notify` (which hold no credential), with the command to detach it. They read `integrations list` (`auto:all`, `vm:` the VM, or `tag:` one of its tags). If that fails they say so and carry on. Detaching from `auto:all` affects the account's other VMs too; attach an integration to a tag to keep it for them. `BOXERY_EXE_ALLOW_INTEGRATIONS=NAME,…` stops the warning for integrations you mean VMs to use
- First connections trust exe.dev's host keys on first use (`$BOXERY_EXE_DIR/known_hosts`). exe.dev documents its fingerprint as `SHA256:JJOP/lwiBGOMilfONPWZCXUrfK154cnJFXcqlsi6lPo`.

Live runs, 2026-11-04, Personal plan (2 vCPU / 4 GB, region nyc), code from #167–#172 merged on master. Two hosts, each with its own profile and prefix:

| | macOS 26.6 arm64, launchd | Ubuntu 24.04 arm64 (a Lima VM), systemd 255 user units, linger |
|---|---|---|
| `build-base` | 131 s | 111 s |
| `e2e.sh` | 43 of 44 (183 s). The one failure: opencode, because ollama wasn't running on the host. The laptop pi extension check was skipped (this Mac's `pi`, with its own extensions, didn't answer) | create, isolation, desktop, MCP, grants, leases (pool join, lease fencing, reset on release), fork, rm pass. No LLM provider was set up on this host, so pi, opencode and the checks that use pi's file (revert, `new --from`) were covered by hand instead |
| `new` / `new --headless` | 15–17 s / 63 s | 14 s / 62 s |
| snap / revert / `new --from` | 9 / 14 / 15 s | 10 / 15 / 14 s |
| fork into 2 | 33 s | 24 s |
| stop / start | 4 / 9 s | 4 / 8 s |
| resize (running: down, disk grown; back up; stopped then start) | 14–15 s each, seen in the guest | 10 s each, seen in the guest |
| `BOXERY_EXE_EGRESS=integrations` | | new, headless new, an image and `new --from` all `integrations`; no internet (`000`), the proxy and integrations answer |
| Host reboot, 2 VMs running | not tried (this was the user's Mac; LaunchAgents load at login) | host back in 20 s, both tunnels `up` 21 s after the reboot without a login, files and LLM bridge intact |
| Interrupted create and capture | killed 0, 0.7, 1.0 and 1.5 s after `cp` was sent, and a capture at 1.0 s: from 1.0 s on, the copy had landed when the CLI died; `reconcile` removed every one, and nothing appeared in the next 3–5 minutes | |
| Account limits | `billing plan` read at init (2 / 4 / 1024). exe.dev refuses a VM beyond them (`--memory cannot exceed 4 GB…`), so boxery caps there | same |

Found and fixed in these runs: a failed revert finished later by recovery (#169), a fork's parallel creates racing on the cached account limits so that one failed (#168), the agent's git identity never set on exeuntu (#171), `e2e.sh` on macOS controllers and small plans (#172).

Not verified yet:
- Work-plan pools (`BOXERY_EXE_POOL`): the account is Personal, which has no pools. What `billing plan` reports for a pool VM's limit is unknown too.
- A host reboot on macOS (launchd).
- The integration warning firing live (it was checked against the fake exe.dev; on the live account, `llm` was already detached and nothing else that holds credentials was attached).

Not supported:
- No HTTPS (`POST /exec`) transport: its 30 s limit is too short for `cp`, `new` and the build, and the tunnels need ssh anyway.

Scorecard against [what stable means](#what-stable-means-for-a-backend) (2026-11-04, after the live runs above; every PR it names is merged, so exe is stable):

| # | | exe | Evidence and what's left |
|---|---|---|---|
| 1 | Contract | pass | Every VM, image and recovery function. No `backend_host_key`: the endpoint is the tunnel's loopback end, trusted like Lima's |
| 2 | Offline tests | pass | `test/shell/host-exe-backend.sh` (125 checks on master, 187 with #167–#171) covers every item on the list |
| 3 | Declaration | pass | `tested`: macOS arm64 and Linux arm64 (2026-11-04). No x64 host was tried |
| 4 | Live: e2e | pass | Both host OSes, on current code (table above). The gaps came from the test hosts, not exe: no ollama on either, no LLM provider on the Linux host (those checks were done by hand there) |
| 5 | Live: lifecycle | pass | Every item on both hosts, and `BOXERY_EXE_EGRESS`. `BOXERY_EXE_POOL` (Work plans only) stays listed as unverified |
| 6 | Live: interruptions | pass | Creates killed at four points and a capture: every copy that landed was removed by `reconcile`, and none landed later (with #169) |
| 7 | Live: host reboot | pass | Linux (systemd, linger): tunnels back 21 s after the reboot, no login. macOS: not tried; LaunchAgents load at login, which the docs now say |
| 8 | Security | pass (#167) | exe.dev integrations are the owner's to use: `init`, `new`, `start` and reimages warn about each credential one that reaches a VM, with the detach command. New accounts attach `llm` to every VM, so until it's detached agents can use the account's LLM credits without a grant |
| 9 | Security notes | pass (#167) | `docs/security.md` has the exe.dev entry |
| 10 | Docs | pass (#168, #178) | Sizes fixed in #168, live results above. An always-on host is a declared requirement and `init` warns on a battery (#178, deciding [#42](https://github.com/modem-dev/boxery/issues/42)) |
| 11 | One answer | pass | Stable everywhere it's stated |

## boat (experimental)

Runs VMs as [boat.dev](https://boat.dev) sandboxes: full Ubuntu 24.04 VMs (own kernel, systemd, Docker, `/dev/kvm`) driven through boat's REST API: `lib/boat.mjs` (Node, running `curl`) makes every request; `boat.sh` keeps the backend contract, ssh and the tunnels. Like `exe`, the host can be any machine with ssh, a Mac included. Offline tests: `test/shell/host-boat-backend.sh` (a fake API). Verified on a real free-trial account (November 2026, macOS host): `init`, the base build (on boat's image, next to its own desktop on `:0`), identity, the launchd tunnel with ssh as `agent` through it (and refused on the public endpoint), the desktop on `:1` and screenshots, the LLM proxy through the `-R` end, `stop`/`start`, and `resize` onto a new machine (re-pinned, hostname restored). Everything that needs a saved snapshot is not, because the trial can't make one: saving the base, `new`, `snap`/`revert`, `fork`, `new --from`, pool resets.

```bash
BOAT_API_KEY=boat_… ./host/boxery-host init --backend boat   # key saved in $BOXERY_BOAT_DIR/api-key (0600), never in host.env
./host/boxery-host build-base                                # builder sandbox → saved snapshot PREFIXbase → builder deleted
```

| | |
|---|---|
| Account | A **paid** plan: images are saved snapshots, which the free trial refuses (`403 trial_saved_snapshots_not_allowed`), so on the trial `build-base` builds but can't save the base. The trial also refuses sandboxes without auto-stop: set `BOXERY_BOAT_TTL` (≤ 7200 s there) and a VM stops by itself that long after each create or start. Every image counts against the plan's saved snapshots (10 per $20), and each one beyond that costs $0.056 a day: the base, each `snap`, each pool VM's clean state, and a fork's image while the fork runs. `BOXERY_BOAT_ORG` bills an organization (`X-Boat-Org`) |
| Names | Sandboxes are named `PREFIXvm-NAME` (VMs) and `PREFIXbuild` (the builder), and found with `GET /sandboxes?namePrefix=PREFIX`. `PREFIX` is `BOXERY_BOAT_PREFIX`, default `bx<6 hex>-`, generated by `init`. Names rather than labels, because boat drops the labels of a sandbox created from a saved snapshot. A VM's current sandbox id (`bx_…`) is `$BOXERY_BOAT_DIR/vms/NAME/id`; it changes with every disk the VM gets (new, revert, pool reset). The account's other sandboxes are never listed or touched |
| Sandboxes | Created with `ttlSeconds: null` (no auto-stop; boat's default stops a sandbox after 1 h; `BOXERY_BOAT_TTL` sets one) and `noEnv: true`, so a VM gets none of the account's environment variables, secret files, logins or CLI token and can't act on the account. Credentials stay on the host, as with every backend |
| ssh | boat runs a real sshd as user `user` (passwordless sudo). The host's admin key (`$BOXERY_BOAT_DIR/id_ed25519`) is authorized through `POST /sandboxes/ID/sshkey`, which also answers the address (`machineIp`, or `sshEndpoint` when the machine has no IPv4) and the host key, pinned per VM in `vms/NAME/known_hosts` under `HostKeyAlias=ID`. Nothing is trusted on first use. Every resume puts the sandbox on a new machine (new address and host key), so `start` re-pins it and rewrites the tunnel. boat's sshd has to stay public, so the base adds `Match User agent Address *,!127.0.0.1,!::1` → `DenyUsers agent` (checked with `sshd -t` in the build) |
| Tunnels | [`lib/remote.sh`](#remote-backends): systemd unit `boxery-boat-PREFIXNAME.service` / launchd agent `dev.boxery-boat-PREFIXNAME`, host port `BOXERY_BOAT_PORT_BASE` + slot (default 22601…), guest port `BOXERY_BOAT_GUEST_PORT` (18088) for the proxy. `ls` shows each tunnel's state and the waits fail with its last error, as for exe (`BOXERY_BOAT_TUNNEL_WAIT`, 120 s). As with exe, run the host on an always-on machine: the tunnel is each VM's only route to the LLM proxy |
| Identity | No cloud-init, so the backend replays the cloud-config as root over ssh (`remote_identity_script`): keys replaced, fresh machine-id, hostname. boat's snapshots don't keep the hostname, so `boxery-boat-boot.service` restores it at every boot. The shared build deletes the ssh host keys, so boat's post-build makes new ones before `sshd -t` (boat's API reports the machine's live key, so pinning follows); per VM, boat's host keys are kept: they are unique per machine and pinned through the API. boat ships its own `pi` and `opencode`; the build removes their `/usr/local/bin` links so boxery's npm installs win ("deleted files stay deleted" across boat's snapshots) |
| Lifecycle | `stop` = `sync`, `POST /stop`, then wait for `archived`: boat saves the disk and removes the machine, so memory and tmpfs grants are gone as after a shutdown. **boat deletes a stopped sandbox's disk after 14 days**; `boxery snap` it to keep it. `start` = `POST /resume` (with `ttlSeconds: null`) |
| Sizes | boat has four fixed types: `small` 2 vCPU / 4 GB / 51 GB, `default` 4 / 8 / 51, `large` 8 / 16 / 111, `xlarge` 16 / 32 / 251. A VM gets the smallest type that fits its CPUs, memory and disk (boxery's `small`, `standard` and `large` shapes map one to one), capped at `BOXERY_BOAT_MAX_TYPE` (`large`; `xlarge` needs boat's $100 plan). `backend_size` reports the type's size. `backend_list` reports the type's disk and no usage. `resize` picks a type the same way (without `--disk`, a VM can move down a type: boat refuses with `type_too_small` when its data doesn't fit, and the VM stays as it was); a running VM is stopped and resumed with the new type, a stopped one gets it at its next start |
| Images | `capture` = guest `sync`, `POST /saved-snapshots` (`PREFIXi-HASH`; saved snapshot names are `[a-z0-9-]`), poll until `ready`. The image name and disk size are recorded in `$BOXERY_BOAT_DIR/images/` first, so an interrupted save is still found. The source keeps running. A saved snapshot is a full copy that outlives its sandbox, so images are independent. boat's automatic per-minute snapshots and `fork` aren't used: a fork of a running sandbox can miss its last minute of writes |
| Recovery | `cleanup_new` stops and removes the tunnel and deletes every sandbox with the VM's boat name; `cleanup_image` deletes the saved snapshot (saving or ready) and its record; `recover_capture` has no source state to restore. `reimage` creates the new sandbox before deleting anything, so a retry leaves exactly one; `ls` ignores a sandbox that isn't the recorded one. Inventory fails if boat's API is unreachable rather than reporting an empty account |
| API | `lib/boat.mjs` (`boat.sh` calls it as `boat COMMAND …`): the key and any body go to curl in a config on its stdin (`-K -`), never on the command line or disk; a 2xx that isn't boat's JSON is an error. Retries 429 `rate_limited` and 503 (`no_ready_machine`: nothing was created) on every method, and 5xx on GET/DELETE; creates send an `Idempotency-Key`. Deletes send `X-Ascii-Confirm-Delete` |

Not verified yet (needs a paid account):
- everything built on saved snapshots: saving the base, `new`, `snap`/`revert`, `fork`, `new --from`, pool resets, and the JSON of `GET /saved-snapshots/NAME` while saving
- saves of busy running VMs (boat: a write right after the save call can still land in it)
- a Linux host (systemd user units); the live run used a Mac (launchd)

Not supported:
- Headless VMs (`new --headless`): `backend_new … stock` refuses. boat's own image would work as the stock image (boat sandboxes ship Node, git and build tools); it needs the plumbing + devtools layers run over ssh, as exe does.
- boat's desktop streaming, public URLs, integrated agents, environments and AI credentials proxy. boxery uses its own desktop (noVNC through the tunnel) and LLM proxy.