# c8ctl-plugin-nano A [c8ctl](https://github.com/camunda/c8ctl) plugin that starts, inspects, and stops a local Nano BPM (`nanobpmn`) cluster. ## About Nano BPM **A Rust research engine exploring high-performance BPMN execution and Camunda 8 compatibility.** Nano BPM (`nanobpmn`) is a single self-contained binary that runs BPMN processes behind a **Camunda 8-compatible v2 REST API**. It embeds a deterministic, event-sourced BPMN engine (`engine-core`), an append-only journal for crash durability, an SQLite-backed read model, optional multi-node Raft replication, and a built-in web console — all in one executable with no runtime dependencies. It is an advanced research prototype: a place to explore what a faster, smaller, faster-to-iterate-on process engine can do, while staying API-compatible with existing Camunda 8 clients and tooling. This plugin is the easiest way to run and manage it — single node or a whole cluster — shipping a prebuilt binary for your platform so there is nothing to compile. It adds a single `nano` command: ```bash c8ctl nano start|status|stop|restart|logs|pause|resume|clean|set|config|update c8ctl nano hire|assign|work # hire/assign manage agent profiles; work runs one as a Nano job worker ``` `nano start N` spawns **N** nanobpmn node processes wired to talk to each other on `localhost` (round-robin partition ownership), tracks them in a state file, and waits until every node is reachable. ## Installation This is a plugin for the [Camunda 8 CLI](https://www.npmjs.com/package/@camunda8/cli) (`c8ctl`). Install the CLI, then load the plugin from npm: ```bash # 1. Install the Camunda 8 CLI (once); requires Node.js 18+ npm install -g @camunda8/cli # 2. Load this plugin from the npm registry c8ctl load plugin c8ctl-plugin-nano # 3. Verify it's available c8ctl nano --help ``` The prebuilt Nano BPM server binary for your platform is pulled in automatically as an npm `optionalDependency`, so there is nothing to compile. To pull a newer release later, run `c8ctl nano update` (see [Updating to a new release](#updating-to-a-new-release-update)). > Loading from a local checkout instead? Use > `c8ctl load plugin --from file:///path/to/c8ctl-nano`. ## Usage ```bash # Start a single-node cluster (port 8080) c8ctl nano start # Start a 3-node cluster (ports 8080, 8081, 8082) c8ctl nano start 3 # Start a 3-node Raft-replicated cluster (RF=3 enables Raft automatically) c8ctl nano start 3 --rf 3 # Choose a different base port (nodes -> 9000, 9001, 9002) c8ctl nano start 3 --port 9000 # Override the partition count (default = node count) c8ctl nano start 3 --partitions 6 # Show cluster status and per-node health c8ctl nano status # Inspect a cluster c8ctl did NOT start (queries /v2/topology on the given port) c8ctl nano status --port 8080 # Tail a node's log (-f / --follow to stream). Node ids are 0-indexed, # so a single-node cluster is node 0. c8ctl nano logs 0 --follow # Simulate a node failing (freeze it) and recovering (resume it) c8ctl nano pause 0 c8ctl nano resume 0 # Stop the cluster (engine data is retained) c8ctl nano stop # Stop the cluster and delete per-node engine data c8ctl nano stop --purge # Stop then start fresh c8ctl nano restart 3 # Restart from a clean slate (delete engine data, keep models & workers) c8ctl nano restart --purge # Wipe journal/data + logs from disk (keeps models & workers) c8ctl nano clean # Persist settings c8ctl nano set bin ~/workspace/nanobpmn/server/target/release/nanobpm-gateway-rest-server c8ctl nano set model-dir ~/bpmn-workspace # Clear a custom setting (back to the managed/release binary, or default workspace) c8ctl nano unset bin c8ctl nano unset model-dir # Show current configuration and on-disk locations c8ctl nano config ``` ## Persistent assets: models & workers Nano BPM separates **persistent authoring assets** (BPMN models and worker code) from **ephemeral engine data** (journal, snapshots, variable spill): - **Workspace** (`NANOBPMN_WORKSPACE_DIR`) — holds `models/` and `workers/`. It is the authoring source of truth, **shared by every node**, and is **never** deleted by `stop` or `clean`. - **Engine data** (`NANOBPMN_DATA_DIR`) — per-node journal/snapshots/spill. Ephemeral; removed by `stop --purge` and `clean`. The plugin points every node at one shared workspace so a model deployed once is visible cluster-wide and survives restarts. By default it lives at `/workspace`; change it with: ```bash c8ctl nano set model-dir ~/bpmn-workspace ``` This creates `~/bpmn-workspace/models/` and `~/bpmn-workspace/workers/`. Restart a running cluster for a workspace change to take effect. ## CLI agent workers: `hire` / `work` Beyond BPMN service-task workers (code in the workspace `workers/` dir), the plugin can turn an interactive **CLI agent harness** (Copilot CLI, Claude CLI, `pi`, "little coder", …) into a Nano job worker. **`hire`** persists an agent *profile* — a name, a **rank** (`principal|senior|junior|decider`), the **command** that starts the CLI, a **model** name, and a list of **capabilities**: ```bash # Interactive c8ctl nano hire # Or non-interactively c8ctl nano hire --name reviewer --rank senior --command copilot \ --model gpt-5 --capabilities code-review,testing # Give the harness command-line switches (e.g. run copilot with --allow-all) c8ctl nano hire --name coder --rank senior --command copilot --arg --allow-all # Opt a role into the ACP harness mode (JSON-RPC/stdio) — see "ACP harness mode" below c8ctl nano hire --name coder --rank senior --command copilot --protocol acp # List profiles c8ctl nano hire --list ``` **`assign [capabilities...]`** grants new capabilities (roles) to an existing hire without re-running `hire`. Capabilities are **added** to (unioned with) the profile's current set — `assign` never removes a role — and the updated job-type matrix is printed. Restart the profile's workers so they pick up the new job types: ```bash # Give an existing reviewer two more capabilities c8ctl nano assign reviewer triage refactoring # --capabilities works too (comma-separated), equivalent to the positionals above c8ctl nano assign reviewer --capabilities triage,refactoring # then restart its workers to service the new job types c8ctl nano work reviewer ``` **`work `** loads the profile, connects with the c8ctl SDK client, and registers one job worker per token in the **rank × capability matrix**, then polls for work in the foreground until Ctrl-C. For rank `senior` and capabilities `code-review, testing` the matrix is: | Token | Meaning | | --- | --- | | `senior` | rank alone | | `senior:code-review` | rank + one capability (spread) | | `senior:testing` | rank + one capability (spread) | | `senior:code-review+testing` | rank + all capabilities, sorted (combined) | so a BPMN service task can target a worker at any granularity by setting its job type to the matching token. **Choosing which engine a worker connects to.** By default `work` (and `supervisor start --worker`) connects to your **active** c8ctl session profile (`c8ctl use profile `). To point a single invocation at a different cluster without switching the active session, pass c8ctl's global `--profile ` — it is honoured the same way it is for core c8ctl commands: ```bash c8ctl nano work fleet --profile nano-validate # this worker only → nano-validate's engine c8ctl nano supervisor start --worker fleet --profile nano-validate # the whole fleet → nano-validate ``` The named profile wins over the active session for that run only; a supervised fleet is pinned to it (the daemon forwards it to every worker it spawns), and `supervisor status` reports the matching `ENGINE`. To also service a job type the matrix can't express — for example a code-first [`@nanobpm/workflow`](https://www.npmjs.com/package/@nanobpm/workflow) flow whose external task type is `:`, or any bespoke token — add one or more `--job-type ` flags. They are serviced **in addition to** the rank×capability matrix, so a single hired reviewer can drive both a model-first `senior:pr-review` task and a code-first flow's task without re-hiring: ```bash c8ctl nano work reviewer --job-type convergence-loop:review-round c8ctl nano work reviewer --job-type senior:pr-review --job-type senior:triage ``` ```bash c8ctl nano work reviewer # poll for work until Ctrl-C c8ctl nano work reviewer --recovery-window 300000 c8ctl nano work reviewer --name reviewer-eu # name this worker (else auto ‹host›-‹profile›-‹random›) ``` #### Zero-config enrolment: `--auto` (serve every deployed agent job type) `--auto` is the **"Borland Delphi on your machine"** onboarding ramp: skip the capability wiring entirely and subscribe the worker to **all deployed *agent* job types**, read straight from the engine. ```bash c8ctl nano work coder --auto # serve every agent job type on the engine c8ctl nano work coder --auto --auto-scope my-app # scope to one app/network (process-id prefix) ``` How it works and why it needs no wiring: - **Engine-read demand.** The worker already holds the engine (C8 REST) endpoint from its c8ctl profile. `--auto` enumerates the deployed process definitions (`process-definitions/search` → `/{key}/xml`) and scans their `` leaves for the job types the engine matches. The engine is the guaranteed shared rendezvous: if a worker can execute an app's agent jobs at all, it and the app are already on the same engine, so *what agent job types exist* is answerable from that engine alone — **no cross-machine app discovery, no app enrol endpoint, no channel connection**. - **Agent-task marker filter.** Not every service task is an agent task — connectors and record-keepers (e.g. `pr.record-plan`) are plain workers. `--auto` keeps only leaves whose service task carries the single-convention external-agent marker **``** (the same marker every external agent task already declares). A task can **opt out** of `--auto` with `` — it is then served only by explicit `--job-type`/profile subscription. - **One poller per agent job type, reconciled on change.** It opens one poller per agent job type and re-reads the engine periodically, adding pollers for newly deployed agent processes and draining pollers for undeployed ones — the same in-place reconcile the profile watch uses, sourced from the engine instead of the profile. - **Raw job-type grammar.** The job type the engine matches (`senior:plan`) is advertised **verbatim** — colon-named types are not forced through any dot-grammar. `--auto` is **mutually exclusive** with capability-resolved serving: it bypasses the rank×capability matrix (any `--job-type` extras are still added). The prompt a worker needs already rides the job header (`io.nanobpm.agentTask.task.prompt`) plus per-instance context, so a generic `--auto` worker needs no baked specialisation. > **Trust.** Engine-read has **no capability gate** — an `--auto` worker will > serve *any* deployed agent job on its engine. That is the accepted trade for the > local/zero-config target; capability-gated serving is the specialised > (capability-declared) enrolment path. Use `--auto-scope ` > to narrow the blast radius to one app/network. > **Dual-stack / IPv6-first engine hosts.** The engine client races IPv4 and IPv6 > (Happy-Eyeballs, RFC 8305) at connect time, so a worker whose engine host > resolves to an **unreachable IPv6 address first** — common with macOS mDNS > (`merlin.local → fe80::… (dead) → 192.168.x.x`), a stray/misordered `AAAA`, or > an IPv6-advertised host — transparently falls back to IPv4 instead of failing > the activation / `--auto` engine read (`fetch failed` / `write EPIPE` / > `UND_ERR_CONNECT_TIMEOUT`). Without this, the `--auto` autoscaler would read > `0` job types and **silently scale the fleet to zero** (workers vanish from the > Nano console) even though `curl`/`fetch` reach the same host fine > (jwulf/c8ctl-plugin-nano#139). No configuration is needed; if you still want to > pin a family you can set `NODE_OPTIONS=--dns-result-order=ipv4first` or point > the engine URL at the IPv4 literal. The optional `--name` sets **this worker's name** — the `workerName` it registers under at the broker (`‹name›:‹jobType›`) and how it shows up in supervisor status/logs. Omit it and a distinct `‹host›-‹profile›-‹random›` name is generated, so two `work reviewer` processes never collide at the broker. (`--name` names the worker; the profile to run is always the positional argument.) ### Live agent visibility: the `/agentic` channel + live terminals A running worker can appear **live on the Workforce visibility page** and stream its agents' terminals to an operator's cockpit. This rides the app's agentic channel (ADR 0056), served **same-port** on the app's own HTTP base URL at path **`/agentic`** — not a sidecar, so there's no extra port to open. **Connecting — on by default (LAN-first).** Nano is designed to run on a trusted network, so visibility is **on by default**. Run a worker against a Nano app and it appears live with **zero configuration** — it joins the channel with a well-known token in **LOCAL mode** (no secret). The worker presents this token to whatever `NANO_AGENTIC_URL` you point it at; the hub honours the well-known LOCAL token from **any origin** (matching the open trusted-LAN posture of the engine itself), so a worker on another box on the LAN appears live too — exposure is governed by the server's bind address, not by a shared secret: ```bash # LOCAL mode (default): appears live with no secrets export NANO_AGENTIC_URL=http://localhost:8080 # app base URL; channel is served at /agentic c8ctl nano work reviewer # agentic channel (local): announcing presence as ‹worker› on ws://localhost:8080/agentic ``` **Zero-config hub auto-discovery.** You usually don't even set `NANO_AGENTIC_URL`. When nwf runs **embedded**, the engine (`:8080`) serves the console but the `/agentic` channel is served by the **embedded app on its own port** (e.g. `:3000`); the engine's console proxy deliberately refuses WebSocket upgrades (nanobpmn ADR 0057 §3 → `501`), so the channel is unreachable via the engine URL. With **no** agentic target configured, `work` therefore **auto-discovers** it: it reads `GET /console/api/projects` and, for each running app that advertises an agentic UI port (`appUi.enabled === true` and `appUi.port`), probes that app's direct `ws://:/agentic`. Discovery runs **against the engine's own host** — a local engine keeps probing `127.0.0.1`, while a remote/LAN engine (e.g. `merlin.local:8080`) steers the probe back at *itself* (`merlin.local:`), never at the worker's own loopback services. It is **time-bounded** (≤2s) and never meaningfully delays job polling. The discovered host and port are printed for debugging. (An IPv6 literal engine host is bracketed in the URL authority, e.g. `ws://[2001:db8::1]:3000/agentic`.) - **Exactly one app →** the worker connects directly to `ws://:/agentic` (bypassing the WS-incapable console proxy) and appears live with **zero configuration** — including cross-machine on a trusted LAN. - **Two or more apps →** the worker **does not guess**: it prints an `ambiguous` error naming each discovered `project → :port` and **stops**. Pin the one you want and re-run: `export NANO_AGENTIC_URL=http://:` (or persist `agenticUrl`). - **Nothing discoverable (e.g. pointed at Camunda, or an API-only gateway) →** the worker prints a one-line advisory naming `NANO_AGENTIC_URL` and **continues doing real work** with the channel simply absent — discovery never fails the worker's actual job. Setting `NANO_AGENTIC_URL` (or persisted `agenticUrl`) **skips discovery** and is used **verbatim** — so an explicit target always wins, and it's also how you disambiguate when several apps are running. `NANO_AGENTIC=off` disables the channel entirely and attempts **no** discovery. **Seeing it work — `supervisor status`.** Supervised workers report both the engine they poll and their agentic-channel state to `c8ctl nano supervisor status` (and the interactive console), so you don't have to read raw worker logs to tell whether presence actually reached the Workforce hub. The table gains an `ENGINE` column (the engine's `host:port`) and an `AGENTIC` column whose value is one of: | AGENTIC | meaning | | --- | --- | | `starting` | transient: the worker just spawned and hasn't resolved its channel target yet (pre-`connecting`) | | `connected` | presence is live on the hub — you should see this worker in the Cockpit | | `connecting` | resolved a hub, socket not open yet (or the hub is unreachable) | | `disconnected` | an established channel dropped (hub restart/outage) — it auto-reconnects, and a worker-side **liveness watchdog** force-re-discovers the hub if it stays down (see below); also set if the channel failed to start (bad URL/refused socket), which is **not** auto-recovered by the watchdog (it only guards a channel that has connected) — fix the target and restart the worker | | `advisory` | nothing discoverable at the engine — **not** in the Cockpit; set `NANO_AGENTIC_URL` | | `off` | visibility disabled (`NANO_AGENTIC=off`) | | `?` | a live worker not yet reporting, or an older build predating these fields | If workers show `advisory` (or stay `connecting`) while jobs still run, that's the "connected to the engine but empty Cockpit" case: point them at the app with `export NANO_AGENTIC_URL=http://:` (e.g. `:3000`). **`settlement-pending` in the `JOB` column.** A worker's `JOB` cell normally shows the job key it is running (or `idle`). If an agent finished its external side effect but the worker then lost its activation lease around `completeJob`/`failJob` — so the fenced settle failed and the engine may still project the job as `CREATED` — the cell instead shows ` settlement-pending ()`, where `` is how long settlement has been pending (measured from the failed settle, not the job's start). This is **observational only**: recovery is already handled by the lease-fence + transcript-resume path, and such a job is *not* counted as busy/in-flight (it doesn't hold up a drain). The state lets you *see* a job caught between side effect and settlement rather than inferring it from a silently-idle worker plus a stuck `CREATED` job; it clears when the job is re-activated on this worker, self-expires after a bounded TTL (so a re-activation that lands on a *different* worker can't leave a ghost lingering here), or the worker restarts. **Liveness watchdog (auto-recovery from a wedged channel).** If the nano server restarts, crashes, or a network partition drops the connection *without* a clean close (a **half-open** socket), a worker's channel client can sit `disconnected` forever — the worker vanishes from the Nano Workers view / Cockpit and, before this, only a supervisor restart brought it back. Each worker now runs a belt-and-suspenders watchdog: once a channel that had connected stays down past a threshold (the client library's own reconnect never recovered it), the worker tears the wedged channel down and re-runs full hub discovery + reopen — no restart needed. The thresholds are tunable via env (sensible defaults; you rarely need these): ```bash export NANO_AGENTIC_STALE_MS=60000 # force re-discovery if a drop hasn't recovered within 60s (default) export NANO_AGENTIC_WATCHDOG_MS=15000 # how often the watchdog checks channel liveness (default) ``` **Lossy-link hardening (reconnect churn that never re-lands presence).** A distinct failure mode shows up on a **lossy/roaming WiFi (or NAT) link**: the drop *is* detected (an abnormal closure, code `1006`), the client *does* reconnect and re-announce — yet presence never re-lands on the hub, so the worker stays absent even though its log shows it re-announcing. The sustained-drop watchdog above does not catch this, because each brief reconnect keeps resetting its drop clock. Two extra safeguards close the gap: - **Presence-keyed watchdog trigger.** A reconnect only counts as recovered once the socket *holds* for a short grace window (so a `1006` blip that immediately re-drops does not mask an unrecovered presence). When presence has not been confirmed within a threshold — regardless of the reconnect flapping — the watchdog forces the same full re-discovery + reopen. - **Jittered reconnect backoff.** Reconnect attempts are spread with equal-jitter backoff so a fleet dropped on the same link does not reconnect in lockstep and re-congest it. ```bash export NANO_AGENTIC_PRESENCE_STALE_MS=60000 # force re-discovery if presence isn't re-confirmed within 60s (default) export NANO_AGENTIC_PRESENCE_GRACE_MS=5000 # how long a reconnect must hold before presence counts as landed (default) ``` **Secure mode (opt-in).** For a deployment where you want the visibility channel authenticated (rather than open on the LAN), start the server **and** every worker box with the **same** `NANO_AGENTIC_SECRET` — same env-var name, same value on both sides (Tab A → Slot A). The worker presents it as its identity token and the hub verifies it against its own `NANO_AGENTIC_SECRET`. Setting it switches the worker into SECURE mode: ```bash # SECURE mode: same NANO_AGENTIC_SECRET on the server and every worker box export NANO_AGENTIC_URL=http://localhost:8080 export NANO_AGENTIC_SECRET= # must equal the server's NANO_AGENTIC_SECRET c8ctl nano work reviewer # agentic channel (secure): announcing presence as ‹worker› on ws://localhost:8080/agentic ``` The shared secret can also be **persisted** (via config) as `agenticSecret`, so it need not be exported into the environment on every run; env still wins over the persisted value. The legacy `NANO_AGENTIC_TOKEN` env var (and persisted `agenticToken`) is still accepted as a **deprecated alias** for the shared secret. The **capability credential** (`NANO_AGENTIC_CREDENTIAL`) is no longer required — it was removed from the hub contract (it was accept-any, pure friction). It remains **optional** and is forwarded only if still configured, for forward-compatibility with a future per-peer capability verifier. To opt out entirely, set `NANO_AGENTIC=off` (or persisted `agentic: false`) — the worker then runs with **no visibility, no relay, nothing else changed**. In secure mode a matching secret connects; a wrong secret is rejected (unauthorized). **How presence appears.** On connect the worker **announces** its identity, its `host`, and the set of `jobKeys` it is currently running, then **heartbeats** to stay live and **re-announces** after a reconnect so its row survives a hub restart. When `work` stops it **deregisters cleanly**, so the worker disappears from the visibility page on exit. **Opting a role into a live terminal (PTY vs pipe).** Each role chooses whether its agent harness runs on a full **PTY** (a real terminal — streamed on the relay lane and **steerable**: an operator's keystrokes reach the running agent) or a plain **pipe** (streamed, not interactive). It's a **per-role opt-in**, set on the hire profile and defaulting to `pipe`: ```bash # Hire a role whose harness runs on a full, steerable PTY c8ctl nano hire --name coder --rank senior --command copilot --terminal pty # Override the mode for a one-off worker without re-hiring NANO_AGENTIC_TERMINAL=pipe c8ctl nano work coder ``` A PTY needs the optional native `node-pty` dependency; when it is **unavailable** (not installed, or Windows) a `pty` role **gracefully falls back to a pipe** that still relays. (If `node-pty` is present but a PTY can't be spawned at runtime, that job fails rather than falling back.) Each job's terminal streams on its own relay stream named `job:‹jobKey›`, so output and steer-in are routed from the `jobKey` alone. **Surviving a hub outage.** A worker that starts **before** the app, or survives a **hub restart**, buffers its outbound frames in a **bounded** local ring and **drains them in order** on reconnect (no loss or reorder within the bound). The bound is operator-tunable for a long expected outage; raise it with `NANO_AGENTIC_BUFFER_CAPACITY` (frames). When the bound is hit the worker warns rather than silently shedding. ### ACP harness mode (opt-in) Alongside the `pipe`/`pty` terminal choice above, a role can opt into driving its harness over the **[Agent Client Protocol (ACP)](https://agentclientprotocol.com)** — JSON-RPC 2.0 over stdio — instead of the default stdin/scrape **pipe**. ACP is **additive, not a switch**: `pipe` stays the default floor, so every CLI harness (and non-agent record-keepers) keeps working unchanged, and **YOLO stays the default** via auto-approve. You turn ACP on per role, exactly like `--terminal`: ```bash # Drive a role's harness over ACP (JSON-RPC/stdio) instead of the pipe c8ctl nano hire --name coder --rank senior --command copilot --protocol acp # Override protocol/permission for a one-off worker without re-hiring NANO_AGENTIC_PROTOCOL=pipe c8ctl nano work coder NANO_AGENTIC_PERMISSION=escalate c8ctl nano work coder ``` - `--protocol pipe|acp` (default `pipe`) selects the harness protocol. `NANO_AGENTIC_PROTOCOL` overrides it at work time (mirroring `NANO_AGENTIC_TERMINAL`). - `--permission yolo|escalate|filter` (default `yolo`) selects the ACP permission policy. `NANO_AGENTIC_PERMISSION` overrides it at work time. **What ACP unlocks.** Because the harness speaks a structured protocol rather than a scraped terminal, the ACP path gives you a **structured turn/tool event stream** (today serialized to text chunks on the relay lane — a *minimal* mode, not yet typed turn/tool envelopes), **native permission handling**, and **PTY-free steering** — an operator's steer text is delivered as a `session/prompt` and an interrupt as a `session/cancel`, with no keystroke injection. The ACP path therefore does **not** need the optional native `node-pty` dependency at all. **Permission policies — mind the status.** Only `yolo` is enforced today: - **`yolo`** *(default, the only enforced policy today)* — auto-allows every permission request the agent raises: full speed, no human in the loop. This is the same auto-approve posture the pipe/PTY paths already run with. - **`escalate`** and **`filter`** are **RESERVED / not-yet-active** in this build. They are **accepted and persisted** for forward-compatibility, but they are **not yet enforced**: at work time they fall back to a **safe interim policy** (currently auto-allow, like `yolo`) and the CLI **emits a warning** so an operator is never misled into thinking destructive operations are gated. Their intended future behavior — **`escalate`** blocking a permission request until a human answers it, and **`filter`** auto-allowing reads/edits while escalating destructive operations — is **not available yet**; it lands once the companion permission-event + escalation bridge (nanobpm/nano-workforce#559) ships. **Per-CLI hire examples.** Some CLIs speak ACP natively; others ride a thin adapter binary. In every case the profile's command (plus any `--arg`s) assembles the ACP invocation; a default `--acp` switch is appended only when the assembled command line doesn't already select ACP, so a native/adapter invocation is never doubled. All of these run with the enforced default `yolo` policy: ```bash # Copilot CLI — native ACP (assembles `copilot --acp`) c8ctl nano hire --name coder --rank senior --command copilot --protocol acp --permission yolo # OpenCode — native ACP server (assembles `opencode acp`) c8ctl nano hire --name coder --rank senior --command opencode --arg acp --protocol acp --permission yolo # Qwen — native ACP behind a HIDDEN flag (`qwen --experimental-acp`; present in # the shipped cli.js but not in `qwen --help`). Detection recognises the # switch, so no second `--acp` is appended. c8ctl nano hire --name coder --rank senior --command qwen --arg --experimental-acp --protocol acp --permission yolo # Claude Code — via the `claude-code-acp` adapter (npm @zed-industries/claude-code-acp) c8ctl nano hire --name coder --rank senior --command claude-code-acp --protocol acp --permission yolo # Pi — via the `pi-acp` adapter c8ctl nano hire --name coder --rank senior --command pi-acp --protocol acp --permission yolo ``` > `--permission yolo` is the default, so you can omit it. You **may** hire with > `--permission escalate` or `--permission filter` today — the value is persisted > — but it is **reserved / not-yet-active** (pending nanobpm/nano-workforce#559) > and currently behaves as the safe interim policy with a warning, so do **not** > rely on it to gate destructive operations yet. **Non-goals.** ACP does not replace anything: the **pipe/PTY** surface stays the default floor and every existing harness keeps working unchanged (ACP is enforced on the host executor; container sandboxes remain **pipe-only** for now). There is **no change to the Camunda-8 worker⇄engine job protocol** — ACP governs only how a worker drives its local agent harness, not how it talks to the engine. **Selecting the model under ACP.** `--model` only exports `AGENT_MODEL` into the harness environment; it is **not** injected into the harness argv. And structured `--arg` values are POSIX single-quoted by the plugin, so a literal `$AGENT_MODEL` in an `--arg` does **not** expand (only the `--command` string is shell-interpolated, since the harness is spawned with `shell: true`). The deterministic, supported way to pin a model for an ACP hire is therefore to **bake it into `--command`**, where the shell does interpolate: ```bash # The --command string IS shell-interpolated, so $AGENT_MODEL (or a literal name) works here: c8ctl nano hire --name coder --rank senior --command 'copilot --acp --model gpt-5.4' --protocol acp ``` (Equivalently, set a harness-specific env var with `--env` when the harness reads the model from its environment.) This is the seam the nano-workforce install script uses to route a detected model to each harness deterministically. ### Live profile reload (no restart on `assign`) A running `c8ctl nano work ` **watches** the profile it is servicing. When you extend or reduce that profile's capabilities in another terminal — ```bash c8ctl nano assign reviewer fix-ci # add a capability to the live profile ``` — the supervisor reconciles its pollers in place: it **starts** pollers for the newly added rank×capability job types and **gracefully drains** the pollers for removed types — best-effort: each draining poller is given a bounded grace window (`STOP_GRACE_MS`) for its in-flight jobs to finish before it is stopped, so long-running work exceeding that window may still be interrupted. Unchanged job types keep running undisturbed, so there is no need to stop and restart the worker. Only **job types** (rank + capabilities, plus any `--job-type` extras) reconcile live. Changes to the profile's `command`, `model`, `sandbox`/`image`, or `env` still require a restart to take effect. If the profile is deleted or the config file is mid-write when the reload fires, the running workers are **kept** (never torn down) and a warning is logged. Each activated job runs the profile's command **once** (one-shot): the job is serialized to JSON and piped to the CLI's **stdin** — ```json { "jobKey": "...", "jobType": "senior:code-review", "processInstanceKey": "...", "prompt": "", "variables": {}, "profile": { "name": "reviewer", "rank": "senior", "model": "gpt-5", "capabilities": ["code-review", "testing"] } } ``` and the profile/model are also exported as `AGENT_PROFILE`, `AGENT_RANK`, `AGENT_MODEL`, `AGENT_CAPABILITIES`, `AGENT_JOB_TYPE` env vars. On exit `0` the job is **completed** with `{ output: , exitCode: 0 }` (captured output is capped at 1 MiB, with a `truncated` flag when exceeded); any other exit **fails** the job with a decremented retry count. Profiles are stored in the plugin's `config.json` (see `c8ctl nano config`). > **Self-managing activation lock (no hardcoded job timeout).** An agent job's > duration is unpredictable, so the worker does **not** ask you to pick a fixed > timeout up front. It keeps the broker's job-activation lock a bounded > `--recovery-window` (default `300000`ms) ahead of *now*, refreshing it every > ~1/3 of that window for as long as the harness is alive **and** producing > output. Consequences: > - **Long jobs never lose their lock.** A run that takes hours keeps going — the > lock is continuously extended, so the broker never re-activates the job while > you're still working on it (which would start a second agent and get the stale > `complete`/`fail` rejected with a 409 "job cannot be failed in the current > state"). > - **Fast recovery on death.** Because each refresh *sets* the deadline to > now+window (the `UpdateJobTimeout` contract is a duration-from-now, not a > cumulative delta), the moment the worker stops refreshing — the process dies, > the node is lost, the harness is idle-killed, or it hits the hard cap — the > lock lapses within one `--recovery-window` and the broker reclaims the job. > This is deliberately optimised for quick reclaim, not for holding a stale lock. > - **`--idle-timeout`** (default `300000`ms) is the liveness gate — but it is > **progress-aware**, not a blunt silence timer. When the window elapses the > worker probes the harness's process subtree: if a descendant is still doing > work (aggregate CPU time across the tree is advancing — e.g. a long, quiet > `./mvnw -q` build that emits nothing for minutes) the agent is treated as > **live** and the kill is deferred. Only a genuinely quiescent tree — no live > descendant, or descendants burning no CPU across the window — is killed as > wedged, so a hung agent still can't hold a job forever while a silent-but-busy > one keeps the cheapest, zero-token build pattern. Reclaim timing: because a > single CPU sample can't reveal progress, the first elapse with live > descendants only *baselines* and re-probes — the kill-vs-defer decision then > lands on the following `--recovery-window`, so a quiescent-but-live subtree > ages out roughly `--idle-timeout` **plus one recovery window** after the last > output, not the instant the idle window first elapses. (A tree with no live > descendants is killed immediately when the idle window elapses.) A task can > widen this per-task class via the envelope's `task.idleTimeoutMs` / > `task.recoveryWindowMs` (see the task envelope below) instead of a global > flag change. > - **`--job-timeout`** is now an *optional* absolute hard cap on total harness > runtime (default `0` = unlimited), for when you want a ceiling regardless of > output. `--lock-grace` is **deprecated and ignored** — the lock is auto-managed. > **Long-poll window.** `--poll-timeout` (default `30000`ms) is how long the > broker holds each `activateJobs` request open waiting for work before returning > empty. A longer window keeps an idle worker on **one** connection for that whole > window instead of reconnecting every few seconds — cutting the number of > connection establishments, and thus the chances of hitting a transient connect > error (`ECONNREFUSED` / connect-timeout) on a flaky link. It maps straight to > the SDK's `pollTimeoutMs` → the broker's `requestTimeout`: `0` selects the > broker's own default (~5s) and a negative value returns immediately when no job > is available. > **Trust boundary.** The profile `command` is run through a shell so you can > write a full invocation (args, pipes, multi-word commands). It is > **operator-authored** — only what you put in your own `config.json` is > shell-interpreted. Untrusted job data reaches the harness solely as stdin JSON > and `AGENT_*` env vars, never interpolated into the command line, so process > variables cannot inject shell commands. > **Resume on re-activation — a continuation, not a duplicate.** When the broker > re-activates an agent job (its lock lapsed after a worker death, node loss, or > idle-kill — see the activation-lock note above), the new worker does **not** > cold-rerun from scratch. Instead it **resumes from the previous agent's state**, > so at-least-once delivery stops being harmful — a re-activation becomes a > *continuation* rather than a duplicate of the agent's external side effects. On > re-activation, before spawning the harness, the worker: > - Fetches the prior **engine-native `AgentInstance` transcript** for this > `elementInstanceKey` (the durable, append-only `AgentHistory` minted while the > previous instance ran). Because it is **engine-backed it is cross-machine** — > the resumed agent can pick up on a completely different worker box, unlike a > local same-host journal. > - **Seeds the harness prompt** with a rendered continuation of that transcript, > so the new agent continues from where the prior one left off and does not > repeat already-completed steps or re-perform side effects (comments, pushes, > PRs). > > **Recovery scope (what survives a re-activation):** > - **Committed work is durable *only when this activation re-checks-out the exact > branch the prior run pushed onto*** — a single invariant: `repository.ref` names a > stable non-base branch **and** `branch.create` names that **same** branch > (`create === ref`). The clone lands the workspace on `ref` (prior commits present), > and `provisionRepo`'s honored `git checkout -B ` is then a no-op that keeps > the workspace on it and pushes it back, so the resumed agent can inspect it for the > **last pushed commit** (`git log` / the open PR for what landed). Every other shape > is **not** durable and is classified **transcript-only**: a `ref`-only job with no > `branch.create` (the PR-based review/fix-ci/rebase shape) is committed onto a > **per-run `nano/agent-work/-` fallback branch** the next clone of `ref` > never sees; a `branch.create` that differs from `ref` does `checkout -B ` > off the freshly re-cloned base and never fetches the existing remote ``; and > a base-like ref/create, a bare-URL / base-only clone, `branch.push=false`, or a > `repository.sha`-detached checkout likewise recover nothing. For every transcript-only > run the resume preamble points at the transcript (VERIFY-first) rather than promising > a branch to check out. > - **Uncommitted working-tree changes are *not* recovered** in this increment — > the throwaway workspace does not persist across activations, so any delta the > previous run had not committed is lost and the resumed agent re-derives it. The > resume preamble states this explicitly. (Persisting the workspace / microVM > across activations to recover uncommitted work is a later isolated-context > increment.) > > Resume is **best-effort and gated to external agent jobs** (the only ones with a > durable transcript). A read failure, an SDK without an AgentInstance read > surface, no prior work to continue, or the `NANO_AGENT_RESUME=off` kill switch > all fall through to the legacy cold rerun with no behaviour change. > > **Agent plans.** When the harness publishes an ACP `plan` (e.g. rusty-harness's > `plan_*` tools), each distinct plan is recorded in the AgentInstance history as an > assistant checklist turn. A resumed run gets the latest plan restated in the resume > prompt (VERIFY-first, like the transcript), and a harness that advertises > `agentCapabilities._meta.planSeed` also receives it in `session/new` `_meta.plan` > so it can keep working the same plan. `NANO_AGENT_PLAN=off` disables both. ### Task envelope, sandboxes & disk hygiene For **agentic** jobs (an agent that clones a repo, works a task, pushes a branch) the job carries a structured **task envelope** under the reserved `io.nanobpm.agentTask` namespace. It is assembled from the job's static `customHeaders` (model-authored defaults) deep-merged with per-instance `variables` (**overrides win**), then normalized to schema v1 and included in the stdin payload as `task`: ```jsonc { "io.nanobpm.agentTask.repository.url": "https://github.com/o/r.git", // header "io.nanobpm.agentTask.repository.ref": "main", "io.nanobpm.agentTask.branch.push": "true", "io.nanobpm.agentTask.task.allowPr": "false" } ``` Element templates emit flat dotpath header keys (strings); the plugin expands them into a nested object and coerces `"true"/"false"` → bool and numeric strings → int. The normalized shape is `{ schemaVersion, repository{provider,url,ref,sha,depth,singleBranch,filter,baseRef,baseSha,cloneTimeoutMs,submodules,authRef}, branch{base,create,push}, setup{commands,env,secretRefs}, task{prompt,promptFile,maxIterations,timeoutMs,idleTimeoutMs,recoveryWindowMs,allowPr,prBase} }`. **Per-task liveness overrides.** `task.idleTimeoutMs` and `task.recoveryWindowMs` let a specific task class (e.g. an implementation job on a large monorepo) widen the idle-liveness / broker-lock recovery window without touching the worker's global `--idle-timeout` / `--recovery-window` flags, and `task.timeoutMs` raises the absolute hard cap (`--job-timeout`). Precedence is **envelope override → worker flag → built-in default**, each clamped to a sane maximum so a task can't request an unbounded window; absent fields leave the worker-flag behaviour unchanged. **Prompt = base + optional verbatim append.** The agent's prompt resolves to `task.prompt` (typically a model header filled at deploy time), falling back to a plain `prompt`/`task` variable. Because a header-delivered base prompt can't be composed in FEEL, a task may supply per-instance context via **`task.appendPrompt`** (reserved) or a plain **`appendPrompt`** variable — it is concatenated onto the base **verbatim, with no injected separator** (the model's ioMapping owns any leading separator/preamble), so a null/empty append leaves the base untouched. This lets the static prompt live in a model header/side-car while the dynamic tail (e.g. plan-revision feedback, a per-task brief) is built per instance. **Live prompts via linked resources.** A service task can declare a Zeebe/Camunda-parity **linked resource** for its prompt instead of baking it into a model header: ```xml ``` At job activation the engine resolves the `resourceId` to the **latest** deployed key and delivers a `linkedResources` custom header (`[{resourceKey, resourceType, linkName}]`). The header carries the **key, not the content** — the worker fetches the bytes over the broker REST API (`GET /v2/resources/{resourceKey}/content/binary`, reusing the same nano endpoint the worker already talks to; override with `NANO_REST_URL`/`NANO_REST_TOKEN`) and uses the UTF-8 content as the **base prompt**. The entry whose `linkName` is `prompt` wins over the header-baked `task.prompt` chain; `appendPrompt` still composes onto it. Redeploying just the resource updates the prompt for the **next activation** — no process redeploy, no worker restart. Jobs without `linkedResources` behave exactly as before (fallback chain). A declared prompt resource that can't be fetched **fails the job** (retryable provisioning error) rather than running an agent with an empty prompt, and the resolved `resourceKey` is logged and echoed on the output envelope as `promptResourceKey` for audit (the engine keeps only `latest`, so the key is the reproducibility handle). On completion the plugin writes an **output envelope** back under `io.nanobpm.agentResult` (`{schemaVersion, status, sandbox, image, output, truncated, stderrTruncated, exitCode, signal, error, promptResourceKey?}`). When a repository was provisioned (below) it also carries `{repository, branch, baseSha, headSha, commits[], pushed, pushError?, pushFailed?, strandedCommits?, branchMismatch?, scanError?, gitError?, pr?}`. `pushFailed` is the explicit "push failed" flag — set for a non-zero `git push` **whose result could not be confirmed as landed at the remote** (a non-fast-forward rejection, or an auth, hook, or network error), not only a server rejection. **`pushFailed` does not by itself prove a remote push was attempted or rejected:** `finalizeGit` also sets it (with `strandedCommits` and a `branchMismatch` `{expected, actual}`) when it *refuses to push at all* because the harness moved HEAD off the provisioned work branch — or left commits abandoned on another local branch — so pushing the work branch would publish stale work and strand those commits (no `git push` runs, so `pushError` is absent in that case). As a guard against false strands, a non-zero push is re-checked with `ls-remote`: if `origin/` already points at `headSha` — or the remote tip is a **descendant** of `headSha` (another actor pushed a further commit after ours landed) — the push is treated as a transport hiccup that landed after the ref was accepted (`pushed: true`, no `pushFailed`/`pushError`); otherwise `pushFailed` is set and `strandedCommits` lists the SHAs of the new commits left UNPUSHED in the throwaway workspace — together they are the recovery handle for a failed push, so consumers must not treat `pushed: false` alone as the only signal. On such a failure (a rejected push **or** a branch-mismatch refusal) the throwaway workspace is preserved **best-effort** (even under the default `--keep-runs=false`) so those SHAs stay recoverable, but this is a *recovery window, not a durable archive*: the run-directory reaper still ages it out by mtime and worker shutdown removes the namespace — copy the stranded commits out promptly (or run with `--keep-runs`). **`scanError`** flags a **best-effort/PARTIAL** `strandedCommits` list: a commit or remote-reachability scan did **not** complete, so a consumer should recover the whole preserved workspace rather than trust the list exactly. It arises two ways: (a) a **pre-push** commit-enumeration scan failed, so `finalizeGit` refused to push on an incomplete graph — **no push was attempted** (`pushError` is absent) and `strandedCommits` is a partial surfacing; or (b) a push was **ATTEMPTED and rejected** but the remote-reachability FILTER that trims already-published commits from the strand set itself failed — so `scanError` is present **alongside** `pushError`, marking the rejected-push strand list as best-effort (it may falsely include an already-landed commit). Either way `pushFailed` stays authoritative; `scanError` is how a consumer knows the strand list is inexact (recover the whole workspace) versus an exact list (a rejected push with no `scanError`). **Git provisioning (host).** When `--sandbox none` (the default) and the envelope carries a `repository.url`, the plugin provisions a workspace on the host around the harness: 1. resolve the optional repo credential (`repository.authRef`, or `GITHUB_TOKEN` for GitHub) — absent ⇒ anonymous clone; 2. `git clone` (honouring `depth`/`submodules`, and `repository.ref`/`branch.base` as the checkout target). **`ref` is always a branch/tag name** (there is no hex heuristic, so a legitimately hex-named branch like `deadbeef` is cloned via `--branch`, never misread as a commit); to pin a **raw commit** use the dedicated **`repository.sha`** field, which clones `ref`/`branch.base` (if any) then fetches + checks the commit out as a detached HEAD. The clone lands in a throwaway workspace under `/agent-runs/run-*`. For a **huge monorepo** the clone envelope can be scoped so it finishes inside the clone timeout: **`singleBranch`** adds `--single-branch` (fetch only `ref`, not every branch — a plain `clone --branch` still pulls all branches/history); **`filter`** (e.g. `"blob:none"`) adds `--filter=` for a partial/treeless clone (full commit graph, lazy blobs — so `merge-base`/`git diff base...head` still work); **`baseRef`**/**`baseSha`** additionally `git fetch` the base (respecting `depth`/`filter`) so a single-branch/shallow clone can still diff `base...head` (exported as `AGENT_REPO_BASE`). **`baseRef`** is always treated as a branch/tag name (even one that looks hex-like) and is mapped into `refs/remotes/origin/`; **`baseSha`** is the field for a raw commit SHA (fetched by id, exposed as the SHA itself). `baseRef` and `baseSha` are **mutually exclusive** — setting both is ambiguous, so the base fetch is skipped and a non-fatal `baseFetchError` records the misconfiguration. A `--depth 1 --single-branch` of only the head otherwise has NO base and NO merge-base, so a naive `git diff main` fails. A failed base fetch is **non-fatal** — the head clone still succeeds and the failure is logged. **`cloneTimeoutMs`** overrides the clone/fetch timeout per envelope (default 120s, or the `--clone-timeout` worker flag) as a backstop for repos big enough to approach the cap even when shallow; a timeout is now reported *as a timeout* rather than an opaque `exit 128`; 3. create `branch.create` (if set) off that target — or, when `branch.create` is **absent** (or names the effective base itself) **and push is enabled**, cut a generated fallback work branch `nano/agent-work/-` so commits are never made directly on the base branch; the branch actually used (configured or generated) rides back in the result envelope (`branch`) and is exported to the harness as `AGENT_REPO_BRANCH`. **Exception — a detached tag/SHA checkout with no `branch.create`:** with no symbolic branch to name a fallback from, it stays `branch: null` and its commits are **not** pushed even when push is enabled — this holds for *any* detached no-`branch.create` checkout, **including** when a `branch.base` is configured (the base name is not used to synthesize a fallback here). Supply `branch.create` to publish work committed off a tag/SHA base; 4. set a **committer identity** on the workspace, preferring the operator's own (`GIT_AUTHOR_*` env → global `git config user.name/email` → the `gh`-authenticated GitHub user), and only falling back to `nano-agent` when none resolve — so autonomous commits are authored by the human running the fleet (who has signed any CLA/DCO), not an anonymous bot; 5. run the harness **in the workspace** (`cwd`), with `AGENT_WORKSPACE`, `AGENT_REPO_URL`, `AGENT_REPO_BRANCH`, `AGENT_REPO_REF`, `AGENT_REPO_BASE` exported and the job envelope on stdin; 6. on success, enumerate new commits, `git push` the branch when `branch.push` (default true), and — when `task.allowPr` — **reconcile the PR the agent opened** for the branch (`gh pr list --head `; `openedBy` reports the PR's actual author login, or `null` when none is found), then post a one-time attribution comment recording that the change was agent-generated (marker- guarded so convergence rounds don't repeat it; disable with `NANO_AGENT_ATTRIBUTION=0`, rename the agent with `NANO_AGENT_NAME`). The token is delivered to git via `GIT_ASKPASS` (env), never on argv or in the remote URL, and is redacted from all logs/results. Credential helpers are **always** disabled for the clone/fetch/push (`-c credential.helper=`), even when a token is present, so a helper like `store`/keychain can never persist the job's token to disk — `GIT_ASKPASS` supplies the secret directly. When **no** token is resolved the clone is *additionally* anonymous **for HTTPS remotes**: inherited `GIT_ASKPASS`/`SSH_ASKPASS` are cleared, and the operator's global git config is neutralized (`GIT_CONFIG_GLOBAL` → the platform null device, `/dev/null` or `NUL` on Windows) so knobs like `http.*.extraHeader` or `url.*.insteadOf` can't silently inject operator credentials. (An **SSH** remote — `git@…`/`ssh://…` — can still authenticate via the host's SSH agent/config; use HTTPS URLs if you need a guaranteed-anonymous clone.) Token-backed jobs keep global config (e.g. `http.proxy`). **`pushFailed` is the authoritative "work is stranded, workspace preserved" flag** (the job still completes, so a later BPMN step can drive the merge); a clone/checkout failure sheds the job (retryable). `pushError` is present **only** for a push that was ATTEMPTED and rejected/unconfirmed — when `finalizeGit` refuses to push at all (a branch mismatch, or a partial/incomplete commit scan), it sets `pushFailed` (with `branchMismatch` or `scanError`) but **no** `pushError`, so consumers must key recovery off `pushFailed`, not `pushError`. **`scanError` flags a best-effort/PARTIAL `strandedCommits` list** — a commit/reachability scan did not complete, so the list may omit or (after a rejected push) falsely include commits and a consumer should recover the whole preserved workspace rather than trust the list exactly. It arises two ways: (a) a pre-push commit-enumeration scan failed, so `finalizeGit` refused to push on an incomplete graph (**no** `git push` ran, so `pushError` is absent); or (b) a push was ATTEMPTED and rejected but the remote-reachability FILTER that trims already-published commits from the strand set itself failed — so `scanError` is present **alongside** `pushError`, marking the otherwise-complete rejected-push strand list as best-effort. Either way `pushFailed` stays authoritative. Workspaces are deleted after each job (keep them with `--keep-runs`) — **except** a job whose `git push` **could not be confirmed as landed** (a non-fast-forward rejection, or an auth/hook/network error whose `ls-remote` re-check did not find `origin/` at or ahead of `headSha` — `pushFailed` + `pushError`), **or** one where `finalizeGit` refused to push (no `git push` ran, so `pushError` is absent) because HEAD moved off the provisioned work branch — or left commits on another local branch or an abandoned detached HEAD reachable only via the reflog (a `branchMismatch` strand) — **or** because a critical local commit scan did not complete (a `scanError` strand, with a best-effort PARTIAL `strandedCommits`), whose workspace is preserved best-effort so its `strandedCommits` stay recoverable; that preservation is still age-gated by the reaper and cleared on worker shutdown, so recover the SHAs promptly. ```bash # The harness sees a cloned repo at $AGENT_WORKSPACE; branch/push/PR are handled for it. c8ctl nano work coder # sandbox=none: repository-bearing jobs are provisioned on the host ``` **Repo-less jobs run in a throwaway temp cwd (host).** When `--sandbox none` and the envelope carries **no** `repository` block at all, there is nothing to clone, but the job still gets a fresh, **empty** `run-*` workspace under `/agent-runs/` as its `cwd` (tracked and auto-reaped by the same run-dir lifecycle as a provisioned clone). This makes *"each job runs in a throwaway per-job workspace"* true **by default**, not only for repo-bearing tasks: the worker's launch dir is never polluted with scratch files or a stray `git init`, and an agent that must clone the repo itself gets a clean base. **This is hygiene, not a sandbox** — the harness runs as your user with full filesystem access, so an empty `cwd` does **not** *confine* it (an agent can still `cd` to a known absolute path); true confinement remains the container increment, and a provisioned `repository` envelope stays the preferred path (it also avoids a full-monorepo self-clone, defeating the `singleBranch`+`filter` shaping above). A job whose envelope carries a `repository` block that declares intent (any `repository.*` field set) but whose **`url` is absent or not a usable clone target** is treated as an orchestrator bug and **fails closed** (retryable) rather than silently degrading to the temp cwd — surfacing the half-specified envelope loudly instead of running an agent with no repo where one was expected. **Sandbox.** By default the command runs on the host (`--sandbox none`). Pass `--sandbox docker` (or `podman`) with an `--image` to run **each job in a throwaway container** instead: ```bash c8ctl nano hire --name coder --rank senior --command "agent-harness" \ --sandbox docker --image ghcr.io/acme/agent:1 c8ctl nano work coder # uses the profile's sandbox/image c8ctl nano work coder --sandbox docker --image ghcr.io/acme/agent:1 # or override ``` Containers are labelled (`nano.managed=1`, `nano.worker`, `nano.jobKey`, `nano.run=`), log-capped (`max-size=10m max-file=3`), run with `--rm`, and a run that is idle-killed (or that outlives an optional `--job-timeout` hard cap) is force-removed. The envelope is piped on the container's stdin exactly as on the host. (Container-side git provisioning — strong isolation — is a later increment; container jobs don't clone yet.) **Host workers inherit your credentials.** A host worker (`--sandbox none`, the default) runs as your user and inherits your full environment and `$HOME`, so your existing `gh` CLI login (from `gh auth login`) or a `GH_TOKEN`/`GITHUB_TOKEN` env var is available to the harness with **no extra setup** — handy when the agent command shells out to `gh`. A **container** sandbox is isolated and does **not** inherit that host login; provide the token explicitly via `setup.secretRefs` / `--secret-resolver host` (see **Secrets** below) instead. **Secrets.** Secrets are referenced by **name**, never value. `setup.secretRefs` (and the repo/PR credential when `task.allowPr` is set — defaulting to `GITHUB_TOKEN` for GitHub) are resolved via a pluggable `--secret-resolver` (only `host`, reading `process.env`, is implemented) and forwarded into the container by name (`-e NAME`) so values never appear in argv or `docker inspect`. A missing required secret fails the job with a clear provisioning message. **Harness env (non-secret).** A harness often needs static startup configuration — e.g. a permission toggle to start a coding CLI with its tools enabled. Persist these on the profile at hire time and/or add them at work time (repeatable `--env NAME=VALUE`); work-time values extend/override the profile's: ```bash c8ctl nano hire --name coder --rank senior --command copilot --env COPILOT_ENABLE_ALL_TOOLS=1 c8ctl nano work coder --env EXTRA_FLAG=on # extends/overrides the profile env ``` Interactive `hire` (no `--env`) prompts for these one `NAME=VALUE` at a time (blank to finish), so values may safely contain `=` or `,`. They apply on both the host and container paths. Per-job `setup.env` from the envelope layers on top (job-specific tuning wins), and the reserved `AGENT_*` variables and resolved secrets always win over user-supplied env so they can't be shadowed. For **secret** values use `secretRefs`, not `--env`. **Command-line switches.** Some harnesses take switches rather than env vars — e.g. `copilot --allow-all`. Append them to the harness command with a repeatable `--arg` (each `--arg` is one argv token). They are persisted on the profile at hire time and can be extended at work time: ```bash c8ctl nano hire --name coder --rank senior --command copilot --arg --allow-all c8ctl nano work coder --arg --verbose # appends to the profile args ``` The command is spawned through a shell (so `command` still resolves on `PATH`), but each `--arg` is shell-quoted as a single literal token, so a value with spaces or shell metacharacters can't break out or inject. They apply on the container path on any OS, and on the host path on POSIX systems. **On a Windows host (`sandbox=none`), `--arg` is rejected** with a clear error — the POSIX quoting isn't honoured by `cmd.exe` — so use a container sandbox (`--sandbox docker|podman`) or bake the switches into `--command` there. **Disk hygiene.** Host job **workspaces** and container sandboxes both get automatic cleanup so leaked artifacts can't fill the disk. Each worker **process** gets its own private namespace under `/agent-runs/worker-/` (a fresh incarnation id every process start, published with an immutable `owner.json` before any child dir appears); its `run-*` job workspaces and `res-*` result channels live there and are removed after each job and swept at startup + on `--reap-interval` (leftovers older than `--reap-age`, in-flight dirs skipped). `--keep-runs` only skips the *per-job* deletion (so a finished job's workspace survives for inspection); the age-based owner-scoped sweep still applies, so a kept dir is eventually reaped once it ages past `--reap-age`. That ordinary sweep is **owner-scoped** — a worker only ever reaps *its own* namespace, so it can never delete a sibling worker's active checkout or result channel out from under an in-flight job (the cross-worker data-loss defect fixed in [#205](https://github.com/jwulf/c8ctl-plugin-nano/issues/205); age is **not** evidence of completion — editing files inside a checkout does not refresh the enclosing dir's mtime). Reclaiming an *abandoned* namespace left by a crashed worker is a **separate, cross-process-safe** operation: it deletes only a namespace whose owning process is *provably* dead (PID-reuse-safe, via a recorded process-start token) **and** has no surviving harness, under an exclusive lock with a final recheck. Anything uncertain — a live/unknown owner, a possibly-alive harness, missing/malformed ownership, a lock held by another reclaimer — is **retained with a diagnostic**, never guessed away. > **Mixed-version rollout.** The `worker-*` namespace is deliberately invisible > to the old flat `run-*`/`res-*` sweep, and the new reclaimer never deletes > unowned legacy flat `run-*`/`res-*` directories. This makes an upgrade safe > while **old** worker processes are still running the pre-#205 code. Updating the > package on disk does **not** replace code already loaded by a running worker: > every old worker must be **drained/restarted onto the fixed version** before any > leftover legacy flat directories can be cleaned up, and legacy flat dirs whose > owner cannot be proven dead are never auto-migrated or deleted. For container sandboxes a **label-scoped** reaper runs at worker startup and on an interval (`--reap-interval`, **milliseconds**, default `300000` = 5m), removing finished/`exited` containers older than `--reap-age` (**milliseconds**, default `3600000` = 1h) while **skipping any run still in flight** — it never touches containers it didn't create and never `system prune`s. A **disk-budget admission shed** fails (retryable) new jobs when the engine data root has less than `--min-free-mb` MB free (default `1024`). > Container-side git provisioning (strong isolation) and the > Vercel/Sandcastle provider are **later increments** — the envelope names above > are frozen so the [nano-ide element-template pack](https://github.com/jwulf/nano-ide/issues/37) > can be built against this contract. ## Supervising a fleet of workers: `supervisor` Running several workers means several `nano work` foreground processes — one terminal each, none of them restarted if they crash. The **`supervisor`** runs and manages a whole fleet from a **single terminal**: a detached daemon spawns one `nano work ` child per worker, restarts a crashed child with capped backoff, and is driven either interactively (a console you can **detach from**, leaving it running) or non-interactively with plain subcommands. ```bash # Start a detached supervisor managing several workers at once c8ctl nano supervisor start --worker reviewer --worker coder --worker decider # Attach an interactive console (starts the daemon if needed). # Detach with `detach` or Ctrl-D — the daemon KEEPS RUNNING. `stop` tears it down. c8ctl nano supervisor # Manage the fleet without the console (any terminal, any time): c8ctl nano supervisor status # id, state, pid, restarts, uptime c8ctl nano supervisor add reviewer # add + spawn a worker (forwards work flags) c8ctl nano supervisor add reviewer --name reviewer-2 # a SECOND reviewer, named so it stays distinct c8ctl nano supervisor add reviewer --instances 3 # add 3 distinct auto-named reviewers in one call c8ctl nano supervisor restart reviewer # by worker id or profile name c8ctl nano supervisor reload # adopt new code fleet-wide (rolling drain+respawn, zero downtime) c8ctl nano supervisor reload reviewer # reload just one worker/profile c8ctl nano supervisor remove coder # stop + drop a worker (also: `all`) c8ctl nano supervisor logs reviewer --follow # tail a worker's log (or the daemon's) c8ctl nano supervisor stop # stop the daemon and every worker ``` Each worker has a **name** — its supervisor id and the broker `workerName` it registers under. Pass `--name` on `supervisor add` (or `work`) to set it; omit it and one is auto-generated as `‹host›-‹profile›-‹random›`, so you can run **several instances of the same profile** and they stay distinct end-to-end (status, logs, and at the broker). To scale a hire to several instances in one call, pass `--instances N` on `supervisor add` — it spawns N distinct auto-named workers of the profile at once (default 1, capped per call). Because each instance needs its own distinct name, `--instances N` (for N > 1) cannot be combined with `--name`; omit `--name` to let them auto-name. `restart`/`remove` accept either a worker id **or** a profile name — targeting a profile affects *every* instance of it. Each worker takes the **same flags as `nano work`** (`--recovery-window`, `--idle-timeout`, `--job-timeout`, `--poll-timeout`, `--sandbox`/`--image`, `--job-type`, `--env`, `--arg`, …); they are forwarded to the spawned child (reconstructed via `reconstructWorkArgs`, which normalizes ordering and coerces booleans), so a supervised worker is semantically equivalent to a hand-run `nano work`. In the interactive console, type the flags after the profile: `add reviewer --recovery-window 300000`. How it works and where things live: - The daemon runs **detached + `unref`'d** (like `nano start` nodes), so it outlives the CLI invocation that launched it — that is what "detach" means. - A JSON state file `supervisor.json` records `{ pid, socket, workers:[…] }`; management commands talk to the daemon over a **control socket** (a Unix domain socket, or a named pipe on Windows) and fall back to the state file when the socket is unreachable (to report a stale/dead daemon). - Per-worker and daemon logs live under `logs/supervisor/` in the state home (`worker-.log`, `daemon.log`). - **Per-worker log cap:** each `worker-.log` is bounded (default **10 MB**) — the daemon pipes the child's stdout/stderr through a rotating ring, keeping the **newest** output and rolling the previous fill to `worker-.log.1` (so on-disk usage stays ~2× the cap). `nano supervisor logs ` still tails the live output across a rotation. Set `NANO_SUPERVISOR_LOG_MAX_BYTES` to change the cap (bytes); `0` or a negative value disables the cap (unbounded append). An unset/invalid value falls back to the 10 MB default. - **Restart policy:** a crashed child is restarted with exponential backoff (1s → 30s cap); a child that stayed up ≥60s resets its backoff. `remove`/`stop` cancel any pending restart, and a `restart` cleanly swaps the child (a late exit from the old process is never mis-counted against the new one). - Stopping is SIGTERM → grace → SIGKILL, per worker and for the daemon; `stop` always clears `supervisor.json` so a stale marker never wedges a future start. ### Hot code reload: `supervisor reload` / `workforce reload` When you update the plugin on a machine that's already running a fleet (`c8ctl nano update`, or otherwise replacing the installed `c8ctl-plugin-nano`), `supervisor reload` adopts the new code **without stopping the fleet**: ```bash c8ctl nano update # pull the new harness onto disk c8ctl nano supervisor reload # roll it into the running fleet, zero downtime ``` - Each supervised worker is a **separate `nano work` process** that reads the plugin from disk when it starts, so the daemon adopts new code by **rolling through the workers one at a time** — gracefully draining each (the same `SIGUSR2` quiesce as `stop`: it stops leasing new jobs, finishes the ones in flight, and exits) and respawning it, which re-reads the updated `c8ctl-plugin.js`, its sidecars, and `supervisor.dist.js`. Because only one worker is down at a time, the rest of the fleet keeps serving — **zero fleet downtime**. To keep it genuinely one-at-a-time, after respawning a worker the daemon **waits for the replacement to report ready** (its activation loop is up and leasing) before draining the next one — a bare spawn/PID is not readiness, so on the normal readiness path a slow replacement can never leave two workers down at once. That wait is **bounded** (`NANO_SUPERVISOR_RELOAD_READY_TIMEOUT_MS`, default 30s): a never-ready replacement can't wedge the roll — the daemon advances anyway. That timeout fallback is the one exception to the guarantee above: when a replacement never reports ready the daemon drains the next worker while the previous one is still unready, so **two (or more) workers can be temporarily unavailable** until the slow replacement catches up. This is a *fleet-level* guarantee: a worker is drained **before** its replacement spawns, so a **single-worker fleet** (or a job type served by only one worker) does lose that capacity for the drain+boot window. Run more than one worker for a type if you need it served continuously across a reload. - It **never kills in-flight work**: a reload waits indefinitely for each worker's jobs to finish (adopting new code is never worth losing a running job). The command **streams progress** and **Ctrl-C detaches** — the daemon keeps rolling in the background (rerun `supervisor status` to check). A reload is refused while another is already in progress. - It **stops the roll on a failed reload** (a canary): if a worker's replacement crashes, fails to spawn, or is swapped out from under the roll — i.e. it has no confirmed serving child — the daemon aborts the remaining pass rather than drain the next worker on top of that gap (which would both break one-at-a-time and risk rolling a broken replacement across the whole fleet). The remaining workers are reported **skipped** and the terminal frame reports a partial failure (`ok:false`), so automation sees it. A readiness *timeout* on a still-live replacement is **not** a failure — it counts as reloaded and the roll continues. - `reload [target]` defaults to the whole fleet; pass a worker id or profile to reload just those. `workforce reload` rolls only the workers a manifest owns. - **Not supported on Windows.** The graceful drain relies on `SIGUSR2` to quiesce each worker, which Windows cannot deliver (Node maps a non-zero signal there to a forceful, SIGKILL-like termination, so the child's drain handler never fires). The daemon therefore **rejects `supervisor reload` / `workforce reload` on Windows** rather than hang or hard-kill in-flight work — use `supervisor restart `, or a full `supervisor stop` + `start`, to adopt new code there. - **Scope — workers, not the daemon.** A reload adopts all **worker-side** code (job running, agent instances, git/container provisioning, the agentic connection, the Effect runtime workers load — the bulk of the harness). The supervisor **daemon** keeps running the code it started with: its workers are its children and watch its pid, so re-exec'ing the daemon would take the fleet down with it. To adopt new **supervisor** code, do a full restart (`c8ctl nano supervisor stop && c8ctl nano supervisor start`) — a rare event, since the daemon is a thin process manager. `supervisor status` shows an `on disk:` line flagging "update available" whenever the code on disk has advanced past the running daemon, so you know when a reload (or restart) is worthwhile. ### Surviving SSH logout: `supervisor install` / `uninstall` On **macOS**, a supervisor started over SSH is bound to your SSH login session's launchd/bootstrap context. When you **log out of that SSH session**, macOS tears the per-session context down and the orphaned daemon + workers lose their network / mDNS resolution path — the fleet does **not** exit cleanly, it **wedges**: the activation loop spins on `SDK activateJobs failed: fetch failed` forever and claims **zero** jobs (workers may still show `running` / agentic `disconnected`). `setsid`/double-fork detachment is not enough there — the daemon must live in a **persistent per-user launchd domain** (`gui/$UID`). **Linux is unaffected**: under systemd-logind with the default `KillUserProcesses=no` a `setsid`'d daemon keeps full network access after logout. Give the supervisor a session-independent launch path so the fleet survives logout and comes back at login/reboot: ```bash c8ctl nano supervisor install # install + start the service c8ctl nano supervisor uninstall # stop + remove it ``` - **macOS** — writes a per-user **LaunchAgent** and bootstraps it into `gui/$UID` (`launchctl bootstrap gui/$UID …`, `RunAtLoad`, crash-only `KeepAlive`). The plist lives at `~/Library/LaunchAgents/io.nanobpm.c8ctl-nano.supervisor..plist` (the `` is derived from the state home, so distinct `C8CTL_NANO_HOME` instances get distinct services). - **Linux** — writes a `systemd --user` unit (`~/.config/systemd/user/c8ctl-nano-supervisor-.service`), enables + starts it, and turns on **lingering** (`loginctl enable-linger`) so it survives logout even where `KillUserProcesses=yes`. Where `systemd --user` is unavailable, no service is needed — the existing `setsid` daemon already survives logout — and `install` says so. - The service inherits only a **curated env** (`PATH`, `HOME`, `LANG`, `C8CTL_NANO_HOME`, the CLI entry) — never your whole SSH environment — so short-lived tokens are not persisted into a plist/unit that outlives the session. - `KeepAlive`/`Restart` are **crash-only**: a `supervisor stop` (clean exit) stays down; only a crash is restarted. When the service **is installed**, every path that starts the daemon (`supervisor start`, and a bare `supervisor` / `supervisor attach`) brings it up **through the service**: it kickstarts the LaunchAgent when a clean `stop` left it down (a plain `kickstart`, never `-k`, so a running fleet is not bounced) and then **adopts** that service-owned daemon instead of spawning a second, session-bound one. Only if the service cannot be started does it fall back to a detached spawn (saying so). When you run `supervisor start` or `attach` **over SSH on macOS without the installed service**, the CLI **auto-reparents** the daemon into the `gui/$UID` launchd domain (equivalent to `install`) so it survives logout, and tells you. If it cannot (e.g. `launchctl` is unavailable, or you set `C8CTL_NANO_NO_LAUNCHD=1`), it prints a prominent **warning** pointing at `supervisor install` instead of silently leaving a fleet that will wedge on logout. > **macOS Local Network Privacy + a LAN engine (macOS 15 Sequoia / 26 Tahoe).** > The `gui/$UID` LaunchAgent that keeps the fleet alive across SSH logout is a > **distinct TCC identity** — and on macOS 15+ it is **not** granted **Local > Network** access, which an interactive SSH/Terminal session inherits for free. > So a service-owned fleet whose engine is on the **LAN** (`merlin.local`, > `192.168.x.x`, `10.x`, …) hits the **same wedge** as the logout case — > `activateJobs failed: fetch failed`, `reconcile skipped — fetch failed`, > `listening on 0 job type(s)`, agentic `disconnected` — **even while you are > still logged in**. The tell is `EHOSTUNREACH` (or a silent connect failure) to > the engine's **LAN IP** from the service, while the *same* fetch works from an > interactive SSH shell and **internet-reachable** hosts still work from the > service. This is *not* the IPv6/mDNS case above — it fails even against the raw > IPv4 literal, because the OS is blocking LAN egress for the launchd identity. > > There is **no `tccutil` / CLI grant** for a headless launchd tool. Pick one: > - **Grant the launch program Local Network access (simplest).** On the Mac's > GUI, **System Settings → Privacy & Security → Local Network**, enable the > entry for the service's program — the **Node.js** runtime (it may surface as > *“Node.js Foundation”* / *“App Background Activity”*). Then > `c8ctl nano supervisor stop && c8ctl nano supervisor start`. The grant is > keyed to the node binary, so it persists across restarts. > - **Route over Tailscale (keeps logout survival, no LAN grant needed).** Traffic > over the Tailscale `utun` interface is **not** classified as “local network”, > so the launchd service reaches it fine. Point the engine at the tailnet > address (e.g. `NANO_REST_URL=http://..ts.net:8080`, or bake it > into the active profile so the service inherits it). > - **Run in the SSH session instead** (`supervisor uninstall`) — inherits the > Terminal grant, but reverts to dying on logout unless you pin a `tmux`/SSH > session. Note that on macOS `supervisor start` over SSH **auto-reparents** > back into the `gui/$UID` launchd identity (reinstalling the LaunchAgent) > unless you set `C8CTL_NANO_NO_LAUNCHD=1` — without that opt-out you land in > the same blocked identity. Start from a **local Terminal** session (or export > `C8CTL_NANO_NO_LAUNCHD=1`) to keep the interactive grant. > - **Run the daemon as root** — root retains LAN access, but running agent > harnesses as root is a poor trade; prefer the options above. ## Composing a workforce: `workforce` `supervisor` is imperative — you compose a fleet with a `start --worker …` plus a pile of `add … --instances N` calls that live only in your shell history. A **workforce manifest** makes that fleet a **reusable, inspectable artifact**: a named set of supervised workers you compose once and bring up convergently with one command. ```bash # Compose a fleet (each add creates/updates one entry; no supervisor needed yet) c8ctl nano workforce add copilot --instances 5 --auto c8ctl nano workforce add claude --instances 1 --auto c8ctl nano workforce add qwen --instances 2 --roles pr-review,feature c8ctl nano workforce list # print the manifest (+ --json) c8ctl nano workforce start # ensure the daemon is up, then reconcile c8ctl nano workforce status # desired vs actual, per worker (+ --json) c8ctl nano workforce stop # remove this manifest's workers (+ stop an empty daemon) c8ctl nano workforce reload # hot-adopt new code into this manifest's workers (rolling, zero downtime) c8ctl nano workforce remove qwen # drop an entry ("all" clears the manifest) ``` Every subcommand takes `--manifest ` (default `default`) to select which manifest it operates on, so you can keep several — `--manifest review-only`, `--manifest full-fleet` — side by side. (This flag was renamed from `--profile`, which now collides with c8ctl's global connection-profile flag.) ### The manifest A manifest is a **separate JSON document per name**, stored at `/workforce/.json` (`` is `C8CTL_NANO_HOME` or the per-OS app dir). It is deliberately **not** a key in `config.json`: manifests are user-curated documents meant to be read, edited by hand, diffed and copied between machines; `config.json` is plugin state. ```jsonc { "version": 1, "name": "default", "workers": [ { "profile": "copilot", "instances": 5, "roles": "auto" }, { "profile": "claude", "instances": 1, "roles": "auto" }, { "profile": "qwen", "instances": 2, "roles": ["pr-review", "feature"] } ] } ``` | Field | Meaning | | --- | --- | | `profile` | A hired profile name (must exist in `hires`; validated at `add` and again at `start`). | | `instances` | How many workers to run for this entry (`1`..`MAX_ADD_INSTANCES`). | | `roles` | `"auto"` → run the worker with `--auto`; **or** an array of capability names. | | `autoScope` | Optional, only with `roles: "auto"` → forwards `--auto-scope `. | | `args` | Optional array of extra `work` flags forwarded verbatim (escape hatch: `--sandbox`, `--stream`, …). | ### `roles` → job types `roles` is resolved **at start time from the manifest**, independent of what the profile was hired with (the install script hires with `--capabilities ""`): - `"auto"` → `c8ctl nano work --auto [--auto-scope X]`. No capability gate; serves every deployed agent job type. This is what the install script sets, and the default when you pass neither `--auto` nor `--roles` to `add`. - `["pr-review","feature"]` → repeatable `--job-type :`, e.g. for a `senior` hire: `--job-type senior:pr-review --job-type senior:feature`. `--job-type` is chosen deliberately over `nano assign`: it uses an existing work-time flag (no new override surface on `work`) and **does not mutate the profile** — the same profile can appear in two manifests with different role sets. `--auto` and `--roles` are mutually exclusive; `--auto-scope` requires `--auto`. ### Reconcile semantics — `start` is convergent, not additive `start` ensures the supervisor daemon is running (starting it if needed), then **reconciles** the running workers to the manifest: - Start workers that are missing. - Stop workers that belong to this manifest but are no longer desired (entry removed, or `instances` reduced). - Leave already-running, still-desired workers **untouched** — no restart churn, no job interruption. A second `start` with an unchanged manifest starts nothing, stops nothing and restarts nothing. - Workers **not** owned by this manifest (added by hand with `supervisor add`, or owned by another manifest) are never touched. The enabling detail is **deterministic worker names**: workforce-owned workers are named `wf---` (`wf-default-copilot-1` … `wf-default-copilot-5`). Since `--instances` can't combine with `--name`, `start` issues N single adds with an explicit `--name` each. The `wf--` prefix is also how `status`/`stop` identify ownership; a name collision with a hand-added worker running a different profile is skipped with a warning rather than clobbered. ### Portability A manifest is portable: copy `default.json` to another machine, `hire` the same profile names there, and `workforce start`. Because `roles` resolves from the manifest (not the hire), the fleet comes up identically even if the profiles were hired with `--capabilities ""`. ### Validation & errors - `add`: the profile must exist in `hires` (otherwise an error pointing at `c8ctl nano hire`); `instances` in `1..MAX_ADD_INSTANCES`; `--roles`/`--auto` mutually exclusive; `--auto-scope` requires `--auto`. - `start`: an entry whose profile was since deleted → a clear error, the entry is skipped, the rest continue, and the command exits non-zero (a partial start never leaves a half-reconciled fleet silently). - A malformed/torn manifest, or an unknown `version`, is refused with an explicit error naming the file path — never silently treated as empty. - `start` with an empty/absent manifest → a friendly pointer at `workforce add`, exit 0. `--json` on `list`/`status` emits machine-readable output (via c8ctl's global `--json` flag, whose parsed value is passed through to the handler) so the install script and CI can consume it. ## Cleaning up disk ```bash c8ctl nano clean # remove engine data + logs (cluster must be stopped) c8ctl nano clean --workspace # ALSO delete models & workers (destructive!) c8ctl nano stop --purge # stop and remove engine data in one step ``` `clean` refuses to run while any node is alive. ### Stress / throughput runs: bounding disk and RAM The engine journal (`journal.jsonl`) is **append-only** — there is currently no compaction or rotation — and the read-model retains every terminal instance by default. Under sustained high load (tens of thousands of PI/s) this fills the disk quickly. Two `start` flags keep a long run bounded: ```bash # Pure throughput: no journal / read-model on disk at all (in-memory engine). # State is lost on stop/restart, and instances live in RAM — so cap them. c8ctl nano start --in-memory --history-max 50000 # Exercise the disk path but cap the read model's terminal-instance history. # (The journal still grows append-only; watch free space.) c8ctl nano start --history-max 50000 ``` - `--in-memory` (alias `--no-journal`) routes the engine to an in-memory journal and a `:memory:` read store — nothing is written under `NANOBPMN_DATA_DIR`. - `--history-max ` sets `NANOBPMN_HISTORY_MAX_INSTANCES`, continuously pruning all but the most recent *n* terminal instances from the read model (`0`/unset = unbounded). Works in both storage modes. `c8ctl nano status` reports the active storage mode (`in-memory` vs `on-disk`) and the history cap. > ⚠️ With `--in-memory`, restart recovers nothing, and Raft/replicated logs are > not persisted. Use it for stress/throughput testing, not durability testing. ## Console profile (`--console`) The server ships a browser console. Pick how much of it is exposed at runtime: ```bash c8ctl nano start # studio (default): full IDE + authoring API c8ctl nano start --console observe # observability views only; authoring refused (403) c8ctl nano start --console off # headless: no console router at all ``` - Values: `studio` (default), `observe`, `off`. An inherited `NANOBPMN_CONSOLE` env var is honored when the flag is not passed. The plugin passes the choice through as `NANOBPMN_CONSOLE` on every node. ## Configuration (`set` / `unset` / `config`) Persistent settings are stored in `/config.json`: | Setting | Env mapping | Set with | Clear with | |---------------------|--------------------------|-----------------------------------|-------------------------------| | Binary path | (used to launch nodes) | `c8ctl nano set bin ` | `c8ctl nano unset bin` | | Workspace directory | `NANOBPMN_WORKSPACE_DIR` | `c8ctl nano set model-dir ` | `c8ctl nano unset model-dir` | Show the effective configuration and all on-disk locations with `c8ctl nano config`. `unset bin` clears a custom binary path so node launches fall back to the managed platform binary — i.e. back on the release train that `c8ctl nano update` tracks. (Note: a `NANOBPMN_BINARY` environment variable still overrides even after `unset`.) `unset model-dir` returns the workspace to its default (`/workspace`). ## Updating to a new release (`update`) The plugin and the bundled server binary (delivered via the matching platform package) ship together on npm as `c8ctl-plugin-nano`. To pull a new nanobpmn release onto a machine that already has nano installed: ```bash c8ctl nano update # check npm for a newer release and install it c8ctl nano update --check # report whether an update is available (no install) ``` `update` compares the installed plugin version against the latest published on npm. When a newer release exists it reinstalls the package globally (`npm install -g c8ctl-plugin-nano@latest`), which brings the new server binary with it. It only ever drives npm — it never touches the private upstream source — so it works for any npm-installed user. After updating, restart any running cluster (`c8ctl nano restart`) so it picks up the new binary. Whenever an update is available, `update` (and `update --check`) also prints a **changelog of what changed since the installed release** — the per-version "Features" / "Bug Fixes" notes pulled from the plugin's public [GitHub Releases](https://github.com/jwulf/c8ctl-plugin-nano/releases) (where semantic-release records them). This lookup is best-effort and non-blocking: if GitHub is unreachable or rate-limited it degrades to a link to the releases page and the update proceeds normally. Set `GH_TOKEN` (or `GITHUB_TOKEN`) to raise the unauthenticated API rate limit. If the plugin is running from a local checkout rather than a global npm install, `update` prints the manual command instead of reinstalling in place. ### Automatic "update available" notice You don't have to remember to run `update --check`: any `nano` or `processos` command also surfaces a one-line notice when a newer release is published. It is deliberately unobtrusive: - The registry lookup runs in a **detached background process**, so a command is never slowed down — the fresh result is used on the next invocation. - npm is queried at most **once per day**, and the notice is shown at most **once per day** (state is cached under the plugin's state home in `update-check.json`). - The notice prints to **stderr**, so it never corrupts machine-readable stdout, and is suppressed when stdout is not a TTY (piped/scripted) or when `CI` is set. To turn it off entirely, set `NANO_NO_UPDATE_NOTIFIER=1` (or the conventional `NO_UPDATE_NOTIFIER=1`). The explicit `c8ctl nano update` command is unaffected. ## Checking status `c8ctl nano status` queries each node's always-on `GET /v2/topology`, which is the authoritative cluster view. Because of this it works in three situations: - **c8ctl-managed cluster** — shows per-node process liveness (PID), reachability, and the live topology (partition leadership). - **External cluster** — a Nano BPM cluster started outside c8ctl (e.g. by hand, a script, or another tool). With no recorded state, status probes `http://127.0.0.1:/v2/topology` and reports what it finds, labelled `(external — not started by c8ctl)`. - **Nothing running** — reports `stopped`. Point status at a specific endpoint with `--port`: ```bash c8ctl nano status # default: managed cluster, else probe port 8080 c8ctl nano status --port 9000 ``` ### Camunda vs Nano detection Nano advertises itself in `GET /v2/topology` with a `nano` object (`engine: "nanobpmn"`) — a superset of the Camunda Orchestration Cluster API. A stock Camunda gateway answers the same endpoint without it, so `status` can tell the two apart and prints a `product:` line (`Nano BPM` or `Camunda`) with the version. If `status` finds a Camunda gateway on the probed port it says so explicitly rather than pretending it is a Nano cluster. For the same reason, `c8ctl nano start` refuses to launch on top of an existing gateway. If any chosen port is already serving a Camunda (or Nano) endpoint it reports exactly what is running and exits without starting: ``` ✗ Port 8080 is already serving a Camunda gateway (v8.6.0). ✗ Refusing to start Nano on top of a running Camunda instance. Start on a free base port instead, e.g. "c8ctl nano start 1 --port 8180". ``` To run Nano alongside a local Camunda, give it a different base port (`--port`); the collision check only applies to the ports Nano would bind. ## Fault injection: pause / resume a node `c8ctl nano pause ` and `c8ctl nano resume ` let you simulate a node failing and coming back online, to exercise Raft failover and recovery on a local cluster: ```bash c8ctl nano start 3 --rf 3 # 3-node Raft-replicated cluster c8ctl nano pause 1 # freeze node 1 (SIGSTOP) — like a hang or partition c8ctl nano status # node 1 shows "paused"; the cluster is "degraded" c8ctl nano resume 1 # unfreeze node 1 (SIGCONT) — it rejoins ``` - **pause** sends `SIGSTOP`, which halts the process instantly and *cannot be caught or ignored* — so the node stops responding without losing its PID or its on-disk state, faithfully mimicking a hung/partitioned node. - **resume** sends `SIGCONT`, and the process continues exactly where it left off. - A paused node is reported as `paused` in `c8ctl nano status` and counts as unhealthy, so the cluster shows `degraded`. - `c8ctl nano stop` automatically resumes any paused node first, so it can shut down gracefully rather than being force-killed. ## Trace capture for historical replay (`--capture`) Start a cluster with `--capture` to record every instance's inputs so runs can be replayed and analysed later: ```bash c8ctl nano start 3 --capture c8ctl nano status # shows "trace capture: on" ``` `--capture` sets `NANOBPMN_TRACE_STIMULI=1` on **every** node. That single flag enables the Tier 2 recorded-input (stimuli) log *and* auto-enables Tier 1 variable capture. It must be set on all nodes because each node's `TraceStore` only sees instances on its own partitions. Read a trace back from any node: ``` GET /console/api/traces/{instanceKey} → { creationVariables, stimuli[], } ``` Optional tuning is done with environment variables, which pass through from your shell automatically (no dedicated flags): | Env var | Default | Purpose | |------------------------------------|---------|--------------------------------------| | `NANOBPMN_TRACE_VARIABLES_MAX_BYTES` | 16384 | Max captured variable payload bytes | | `NANOBPMN_TRACE_STIMULI_MAX` | 1024 | Max recorded stimuli per instance | | `NANOBPMN_TRACE_CAPACITY` | 2000 | Max traced instances retained | > Setting `NANOBPMN_TRACE_VARIABLES=1` alone enables only Tier 1 (variables); use > `--capture` for full recorded-input replay. ## How nodes are configured Each node is the single `nanobpmn` server binary, configured entirely through environment variables. For `nano start 3` the plugin spawns: | Node | `PORT` | `NANOBPMN_NODE_ID` | `NANOBPMN_NODES` | |------|--------|--------------------|--------------------------------------------------------------------| | 0 | 8080 | 0 | `http://127.0.0.1:8080,http://127.0.0.1:8081,http://127.0.0.1:8082` | | 1 | 8081 | 1 | (same) | | 2 | 8082 | 2 | (same) | Additionally every node gets: - `NANOBPMN_PARTITIONS` — total partitions (default = node count) - `NANOBPMN_RF` — replication factor (default `1`) - `NANOBPMN_RAFT=1` — set automatically when `RF > 1` (or via `--raft`) - `NANOBPMN_DATA_DIR` — a per-node engine data directory - `NANOBPMN_DURABILITY=async` — set by default for throughput; override by exporting `NANOBPMN_DURABILITY` (e.g. `sync`) before `nano start` - `NANOBPMN_REPLICATE_ACTIVATION=digest` — set by default so activated-job state is observable across the cluster; override by exporting `NANOBPMN_REPLICATE_ACTIVATION` before `nano start` - `NANOBPMN_REPLICATION=leader-durable` — set by default; override by exporting `NANOBPMN_REPLICATION` before `nano start` - `NANOBPMN_WORKSPACE_DIR` — the shared workspace (models & workers) - `NANOBPMN_TRACE_STIMULI=1` — set on every node when `--capture` is passed Partition ownership is deterministic (`partition_id % num_nodes`), so the nodes agree on the cluster map with no coordinator. With `RF=1` each partition lives on one node and the others forward to it; with `RF>1` partitions are Raft-replicated across nodes. ## Locating the binary The plugin needs a built `nanobpmn` server binary. Resolution order: 1. `--binary ` 2. configured path (`c8ctl nano set bin `) 3. `NANOBPMN_BINARY=` 4. the matching **platform package** (`@nanobpm/c8ctl-plugin-nano--`), installed automatically as an `optionalDependency` when you install the plugin from npm 5. `release` build under the nanobpmn repo 6. `debug` build under the nanobpmn repo Most users never need a local build: installing the plugin from npm pulls in the prebuilt binary for their platform (step 4). Steps 5–6 are the local-dev path. The repo root defaults to `~/workspace/nanobpmn` and can be overridden with `NANOBPMN_REPO`. Build a binary with: ```bash cd ~/workspace/nanobpmn && make release-gateway # API-only gateway # or cd ~/workspace/nanobpmn && make release # includes the web console ``` ## State & data locations State, config, logs, per-node data, and the workspace live under a per-user directory (override with `C8CTL_NANO_HOME`): - **macOS**: `~/Library/Application Support/c8ctl-nano` - **Linux**: `$XDG_DATA_HOME/c8ctl-nano` (or `~/.local/share/c8ctl-nano`) - **Windows**: `%LOCALAPPDATA%\c8ctl-nano` ``` /config.json # persistent settings (binary path, workspace dir) /cluster.json # tracked cluster: nodes, pids, ports, config /data/node-/ # per-node engine data (journal, spill, snapshots) — ephemeral /logs/node-.log # per-node stdout/stderr /workspace/ # default shared workspace (models/, workers/) — persistent ``` `nano stop` removes the state file but keeps `data/` by default so you can stop a cluster and keep your journals; pass `--purge` to delete engine data too. The workspace is never removed except by `nano clean --workspace`. ## Flags | Flag | Applies to | Description | |----------------|------------|----------------------------------------------------------| | `--port` | start | Base HTTP port; node *i* listens on `basePort+i` (8080) | | `--partitions` | start | Total partitions across the cluster (default node count) | | `--rf` | start | Replication factor; `>1` enables Raft (default `1`) | | `--raft` | start | Force Raft on (default: on iff `rf>1`) | | `--capture` | start | Enable trace capture (recorded-input replay) on every node | | `--binary` | start | Path to the nanobpmn server binary (overrides `set bin`) | | `--force` | start | Stop any existing cluster first | | `--purge` | stop | Also delete per-node engine data | | `--workspace` | clean | Also delete the workspace (models + workers) | | `--follow`,`-f`| logs | Stream log output (`tail -F`) | ProcessOS flags (`processos` command): | Flag | Applies to | Description | |----------------|------------|----------------------------------------------------------| | `--port` | start | ProcessOS listen port (default 8090) | | `--nano-url` | start | Target Nano BPM engine URL (default `http://localhost:8080`) | | `--binary` | start | Path to the ProcessOS binary (overrides `set bin`) | | `--spawn-nano` | start | Force spawning a pilot Nano engine (default on when a nano binary is available) | | `--no-spawn-nano` | start | Don't spawn a pilot engine; reuse the `--nano-url` engine | | `--force` | start | Stop any existing ProcessOS instance first | | `--follow`,`-f`| logs | Stream log output (`tail -F`) | ## Managing ProcessOS (`processos`) ProcessOS is the optimization-plane server that analyses a running Nano BPM engine. The plugin can manage a single local ProcessOS instance with the same start/stop/status/logs lifecycle as `nano`. > **ProcessOS is a closed beta.** The operational commands (`start`, `stop`, > `status`, `logs`, `restart`) stay locked with a *"not available yet"* notice > until you opt in. Only `set` and `config` work before then. Opt in either by > setting the download URL you were given by the Nano BPM team, or by pointing > the plugin at a binary you already have. ### Quick install (closed-beta invitees) If you were given a ProcessOS download URL, this one-liner installs the Camunda 8 CLI (`@camunda8/cli`) and this plugin, then configures the download URL: ```bash curl -fsSL https://gist.githubusercontent.com/jwulf/9015a7c660b274c568d80e85c3914161/raw/install-processos.sh \ | bash -s -- "" ``` (Requires Node.js 18+. The canonical script lives at [`install-processos.sh`](./install-processos.sh).) Then run `c8ctl processos start`. ```bash # Closed-beta channel: persist the download URL, then start c8ctl processos set download-url c8ctl processos start # fetches processos-- on first run # …or point the plugin at a binary you already have c8ctl processos set bin ~/Downloads/processos c8ctl processos start ``` The download URL is the prefix the release binaries live under (e.g. the `…/processos/latest/` bucket URL). Persist it with `c8ctl processos set download-url `, or set `PROCESSOS_DOWNLOAD_URL` in your environment (the env var wins when both are present). The plugin appends the per-platform asset name (`processos-darwin-arm64`, `processos-linux-x64`, `processos-win32-x64.exe`, …), downloads it to `/bin/`, marks it executable, and runs it. The cached download is reused on subsequent starts. ```bash # Start ProcessOS against the local Nano BPM engine (http://localhost:8080) c8ctl processos start # Or against a specific engine, on a specific port c8ctl processos start --nano-url http://localhost:8080 --port 8090 # Inspect / stream logs / stop c8ctl processos status c8ctl processos logs --follow c8ctl processos stop ``` ### Automatic update notice When you're on the closed-beta channel (download URL configured), the plugin checks for newer ProcessOS builds in the background and prints a short one-line notice (at most **once per day**) when the published version is newer than the one you're running. It compares your installed binary's version against the `version.json` the release pipeline publishes next to the binaries, never blocks the command (the check runs detached), and is suppressed on non-interactive shells, in CI, and when `NO_UPDATE_NOTIFIER` / `NANO_NO_UPDATE_NOTIFIER` is set. To update, stop and start ProcessOS again — a downloaded binary re-fetches the latest build; a `set bin` binary updates itself. ### Pre-upgrade read-model backup & restore A schema-changing gateway release can reproject the SQLite read model and, in the worst case, silently drop completed process-instance history (root cause and durable fix tracked in `nano-bpm#831`). As a safety net, whenever `start` downloads a **different** ProcessOS version over an existing cached binary — a true upgrade — the launcher first snapshots each per-node read model **before** swapping the binary. First installs (no cached copy yet) are not upgrades, so they skip the backup. For every `…/data/node-/` that has a `read-model.sqlite`, the launcher copies it — together with its `-wal` sidecar (WAL mode keeps uncheckpointed pages there) and `-shm` index, plus the coherent point-in-time set `journal.head` and `snapshot.*.bin` — into a `read-model-backups/` subdir under that node, named `read-model.pre-upgrade---.sqlite` (the `` token keeps two backups that land in the same millisecond from colliding). The backup path is logged at INFO, and a bounded ring (the last **5** upgrades per node) is retained; older sets are pruned. The backup is best-effort: a failure is logged and never blocks the upgrade. To restore a node's read model from a backup (do this while the node is stopped): ```bash # 1. Stop the cluster so nothing is writing the read model. c8ctl nano stop # 2. Pick the pre-upgrade backup you want to restore (newest shown first). NODE=~/Library/Application\ Support/c8ctl-nano/data/node-0 # adjust per platform/node ls -t "$NODE/read-model-backups"/read-model.pre-upgrade-*.sqlite # 3. Replace the live read-model files with the chosen backup set. Remove the # stale WAL/SHM first so SQLite does not replay them over the restored DB. STEM="$NODE/read-model-backups/read-model.pre-upgrade---" rm -f "$NODE/read-model.sqlite" "$NODE/read-model.sqlite-wal" "$NODE/read-model.sqlite-shm" cp "$STEM.sqlite" "$NODE/read-model.sqlite" [ -f "$STEM.sqlite-wal" ] && cp "$STEM.sqlite-wal" "$NODE/read-model.sqlite-wal" # 4. Start the cluster again. c8ctl nano start ``` > On Linux the data dir defaults to `~/.local/share/c8ctl-nano/data`, on Windows > to `%LOCALAPPDATA%\c8ctl-nano\data` (override the root with `C8CTL_NANO_HOME`). On a successful `start` the summary leads with the landing page: ``` Start here http://127.0.0.1:8090/ (landing) Cockpit http://127.0.0.1:8090/cockpit Health http://127.0.0.1:8090/health Target Nano http://localhost:8080 ``` ### Pilot engine (spawned by default) ProcessOS uses a Nano engine in two roles: the **target** engine it analyses (read-only, set with `--nano-url`), and its **own** internal "pilot" engine where it runs experiments. **By default ProcessOS spawns its own pilot engine** as a child process, so it never disturbs the target: ```bash c8ctl processos start # spawns a pilot engine automatically c8ctl processos start --no-spawn-nano # reuse the --nano-url engine for the pilot too ``` ProcessOS spawns its pilot engine from a Nano gateway binary given in `PROCESSOS_NANO_BIN`. The plugin **auto-wires `PROCESSOS_NANO_BIN`** from the same binary `c8ctl nano` uses (`--binary` / `nano set bin` / `$NANOBPMN_BINARY` / the platform package / a repo build). A console-enabled nano build is required — the npm-distributed binaries qualify. The spawned engine is torn down when ProcessOS stops. If no nano binary can be found, ProcessOS falls back to `--no-spawn-nano` automatically (using the target engine as the pilot) and prints a warning. Force the behaviour explicitly with `--spawn-nano` / `--no-spawn-nano`, override the binary with `c8ctl processos set env PROCESSOS_NANO_BIN=`, or set the mode persistently with `c8ctl processos set env PROCESSOS_SPAWN_NANO=false`. ### ProcessOS configuration Settings persist under a `processos` key in the same `config.json` as `nano`. ```bash c8ctl processos set bin # path to the downloaded ProcessOS binary c8ctl processos set download-url # closed-beta binary download URL (enables ProcessOS) c8ctl processos set port # listen port (default 8090) c8ctl processos set nano-url # target Nano BPM engine (default http://localhost:8080) c8ctl processos set data-dir # PROCESSOS_DATA_DIR (default /processos-data) c8ctl processos set env KEY=VALUE # set any passthrough env var (e.g. PROCESSOS_LLM_MODEL) c8ctl processos set env KEY= # unset a passthrough env var c8ctl processos config # show current settings and on-disk paths ``` The binary is resolved in this order: `--binary` flag → `set bin` → `$PROCESSOS_BINARY` → a cached download under `/bin/` → a local `processos/target/{release,debug}/processos` build → a fresh download from the configured download URL (`set download-url` / `$PROCESSOS_DOWNLOAD_URL`). Typed settings (`port`, `nano-url`, `data-dir`) always win over generic `env` passthrough values when launching. ## Installing from a local checkout (development) ```bash c8ctl load plugin --from file:///path/to/c8ctl-nano ``` Then verify it shows up: ```bash c8ctl help | grep nano ``` For the normal npm install, see [Installation](#installation) above. ## Distribution & releasing Releases are automated with **semantic-release** (`.github/workflows/release.yml`, `release.config.cjs`). Pushing conventional commits to `main` cuts a version, publishes to npm, and creates a GitHub Release. ### Platform packages The server binary is shipped as a set of platform-specific npm packages, one per target, gated by npm's `os`/`cpu` fields. They are **scoped under `@nanobpm`** (a scope we own) so the names can never be squatted or npm-security-held: | package | os | cpu | |--------------------------------------------|--------|-------| | `@nanobpm/c8ctl-plugin-nano-darwin-arm64` | darwin | arm64 | | `@nanobpm/c8ctl-plugin-nano-darwin-x64` | darwin | x64 | | `@nanobpm/c8ctl-plugin-nano-linux-x64` | linux | x64 | | `@nanobpm/c8ctl-plugin-nano-linux-arm64` | linux | arm64 | | `@nanobpm/c8ctl-plugin-nano-linux-armv7` | linux | arm (v7) | | `@nanobpm/c8ctl-plugin-nano-linux-armv6` | linux | arm (v6) | | `@nanobpm/c8ctl-plugin-nano-win32-x64` | win32 | x64 | The root `c8ctl-plugin-nano` lists all of these as `optionalDependencies` (pinned to the exact release version, injected into the published tarball at release time). npm installs only those matching the host, so each user downloads a single binary — **except on 32-bit ARM**, where npm's `cpu` field is just `arm` for both armv6 and armv7, so both install and `platformForHost` picks the right one at runtime via the host's `arm_version` (falling back to the armv6 build, which also runs on armv7). The mapping lives in `platforms.mjs` — the single source of truth shared by the build/publish scripts and the plugin's runtime resolution. ### Binary delivery contract (upstream CI) This repo never builds or references the private Nano BPM source. Instead, the upstream cross-compile pipeline uploads prebuilt binaries as assets on a rolling GitHub Release named **`binaries`** in this repo. The release workflow downloads them (`gh release download binaries`) and packs them into the platform packages. Each asset must be named exactly (see `PLATFORMS[].asset` in `platforms.mjs`): ``` nanobpm-gateway-rest-server-darwin-arm64 nanobpm-gateway-rest-server-darwin-x64 nanobpm-gateway-rest-server-linux-x64 nanobpm-gateway-rest-server-linux-arm64 nanobpm-gateway-rest-server-linux-armv7 nanobpm-gateway-rest-server-linux-armv6 nanobpm-gateway-rest-server-win32-x64.exe ``` The upstream job needs a token with `contents: write` on this repo and can upload with e.g. `gh release upload binaries --clobber`. ### What triggers a release The plugin's npm version is **decoupled** from the nanobpmn binary version, so uploading new binaries does **not** by itself publish a new npm version — `semantic-release` only releases on releasable commits to `main`. To make a binary update ship, the upstream pipeline (after uploading the assets) rewrites the tracked marker file **`nanobpmn-binary.json`** in this repo with the new nanobpmn version/commit and pushes it as a `fix(binary): …` commit. That commit triggers the release workflow, which downloads the just-uploaded binaries and publishes a patch release. The marker is surfaced to users in `nano config` (`bundled nano `). The push is a no-op when the marker is unchanged. ```json // nanobpmn-binary.json — overwritten by upstream CI; "0.0.0-dev" = local checkout { "version": "v1.4.2", "commit": "cdeb390", "updated": "2026-06-27T11:00:00Z" } ``` ### OIDC / Trusted Publishing The workflow is set up for npm **Trusted Publishing** (OIDC, `id-token: write`) with provenance (`NPM_CONFIG_PROVENANCE: true`, requires this repo to be public). Trusted Publishing is per-package and requires the package to already exist, so: 1. **Bootstrap** the first release with a granular-automation `NPM_TOKEN` secret — it is used automatically and creates all six packages. The token must have **publish rights to the `@nanobpm` scope** (the platform packages are scoped). 2. On npmjs.com, add a **Trusted Publisher** (this repo + `release.yml`) for the root package and each of the five `@nanobpm/c8ctl-plugin-nano-*` platform packages. 3. Remove the `NPM_TOKEN` secret; subsequent releases authenticate via OIDC. > Note: because the platform packages are **scoped** (`@nanobpm/…`), they sidestep > the unscoped-name squatting/`0.0.1-security` hold that previously blocked > `c8ctl-plugin-nano-win32-x64`. If you ever add a new platform, its scoped name is > yours to publish immediately.