# Module Primitives — the remote-ops toolkit for module hooks

**The one rule: modules never hand-build SSH.** A module hook that needs to touch
a deployed box does it through a typed primitive from `@celilo/capabilities` —
never a raw `ssh root@…` string, never `node:child_process`. SSH lives in exactly
one place (the `remoteExec` seam) and in Ansible's transport. A raw ssh/exec in
`modules/**/scripts/` is a defect (and a publish/lint gate flags it).

Why: hand-rolled ssh strings re-invent host-key handling, escaping, timeouts, and
secret-hygiene every time — and can't be unit-tested. The primitives give you one
correct, mockable implementation of each shape of remote work.

```ts
import { probe, serviceCtl, applyRenderedConfig } from '@celilo/capabilities';
```

---

## The two things every primitive shares

**Target** — a `RemoteTarget` is just `{ ipv4_address: string }`. A
`DeployedSystem` (what your hook gets in `context.systems`) satisfies it directly;
for a machine addressed by config, pass `{ ipv4_address: ip }`.

**Runner** — every primitive takes an optional trailing `runner` (default
`execRunner`, which really shells out). In unit tests pass `createMockRunner([…])`
so you assert on the command strings/stdin without touching a box. A non-zero
remote exit comes back as `{ ok: false, … }` — **failure is data, never a throw** —
so your hook decides whether it's fatal (a deploy) or best-effort (a teardown).

```ts
// test
import { createMockRunner } from '@celilo/capabilities';
const { run, calls } = createMockRunner([
  { match: 'systemctl is-active caddy', result: { ok: true, stdout: 'active', stderr: '' } },
]);
expect(probe(sys, { kind: 'systemd', unit: 'caddy' }, run).healthy).toBe(true);
```

---

## Which primitive when

| You want to… | Use | Not |
|---|---|---|
| Check a box is healthy (read-only) | `probe` | `ssh … systemctl is-active` |
| Start/stop/restart a systemd unit | `serviceCtl` | `ssh … systemctl restart` |
| Push a config whose content you computed from celilo's DB | `applyRenderedConfig` (converge) | `ssh … cat > file` |
| Wait for a condition to come true | `waitFor` | `sleep` / `setTimeout` |
| Back up / restore app state (binary) | `streamBackup` / `streamRestore` | `ssh … | base64` |
| Read a unit's journal | `tailLog` / `grepLog` | `ssh … journalctl` |
| Run a secret on a command line | `runAppCommandWithSecret` | secret on argv |
| Anything else with no capability/HTTP/converge path | `runAppCommand` (+ `// escape-hatch:`) | a raw ssh string |

**Reach for a capability method first.** If another module already provides the
thing (open a port → `firewall.exposeService`; register a route →
`public_web.registerReverseProxy`; mint a token → `idp`/`source_forge`), call
that, not a primitive. The primitives are for the module's *own* box.

---

## The primitives

### `probe(target, check, runner?, opts?) → { healthy, detail }`
Read-only health check. Never mutates. `check` is one of:
- `{ kind: 'systemd', unit }` — unit is `active`.
- `{ kind: 'command', command, expectStdoutIncludes? }`.

For HTTP use `probeHttp`, not `probe` — see below.
```ts
probe(sys, { kind: 'systemd', unit: 'caddy' }, run);
```

### `probeHttp(target, { port, path?, expectStatus?, headers?, timeoutMs? }) → Promise<{ healthy, detail, failure? }>`
HTTP health check, run **from the management server** over `fetch`. Nothing runs
on the target — no SSH, no curl (the production LXC image ships none, and a
missing binary used to be indistinguishable from a dead service). There is
deliberately **no `url`**: you give a port and a path, and the primitive builds
the address from the target's `ipv4_address`, so a check cannot name `localhost`
and end up probing celilo-mgr itself. `expectStatus` defaults to 200; pass a
list (e.g. `[200, 308]`) or `headers: { Host: 'app.example.com' }` for a vhost.
Redirects are **observed, not followed**. On failure, `failure` is
`'unreachable'` (could not connect) or `'status'` (answered with the wrong code).
```ts
await probeHttp(sys, { port: 80, headers: { Host: fqdn }, expectStatus: [200, 308] });
```

### `serviceCtl(target, unit, action, runner?, opts?) → RunResult`
systemd control. `action ∈ start|stop|restart|reload|enable|disable`. Mutating —
for liveness use `probe`.
```ts
serviceCtl(sys, 'caddy', 'reload', run);
```

### `applyRenderedConfig({ target, path, content, validate, apply, statePath?, runner?, timeoutMs? }) → RunResult`
**The converge primitive.** For config whose *content* your hook computes at
deploy-time from celilo's DB (caddy's Caddyfile from `web_routes`, knot's views
from the DNS ledger, the firewall ruleset from the port-forward registry) —
content a static Ansible render can't know. One atomic on-box script, one SSH
round-trip: back up → write (`content` rides **stdin**, newline-safe) → `validate`
(rolled back + not activated if it fails) → `apply` (rolled back if it fails).
`{path}` is substituted into `validate`/`apply`.

By default, rollback and last-render state are retained beside `path` as
`.celilo-bak` and `.celilo-prev`. Set `statePath` to a prefix outside daemon
include directories (such as `/etc/dnsmasq.d`) when the daemon parses every
neighboring file.
```ts
applyRenderedConfig({
  target: sys, path: '/etc/caddy/Caddyfile', content: rendered,
  validate: 'caddy validate --config {path} --adapter caddyfile',
  apply: 'caddy reload --config {path} --force', runner: run,
});
```

### `waitFor(predicate, { attempts?, intervalMs?, onAttempt? }) → Promise<boolean>`
Poll an async predicate until true or exhausted (default 30 × 5s). A combinator —
it never touches the remote; wrap a `probe`/`probeHttp`/`runAppCommand` thunk. Use
`onAttempt` to emit a heartbeat so a long wait doesn't trip the hook idle-timeout.
Never `sleep`.
```ts
await waitFor(async () => (await probeHttp(sys, { port: 3000 })).healthy,
  { onAttempt: (n) => logger.info(`waiting… ${n * 5}s`) });
```

### `tailLog(q & { lines? }) / grepLog(q & { grep }) → RunResult`
Read a systemd unit's journal. `q = { target, unit, grep?, regex?, ignoreCase? }`.
`grepLog` requires `grep`; `regex: true` uses `grep -E`.
```ts
grepLog({ target: sys, unit: 'caddy', grep: 'certificate obtained', regex: true });
```

### `streamBackup / streamRestore / fetchFile / pushFile → RunResult`
Binary-safe streaming — bytes flow through the local shell redirect/pipe, never
Node's string capture, so tar/pg_dump/sqlite snapshots survive byte-for-byte.
```ts
streamBackup(sys, 'tar -cf - -C /var/lib/forgejo repositories', '/backup/repos.tar', run);
streamRestore(sys, '/restore/repos.tar', 'tar -xf - -C /var/lib/forgejo', run);
fetchFile(sys, '/etc/app/id', '/local/id', run);   // pull
pushFile(sys, '/local/cert.pem', '/etc/app/cert.pem', run);  // push
```

### `runAppCommandWithSecret(target, command, secret, runner?, opts?) → RunResult`
For a command needing a secret on its **command line** (token mint, credential
register) where the app has no HTTP API. The secret rides **stdin** into shell var
`SECRET` — never argv, so it's absent from logs/history/the recorded call.
Reference it as `"$SECRET"`. Escape-hatch discipline applies (comment required).
```ts
// escape-hatch: act_runner registers only via CLI; no HTTP API.
runAppCommandWithSecret(sys, 'act_runner register --token "$SECRET"', token, run);
```

### `runAppCommand(target, command, runner?, opts?) → RunResult`
**The escape hatch.** Only when the work is neither a capability, HTTP, converge,
nor a service action (e.g. removing app state on teardown). **Every call site MUST
carry an inline `// escape-hatch: <why no capability/HTTP/converge path>` comment**
— the recurrence-gate lint flags an uncommented one.
```ts
// escape-hatch: purge app data dir on teardown; no capability owns this.
runAppCommand(sys, 'rm -rf /var/lib/app/*', run);
```

### `remoteExec(target, command, opts?, runner?) → RunResult`
The single SSH seam every other primitive builds on. You almost never call this
directly — prefer the specific primitive. `opts` is `{ input?, timeoutMs? }`.

---

## Rules of thumb

- **Failure is data.** Check `.ok`; decide fatal vs best-effort. Don't wrap in
  try/catch expecting throws.
- **Unit-test with `createMockRunner`.** Assert on `calls[].cmd` and `calls[].input`.
  Controlled identifiers (systemd unit names) are passed unescaped; the outer
  `remoteExec` escapes the whole command, so assert on bare substrings.
- **Secrets never on argv** — `runAppCommandWithSecret`, or a capability method.
- **No `sleep`** — `waitFor` on the real condition.
- **Computed config → `applyRenderedConfig`**, not `ssh cat >`. It validates and
  rolls back; a broken config never goes live.

See also: `CELILO_SUBSYSTEMS.md` (the primitive impls + the firewall converge),
`openspec/changes/unified-management-no-ssh/proposal.md` (the design), `reference/MODULE_DEVELOPMENT_GUIDE.md`.