# Celilo Subsystem Inventory **What this is:** a map of infrastructure primitives celilo **already implements**, with the entry-point file for each. It exists so agents stop reinventing features (the trigger case: IP/VMID allocation, which is already IPAM). **The rule:** Assume celilo already provides any infrastructure primitive (IP/VMID allocation, secrets, DNS, firewall, cross-module data, packaging, event scheduling). Before proposing to build one — or asking the operator how to do it — grep this inventory / the codebase and cite what you found. Only raise a question if the search is genuinely empty, or the real choice is between existing mechanisms. This is an *implemented-feature* map. For terminology see `GLOSSARY.md`; for architecture see `openspec/specs/`. Companion doc: [CELILO_CORE_MODULES.md](./CELILO_CORE_MODULES.md) (the production modules celilo ships). > **All file paths below are relative to the celilo source-repo root** (e.g. > `apps/celilo/src/ipam/allocator.ts`). They are references into the source tree, not links > relative to this file — this doc also ships inside the `@celilo/cli` npm package, where the > wider repo isn't present. Paths drift; if an entry looks wrong, re-grep and fix it here. --- ## Allocation & infrastructure selection - **IPAM (IP/VMID allocation)** — `apps/celilo/src/ipam/allocator.ts` — `allocateIPFromSubnet`, `allocateVMID`, `reserveIP`/`unreserveIP`, `inferZoneFromIP`, `getAllocation`. Auto-wrapper: `apps/celilo/src/ipam/auto-allocator.ts` — `allocateForModule` / `deallocateForModule`. - **Ingress IPs (how the LAN reaches a service in a segmented zone)** — `ensureIngressIps` in `apps/celilo/src/templates/generator.ts`, called from `generateTemplates`. A module opts in by declaring an infrastructure variable whose name ENDS IN `ingress_ip` (`dns_ingress_ip` on the resolver, `ingress_ip` on `caddy-internal`); generate allocates and RESERVES a free `internal`-subnet address once and reuses it forever after, and the module's `on_install` passes it to `firewall.exposeService({ ingressIp })`. That renders ONE DNAT on the firewall's internal side and nothing on the external interface — it is not a public port-forward, and conflating the two is what pinned `caddy-internal` into the `internal` zone until celilo#879. Idempotence is load-bearing: re-allocating on a later generate silently moves the address clients use, with every command still reporting success (`apps/celilo/src/templates/ingress-ip.test.ts`). - **Infrastructure selection (container-service vs machine pool)** — `apps/celilo/src/services/machine-pool.ts` (`getMachineByHostname`, `addMachine`, `assignModuleToMachine`) and `apps/celilo/src/services/container-service.ts` (`getContainerServiceByName`, `addContainerService`, `verifyContainerService`). Provider API clients: `apps/celilo/src/api-clients/proxmox.ts`, `apps/celilo/src/api-clients/digitalocean.ts`. - **Zone detection / system config** — `apps/celilo/src/services/zone-detector.ts` — `detectZoneFromIp` reads `network..subnet` from the `systemConfig` table and returns `NetworkZone | 'unknown'`. It answers CONTAINMENT ONLY. It used to return `'external'` on no-match, which conflated "no declared subnet contains this" with "the internet can route to this" — on a firewall with five RFC1918 legs that reported four of them as facing the internet. `'unknown'` is the honest answer; the caller resolves it (see `machine add`: publicly routable → `external`, otherwise fail asking for `--zone`). The subnet-backed zone list is derived from `NETWORK_ZONES` minus `external`, which has no subnet and must never be given one. - **Interface classification** — `packages/capabilities/src/interface-classification.ts` — THE shared classifier, used by the backend and every firewall provider module so the two cannot drift apart again. `isPubliclyRoutable(ip)` is a property of the address alone (false for RFC 1918, RFC 6598 carrier-grade NAT, loopback, link-local, multicast, reserved). `classifyInterfaces(interfaces, zones)` assigns each interface `zone → external → alien`, first match winning, where `external` is the RESIDUAL — routable and claimed by no declared zone — and is never subnet-matched. `externalEdge()` returns none/single/**ambiguous** rather than silently picking the first public address. `defaultRouteFinding()` enforces the invariant that the default route leaves through `internal` or `external`. **`subnetContains(cidr, ip)` lives here and is the ONLY implementation** — three existed and disagreed (the backend's mishandled `/0`); the other two are deleted, not aliased. Design: `openspec/changes/firewall-interface-classification/design.md`. - **Declared networks (classification input)** — `readDeclaredNetworks(db)` in `apps/celilo/src/hooks/capability-loader.ts` — every `network..subnet` in system config, which is what an interface is attributed against. Read from the CONFIG, not from `NETWORK_ZONES`: celilo holds networks that are not placement zones (`network.control-plane-vpn.subnet`, which `wireguard` requires and reads). Injected into the firewall capability as a LIVE reader (`declaredNetworks`) — that liveness was a mitigation for values written mid-run by a module hook, which `network-declaration` removes; see that spec before assuming a snapshot is still unsafe. - **Network requirement + ensure (celilo owns the namespace)** — `apps/celilo/src/services/network-ensure.ts` (`ensureRequiredNetworks`), `NetworkRequirementSchema` / `getRequiredNetworkNames` in `apps/celilo/src/manifest/schema.ts`. A module declares `requires.networks: [{name}]` — a NAME, never a value; the schema is `.strict()` so a `subnet:` on the requirement is rejected with a message saying why. The deploy calls `ensureRequiredNetworks` in its interview phase, BEFORE generation and before any hook, and asks over the generic bus interview (`askText`, so it is answerable headless) for anything undefined. Which attributes a network has is celilo's answer, taken from `apps/celilo/schemas/system_config.json`: that file declares `network..gateway` for the routed segments and omits it for `control-plane-vpn`, so a gateway is never asked for a network that has none. Well-known names carry a `suggested` range there — deliberately NOT `default`, which `getDefaultConfiguration()` would seed at `system init`. Spec: `openspec/specs/network-declaration/spec.md`. - **The network write path (closed) + celilo's own discovery** — `celilo system apply-config` (`apps/celilo/src/cli/commands/system-apply-config.ts`) REFUSES the whole `network.` namespace, `network.bridge` excepted (a Proxmox bridge name is not addressing, and it is the one network key with a schema default). That is the automation surface a module hook shells out to, so closing it there is what makes "networks are celilo's" an authority rather than a convention every module has to remember. The refusal names the alternative — declare the network, read it with `$system:` — because a bare rejection sends a module author hunting for a typo. The one write that legitimately needed the surface moved INTO celilo: `celilo system discover-network` (`apps/celilo/src/services/network-discovery.ts`, `cli/commands/system-discover-network.ts`) parses the box's own `ip route` and records `network.internal.*` — or `network.secure-mgmt.*` when the box is off the internal LAN (#300). `celilo-mgmt` calls it and decides nothing; it used to parse and write this itself. Idempotent and never overwrites addressing already set. Recurrence gate: `test-integration/module/no-module-writes-networks.test.ts`. - **Firewall interface audit** — `apps/celilo/src/services/audit/interface-classification.ts` — reports per-firewall classification in `celilo audit`: alien interfaces by name and address (drift), and the blocking findings a converge refuses on — a carrier-grade NAT leg, an ambiguous external edge, a default route on the wrong leg. - **Zone taxonomy (canonical list)** — `apps/celilo/src/db/schema.ts` — `NETWORK_ZONES` is the single array; `NetworkZone` is DERIVED from it. Never hand-maintain a second copy: a duplicate that dropped a member made zone validation return null and silently fall back to a wrong-but-valid zone. Two copies are unavoidable and BOTH are guarded: `modules/iptables`' `zones` picker (`test-integration/module/zone-options-cover-network-zones.test.ts` fails on any drift, and did when `isp-transit` was added to core alone) and the generated `schemas/module-manifest.schema.json` (`bun run check:schema`, run from `apps/celilo` — a FOURTH gate CI runs that the "Validation gates" list in CLAUDE.md does not name). - **`isp-transit`** — the private segment between a celilo firewall and the router upstream of it. Non-placement and non-allocatable (it joins `external` and `control-plane-vpn` in the excluded set) because that router owns the addressing. It exists because on a DOWNSTREAM firewall the default route and the workload network share one interface, so `internal` meant both at once and every egress rule also reasoned about the workload network. A firewall with no separate transit leg declares `default_route_zone: internal` and behaves as it always did. - **Declared egress zone** — `modules/iptables` `default_route_zone`, consumed by `defaultRouteFinding` in `packages/capabilities/src/interface-classification.ts` — celilo already classifies whichever interface carries the default route; what it could not do is tell an INTENDED egress leg from a drifted one, so the comparison was hardcoded to "internal or external". Declaring the intent makes a wandered default route a finding. Defaults to `internal`. - **Capability-secret access is reference-driven** — `apps/celilo/src/capabilities/validation.ts` — `validateCapabilityAccess` fires on the `$capability:.` paths the CONSUMER references, collected from its manifest AND from the templates `validateModuleTemplates` already parses at import. It used to iterate the secrets the PROVIDER declares, so requiring a capability that merely carries a secret put a module on the hook for that secret's `readable_by` list (celilo#854, celilo#1038). Reading a secret still goes through `checkCapabilitySecretAccess` at resolution time. - **Zone-first capability provider selection** — `apps/celilo/src/capabilities/lookup.ts` — one shared selection function used by both the direct lookup and the hook capability loader, which had drifted apart. Well-known-provider uniqueness is scoped per zone, which is what lets a second `dhcp_server` provider exist at all while `axon`/`greenwave` claim it fleet-wide. - **Control-plane network (`secure-mgmt`)** — `apps/celilo/src/hooks/capability-loader.ts` — `loadControlPlaneSubnet` returns the subnet of the zone `celilo-mgmt` is deployed in. `secure-mgmt` is a placement zone AND the control-plane tier, deliberately NOT in `ZONE_TIER_ORDER` (it is not part of the `dmz → app → secure` data-plane chain; it reaches every tier by trust). The firewall's `trustedSubnets` derives from this rather than assuming celilo-mgr sits on `internal`. Reported as an actionable gap by `checkControlPlaneNetwork` in `apps/celilo/src/services/fleet-checks.ts` when the management address matches no configured subnet. - **A deployed system's zone** — `apps/celilo/src/services/deployed-systems.ts` — for machine-pool deploys the zone recorded is the ZONE OF THE MACHINE, not `requires.system.zone` (which is only the minimum used to *select* a host, as with sizing). Three writers must agree: `recordDeployedSystemForModule`, `backfillModuleSystems`, and `apps/celilo/src/variables/context.ts` — the last runs latest and will overwrite the others. - **Host discovery ("which host serves module X?")** — `apps/celilo/src/cli/commands/module-where.ts` (`celilo module where [--json]`, MCP `celilo_module_where`) — reads deployed hosts from `module_systems` via `getModuleSystems`, reconciles the live Proxmox node via `reconcilePlacement`, and adds a role-based reachability hint per zone. CI/build infra (builder VM, Forgejo runners) is out of scope (not in `module_systems`). - **Daemon-journal diagnostic ("what does the module's daemon know that celilo doesn't?")** — `apps/celilo/src/services/module-journal.ts` + `apps/celilo/src/cli/commands/module-journal.ts` (`celilo module journal [--unit|--lines|--since|--grep|--json]`, MCP `celilo_module_journal` on the RO principal) — resolves hosts via `getModuleSystems` and reads each one's systemd journal through the `tailLog` remote primitive. General to every module, not one. **READ-ONLY by construction**: `planJournalRead` is pure and the only remote command the op can emit is a `journalctl` read, so it cannot reconfigure a module, send anything, or consume inbound messages the collection path is waiting for (`module-journal.test.ts` asserts on the exact command reaching the SSH seam). Sibling to `celilo module logs`, which reads the LOCAL Ansible deploy log, not the remote daemon. Unit defaults to the glob `*`. An unreachable host reports UNREADABLE, never as an empty journal. Rationale: openspec/changes/fix-signal-inbound-delivery Decision 5. ## Capability system (cross-module data & functions) - **Define a capability function** — `packages/capabilities/src/define-capability-function.ts` — `defineCapabilityFunction`. - **Canonical capability registry** — `packages/capabilities/src/capability-registry.ts` — `KNOWN_CAPABILITY_NAMES` (the authoritative list), `CapabilityRegistry` type. Also `PROVIDER_VIEW_CAPABILITIES` / `isProviderView` / `ProviderViewCapabilities`: the registry entries (`web_routes`, `firewall_registry`) that are framework-injected PROVIDER VIEWS rather than capabilities a module can declare — in the registry so audits and type checks reach them, named separately so author-facing surfaces leave them out. `ProviderViewCapabilities` is what puts them on a hook's `capabilities` object. Public surface: `packages/capabilities/src/index.ts`. - **Loader (wires provider factories into hook contexts)** — `apps/celilo/src/hooks/capability-loader.ts` — `loadCapabilityFunctions`, `resolveFirewallNatIp`. - **Which provider a consumer actually resolved to (`capability_bindings`)** — `apps/celilo/src/services/capability-bindings.ts` — `recordCapabilityBinding`, `listCapabilityBindings`, `withBindingRecord`, written from the single return of `loadCapabilityFunctions`. The `capabilities` table is provider-side only, so celilo could answer "who COULD provide this" and never "who does this module actually use"; every caller needing the second question rebuilt the same approximation (the consumer's `requires` + `optional` crossed with `capabilities`), which names every provider a module MIGHT have bound to. `tango-nexus` declares four optional capabilities, is deployed against one off-fleet host, and that approximation carries three edges that do not exist. **The CALL is the binding, not the resolution** — the loader injects every registered capability regardless of what the consumer declared (its own comment: "not just required ones"), so recording what it resolves is a SUPERSET of even the permissive set. A consumer's hook invoking a method is the only event that separates the optional it uses from the ones it merely declares, which is why this matters most for the capabilities that mint nothing (`external_web`, `idp`, `source_forge`, `registry_publish`, `control_plane_vpn`, `apt_publish`) and therefore leave no artefact anywhere else. Recorded on ATTEMPT rather than success: a consumer that called a provider and got an error is standing on that provider, and hiding the edge exactly when the call fails hides it when it matters most. Unique on (consumer, capability) so a redeploy re-asserts and a provider swap rewrites in place; `bound_at` is LAST-SEEN, so a binding last exercised forty deploys ago is a queryable fact rather than a silent lie. Cascades on the consumer. It runs BESIDE `planConsumerCleanup`, never narrowing it — cleanup must still reach every provider that might hold minted state, including one whose binding row was never written. See celilo#1072. - **Ledger wrappers (stateful capabilities)** — `apps/celilo/src/services/dns-registrations.ts` (`withDnsRegistrationLedger`), `apps/celilo/src/services/dns-internal-records.ts` (`withDnsInternalLedger`). - **Public DNS ledger (`dns_registrations` + `dns_registration_consumers`)** — `apps/celilo/src/services/dns-registrations.ts` — records WHICH MODULE ASKED FOR WHICH NAME, and deliberately **no address**: the published value is source-detected on every assert and observable on demand from public DNS, and a stored copy is one careless read away from becoming an instruction again (celilo#626 — the 15-minute `refresh_registrations` replayed addresses written by an older celilo and took five public names dark for nine days while every in-fleet check stayed green). Consumers are a SET, not a column: the FK cascade on a single `consumer_module_id` decided when a LIVE record was forgotten, so a row now survives until its LAST consumer is removed (`openspec/changes/public-dns-reachability/design.md` D5). Orphaned rows are pruned on read (`pruneOrphanedRegistrations`) rather than by trigger — SQLite fires delete triggers for FK-cascaded deletes only with `recursive_triggers` on. `companion: true` marks a name celilo claimed on a module's behalf (`www.` ↔ ``) rather than one a module asked for; the provider reports it as `outputs.companion_fqdn` whether or not the claim reported success, because a provider's success response is not evidence of publication. Operator surface: `celilo dns registrations`. ### Known capabilities (impl → provider module) | capability | impl | provider module(s) | |---|---|---| | `public_web` | `packages/capabilities/src/public-web.ts` (`createPublicWeb`). **Route lifecycle** is framework-owned at both ends, not per-module: removing a consumer dispatches caddy's `on_consumer_removed` hook (re-render the Caddyfile without that module's routes → reclaim `/srv/www/`) via the generic **consumer-removal cleanup** below, whether or not the module has an `on_uninstall`; deploying a provider runs `apps/celilo/src/services/public-web-republish.ts` (re-run every static consumer's `on_install`) so a provider rebuild refills the web roots it destroyed. There is no consumer-facing `unregister_routes` — withdrawal is the provider's, by design (openspec/changes/consumer-removal-cleanup D11). | caddy | | `idp` | `packages/capabilities/src/idp.ts` | authentik | | `dns_registrar` | `packages/capabilities/src/dns-registrar.ts` (`registerHost({ fqdn })` — **there is no way to supply an address**; the provider re-derives it from its own update's source IP on every assert) | namecheap | | `external_web` | `packages/capabilities/src/external-web.ts` (`publishStaticSite`). **OFF-FLEET static publishing** — the counterpart to `public_web`. See the in-fleet vs off-fleet note below. | generic-cpanel-hosting-provider | | `firewall` | `packages/capabilities/src/firewall.ts` (`exposeService`, `unexposeService`, `listExposedServices`). **Converge model**: register into the shared-core port-forward registry → render the complete ruleset → apply atomically (see the firewall converge note below). | greenwave, iptables | | `dns_internal` | `packages/capabilities/src/dns-internal.ts` | knot-unbound-internal, technitium | | `dhcp_server` | `packages/capabilities/src/dhcp-server.ts` | greenwave | | `source_forge` | `packages/capabilities/src/source-forge.ts` | forgejo | | `registry_publish` | `packages/capabilities/src/registry-publish.ts` | celilo-registry | | `notification` | `packages/capabilities/src/notification.ts` (`send`, optional `receive`) | signal (planned) — transports are ordinary modules, both self-hosted and credential-only | | `cross_module_read` | `packages/capabilities/src/cross-module-read.ts` | framework (read other modules' capability data) | | `control_plane_vpn` | `packages/capabilities/src/control-plane-vpn.ts` (`registerClient`, `revokeClient`, `listClients`, `getEndpoint`). **Its state is NOT in a celilo table** — registrations live in the PROVIDER's own module config under `registered_peers`, kept separate from the operator's `peers` so a machine can never rewrite what an operator typed, and so `revokeClient` structurally cannot reach a declared peer. Do not go looking for a `vpn_clients` table; there isn't one and that is deliberate. Addresses are allocated by the CONSUMER from `client_pool` — celilo's IPAM deliberately does not cover VPN clients. ⚠️ Every client registered here reaches every managed zone: the tunnel's client subnet is a registered trusted source. | wireguard | | `private_web` | `packages/capabilities/src/private-web.ts` (`publishStaticSite`, `registerReverseProxy`, `unregisterRoutes`, `getCaCertificate`). **Fleet-only HTTP ingress** — internal DNS, an internally-issued certificate, and no public exposure. A SIBLING of `public_web` rather than a flag on it, because `public_web` treats an unreachable route as a deploy failure and publishes a public record to prevent one. Reuses `public_web`'s request types so a consumer can write `private_web ?? public_web` and call through the union without branching. **MODULE-provided**, unlike `public_web`: the implementation is `modules/caddy-internal/scripts/private-web-functions.ts` (wired via `CAPABILITY_MODULE_MAP`), and the route table lives in that module's own config rather than celilo's `web_routes` — a private route in `web_routes` would be picked up and served by the PUBLIC caddy, which derives its served hostnames from every row (celilo#846). | caddy-internal | ### In-fleet (`public_web`) vs off-fleet (`external_web`) Both serve a static site; the difference is **control, not file serving**. A system is *in-fleet* when celilo governs it — it can run a managed daemon, bind a port, drive the firewall, put the box on the VPN. It is **off-fleet** when celilo can only place files there: an unprivileged, jailed cPanel account is the canonical case. Off-fleet is a governance boundary, not a deploy boundary. | | `public_web` (caddy) | `external_web` | |---|---|---| | TLS certificate | celilo obtains it (ACME) | already installed by the host | | Public + split-horizon DNS | celilo registers | the host's | | Firewall ingress | celilo opens | not celilo's to open | | `idp` / reverse-proxy composition | yes | no | | Route | any path, incl. `/` | a subfolder only; `/` is REJECTED (the docroot holds the host's own site) | `external_web` is module-owned (loaded through `CAPABILITY_MODULE_MAP` like `dns_registrar`), not framework-owned like `createPublicWeb` — there is no celilo-side machinery to own, only a provider holding credentials. Its `publishStaticSite` signature is a subset of `public_web`'s on purpose, so a content module targets either with `capabilities.external_web ?? capabilities.public_web`. Both are **authoritative**: a publish that doesn't reach clients throws (#328). - **Static-content converge (`/srv/www`, design D10)** — `apps/celilo/src/services/static-content-converge.ts` (pure `planStaticContent`/`buildStaticContentPlan` + `writeStaticContentVars` + the `executeAnsible` caller `convergeStaticContent`). The declared `web_routes` release set (slug, content hash, hostnames, the source dir core derives from `modules.source_path`) is the desired state; the provider's Ansible converge makes `/srv/www` match: content-hashed release dirs `/srv/www/-`, an atomic symlink swap of `/srv/www/`, pruning beyond `static_release_retention` (caddy manifest variable, operator config with default 5 — the role never carries the number). Two callers, one plan: the `public_web` capability's `convergeStaticContent` callback (tag-scoped run through `executeAnsible`, so a publish returns only once the host matches) and the provider's own deploy (vars written pre-Ansible in `module-deploy.ts`, so a rebuilt host recovers with no consumer). The content hash lands on EVERY route row sharing the slug. Replaced the hand-built ssh tar pipe in `upload_static_assets` (celilo#1014); the no-hand-built-SSH gate now scans `@celilo/capabilities` everywhere except its primitive file (`module-script-scan.ts`, `scanCapabilityPackageSource`). e2e: `modules/caddy/e2e/static-content-converge.test.ts`. ## Remote-ops primitives (the SSH seam — modules never hand-build SSH) Module hooks reach a remote box ONLY through these typed primitives (openspec/changes/unified-management-no-ssh/proposal.md); a raw `ssh` string / `node:child_process` in `modules/**/scripts/` is a defect. Impl: `packages/capabilities/src/remote.ts` (exported from `packages/capabilities/src/index.ts`). - **`remoteExec`** — the ONE ssh seam (`ssh @ `); everything else builds on it. `user`/`port`/`identityFile` on the target default to `root`/22/the agent key, so a fleet call is unchanged; an OFF-FLEET account (`external_web`) sets them. - **`probe`** — read-only health checks over SSH: `systemd` / `command`. It carried a third `http` kind that SSHed in and ran `curl`; the production `ubuntu-22.04-standard` LXC ships none, and a missing binary was indistinguishable from a dead service. Deleted — `probeHttp` replaces it (`openspec/changes/probe-http-from-management/`). - **`probeHttp`** — HTTP health check run FROM the management server over `fetch`; nothing on the target. Takes `port` + `path`, never a URL, and builds the address from the target's `ipv4_address` — so a check cannot name `localhost` and silently probe celilo-mgr's own port. Redirects observed, not followed (caddy's 308 is a healthy answer). Typed failure: `'unreachable'` vs `'status'`. - **`serviceCtl`** — systemctl start/stop/restart/reload/enable/disable. - **`runAppCommand` / `runAppCommandWithSecret`** — escape-hatch on-box command; the secret variant feeds the secret on **stdin** (`$SECRET`), never argv. - **`streamBackup` / `streamRestore` / `fetchFile` / `pushFile`** — binary-safe streaming via local shell redirect/pipe. - **`waitFor`** — predicate-poll combinator. - **`applyRenderedConfig`** — converge: write rendered config → validate → apply → rollback (one round-trip). Used by caddy (Caddyfile), knot (views), iptables (ruleset). On success it RETAINS the file it replaced as `.celilo-prev` — the durable "what celilo last rendered" record a whole-file converge diffs against. - **`installAuthorizedKey`** — ONE-TIME credential bootstrap for an off-fleet account: `ssh-copy-id` under a password taken from the child ENV (`SSH_ASKPASS_REQUIRE=force`, so no `sshpass` dependency and the password never lands in a command string). Idempotent; used by `external_web` onboarding so every later publish is key-based. - **`tailLog` / `grepLog`** — journald reads (regex / ignoreCase). Target = `RemoteTarget` (`{ ipv4_address }` plus optional `user` / `port` / `identityFile`; `DeployedSystem` satisfies it). The runner seam (`execRunner` real / `createMockRunner` for tests) lives in `packages/capabilities/src/testing.ts`. Authoring guide: `MODULE_PRIMITIVES.md` (ships in `@celilo/cli`). ## Firewall converge & port-forward registry - **Port-forward registry (desired state)** — `port_forwards` DB table (`apps/celilo/src/db/schema.ts`, migration `0014`) + the injectable `PortForwardStore` (`apps/celilo/src/services/port-forwards.ts`, `buildPortForwardStore`). Shared-core so any firewall provider reconciles against one store; injected by the capability-loader into the firewall factory. - **Ruleset renderer (pure)** — `modules/iptables/scripts/ruleset-renderer.ts` (`renderRuleset`): registry + firewall state → a complete `iptables-restore` file. Default-DROP FORWARD + established/related + egress + the coarse zone-tier matrix (dmz→app, app→secure from `network..subnet`) + per-service DNAT allows. Egress permission and egress TRANSLATION are emitted together or neither: a blanket `-o ` MASQUERADE on a leaf, one `-s -o ` rule per network on a DOWNSTREAM firewall (no external edge). An upstream does not translate on a downstream firewall's behalf. `subnetsNeedingTranslation(state)` is that set — EVERY network behind the firewall (tiers ∪ every declared zone subnet ∪ control plane ∪ registered trusted sources) minus whichever of them CONTAINS the egress interface's own address (`egressIp`, read off the box). Not just the data-plane tiers: a downstream firewall's egress leg IS its LAN leg, so an untranslated network is unreachable from a LAN host as well as from the internet. - **Converge** — `modules/iptables/scripts/firewall-functions.ts` (`converge`): `exposeService`/`unexposeService` register into the store, then render, CHECK, DIFF, and only then apply atomically via `applyRenderedConfig` (`iptables-restore`). Replaces the old per-rule `iptables -A`; the registry (not `iptables -L`) is the source of truth. It does NOT apply unconditionally: a failing check or an unexplainable removal throws, leaving the working ruleset in place (`config.force` overrides, on the record). - **Render-time completeness checks (pure)** — `modules/iptables/scripts/ruleset-checks.ts` (`checkRenderedSet`, `blockingFindings`, `formatConvergeRefusal`): the SEMANTIC counterpart to `iptables-restore --test`, which only validates syntax. Reports a zone permitted to egress with no translation covering it, a control plane that cannot reach a zone it manages, and an unknown control plane (a warning, distinct from "known and absent"). Runs under `config.dryRun` too. - **Pre-apply ruleset diff (pure)** — `modules/iptables/scripts/ruleset-diff.ts` (`parseRuleset`, `diffRuleset`, `driftAgainstLive`, `snapshotCommand`, `parseSnapshot`): what the converge would remove, and whether a registry change explains it. Exact set algebra against the retained previous render, not a heuristic. Live-vs-persisted drift is reported, never decided on. - **Trusted-source registry (desired state)** — `trusted_sources` DB table (migration `0016`) + `apps/celilo/src/services/trusted-sources.ts` (`buildTrustedSourceStore`, `composeTrustedSubnets`). The sibling primitive to port forwards, for the case a port forward cannot express: an ORIGIN subnet permitted to initiate into every managed zone. Registered through the OPTIONAL `registerTrustedSource` / `withdrawTrustedSource` / `listTrustedSources` trio on `packages/capabilities/src/firewall.ts`; `iptables` implements it, `greenwave` deliberately does not. A consumer detects support with `supportsTrustedSources` and fails loudly via `requireTrustedSources` — a silent no-op is prohibited by the contract, because a discarded registration yields a network that believes it has reach it does not have. - **Composed trusted subnets** — `apps/celilo/src/hooks/capability-loader.ts` (`loadTrustedSubnets`): derived control plane + module registrations + the `firewall.trusted_subnets` operator override, deduped and origin-tagged. With no contributors it equals the derived subnet alone, so the rendered ruleset is byte-identical to the pre-composition output. - **Unowned-trusted-network audit** — `apps/celilo/src/services/audit/trusted-sources.ts` + `apps/celilo/src/services/firewall-reach.ts`: reads each firewall's LIVE table (`iptables-save`) and reports reach granted to a network celilo does not recognise, instead of removing it silently at the next converge. Surfaces under `celilo system audit` as the `trusted_sources` category. ## Hooks & deploy - **Hook executor / ABI** — `apps/celilo/src/hooks/executor.ts` (`invokeHook`, `executeHookScript`, `checkRequiredCapabilities`), types in `apps/celilo/src/hooks/types.ts` (`HookContext`, `HookDefinition`, `HookName`). Named-hook runner: `apps/celilo/src/hooks/run-named-hook.ts`. Manifest hook config: `apps/celilo/src/hooks/load-hook-config.ts`. - **Hook process boundary (a hook is a program celilo RUNS)** — `apps/celilo/src/hooks/hook-protocol.ts` (the NDJSON frame union, `HOOK_PROTOCOL_VERSION`, `serializeError`/`deserializeError`), `apps/celilo/src/hooks/broker.ts` (`startBroker`, `capabilityShape`), `apps/celilo/src/hooks/hook-runner-entry.ts` + `apps/celilo/src/hooks/hook-runner.ts` (the spawned shim — the ONLY thing that `import()`s module code; the entry exists to install the advisory lint before the shim's import graph can ESM-load `node:fs`, see `unjailed-lint.ts`). `executeHookScript` spawns `bun hook-runner-entry.ts` over a Unix socket instead of importing; the nine `invokeHook` call sites and `defineHook` are unchanged. **The broker does not know what a capability is**: it sends a shape descriptor built by the same own-string-key walk `wrapWithLogging` does (functions → `methods`, everything else → `data`, which is where `stampProvider`'s `providerModuleId` lives), and the shim rebuilds forwarding proxies from it — so an optional method a provider did not implement is absent rather than present-and-throwing, and `if (cap.registerTrustedSource)` keeps answering correctly. A socket rather than stdout because module scripts spawn subprocesses and a grandchild writing to fd 1 would corrupt the frame stream. Two consequences worth knowing: the child's environment is an **allow-list** (`hookChildEnv` — how to run: `PATH`, `HOME`, `LANG`, `TZ`, `TMPDIR`; whom to trust: `NODE_EXTRA_CA_CERTS`, `SSL_CERT_FILE`, `SSL_CERT_DIR`; the proxy variables; the `CELILO_HOOK_*` channels; `CELILO_DEBUG` — `FORWARDED_ENV` in executor.ts is the source of truth), so a hook reading any other operator variable now gets `undefined`; and a timeout is a real SIGTERM-then-SIGKILL with the broker refusing further capability calls, replacing a `Promise.race` that cancelled nothing and let a "timed out" hook go on writing DNS and firewall state (celilo#1003). Capability PROVIDER factories still load in-process — they ARE the broker's implementation. Stages 1 and 2 of `openspec/changes/hook-process-boundary`; the filesystem is claimed by the jail below, and SSH reachability by the remote-ops broker next. - **Unjailed advisory lint (task 4.7, NOT a security boundary)** — `apps/celilo/src/hooks/unjailed-lint.ts`. Where there is no jail backend the executor passes the run's derived mount set to the shim in the environment (`CELILO_HOOK_MOUNT_SET`), and the shim wraps the path-taking `node:fs` / `node:fs/promises` functions so an access outside the set (or a write to a read-only row) warns through the hook's own logger: "on a jailed host this would fail", and that it is advisory. It is the mount-set derivation's SECOND consumer, so it cannot drift from what the jail enforces. It observes JS-level `node:fs` calls only — module code bypasses it trivially — and it must never be described as a boundary, in code or output. - **Remote-ops broker (reachability scoped by the credential, stage 3 / D12)** — `apps/celilo/src/hooks/remote-broker.ts` (`startRemoteBroker`, `RemoteAccessPolicy`) answering a SECOND socket beside the capability one, with the policy in `apps/celilo/src/services/remote-access.ts` (`remoteAccessPolicy`) and the asking half inside `@celilo/capabilities`' own remote primitives (`packages/capabilities/src/remote.ts`, "The hook remote-ops bridge"). The jail binds no `~/.ssh`, so a hand-built `ssh` cannot authenticate; the primitives detect `CELILO_HOOK_REMOTE_SOCKET`, send each operation as a STRUCTURED request (never a shell string — the broker rebuilds the ssh line itself), and the broker checks the target against `ownedSystemModuleIds` + `getModuleSystems` before running anything, refusing with the module, the target, and the capability route named. Requests carrying the module's OWN credential (an `identityFile` crossing as content and materialised per call, or `installAuthorizedKey`'s password — the cPanel case) are scoped by that credential instead; an explicit non-root user likewise, because the fleet key's authority is root on fleet systems. Stream primitives' LOCAL paths are confined to the run's granted roots (stateDir, screenshots, generated/, declared path inputs), or `streamBackup` would be a write-as-celilo oracle. Attribution is by the module that PERFORMS the operation: a hook's request to the hook's module (the nine `invokeHook` sites build the policy), a provider's transport to the provider's module — providers run in-process and do not cross this socket, and `public_web`'s hand-built upload was replaced by the Ansible static-content converge (capability-owned-tables stage 4, celilo#1014). Residual, recorded rather than papered over: a hook can still `fetch()` any HTTP endpoint directly; only `probeHttp` consults the target check. - **Hook jail (a hook sees the paths it was given, and nothing else)** — `apps/celilo/src/hooks/mount-set.ts` (`deriveMountSet`, `toBwrapArgs`, `forbiddenPaths` — PURE, computes a filesystem view and touches nothing) and `apps/celilo/src/hooks/jail.ts` (`detectJailBackend`, `planJailedSpawn`, `realpathRequest`, `runtimeModulePathsFor`, `recordJailMode` — the half that touches the machine). `executeHookScript` spawns the shim under `bwrap` with the module's tree read-only, `state/` + `generated/` + this run's `screenshots/` read-write on top of it, each contract-declared path input at its declared access, the broker's socket directory, the runtime and the `node_modules` the shim resolves through, and `/tmp` a fresh tmpfs FIRST so it cannot erase the socket or a staged input. `~/.ssh` is deliberately NOT bound (stage 3, D12): withholding the credential is what makes the remote-ops broker's target check a boundary. **The acceptance criterion is absence, not a check**: the module store and the data directory are simply not bound, so `master.key` and a sibling module give `ENOENT`. The set is DERIVED — a module cannot ask for more — and `bwrap` itself is never in it, because the AppArmor profile grants `userns` to `/usr/bin/bwrap` for anyone on the box (design D9). Backend detection RUNS bubblewrap rather than looking for it (four different denials all leave the binary in place), and the resulting mode is written to `hook-jail-mode.json` beside celilo's other per-machine state rather than logged, so a host that stops jailing is readable. The policy is `auto` / `required` / `off`, resolved in a fixed precedence: the `CELILO_HOOK_JAIL` environment variable, then the stored `hooks.jail_policy` system config key (`celilo system config set hooks.jail_policy `, which asks an interview question before writing `off` unless `--force`), then the default, which is **`off`** (ce-rez7). Jailing begins because an operator set a policy, never because an upgrade installed a backend — celilo-mgr's 2026-09-07 move to 1:2.2.1 installed bubblewrap and loaded the AppArmor profile and hooks kept running unjailed, which is the ruling working. `resolveJailPolicy` returns the source alongside the policy and `system doctor` renders it, so an effective value is always locatable. macOS has a `sandbox-exec` backend in the union, but `auto` defers on it (ce-29z: sandbox-exec DENIES undeclared writes where bubblewrap masks them, and D14's declared-path mechanism is not built), so the hook runs unjailed with the mode recorded; `required` bypasses the deferral for an operator who opts in. An `unjailed` record carries `lastJailed` (the jailed record it replaced on the same host), which is what the `hook_jail` self-monitor reads (`services/alerting/hook-jail.ts`, created unsuppressible by `celilo monitor add hook_jail`) to alert on a host that used to jail and has stopped. `celilo system doctor`'s "Hook execution" section (`renderHookExecutionSection`) reports the live mode and, when unjailed, why. - **Deploy pipeline** — `apps/celilo/src/services/module-deploy.ts`. - **Control-plane bootstrap (celilo initialises its own box)** — `apps/celilo/src/services/control-plane-bootstrap.ts` (`bootstrapControlPlane`) with `apps/celilo/src/services/dns-discovery.ts` (`discoverDns`) beside `network-discovery.ts`. Called from `module-deploy.ts` at the point `on_install` runs, for `CONTROL_PLANE_MODULE_ID` (exported from `services/deployed-systems.ts`, replacing three private copies of the string). Reads the box's upstream resolvers, mints the fleet key via `ensureFleetKey`, writes `dns.*` + `ssh.public_key` in ONE `initializeSystem` call, records the network via `discoverAndRecordNetwork`, then polls `checkDispatcher` and FAILS the deploy if no dispatcher answers. **It is not a hook, and that is the point** (celilo#1225): it was `modules/celilo-mgmt/scripts/on_install.ts`, which reached all of this by spawning the `celilo` CLI — impossible inside the jail, whose mount set binds no `/usr/bin`, no `/usr/local/bin` and no shell, so the deploy ran Ansible clean and then died in its own install hook on every host with a jail backend. celilo-mgmt is never deployed to a remote box, so the host being configured is always the host celilo runs on and the hook was a process boundary with celilo on both sides. Self-registering the management host into its own pool did NOT move: celilo places a module only on a machine already in the pool. The dispatcher read is celilo's four-part `checkDispatcher`, not the "is a process up?" probe the hook used. - **DNS provider backfill** (re-emit registrations when a provider deploys) — `apps/celilo/src/services/dns-provider-backfill.ts` — `isDnsInternalProvider`, `backfillProviderDns`. - **Base-module aspects (fan-out across the fleet)** — `apps/celilo/src/services/aspect-runner.ts` — `planAspectFanOut`, `runAspectFanOut`, `maybeRunAspectForTrigger`. Aspect content lives in `modules//base-module-aspect/` (e.g. knot-unbound-internal, technitium). Two directions, with DELIBERATELY OPPOSITE failure semantics: - **Outbound** (`maybeRunAspectForTrigger`, `on_install`): one provider's aspect across the whole fleet, enumerated at that instant. A failure never fails the provider's own deploy — aspects are forward-progress and a partial fleet converges on the next fan-out. - **Inbound** (`reconcileAspectsForSystems`): every approved aspect applied to systems that have just come into existence. Called from `module-deploy.ts` between `waitForSSH` and `executeAnsible` (so a module's playbook and `on_install` see a correctly configured host) and from `machine-add.ts`. Eligibility is `applicable_zones` + approval; the aspect's `triggers` list is deliberately NOT consulted, so convergence is not opt-in per manifest. A failure here IS fatal to the deploy that created the system — the aspect is a prerequisite of that host — while a failure on `machine add` is only a warning. PAUSED providers are skipped, which is the escape hatch for a wedged aspect. Nothing is rolled back: rows, guest and IPAM allocation persist and a re-run converges on the same system. Fixes celilo#902, where a system provisioned after its provider deployed silently never received the aspect. See `openspec/changes/aspect-fanout-new-systems/`. - **Coverage verification** — `verifyAspectCoverage` + `celilo system doctor --deep [--fix]`. Answers "is any system missing an aspect its zone entitles it to" WITHOUT stored state: the entitled set is `planAspectFanOut` itself, and coverage is measured by evaluating the role against the host in Ansible check mode (`executeAnsible({check:true})` + `parseAnsibleRecap`). Three outcomes, not two — `changed=0,skipped=0` applied, `changed>0` missing, **`skipped>0` unknown**, because check mode SKIPS a task it cannot evaluate and a `command`/`shell` role would otherwise report clean having never run. - **In-flight operation lock** — `apps/celilo/src/services/module-operations.ts` — `startOperation`/`completeOperation`/`failOperation` record deploy/uninstall/backup/restore in `module_operations`; `refuseIfInFlight`/`checkInFlight` are what backup and restore consult. Deploy and uninstall REGISTER but never check: it is a one-way guard protecting backup/restore consistency, not a general mutex (`openspec/specs/management-server-backup/spec.md` "In-flight operation refusal"). A row stops holding the lock once it is GONE, STOPPED/zombie (`isPidRunnable`, `ps -o state=` — `kill(pid,0)` calls a Ctrl-Z'd process alive), or older than `OPERATION_TTL_MS` (2h). The TTL is not redundancy: a pid is a recycled number, and once the pid space wraps an old row names an unrelated healthy process. Operator surface: `celilo module operations [list|clear] [--abandoned] [--all]` (`apps/celilo/src/cli/commands/module-operations.ts`); `list` shows only what holds the lock, abandoned rows are summarised unless `--abandoned`. Abandoned rows are reclaimed hourly by the `celilo-operations-sweep` bus subscriber (`timer.tick.1h` → `celilo module operations clear`, armed by `ensureOperationsSweepSubscriber` from module registration and `celilo system migrate`). `clear` MARKS rows failed rather than deleting them, and that is load-bearing: the `abandoned_operations` audit reads exactly those released rows to notice one module's operation dying over and over. - **Module pause / unpause (control-plane quiescence)** — `apps/celilo/src/services/module-pause.ts` — pure `planPause`/`planUnpause` producing an ordered plan, `executePause`/`executeUnpause` performing it, plus `listPausedModules`/`pausedAmong`/`formatPausedDuration`/`describePausedModule` (the ONE place an age is formatted). CLI: `apps/celilo/src/cli/commands/module-pause.ts` (`celilo module pause|unpause [--cascade] [--stop-infra] [--reason] [--dry-run] [--yes]`). Pausing takes a module out of the CONTROL plane — no dispatched events, no timer hooks, no health checks, alerts suppressed — while leaving the DATA plane running, because capability consumption is deploy-time: every consumer calls `firewall`/`dhcp_server` from `on_install` and nothing calls it while serving. Config, secrets, IPAM/VMID and placement are preserved; `on_uninstall` does NOT run. Quiescence is enforced in two places: pausing drops the module's bus subscriptions (`unregisterModuleSubscriptions`), and `run-named-hook.ts` refuses any non-lifecycle hook for a PAUSED module (`skippedPaused`), which catches the paths that skip the bus — `events resync-subscriptions`, a restore that starts events.db empty, aspect fan-out, public-web republish. `on_install`/`on_uninstall` are exempt by hook NAME (not a caller flag): unpause redeploys through `on_install`, and removing a paused provider needs `on_uninstall`. Unpause always REDEPLOYS (`deployModule`) — that is what rebinds a consumer to a replacement provider and recreates provider-local state from the consumers that own it — and re-registers subscriptions, which a plain deploy does not do. A failed unpause restores `PAUSED` rather than leaving the module live and mis-bound. Cascade order reuses `services/update/dep-graph.ts` unchanged (pause = consumers first, unpause = providers first) and is computed from the GRAPH, never from which modules are currently paused, so a cascade walks THROUGH already-done members and is resumable. See `openspec/specs/module-pause/spec.md`. - **Provider-removal guard** — `apps/celilo/src/services/remove-guard.ts` — `findRemovalBlockers`/`describeRemovalRefusal`, called from `apps/celilo/src/cli/commands/module-remove.ts`. A PAUSED module is not a dependent (unpause cannot return it to service without a redeploy, and a redeploy re-resolves capabilities), which is what makes a provider swap possible at all. A dependent is one declaring the capability under `requires` **or** `optional` — the same relation `dep-graph.ts` uses, so the guard and the cascade agree on the set; the guard previously read `requires` alone, which let a removal silently orphan `technitium`'s `optional` `dhcp_server`. Refusals name each blocker AND which declaration makes it one. It deliberately does NOT exempt a dependent because another provider of the same capability exists (celilo#683). - **Consumer-removal cleanup (every provider is told)** — `apps/celilo/src/services/consumer-cleanup.ts` — pure `planConsumerCleanup` + `loadConsumerCleanupPlan` + `runConsumerCleanup`, called from `apps/celilo/src/cli/commands/module-remove.ts` after `on_uninstall` and before `terraform destroy`. A capability is two-sided: the consumer asks, the provider mints something in ITS world (a caddy site block, a DNAT rule, an OIDC client at authentik, a registered CI runner), and removal only ever touched one side — the FK cascade dropped celilo's row, so the provider's next converge had no way to learn the thing existed. This dispatches the `on_consumer_removed` hook to every provider of every capability the departing module declared under `requires` OR `optional` (the same relation `remove-guard.ts` counts as a dependency edge), **once per provider** rather than once per capability, sorted by provider id. The hook receives one input, `consumer`, and NOTHING else: a provider that cannot answer "what do I hold for this module" without being told has a different defect — the consumer's id was never recorded at mint time. It replaces `services/web-route-cleanup.ts`, which did the same job for exactly one capability, by name, from core. Semantics that are easy to get backwards: **a failed withdrawal never blocks the removal** — the consumer goes and the failing PROVIDER is marked `ERROR` with the departing consumer named in `error_message` (surfaced as a `blocked` finding by `services/audit/undeployed-modules.ts`), because the hook is a full converge and after a failure the provider's state is unknown rather than "one thing missed". **Dispatch continues past a failure**, so one broken provider cannot leave the others holding state. A PAUSED provider is skipped with a warning naming what it keeps (`run-named-hook.ts` refuses non-lifecycle hooks on a paused module, and `on_consumer_removed` must NOT join `LIFECYCLE_HOOKS`), and a never-deployed one is skipped silently. Providers implementing it: caddy, caddy-internal, iptables, greenwave, axon, authentik, forgejo, generic-cpanel-hosting-provider. See `openspec/changes/consumer-removal-cleanup/`. - **Provider-arrival backfill (every consumer is re-run)** — `apps/celilo/src/services/provider-arrival.ts` — pure `planProviderBackfill` + `loadProviderBackfillPlan` + `runProviderBackfill`, called from `module-deploy.ts` at the end of BOTH deploy paths (config-only and full). The mirror of consumer-removal cleanup, and it exists because celilo handled one side of a capability generically and the other by hand: a provider arriving had three hand-written pieces covering two capabilities, and `firewall` — provided by `axon`, `greenwave` AND `iptables` — had none, so its consumers inherited nothing (celilo#1011). When a module deploys, every already-installed module declaring one of its capabilities under `requires` OR `optional` has its `on_install` re-run. **PULL, never push** (design D7): the consumer re-registers through the path that worked the first time, rather than core replaying history into the provider by calling the provider's own hooks on a consumer's behalf — the push shape is a second implementation with its own bug surface, and `backfillWebRouteDns` had already drifted into one. Never fatal to the provider's own deploy: a failed consumer is named with `celilo module deploy ` to retry, and a PAUSED or never-deployed consumer is reported as skipped rather than silently passed over. Shares `PRE_DEPLOY_STATES` with `consumer-cleanup.ts` — one definition, also read by `module-remove.ts`. It replaced `services/public-web-republish.ts` (deleted) and deliberately did NOT replace `services/dns-provider-backfill.ts`, whose docblock records the two reasons: one half replays celilo's HOST inventory rather than a capability set, and the other covers FQDNs published through `public_web` by modules that declare `public_web` and never `dns_internal`. See `openspec/changes/capability-owned-tables/` stage 1. - **Firewall registry ownership** — `apps/celilo/src/services/port-forwards.ts` + `apps/celilo/src/services/trusted-sources.ts`. Both stores are bound to the CONSUMING module and stamp `registered_by` themselves; a caller cannot supply it, so a registration is never attributed to the wrong module. Both writes are **declarative**: `replace()` states a consumer's COMPLETE set for a target, so a port or subnet it previously registered and now omits is withdrawn (celilo#855 — before this, a module that exposed `:8080` and redeployed exposing `:9090` kept both, forever). The owner is IN both unique indexes, not merely beside them: two consumers wanting the same forward are two ROWS, so one leaving cannot delete a rule the other still needs; `modules/iptables/scripts/ruleset-renderer.ts` dedupes on the rule tuple so the pair renders once. Neither column is a FK, so `runConsumerCleanup` deletes these rows explicitly after every provider has converged without them. - **Backup artifact encryption** — `apps/celilo/src/services/backup-cipher.ts` — `encryptFileToFile`/`decryptFileToFile`, file-in/file-out and streamed, used by every backup writer (`backup-create.ts`) and reader (`backup-restore.ts`, `restore-from-file.ts`). Do NOT route artifacts through `secrets/encryption.ts`: that API is string-in/string-out for short DB values, and feeding it a tar cost base64 (1.33x) then hex (2x) then `JSON.stringify` — ~9x the artifact in memory, which OOM-killed forgejo's 774 MB backup, and a hard ~805 MB ceiling from the max string length that no amount of RAM raises. On-disk format is `magic "CELILOBK" (8) | version (1) | iv (16) | ciphertext | GCM tag (16)`; the tag is a trailer because it does not exist until the last byte is encrypted. `decryptFileToFile` still reads the pre-2026-07 JSON-envelope artifacts, discriminating on the magic bytes — the envelope's own `schemaVersion` cannot serve, as it lives inside the encrypted tar. ### Paused modules are conspicuous A pause switches OFF the alerting that would otherwise report the module as down, so the paused-ness itself is the signal. Four surfaces, none optional: `celilo module list` / `celilo status` render `PAUSED (3d)` with the reason; `checkPausedModules` in `apps/celilo/src/services/fleet-checks.ts` makes ANY paused module a `system doctor` FAILURE (no threshold — a pause is a degraded state, and a threshold is just something to tune until the detector stops firing); `findSuppressor` in `apps/celilo/src/services/alerting/suppression.ts` attributes the module's suppressed alerts to the pause rather than dropping them anonymously; and `fleetWarnings()` in `apps/celilo/src/api/serve.ts` stamps a warning naming every paused module and its age onto EVERY management-API result (`ResultMessage.warnings`), including on commands unrelated to those modules. That last one is the important half: it does not depend on the operator choosing to look, which is how a forgotten pause actually gets found. ## Management-host browser runtime - **The browser celilo provisions** — installed by `modules/celilo-mgmt/ansible/roles/celilo-mgmt/tasks/debian.yml` into `/var/lib/celilo/browsers`, behind `install_browser` (a manifest boolean **defaulting to false**, exactly how `install_docker` / `install_terraform` gate third-party-repo tooling: ~170 MB from an external CDN). The pin is `playwright_version` in `modules/celilo-mgmt/manifest.yml` — the fleet's single browser-version declaration. What is UNCONDITIONAL is the *declaration*, so a host that needs a browser and lacks one says so. It provisions `chromium-headless-shell`, **not** full Chromium: both builds on celilo-mgr are headless shells and `chromium.launch()` has been driving one in production the whole time, so the full browser would change what works rather than preserve it. Accepted ceiling: no headed browsing. macOS is an explicit `fail`, not a silent skip. - **The seam** — `packages/capabilities/src/browser.ts` — `resolveBrowser()` returns `{ browser, flavor, executablePath, browserVersion, playwrightVersion, revision }` read from the stable symlink `/var/lib/celilo/browsers/current/chrome` plus the descriptor `current/celilo-browser.json` the install writes. Consumers pass `executablePath` to their launcher, which bypasses Playwright's revision-keyed cache resolution — the revision is a property of the CLIENT version (1.55.1 → 1193, 1.60.0 → 1223), so without it a consumer's client bump silently moves the browser out from under it. **A plain exported function, deliberately NOT a capability and NOT a `HookContext` field**: an event-bus subscriber runs as its own subprocess with no hook context at all, and `lunacycle` — the one consumer — launches from both a hook and a subscriber. There is no `CapabilityRegistry` / `KNOWN_CAPABILITY_NAMES` entry and there should not be one. - **Two failures, two severities** — `BrowserUnavailableError.reason` is `'not_provisioned' | 'unusable'`, and the discriminant is load-bearing. Installation is opt-in, so *absent* is the NORMAL state of a host; a consumer reports it non-fatally. A module throws on any failing check item and celilo maps any fail → unhealthy → the module stays INSTALLED, so a single severity would take every browser-using module permanently unverified on any host that had not flipped a flag it never knew about. *Unusable* — dangling symlink, not executable, unreadable descriptor — is a regression and fails. **Provisioning status is decided from the BINARY, never from a directory existing**: a build directory present with no executable inside it satisfies every path check and then fails at launch. - **`system doctor` rows** — `apps/celilo/src/system/prereqs.ts` — `browser` and `fonts` joined `PREREQUISITES`. `PrerequisiteSpec.command` is the second check kind: an ABSOLUTE path switches presence from `command -v` to exists + executable + reports-a-version. ⚠️ **Never ask Playwright which executable it would use.** `chromium.executablePath()` reports the FULL browser while a headless launch opens the SHELL — measured disagreeing on one machine in one run — so on a shell-only host it validates a path that is not there while the binary that actually runs is fine. Because a not-provisioned browser is deliberately not a check failure, this row is the ONLY place an operator learns the host has none. `fonts` runs `fc-match sans-serif`, which resolves an actual face rather than merely proving fontconfig is installed, and displays the family — that is what decides whether a retained screenshot has legible glyphs. - **Per-run artifact directory** — `moduleArtifactDir(modulePath, runKey)` in the same file is the one definition of the module store's artifact layout, consumed by `apps/celilo/src/hooks/executor.ts` when it creates `HookContext.screenshotDir` and reachable from a bus subscriber that has no context. Per-run because a consumer writing fixed filenames — a reasonable thing to do — would otherwise overwrite itself every run and leave retention nothing to retain. The executor discards the directory when the hook wrote nothing, so an idle module accrues none; note it is created only AFTER every early return, because the capability pre-flight returns outside the `try/finally` that reclaims it and an earlier `mkdir` leaked one empty directory per affected run, forever. - **Artifacts reach the operator** — the executor collects EVERY file written to the run directory (`collectArtifacts`), carried as `HookResult.artifactPaths` (there is no `screenshotPath`; it was deleted, not deprecated). `apps/celilo/src/services/health-runner.ts` carries them onto `HealthCheckResult` — including the hook-FAILED branch, where they matter most because there are no named checks to explain anything — and `failingKeysFromHealthItems` (`services/alerting/keys.ts`) appends them to each failing item's `details`, which is the field that actually reaches an alert. Collecting them and stopping short of `details` would accomplish nothing. `apps/celilo/src/hooks/types.ts` exports `describeArtifacts`, the one renderer the three error-message sites share. - **Retention is by AGE, never by run count** — `apps/celilo/src/hooks/artifact-retention.ts` — 24 h plus a per-module byte ceiling, pruned on write so it needs no scheduler. Count-based pruning is the trap: for a persistent failure the FIRST artifact set carries the original cause and later ones repeat it, so "keep the last 5" discards the useful one within about an hour at a 15-minute cadence. A run is aged by its NEWEST file, not the directory's mtime, so an in-flight run cannot be evicted underneath itself. Artifacts are excluded from module backups (`modules/celilo-mgmt/scripts/on_backup.ts` `EXCLUDE_DIRS`) and preserved across `module update`. - **Pin-drift guardrail** — `apps/celilo/src/services/audit/browser-pin.ts` — the `browser_pin` drift category compares each module's BUNDLED `playwright-core` against the provisioned browser, and it compares **revisions, not version strings**: the revision is what has to match a build, and each client states its own in the `browsers.json` inside the package, so nobody maintains a version→revision table that would rot. It WARNS (`drift`) and never blocks — under D2 the consumer passes an explicit `executablePath`, so a mismatch is protocol skew rather than a failure. Silent when no browser is provisioned (the opt-in default) and when no module bundles a client (almost all of them); a check that fired in either case would put a permanent finding on every host in the fleet. Adding a `DriftCategory` does NOT create a monitor — only 3 of the 21 categories have one — so this is a `system audit` finding, not a page (D4). - Design and the declined alternatives (no probe capability, no fourth `builtin_check`, no concurrency semaphore, no off-box prober — each with its trigger): `openspec/changes/managed-browser-runtime/`. ## Generation & templating - **Generator** — `apps/celilo/src/templates/generator.ts` — `generateTemplates` (orchestration), plus Terraform/Ansible file handling. - **Variable resolution** — `apps/celilo/src/variables/resolver.ts` (parser: `apps/celilo/src/variables/parser.ts`). Supported prefixes: `$self`, `$system`, `$secret`, `$system_secret`, `$capability`, `$infra`. ## Secrets - **Vault / encryption** — `apps/celilo/src/secrets/vault.ts` (`deriveVaultPassword`, `getVaultPassword` — Ansible Vault) and `apps/celilo/src/secrets/encryption.ts` (`encryptSecret`/`decryptSecret`, AES-256-GCM). ## Packaging, registry & publish - **Module packaging** — `apps/celilo/src/module/packaging/` — `build.ts` (`buildModule`), `extract.ts`, `checksum.ts`, `signature.ts` (`signChecksums`/`verifySignature`), `release-metadata.ts`, `audit.ts`, `package-rules.ts`, `generated-plane.ts`, `host-plane.ts` (see **Module integrity** below). - **Publish driver** — `scripts/publish.ts` shims to `apps/celilo/src/cli/commands/publish/` (workspace npm packages via `bun publish` + module registry; preflight stale-version/stale-manifest gates). - **Registry publish-token admin (append-safe)** — `apps/celilo/src/cli/commands/registry-token.ts` — `celilo registry token add/rm ` read-modify-write the celilo-registry `publish_tokens` secret (newline-separated bootstrap/admin list). Avoids the `module secret set` full-overwrite that clobbered other holders. Runs on-mgr where the master key lives; runtime-minted scoped tokens are handled separately by the registry-server. - **Contributor identity tokens (idp-issued, per-user)** — `apps/celilo/src/cli/commands/token.ts` — `celilo token obtain|list|revoke` mints/lists/revokes per-user API tokens via the `idp` capability (`create_token`/`list_tokens`/`revoke_token` on authentik). A module contributor authenticates publishes AS THEMSELVES (SECURE_MODULE_PUBLISH.md §6) — no admin/shared token on their machine; the token feeds `celilo author init`. Shown once at mint; the idp stores it hashed, celilo persists nothing. Runs on-mgr where the idp provider lives. Distinct from `registry token add/rm` (raw bootstrap list, different trust model). - **Registry token verification (opaque + idp introspection)** — `packages/registry-server/src/auth.ts` (`TokenAuth` — opaque SHA-256 publish tokens, admin/per-package scope) + `packages/registry-server/src/introspection.ts` (`IntrospectionVerifier` — RFC 7662 verify-bridge, SECURE_MODULE_PUBLISH.md §5[D-A]). `authorizePackage()` in `server.ts` tries the opaque set first (unchanged), then, for a token unknown to it, `identify()`s it via the idp introspection endpoint using the registry's confidential OIDC client creds (`OIDC_INTROSPECTION_ENDPOINT`/`OIDC_CLIENT_ID`/`OIDC_CLIENT_SECRET`, provisioned on install — ce-7aa), reading `{active, sub, groups, exp}`. Fails CLOSED on any introspection error; never logs tokens/secrets. Instant revocation: revoke at the idp → next publish sees `active:false` → 401. - **Registry module-owner table (hybrid group + owner authz — ce-1ch, D-C)** — `packages/registry-server/src/module-owner-store.ts` (`ModuleOwnerStore` — JSON-persisted `{moduleName, ownerSub, claimedAt, sourceGroup}`, `REGISTRY_OWNERS_FILE`/`dataDir/module-owners.json`). The verified `groups` claim gates *whether* an identity may publish (`REGISTRY_ADMIN_GROUP`→publish/reassign anything; `REGISTRY_PUBLISHER_GROUP`, default `celilo-authors`, configurable→claim+publish owned); the owner table gates *which names*. First-publish-claims: the first verified publisher of an unclaimed name owns it; a *different* publisher is then DENIED (confused-deputy defense — Author-A cannot publish Author-B's module). Admin HTTP endpoints `GET /api/v1/modules/owners`, `GET|POST /api/v1/modules/owners/{name}` (reassign). Operator front door: `celilo registry owner list|show|set` (`apps/celilo/src/cli/commands/registry-owner.ts`), admin token resolved from the local `publish_tokens` bootstrap list. ## Module integrity (the four places a module exists) A module exists in four places at once, and celilo verifies the correspondence between them. Design: `openspec/changes/module-integrity-rigor/design.md`. - **The one classifier** — `apps/celilo/src/module/packaging/package-rules.ts` — `classifyModulePath(relPath)` returns `package` / `derived` / `unknown` and is the SINGLE answer to what belongs to a module. `build.ts#shouldExclude`, `extract.ts#scanDirectory`, `import.ts#copyModuleFiles`, `module-update.ts` and `audit.ts` all route through it. Composes with, and does not restate, `includeNodeModulesPath` (ISS-0046). - **The versioned baseline** — `moduleIntegrity` in `apps/celilo/src/db/schema.ts` — checksums PLUS the `version` they describe. Upserted by `module import` and written by `module-update.ts#updateOne`, so it tracks the installed version instead of freezing at first import. `version` is nullable: NULL means "written before celilo stamped versions", which verify reports rather than papers over. - **Plane 1, installed tree vs baseline** — `apps/celilo/src/module/packaging/audit.ts` — `auditModule(moduleId, db, { deep })`. The entry point for `module verify`. - **Plane 2, generated project vs installed tree** — `apps/celilo/src/module/packaging/generated-plane.ts` — `compareVerbatimRoleAssets` (pure), `readVerbatimRoleAssets`, `refuseIfGeneratedIsStale`. Only `ansible/roles//files/**` has a meaningful expected digest; Ansible templates the rest. Wired into `module verify` and into `module-deploy.ts` as a pre-flight that refuses BEFORE contacting any system (celilo#925). - **Plane 3, host vs generated project** — `apps/celilo/src/module/packaging/host-plane.ts` — `verifyModuleOnHosts` runs the generated playbook through `executeAnsible(..., { check: true })` and classifies each `PLAY RECAP` line as `converged` / `drift` / `unmeasured`. One SSH per system, so `--deep` only. A module opts out with `verify: { deep: false, reason: … }`; the reason is required and is printed. - **Surfaces** — `celilo module verify [--deep] [--json]` (`apps/celilo/src/cli/commands/module-verify.ts`), the `module_integrity` audit category (`apps/celilo/src/services/audit/module-integrity.ts`), and `system doctor`'s fleet section. Same implementation behind all three. - **Dead convergence machinery** — `apps/celilo/src/services/audit/detect-without-converge.ts` — the `detect_without_converge` category reports a module declaring a `reconcile_*` / `refresh_registrations` / `reassert_dhcp_dns` hook that no subscription ever fires (celilo#934). ## Alerting & notifications Observation and delivery: monitors run checks on a schedule, alerts hold what is currently wrong, and routes carry the message to a person's phone. Design: `openspec/changes/add-alerting/design.md`. - **Alert identity** — `apps/celilo/src/services/alerting/keys.ts` — the key grammar (`module:[/check:]`, `builtin:[/:]`) that makes "the same problem" the same alert across runs. `moduleAlertKey`, `moduleCheckAlertKey`, `builtinAlertKey`, `parseAlertKey`. - **Reconciliation** — `apps/celilo/src/services/alerting/reconcile.ts` (`reconcile`) — a successful run's failing-key set is authoritative and resolution is by SET DIFFERENCE (absent ⇒ resolved). A run whose outcome is `error` resolves NOTHING and fires a module-level alert instead: the false-all-clear guard. - **Monitor execution** — `apps/celilo/src/services/alerting/run-monitor.ts` (`runOneMonitor`) + `sweep.ts` (`selectDueMonitors`) + `builtin-monitors.ts` / `health-coverage.ts` / `hook-jail.ts` (the built-in checks, the "module with no health check" coverage check, and the hook-jail regression self-monitor). - **Health-check cadence (one accessor)** — `apps/celilo/src/services/alerting/health-cadence.ts` — `effectiveHealthCheckCadence(manifest, override)`, `isScheduled`, `loadModuleHealthCadences`, `reconcileModuleWatchState`. The manifest's `hooks.health_check.interval` SUGGESTS; the operator's `health_check_interval` decides; both resolve at read time. `null` means nobody named a cadence (a coverage gap); `'manual'` means the operator opted out (a decision — raises no coverage finding, and resolves the module's live alerts on the way down, since nothing will report on them again). - **⚠️ `monitors.intervalMinutes` and `monitors.enabled` are `builtin_check`-only.** A `module_hook` row carries severity, escalation policy and `lastRunAt`; its cadence and whether it is watched resolve through the accessor above. The columns' meaning depending on `kind` is a named smell (design.md D8) — the alternatives are a cached resolved value that rots, or splitting the table, which needs a synthetic monitor identity for `alerts.monitorId`. Gates: `sweep-runner.test.ts` asserts a module row's stored values are NOT consulted; `cadence-migration.test.ts` asserts the same through `loadModuleHealthCadences`. - **Carrying an existing fleet over** — `apps/celilo/src/services/alerting/cadence-migration.ts` (`migrateMonitorCadences`), run from `celilo system migrate` (the `.deb` postinst runs it on every apt upgrade). A monitor row whose cadence diverges from its manifest gets that cadence written as an override; a disabled one gets `manual`. Not bookkeeping: without it the upgrade that ships read-time resolution silently reverts every hand-set cadence to the author's suggestion and resumes watching modules an operator deliberately disabled. Idempotent — writes only where no override exists. - **Scheduled audit categories (`builtin_check` monitors)** — `apps/celilo/src/services/alerting/builtin-source.ts` — `SCHEDULABLE_BUILTIN_CHECKS` is the list of `celilo system audit` categories cheap enough to run every sweep: `machines_reachable`, `backups`, `disk_space`, `abandoned_operations`, and `public_dns` (`apps/celilo/src/services/audit/abandoned-operations.ts` — ≥3 abandonments of the same (module, operation) in 7d, the fingerprint of an operation being killed mid-flight). Everything else in the audit needs the whole world injected (proxmox, terraform, registry) and is not schedulable. Enable one with `celilo monitor add backups --interval 1h`, and re-cadence it later with `celilo monitor set-interval backups 6h` (in place, because the monitor id owns the alert history). `monitor set-interval`/`enable`/`disable` REFUSE a module target and name `celilo module config set health_check_interval` — two ways to set one module's cadence would disagree about what `module status` shows. Targets are tab-completable — `completion.ts` reads `SCHEDULABLE_BUILTIN_CHECKS` directly rather than a hand-copied list, so a newly-schedulable check is completable immediately. - **Disk-space check** — `apps/celilo/src/services/audit/disk-space.ts` (`auditDiskSpace`, pure over measurements) + `apps/celilo/src/services/disk-probe.ts` (`probeDiskUsage`, target set from `diskProbeTargets`). Thresholds: `drift` at 85%, `blocked` at 95% — early enough to act on, since a check that fires at exhaustion reports an outage rather than preventing one. ⚠️ **The local management box is MEASURED, not exempted.** `probeMachines()` deliberately reports the local box reachable without probing it (celilo has no SSH key for itself, and the question is meaningless there); copying that shortcut into a disk check would skip the host most likely to fill — the one that stages backups, caches modules and writes the logs, and the one that DID fill. ⚠️ **It probes SYSTEMS, not just MACHINES.** `machine` is celilo's narrow term for an operator-pre-provisioned box in the machine pool; it excludes every LXC/VM celilo provisioned through a container_service — which is most of the fleet. The probe walked `listMachines()` alone until celilo#1133, so the registry, the forge and the firewall were never measured — and **two of them filled on the same night, wearing different disguises**: celilo-registry (vmid 204) hit 100% with 4 KB free and publishes began returning `Internal Server Error` while every celilo surface stayed green (reads work on a full disk; only writes fail), and git.celilo.computer (vmid 206) hit 100% and its runner logged `database or disk is full` every 2s, claiming no work for ~13 hours — presenting as a DEAD CI RUNNER when the runner was healthy and polling. Scheduling the monitor would not have caught either: the check would have reported all-clear about filesystems it never looked at. `diskProbeTargets` now unions the machine pool with `getProvisionedSystems`, deduplicated by address (co-hosted modules share one filesystem and so one finding), containers probed as `root`. Local reads `statfs`; remote runs `df -P /` over the same bounded SSH. `percentUsed` matches `df`'s capacity semantics (excludes root-reserved blocks) so an alert and an operator's own `df` agree. An unmeasurable host yields a `todo` finding — recorded, never paged, because `machines_reachable` is already paging for that host. Findings are subjected on the **hostname**, not the machine UUID, because suppression resolves a machine's ancestor key from the hostname (see #596, where `machines_reachable` gets this wrong and its alerts therefore never suppress anything). The `backups` roster comes from `apps/celilo/src/services/audit/backup-source.ts` (`loadBackupAuditInfo`), shared with `celilo system audit` so both judge the same fleet. - **Instance sizing (`celilo proxmox vm|ct list|resize`)** — `apps/celilo/src/cli/commands/proxmox-instance-resize.ts` + `proxmox-resize-guards.ts` (`validateResize`/`computeFloor`, pure and unit-tested without Proxmox) + `proxmox-instance-list.ts` (DESIRED vs ACTUAL c/m/d, drift). A celilo-provisioned instance's size is **canonical infrastructure state in `module_systems`**, never module config; `requires.system` is only the floor used to select a host. Sizing flows `module_systems` → `$self:{cores,memory,disk}` (seeded from `requires.system` on first provision, `apps/celilo/src/variables/context.ts`) → the instance Terraform. ⚠️ **Two apply paths.** cpu/memory reconcile declaratively — record the size, redeploy the owning module, and the provider stop/starts the guest, so that path costs a reboot and is gated on the operator approving one. Disk growth goes DIRECT to Proxmox's resize API (`ProxmoxClient.resizeGuestDisk`, `pct resize`/`qm resize`): online, additive, no power change and no redeploy. That is not a shortcut — a Terraform reconcile means a full Ansible pass, and the box being grown is the one that has run out of room to run one (celilo#1133). Guards: floor and disk-shrink are hard, node capacity yields to `--force`. lxc grows its filesystem too; qemu grows only the block device and the guest must extend its own partition, which the command says out loud. - **Public-DNS reachability check (the only check with an OFF-FLEET vantage)** — `apps/celilo/src/services/audit/public-dns.ts` (`auditPublicDns`, pure over an injected probe and the previous run's counters) + `apps/celilo/src/services/public-dns-probe.ts` (the probe) + `audit/public-dns-source.ts` (ledger names + the `public_dns_evidence` counters). Every other check in celilo looks from INSIDE, behind a split-horizon resolver that deliberately answers with an in-zone address — correct for its purpose, and why all of them reported healthy for the nine days of celilo#626. This one resolves every `dns_registrations` FQDN through an **off-fleet resolver** (`public_dns.resolver`, default `1.1.1.1`) and compares it against the address the fleet appears to come from per an independent **echo service** (`public_dns.echo_url`, default `https://api.ipify.org`). Three properties are load-bearing: `assertOffFleetResolver` REFUSES a resolver matching `dns.primary`/`dns.fallback` (a check that quietly used the fleet's resolver would pass forever — the original bug one layer up); the expectation never comes from the registrar's own response (self-agreement, and Namecheap returns `ErrCount 0` for `www` updates it does not apply); and a divergence is a finding only once it OUTLIVES the record's own TTL, measured from the last assert, or it would page on every ISP re-lease. Missing evidence is counted rather than read as success — one undetermined run is silent, N consecutive ones are their own finding (`public_dns_evidence`), which is the hole celilo-website's isitup.org probe demonstrated live. Codes: `public_dns_stale`, `public_dns_missing`, `public_dns_companion_unclaimed`, `public_dns_unverifiable`. Spec: `openspec/specs/public-dns-reachability/spec.md`. - **The sweep** — `apps/celilo/src/services/alerting/sweep-runner.ts` (`runSweep`) — the ordered pass that makes alerting run by itself: run due monitors → promote past-grace alerts → re-evaluate suppression → flush quiet-hours deferrals → notify. Driven by `celilo alerts sweep` on `timer.tick.5m`. Never throws for one bad monitor. - **Suppression (topology-derived, never configured)** — `apps/celilo/src/services/alerting/suppression.ts` — `ancestorKeysFor`/`findSuppressor`/`machineAlertKey`. A firing machine explains its modules' failures; a firing capability provider explains its zone's consumers. Derived from `module_systems`, so it cannot drift from reality. Deploy windows: `deploy-hooks.ts` (`openDeployWindow`/`closeDeployWindows` — closed by module, so a crashed deploy self-heals). - **Escalation & quiet hours** — `escalation.ts` (`decideEscalation`, every reason to stay silent enumerated) + `quiet-hours.ts` (`isWithinQuietHours`, Intl-based and DST-safe). Quiet hours defer the MESSAGE while the escalation clock keeps running. - **Delivery** — `notifier.ts` (`notifyAlert`, `deliverDeferred`, `notifyResolved`, body composition) + `transport-loader.ts` (`loadNotificationTransport` — resolves a route's transport module's `notification` capability). - **Inbound replies** — `tokens.ts` (per-DELIVERY reply tokens, so a reply identifies WHO) + `inbound.ts` (`interpretInbound` — two-factor: valid unexpired token AND sender matching the issuing route) + `inbound-poller.ts` (`pollInbound`, `makeReceiver`). celilo-mgr polls the transport; nothing calls in, so the ack path never depends on the public HTTPS it may be paging about. - **Ack / silence / resolve** — `ack.ts` — three deliberately distinct operations. An ack is broadcast to every other paged route. - **Interviews over a transport** — `interview-responder.ts` (delivery policy; `secret.*` families are REFUSED) + `notification-responder.ts` (queries the bus's events table for unanswered questions: a watch only sees events emitted after it registers, and `alerts poll` is one-shot. Leaves a question alone for 90s so an attached responder — terminal, or the `api-serve` wire bridge — answers first). Alerts and interviews share the send path, the token table, and the inbound path; only `targetId` differs. - **People, routes, policies** — `people.ts` — `createPerson`/`createRoute`/`createPolicy`/`addPolicyStep`. - **Policy resolution** — `apps/celilo/src/services/alerting/notify-deps.ts` — `policyForAlert` (the single answer to "who would this page") + `buildNotifyDeps` (assembles steps, routes, quiet hours, transport, reply token). The policy is read from the MONITOR on every sweep, never snapshotted onto the alert, so `escalation-policy assign` takes effect on alerts that are already firing. - **Persistence** — `apps/celilo/src/services/alerting/store.ts`; tables `people`, `routes`, `escalation_policies`, `escalation_steps`, `monitors`, `monitor_runs`, `suppression_windows`, `alerts`, `notification_deliveries` (`apps/celilo/src/db/schema.ts`, migrations `drizzle/0017_alerting.sql`, `drizzle/0018_drop_alert_policy_snapshot.sql`). - **CLI** — `apps/celilo/src/cli/commands/` — `alerts-list.ts`, `alerts-act.ts` (ack/silence/resolve), `alerts-sweep.ts`, `alerts-poll.ts`, `monitor.ts`, `notify-config.ts` (`person`/`route`/`escalation-policy`). ## Backups Creation, scheduling and freshness. A module declares an `on_backup` hook and SUGGESTS a `backup.schedule`; the operator's override decides; celilo runs it on the resolved cadence and alerts when it stops. - **Creation** — `apps/celilo/src/services/backup-create.ts` — `createModuleBackup` (invokes the module's `on_backup` hook into an encrypted envelope), `createSystemStateBackup` (celilo.db), `findBackupEligibleModules` (returns each module's operator config alongside its manifest, because every caller has to resolve a policy out of the two together), `isBackupDue`. Storage destinations: `backup-storage.ts`. Restore: `backup-restore.ts`. - **Staging celilo's own state** — `apps/celilo/src/services/system-state-stage.ts` — `stageSystemState(rootDir)` and `snapshotDatabase(src, dest)`. Before invoking `on_backup` on a `cross_module_read` module, celilo COPIES its own state (`celilo.db` snapshot, `master.key`, the fleet `ssh/`, and `module_src//` for every module's lean source) into a directory it creates, and passes the path as the contract input `system_state_root`. ⚠️ **This is what lets celilo back ITSELF up without exempting celilo-mgmt from the hook jail** (`openspec/changes/hook-process-boundary`, design D9b): the hook never READ those bytes, it copied them into `backup_dir`, so the framework does the copying and celilo's data directory is in no hook's mount set. Same allow-list as `cross_module_root` — one privilege, one list to audit. ⚠️ **`snapshotDatabase` uses a readonly connection + `serialize()`, never `copyFileSync`**: celilo runs the DB in WAL mode, so the main file is routinely one near-empty page while all the real data sits in `celilo.db-wal` (measured: 4 KB main against 832 KB WAL), and a plain copy produces a snapshot that opens cleanly, contains NOTHING, and is installed by restore. Gate: `services/system-state-stage.test.ts` writes 200 uncheckpointed rows and asserts they survive. - **The fleet SSH key (one accessor)** — `apps/celilo/src/services/fleet-key.ts` — `ensureFleetKey()` (idempotent mint, returns the public half) and `getFleetSshDir()`. Surfaced as `celilo system ensure-fleet-key`, which also records `ssh.public_key`. celilo-mgmt's `on_install` used to mint the keypair itself inside celilo's data directory — a WRITE into the one directory the jail exists to keep out of the mount set (design D9b), which staging does not cover because staging covers copies OUT. ⚠️ **Never re-key**: an existing key is reused, because regenerating strands every machine whose `authorized_keys` holds the old public half, and a redeploy calls this every time. `getFleetSshDir()` follows the DB (`dirname(getDbPath())/.ssh`), not `getDataDir()` — the same directory on a deb install and different when `CELILO_DB_PATH` is overridden; both the mint and `restore-from-file.ts`'s laydown read that one helper so they cannot drift apart. - **Cadence (one accessor)** — `apps/celilo/src/services/backup-schedule.ts` — `effectiveBackupSchedule(manifest, override)`. The manifest SUGGESTS; the operator's `backup_schedule` row in `module_configs` decides; absent from both means `daily`, NOT `manual` (opting out takes an explicit `manual`). Resolution happens at READ time — nothing is materialised at install or deploy — so a corrected manifest reaches every install that has not overridden. The one-argument form was DELETED rather than kept as an overload (Rule 3.9): it would let an un-updated reader compile clean while silently ignoring overrides. Both the freshness audit and the backup sweep must read cadence through this one function, or a module can be alerted-on but never backed up. - **The cadence type** — `apps/celilo/src/services/cadence.ts` — `Cadence` (`{minutes}` | `'manual'`), `parseCadence` / `formatCadence` / `cadenceMs`, and `cadenceSchema({floorMinutes})`. One spelling set for every cadence in celilo: a named period (`hourly`/`daily`/`weekly`/`monthly`), a duration (`6h`, `90m`, `3d`), or `manual`. ⚠️ **The floors are DERIVED from the sweep ticks, never written down**: this file owns `BACKUP_SWEEP_PATTERN` / `ALERTING_SWEEP_PATTERN` and computes `BACKUP_CADENCE_FLOOR_MINUTES` / `MONITOR_INTERVAL_FLOOR_MINUTES` from them (the sweeps import their pattern from here), so changing a tick moves what it can serve in the same edit. A cadence finer than its sweep's tick is REFUSED, not coerced — accepting it leaves the operator believing they configured something that can silently never happen. - **Retention (one accessor)** — `apps/celilo/src/services/backup-retention.ts` — `effectiveBackupRetention(manifest, configs)`, `prunesNothing`, `identifyExpiredBackups`, `pruneBackupsForModule`. Two INDEPENDENT dimensions (copies, age), each resolved override → manifest → **unbounded**. ⚠️ **An unset dimension is unbounded, never a default bound.** `backup.retention` is an optional block, so a manifest omitting it prunes nothing at all and its inner `count: 7` / `max_age_days: 30` defaults never apply; if setting one dimension let the other fall back to those, an operator asking to keep 3 copies would silently arm a 30-day deletion on a module that had been keeping everything. Unbounded is `Infinity`, which `identifyExpiredBackups` needs no special case for. Gate: `services/backup-retention.test.ts`. The four sites that used to read `manifest.backup.retention` directly (`cli/commands/backup-sweep.ts`, `backup-create.ts`, `backup-prune.ts` twice) all go through the accessor — that duplication is what let the schedule readers drift. - **The sweep** — `apps/celilo/src/services/backup-sweep.ts` — `runBackupSweep` (the pass that makes backups run by themselves: reclaim orphaned staging → for each eligible module, is its declared cadence due → back it up → apply declared retention) + `ensureBackupSweepSubscriber`. Driven by `celilo backup sweep` on `timer.tick.1h` — the coarsest tick that can still serve an `hourly` cadence. Armed from BOTH `registerModuleSubscriptions` (any module declaring an `on_backup` hook, so it appears on install or `module update`) AND `celilo system migrate`, which the `.deb` postinst runs on every apt upgrade — registering only from module install/update meant a corrected budget never reached an existing fleet, since the row already exists and module updates can be weeks apart. A run refused by the in-flight operation lock is a skip retried next tick, never a failure. - **⚠️ The sweep's budget is stated, never inherited** — `BACKUP_SWEEP_TIMEOUT_MS` (4h) and `BACKUP_SWEEP_MAX_ATTEMPTS` (1) in `backup-sweep.ts`. The event bus defaults to `timeout_ms: 60000` / `max_attempts: 3`, and inheriting them made scheduled backups structurally impossible: one forgejo backup measured ~5.5 minutes (1.3 GB result, 3.9 GB peak staging), so the dispatcher SIGTERMed it at 60s — three times an hour, for days. The retry count is half the bug, not a detail: an impossible pass retried 3x strands 3x the staging (27 GB in 5.7 hours on celilo-mgr). The hourly tick IS the retry. - **Staging reclamation** — `apps/celilo/src/services/backup-staging.ts` — `reapOrphanedStaging`, `stagingDirFor` (the single source of truth for `/tmp/celilo-backup-`, shared with `backup-create.ts` so writer and reaper cannot drift). `backup-create.ts` removes its staging in a `finally`, which is correct and NOT enough: a `finally` never runs when the process is killed by a signal — dispatcher timeout, OOM, Ctrl-C, reboot — and those strand the LARGEST directories. So reclamation is kill-mode agnostic by construction: it asks "is anyone still using this?", answered from the `backups.pid` column plus `isPidRunnable`, both of which outlive the process. A directory is removed only when its owner is provably gone (record absent, record terminal, pid dead, or past `STAGING_TTL_MS` = 6h — the TTL is the only check surviving pid reuse). A live backup is always kept; an unrecognised name is ignored, never deleted. Reclaiming a record that still claimed `in_progress` also marks it failed with `ABANDONED_BACKUP_MESSAGE`, so `celilo backup list` stops showing phantom in-flight backups and the `backups` drift check cannot read a dead attempt as a fresh backup. - **Freshness audit** — `apps/celilo/src/services/audit/backups.ts` (`auditBackups`, `backupStaleThresholdMs`) — `backup_missing` / `backup_stale` drift findings against the same EFFECTIVE cadence, loaded (with each module's override) by `audit/backup-source.ts`. The stale threshold is a formula, `cadence + max(1h, cadence × 0.1)`, not a lookup table: a table cannot answer for `6h`. An effective cadence of `manual` short-circuits BEFORE the never-backed-up check — a module the operator opted out of used to be reported as missing a backup forever, remediated only by the thing they declined. - **⚠️ Both age measurements read `backups.completedAt`** — `loadBackupHistory` (`backup-metadata.ts`) and `latestSuccessfulBackupByModule` (`audit/backup-source.ts`). The run path used to measure from `startedAt` and the audit from `completedAt`, so the two disagreed by the duration of the backup itself. Gate: `services/backup-age-agreement.test.ts`. - **CLI** — `apps/celilo/src/cli/commands/` — `backup-sweep.ts`, `backup-create.ts` (also `celilo module backup`), `backup-list.ts`, `backup-restore.ts`, `backup-prune.ts`, `backup-delete.ts`, `backup-import.ts`, `backup-pull.ts`, `backup-name.ts`. - **Storage destinations CLI** — `storage-add-local.ts`, `storage-add-s3.ts`, `storage-list.ts`, `storage-verify.ts`, `storage-set-default.ts`, `storage-set-path.ts` (relocate a local destination, migrating existing archives unless `--no-migrate`), `storage-remove.ts`. Any credential change goes through `updateStorageCredentials` in `backup-storage.ts`, which clears the verification stamp — a `✓ Verified` must never describe a destination it was not measured against (#566). ## Per-module operator policy How celilo TREATS a module — its backup cadence, its health-check cadence, its upgrade controls — as opposed to how the module configures itself. - **The keys** — `FRAMEWORK_CONFIG_KEYS` in `apps/celilo/src/cli/commands/module-config.ts` — keys EVERY module accepts whether or not its manifest declares them: `auto_upgrade`, `upgrade_policy`, `backup_schedule`, `health_check_interval`, `backup_retention_count`, `backup_retention_max_age_days`. Each carries `{schema, why}`: a Zod schema, and the reason a wrong value is REFUSED rather than coerced. That reason is per key on purpose — the substance is what a wrong value would silently do (a cadence typo falls back to the manifest's suggestion; an `upgrade_policy` typo falls back to `by-semver`, which skips the pre-deploy backup on a patch), and no validator knows that. - **Storage** — rows in `module_configs`, unique on `(module_id, key)`, cascading on module removal. Nothing deletes them on `module update`, and `backup-create.ts` archives them into backup envelopes while `backup-restore.ts` restores them — so an override survives update, upgrade and restore with no new table and no migration. - **Operator surface** — `celilo module config set|get|unset [value]`. `unset` (→ `deleteModuleConfig` in `services/module-config.ts`) is what returns a module to following its manifest; without it an operator who once set a value could never go back, and later manifest corrections would stop reaching them permanently. Unsetting an absent key reports that and SUCCEEDS — it states a desired end state. All three are MCP tools with no bespoke code, generated from `packages/core/src/command-registry.ts`. - **Display** — `formatCadencePolicy` in `apps/celilo/src/cli/commands/module-status.ts` shows the effective value AND its source, naming the manifest's suggestion when an override is in effect. The stored override alone does not say what it was changed from; the effective value alone does not say who chose it. ## Persistence - **DB schema** — `apps/celilo/src/db/schema.ts`. Client: `apps/celilo/src/db/client.ts`. Migration runner: `apps/celilo/src/db/migrate.ts`. Migrations: `apps/celilo/drizzle/`. - **Frozen-watermark repair** — `runMigrationsOn` (`db/migrate.ts`) is what `createDbClient` calls on open, not drizzle's migrator directly. drizzle is watermark-only, so a DB from the imperative hand-list era — schema applied, `__drizzle_migrations` never told — makes it re-run migrations and die on `duplicate column name`. That throw is inside the OPEN, so every celilo command on that box fails, `celilo system migrate` included: it reaches its own repair through `getDb()`. The repair engages only after the stock migrator has failed AND only when `findSchemaDrift` reports the declared schema entirely present, and it corrects the LEDGER without running SQL — replaying is not an option, since `drizzle/0021_dns_registration_consumers.sql` rebuilds `dns_registrations` by dropping the original and renaming a copy over it. A PARTIALLY applied schema (celilo-mgr's own pre-remediation state) still fails, naming the hand remediation. Gate: `db/migrate.test.ts`. Both entrypoints run `celilo system migrate` — the `.deb` postinst and the `celilo-mgmt` Ansible role, before the dispatcher starts (celilo#169). ## Events - **Migration interrogation** — `apps/celilo/src/db/migration-status.ts` — `getMigrationStatus(sqlite, migrationsFolder)` reports applied count, latest applied migration BY TAG, and pending ones by name (joining `__drizzle_migrations.created_at` to drizzle journal `when`, which is exact). Surface: `celilo system migrate --status`, which opens the DB **read-only** on purpose — `getDb()` auto-migrates on open, so a status routed through it would repair what it claims to report and could never say "pending". Paired with `findSchemaDrift` (`db/schema-introspection.ts`), which is column-aware: a table COUNT cannot distinguish "the column migration applied" from "nothing happened", which is why a rollout asserting `backups.pid` had to reach for `sqlite3` over SSH. `checkSchemaDrift` (`services/fleet-checks.ts`) fails on a missing table, a missing column, OR an unapplied journal migration, and its summary names tables AND columns so the operator can see what was checked. - **Event bus** — `packages/event-bus/src/index.ts` — `Bus`, `openBus`, `defineEvents`, `defineHandler`, `runDispatcher`, pattern matching + timer ticks (`emitDueTimerTicks`, `retentionSweep`). **Exactly one dispatcher per bus**: `runDispatcher` refuses to start while another dispatcher's process is alive (`assertSoleDispatcher` in `dispatcher.ts`, liveness via `bus.liveDispatchers()` — `kill(pid,0)`, not heartbeat age, since a tick blocks for as long as its slowest handler). The exclusion is here rather than in the systemd unit because a stranded dispatcher can sit outside the unit's cgroup where `KillMode` cannot reach it (#580). `bus.health()` reports `dispatcherCount`/`dispatchers` and a `duplicate_dispatcher` status; `checkDispatcher` (`services/fleet-checks.ts`) fails on more than one. **Supervision** (`services/events-daemon.ts`): the unit is named `celilo-events.service` in BOTH the user and system scope, so the two are indistinguishable in every operator-facing string — `install-daemon` (which defaults to **user** scope) therefore refuses when the other scope's unit exists, and `checkDispatcher` compares the live pid against each installed unit's `MainPID` (`unitMainPid`) rather than merely testing that a unit file exists, since a file nobody is running is not supervision (#610). - **Dispatcher supervision (install / restart)** — `apps/celilo/src/services/events-daemon.ts` — `installDaemon`/`planDaemonInstall`/`uninstallDaemon`/`readInstalledUnit` write the systemd unit or launchd plist without touching supervisor state, and `restartDaemon` (+ pure `orphanDispatcherPids`, `resolveRestartScope`, `supervisorCommands`) is the one verb that DOES cycle it. CLI: `celilo events install-daemon|uninstall-daemon|show-daemon|restart-daemon [--system]`. `restart-daemon` exists because the dispatcher runs the code it LOADED: celilo-mgr sat 9 days on event-bus v0.1.8 after apt installed v0.2.0, running the very bug the release fixed (celilo#604). Two things make it non-trivial and both are load-bearing: (1) it stops any live dispatcher the supervisor does not own first — an ORPHAN (PPID 1) is invisible to `systemctl restart`, and `assertSoleDispatcher` then crash-loops the unit while the old code keeps serving; (2) it verifies on the BUS that a NEW pid is live reporting `BUS_VERSION`, never off systemctl's exit code, which returns 0 into exactly that crash loop. System scope shells `sudo systemctl` (the unit is root-owned; celilo runs unprivileged), covered by the scoped `/etc/sudoers.d/celilo-events-restart` conffile that `celilo-bootstrap` ships — a unit test asserts the argv and the grant cannot drift apart. ## Terminal UI (`@celilo/cli-display`) Every CLI-facing primitive celilo renders with. Lives in `packages/cli-display/src/` so module code running in-process (capability functions, hook scripts) can reach it via `@celilo/capabilities`' re-export, and so `@celilo/core` can use it without depending on `@celilo/cli`. `@clack/prompts` was removed tree-wide in celilo#699; nothing here wraps a third-party prompt library. **The stream contract:** stdout carries a command's RESULT and nothing else — no glyph, no box-drawing prefix, no ANSI. Everything else (prompts, progress, banners, log lines, diagnostics) goes to **stderr**. This is what lets `celilo module list | grep '^caddy '` and `celilo events tail | jq` work without a repair step; the previous renderer put chrome on stdout, which cost a full e2e run when a `│ ` prefix read as a missing module (celilo#695) and made ten JSON commands unparseable (celilo#698). - **Prompts** — `packages/cli-display/src/prompt.ts` — `text`, `password`, `confirm`, `select`, `multiselect`, plus `CANCEL`/`isCancel` and `isInteractive`. Raw-mode keypress handling with cursor movement, submit-time validation, and per-keystroke tokenization (one stdin chunk can carry a whole paste). **Refuses a non-TTY stdin** with `PromptUnavailableError` instead of hanging on it or taking the next newline as a considered answer — both real past failures. New interactive decisions must NOT be added here; they belong on the event-bus interview, which a headless responder can answer. - **Message chrome** — `packages/cli-display/src/messages.ts` — `intro`, `outro`, `note`, `cancel`, and `log.{success,error,warn,info,message,step}`. All to stderr; routed into the active `ProgressDisplay` when one is set, so the two renderers never fight over the cursor. - **Colour** — `packages/cli-display/src/colors.ts` — `colorEnabled()`, `paint()`, and the `colors` token object. Gated per use on `NO_COLOR` → `FORCE_COLOR` → `stderr.isTTY`, so colour never reaches a pipe, a file, or protocol mode. - **Progress display** — `packages/cli-display/src/progress-display.ts` — `ProgressDisplay`: a flat append-only stream with a time gutter and one status-glyph vocabulary in `render` mode, or structured `[progress:start|done|fail|sub]` markers in `protocol` mode (what the Remote API server emits and the client re-renders locally). - **Active-display singleton** — `packages/cli-display/src/active-display.ts` — `getActiveDisplay`/`setActiveDisplay`; how a long-running command lets everything down its call chain route output through one display. - **celilo-side wrapper** — `apps/celilo/src/cli/prompts.ts` — `celiloIntro`/`celiloOutro`, `promptText`/`promptPassword`/`promptConfirm`, `showNote`, `log`. Roughly 160 `log.*` call sites route through this one module, which is why swapping the underlying implementation was a one-file change. - **FuelGauge** — `apps/celilo/src/cli/fuel-gauge.ts` — the Cylon-style indicator for operations over ~3s (ESC to background, ^C to cancel); delegates to the active `ProgressDisplay` when one is set. ## Remote API (drive the CLI over the wire) Run any celilo command on celilo-mgr over SSH instead of screen-scraping `ssh celilo …`. Typed, streamed, per-operation authz, mid-run interviews. Design: `openspec/changes/replace-ssh-cli-api/proposal.md`. - **Lightweight core (`@celilo/core`)** — `packages/core/src/` — the transport primitives lifted out of `@celilo/cli` so a consumer (e.g. the MCP server) can reach the wire without dragging Ink/React/drizzle/aws-sdk: `command-registry.ts` (`COMMANDS` + `CommandDef`/`ArgDef`/`FlagDef`), `protocol.ts`, `remote-client.ts`. Public surface: `packages/core/src/index.ts`. - **Wire protocol** — `packages/core/src/protocol.ts` (`@celilo/core`) — versioned NDJSON tagged union (`command`/`progress`/`log`/`result`/`error`/`interview`/`answer`) + `translateOutputLine`. - **Registry serialization** — `apps/celilo/src/cli/commands/commands-json.ts` — `celilo commands --json` prints the full `COMMANDS` tree as JSON; the live source of truth the MCP fetches to generate its tool surface (so it mirrors whatever celilo version the server runs). `service list --json` similarly exposes configured providers for MCP auto-detect. `module list --json` prints the module roster (id/version/state) as stable JSON — the backbone the MCP composite troubleshooting tools correlate `audit --json` findings against. - **Server** — `apps/celilo/src/api/serve.ts` (`apiServeMode`); the `celilo api-serve --principal=` sshd forced-command entry point (dispatched in `apps/celilo/src/cli/index.ts`). Authorizes per principal, runs the command as a protocol-mode child, streams output, audits to stderr. - **Client** — `packages/core/src/remote-client.ts` (`@celilo/core`) — `resolveRemote` (`--remote ` / `CELILO_REMOTE`), `runRemoteClient` (`ssh -T`, renders progress via the local ProgressDisplay, answers interviews via the `@celilo/cli-display` prompts). Refuses to prompt on a non-TTY stdin, replying `unanswerable` rather than submitting a default as if a human had chosen it. - **Access control** — `apps/celilo/src/services/api-access.ts` — `grantPrincipal`, `isAuthorized` (deny-by-default, `command:subcommand` grants), `renderAuthorizedKeys`. Table: `api_principals` (`apps/celilo/src/db/schema.ts`). CLI: `apps/celilo/src/cli/commands/api.ts` (`api grant|list|revoke|authorized-keys|key new`). - **Principal enrolment for a module (`control_plane_api`)** — `apps/celilo/src/services/api-principal-enrolment.ts` — `enrolControlPlanePrincipal`, `revokeControlPlanePrincipal`, and `buildControlPlaneApi`, the method table a consuming module's hooks receive. The consumer generates an ed25519 pair on its own system and presents the public half; nothing here accepts a private key. Grants are DERIVED from `readOnlyGrants(COMMANDS)` and are not a parameter, so a caller cannot ask for more, and a write verb is never granted however it is named. **Framework-granted, so no module provides it** — enrolment writes celilo's own `api_principals` row and a module script may import nothing but `@celilo/capabilities`, which rules out celilo-mgmt as much as anyone else (`web-ui-console` D7b). Injected by `capability-loader.ts` ONLY for a module whose stored manifest declares it under `requires`/`optional`, unlike every other capability the loader hands out, and scoped to that module: a caller may not name a neighbour's principal. Contract: `packages/capabilities/src/control-plane-api.ts`. - **Mid-run interview bridge (`kind:daemon` responder)** — `apps/celilo/src/services/remote-responder.ts` — `startRemoteResponder` bridges bus `interview.required.*` ↔ wire. - **Server provisioning** — the `celilo-bootstrap` deb (`packaging/celilo-bootstrap/scripts/postinst`) creates the non-root `celilo-api` landing account + sshd; membership in the `celilo` group + `/etc/sudoers.d/celilo` (`!use_pty`) gives api-serve DB access via the wrapper's sudo-drop. - **Self-upgrade (apt)** — `celilo apt-upgrade` (`apps/celilo/src/cli/commands/apt-upgrade.ts`) upgrades the deb-installed `celilo`/`celilo-bootstrap` packages (`apt-get update` → `--only-upgrade install`) then spawns a fresh `celilo system migrate` (ISS-0100), then `celilo events restart-daemon` so the dispatcher actually runs the code just installed — a failure there fails the whole command and names which steps DID complete, because "upgraded" while the dispatcher serves stale code is the silent state celilo#604 documents. It's the RW target behind the MCP's registry-derived `celilo_apt_upgrade` tool; the celilo user's two apt invocations are scoped-sudo'd by `/etc/sudoers.d/celilo-apt-upgrade`, shipped by `celilo-bootstrap`. **This upgrades celilo ITSELF — not the modules it manages. For those, see Module auto-upgrade below; the two are routinely confused.** - **Module auto-upgrade (registry-poll CD)** — the *pull* half of continuous deployment: celilo-mgr polls the registry and upgrades opted-in modules unattended. Spec: `openspec/specs/module-auto-upgrade/spec.md`. Entry points: `apps/celilo/src/cli/commands/module-upgrade.ts` — `runRegistryPoll` (the `--poll` path), `selectPollTargets` (pure: `autoUpgrade && latest && change ∉ {up-to-date, ahead}`), `upgradeOneModule` (update → backup → deploy → verify), `needsPreUpgradeBackup`, `pickAutoUpgrade`/`pickUpgradePolicy` (both fail closed/safe); `classifyVersionChange` in `module-update.ts` (treats a registry `+N` revision as a patch); `resolveDeployPosture` in `apps/celilo/src/services/deploy-posture.ts`. Trigger: celilo-mgmt's `registry-poll` subscription (`modules/celilo-mgmt/manifest.yml`) on `timer.tick.15m` with handler **`celilo module upgrade --poll`** — the flag is REQUIRED, since the dispatcher appends the event id positionally and a bare handler would consume it as the optional module name (silent: 3108 deliveries, 0 successes). Operator controls are framework config keys settable on ANY module (`FRAMEWORK_CONFIG_KEYS` in `module-config.ts`): `auto_upgrade` (opt-in, default false) and `upgrade_policy` (`by-semver`|`always-safe`|`always-fast`), validated at set time because both readers fail open. ⚠️ `always-safe` guarantees safe *posture*, NOT a backup — `needsPreUpgradeBackup` also requires the TARGET manifest to declare an `on_backup` hook, else it warns and proceeds. Confirm a data-bearing module declares `on_backup` before enabling `auto_upgrade` on it. The *build* half (app CI publishing a `.netapp` on merge) is not yet shipped — `openspec/changes/build-bus-poll-cd`. - **MCP service (`@celilo/mcp`)** — `packages/mcp/src/` — an operator-facing stdio MCP server (official `@modelcontextprotocol/sdk`, bin `celilo-mcp`) that drives a remote celilo server over the Remote API for an AI client. Two-item config (`config.ts`: `server` + `defaultUser`, env or `~/.config/celilo-mcp/config.json`). Dual-principal auth (`auth.ts`: `celilo-mcp auth setup` enrolls read-only `celilo-mcp-ro` + full `celilo-mcp-rw` ed25519 keypairs, prints the exact `celilo api grant` lines the operator runs server-side). Transport (`transport.ts`): reuses `@celilo/core` `runRemoteClient`, selecting the principal by `ssh -i ` and capturing structured output. Tool surface is generated LIVE from the server's command registry — `registry-fetch.ts` fetches `celilo commands --json` (+ `service list --json` for configured providers) over the RO principal on connect; `tools-from-registry.ts` (pure) projects that into one tool per runnable leaf, grouped by top-level command (`celilo_module_*`, `celilo_proxmox_*`, …), each with a Zod input schema from the leaf's args/flags and a read/write tag → RO/RW routing, plus a generic `celilo_run` escape hatch. Auto-detect hides provider-gated groups (e.g. `celilo_proxmox_*` until a Proxmox service is configured) and re-detects on a timer, emitting `notifications/tools/list_changed` when the surface changes. Coverage gate (`tests/coverage.test.ts`) asserts every registry leaf maps to a tool. Composite RO troubleshooting tools (`troubleshoot.ts` pure correlation + `troubleshoot-tools.ts` thin adapters): `celilo_assess_module ` and `celilo_fleet_status` correlate `celilo audit --json` (the drift backbone) with the `module list --json` roster into a per-module / fleet-wide verdict. Design: `openspec/changes/celilo-mcp-service/proposal.md`. (Distinct from the dev/ops `@celilo/mcp-server` below.) ## Web console (read-mostly operator UI) The fleet drawn by zone, a live module roster, and the alert and backup state that says what needs attention. Design: `openspec/changes/web-ui-console/`. **Not yet deployed** — the SPA, its server, the console read verbs, the acknowledgement path and the `control_plane_api` capability exist; the module's `on_install` that CALLS that capability, and the e2e suite, do not. The capability issues only the derived read-only grants and has no parameter that could widen them (task 6.3b settled that deliberately), so acknowledgement renders a denial until an operator grants `alerts:ack` by hand. - **Console read verbs** — `apps/celilo/src/cli/commands/console.ts` — `celilo console status` (the dashboard's single poll: zone order, per-module systems, observed health, backup freshness) and `celilo console get [--depth N]` (one module's bounded capability closure). Both classify **read-only** under `readOnlyGrants()`, so the console's principal covers them with no hand-maintained list. The verbs are named from the read-verb vocabulary (`status`, `get`) rather than for prose, because the API authorises at two levels and classifies a leaf by its SUBCOMMAND token — `console roster` would read better and classify as a WRITE. - **Console projection** — `apps/celilo/src/console/projection.ts` — the narrow payload, deliberately WITHOUT `manifestData`: `module list --json` returns 156 KB for 23 modules because it embeds every manifest blob, which is the wrong payload for a poll loop. Distinguishes *not deployed* from *not observed*; a module with no system is carried, not omitted. - **Bounded capability closure** — `apps/celilo/src/console/closure.ts` — `computeClosure()` wraps `planConsumerCleanup()`'s edge rather than forking it, adding distance, optionality and cycle termination. Takes a bindings map (`capability_bindings`, celilo#1072) so the walk follows what a module has ACTUALLY called into; without one it follows declarations, which answers "what could this reach" rather than "what does this stand on". - **Read-only classifier** — `packages/core/src/read-only-classifier.ts` — `READ_VERBS`, `isReadOnlyPath`, `opOf`, `flattenLeaves`, `readOnlyGrants()`. Lives beside `COMMANDS` because it is a property of the registry, and both the MCP server's `RO_GRANTS` and the console's principal derive from it. A new read verb is covered automatically; a new write verb is not. - **SSH transport** — `packages/core/src/ssh-transport.ts` — `childProcessTransport`, `sshArgs`, `sshTransportWith`, and the exit reaper. Shared rather than copied because the reaper exists after 656 orphaned ssh clients threw celilo-mgr into MaxStartups throttling (celilo#921), and the console server has the same restart-freely lifecycle. - **Console server** — `apps/console-server/src/upstream.ts` — holds **no database handle**; every fact arrives over the remote API as a principal granted the derived read ops plus `alerts:ack`. An interview is a FAILURE (a browser has no responder), and an unavailable read is reported with a reason (`denied` / `unknown-verb` / `unreachable` / `interview` / `malformed`) rather than returned as an empty result. Failures are not cached, so fixing a grant recovers on the next poll. `Upstream.run` is the one non-read: uncached, unparsed (the ack answers with a sentence), and it drops every cached read afterwards. - **Console acknowledgement** — `apps/console-server/src/verbs.ts` (`readSession`, `ackAlert`), `apps/celilo/src/cli/commands/notify-config.ts` (`celilo person list --json`) — the console's ONLY write. `celilo alerts ack` falls back to `people[0]` when `--as` is absent, which in a browser would credit every acknowledgement to whoever sorts first, so `ackAlert` takes a required `person` and there is no path through it that omits the flag. The person comes from `readSession`, which resolves the identity provider's subject against `person list --json` (display name, then the `sub` claim, case-insensitive); a subject matching nobody resolves to null and the console draws no control. `ackedAt` is re-read from celilo's row rather than stamped by the console server, which is a different machine. Gates: `apps/console-server/tests/ack.test.ts` records the argv, so "refused but written anyway" is visible. - **Protocol** — `packages/console-protocol/` — tsrpc definitions plus the generated `serviceProto`, shared by server and SPA. The repo's single documented exception to ESM-everywhere: no `type` field, because `tsrpc-cli proto` `require()`s the protocol sources. Gated by a test in an ESM directory that round-trips a call. - **SPA** — `apps/console/src/` — Vite + React 19 + atom.io + tsrpc-browser. Three routes (`dashboard`, `alerts`, `backups`); a module's detail is a panel, not a fourth route. Visual constraints live in `shell.css` and four are asserted against the source by `tests/constraints.test.ts`, including "no control for any operation that can raise an interview". Every read goes through one `Panel` that separates *unavailable* from *empty*. - **Zone topology renderer** — `packages/visualizer/src/gen/zone-topology.ts`, `topology-theme.ts`, `column-assignment.ts`, `render/topology-svg.ts` — bands by zone in trust order, columns assigned so a dependency edge falls as a vertical drop, orthogonal routing checked against every module rather than assumed clear, and a report naming anything it could not place. Tuning playground at `bun run dev` in `packages/visualizer`, route `/topology`; it exports a `TopologyTheme` JSON the console commits. ## E2E simulation - **cele2e harness** — `packages/e2e/src/` — `runner.ts`, `container-manager.ts` (`startNetwork`, `reconnectNetwork`), `network-builder.ts` (`NetworkBuilder`). - **Run wrapper** — `infra/scripts/cele2e-run.sh` lives in the separate `infra/` clone, **not** in this repo. See the cele2e section of the repo-root `CLAUDE.md` for the operator workflow. - **Signal simulators** — three containers, three jobs. `docker/Dockerfile.signal-cli` runs the REAL unlinked daemon (`network().withSignalCli()`, reachable at `signal-cli.lab`) so `e2e/tests/signal-contract.test.ts` can re-check celilo's understanding of the JSON-RPC surface against the actual binary — the pebble pattern. `docker/Dockerfile.signal-sim` runs `simulators/signal-cli/server.ts` (`withSignalSim()`, `signal-sim.lab`), the drivable stand-in with a control surface (`/_control/inbound`, `/_control/sent`, `/_control/unlink`). `/_control/inbound` from the LINKED account's own number emits a `syncMessage.sentMessage` transcript rather than a `dataMessage`, because that is the only shape the real daemon delivers a note-to-self in — and a note-to-self is what every reply is in the default single-operator setup (#460). `docker/Dockerfile.signal-release` (`withSignalRelease()`, `signal-release.lab`) serves the signal-cli release tarball so the module's deploy-time download resolves inside the sealed network — the download is served, never skipped. The libsignal aarch64 native is compiled at `build-infra` time from the module's own recipe (`packages/e2e/scripts/stage-libsignal.ts` → `modules/signal/build/`) and staged into the apt-repo pool, so the recipe is exercised for real on every rebuild while the deploy stays fast and the network stays sealed. - **Public-boundary NAT model** — `packages/e2e/config/routing/` — exactly ONE NAT sits between the fleet and the simulated internet: the customer firewall (`fw-main` in `direct-internet`, `fw-isp` in the two-layer default), which MASQUERADEs to `203.0.113.100`. The ISP edge `fw-ext` ROUTES the customer's `203.0.113.0/24` and must never re-NAT it (`-s 203.0.113.0/24 -j RETURN` ahead of its MASQUERADE) — an ISP does not NAT a subscriber that already holds a public address. This is load-bearing, not cosmetic: Namecheap-style DDNS registers the SOURCE address when the caller omits `ip=`, which is how celilo registers public names since #464/#466, so a second NAT here publishes the simulator's own address for every public hostname and quietly breaks ACME, inbound reach, and seeded apex records. Constants: `externalWanIp()` / `externalWanSubnet()` in `src/types.ts`. The corollary: every simulator on `internet-external` must default-route via fw-ext (`100.64.0.1`) so it can reply to the customer's public address — Docker's bridge gateway has no path across networks. `config/routing/public-sim-entrypoint.sh` is the shared two-liner; the DNS hierarchy, pebble, isitup and celilo-website-sim already carried it, and npm-registry / registry / apt-repo / minio / cpanel-host only appeared to work because the double NAT put fw-ext's on-link address in the source field. - **Simulated address plan** — `packages/e2e/src/types.ts` — `SIM_PRIVATE_SUPERNET` (`10.226.0.0/16`), `zoneIp(zone, host)`, `ZONE_SUBNETS`, `ZONE_GATEWAYS`. Every simulated PRIVATE zone is derived from this one table; the compose generator, `zone-classifier.ts` and the harness's `system init` all read it, so renumbering the whole sim is a one-line change. The sim deliberately does NOT reuse a real fleet's zone /24s (#539): the previous plan was byte-identical to production's, so a stack leaked on celilo's own forgejo-builder — which lives in the real dmz — claimed the builder's own subnet and blackholed every containerized CI job's route to the forge for ~135s. Teardown cannot prevent that (a SIGKILL runs no handler), so the addresses moved instead; the second, better reason is that a suite which only passes on production's exact octets is asserting one site's address plan rather than celilo's behaviour. `src/address-plan.test.ts` is the recurrence gate — it fails if a zone leaves the supernet, or if any retired fleet prefix reappears anywhere in `packages/e2e`, `e2e/tests` or `modules/*/e2e`. - **MCP server (agent-driven e2e)** — `packages/mcp-server/src/index.ts` — stdio MCP server exposing `start_run`/`run_status`/`run_result`/`stop_run` (detached cele2e runs read off the event bus, no ANSI scraping) + `env_check` (docker VM, run-lock, shared-infra, mgmt image CLI version, netapps). Dev/ops tool, `private`, not shipped to consumers.