import { existsSync, mkdirSync, readdirSync, rmSync } from 'node:fs'; import { basename, join } from 'node:path'; import { parse as parseYaml, stringify } from 'yaml'; import { normalizeObservers, observerEnv, observerPlacement } from './observer'; import { PROXMOX_SIM_IP, SIMULATOR_IPS } from './simulator-ips'; import type { MachineSpec, NetworkConfig, TopologyPreset, Zone } from './types'; import { ZONE_GATEWAYS, ZONE_SUBNETS, externalWanIp, greenwaveRouterIp, internalResolverIp, zoneIp, } from './types'; const PACKAGE_DIR = join(import.meta.dir, '..'); /** * Is this @celilo/e2e running from the monorepo, or from an npm install? * * `/../../modules` is `/modules` in the monorepo and * `/node_modules/modules` (nonexistent) in a consumer install. */ function isMonorepoInstall(): boolean { return existsSync(join(PACKAGE_DIR, '..', '..', 'modules')); } /** * The host dir bound read-only at the registry's `/uploads` — and therefore * the dir `publishModule` writes a `.netapp` into. The registry rescans it on * every request, so a host-side write is picked up with no container write and * no HTTP handshake. * * Both install shapes bind one. They differ only in WHICH, because they differ * in where standard modules come from: * * - **Monorepo dev** — modules are served live from `/modules` * (`../../modules`, BOOTSTRAP_MODULES_DIR mode), so edits to a module's * source flow through without a manual repack. `/uploads` is therefore a * dedicated drop-zone, emptied on every shared-infra start (see * `prepareRegistryDropZone`) — a leftover `.netapp` would shadow live source. * * - **npm-installed consumer** (lunacycle and friends) — `../../modules` does * not exist, so there is no source to serve. `/uploads` IS the netapp cache * `cele2e build-infra` fills from the public registry (see * stageNetappsFromRegistry in cli/build.ts), and a published module lands * beside the standard ones. Consumers vendor nothing. * * Read this through `registryUploadsHostDir()` rather than re-deriving it. * publishModule used to `docker cp` into the container's `/uploads` instead, * which works only where that path is NOT a mount — so it passed forever in * the monorepo and failed for every consumer against the read-only bind * (celilo#1142). One function now decides the mount and the write target * together, so they cannot disagree again. */ export function registryUploadsHostDir(): string { return join(PACKAGE_DIR, isMonorepoInstall() ? 'uploads' : 'netapps'); } /** * Make the drop-zone ready for a shared-infra bring-up. * * Creates it: Docker silently creates a MISSING bind source as an empty * root-owned directory, which the host then cannot write into. * * Empties it, but only in the monorepo, where `/uploads` is dedicated — a * `.netapp` left by a previous run would shadow the live module source it was * packed from. In a consumer install the same dir IS the build-infra netapp * cache, so clearing it would delete every standard module. (Before * celilo#1142 published netapps lived in the registry's container layer and * died with the container, so starting empty is the behaviour being kept, not * a new one.) */ export function prepareRegistryDropZone(): void { const dir = registryUploadsHostDir(); mkdirSync(dir, { recursive: true }); if (!isMonorepoInstall()) return; for (const file of readdirSync(dir)) { if (file.endsWith('.netapp')) rmSync(join(dir, file), { force: true }); } } function getRegistryVolumes(): string[] { const uploads = `./${basename(registryUploadsHostDir())}:/uploads:ro`; return isMonorepoInstall() ? ['../../modules:/modules:ro', uploads] : [uploads]; } const NETWORK_DRIVER_OPTS = { 'com.docker.network.bridge.enable_ip_masquerade': 'false', 'com.docker.network.endpoint.sysctls': 'net.ipv4.conf.IFNAME.rp_filter=0', }; const ROUTER_SYSCTLS = { 'net.ipv4.ip_forward': '1', 'net.ipv4.conf.all.rp_filter': '0', }; function networkDef(subnet: string, gateway: string) { return { driver: 'bridge', driver_opts: NETWORK_DRIVER_OPTS, ipam: { config: [{ subnet, gateway }] }, }; } /** * A zone's bridge, derived entirely from the address plan in `types.ts`. The * bridge itself takes `.250` (Docker's own gateway address on the network — NOT * the routing gateway, which is the firewall at `ZONE_GATEWAYS[zone]`). */ function zoneNetworkDef(zone: Zone) { return networkDef(ZONE_SUBNETS[zone], zoneIp(zone, 250)); } /** * Image tag prefix for baked e2e images. `cele2e build-infra` creates these * tags; e2e-save/e2e-load persist them across colima restarts. A compose * referencing one of these tags never builds it — a missing tag is a loud * failure naming the remedy (see referencedImages / container-manager). */ const IMAGE_PREFIX = 'celilo-e2e'; /** * The image tag `baseService` assigns a service built from `dockerfile`. * Stable across runs (unlike compose's `-` default), which is * what lets callers rebuild a sim image by tag — e.g. the bake re-building the * registry sim after re-staging tarballs (celilo#1299). */ export function imageTag(dockerfile: string): string { // docker/Dockerfile.management -> celilo-e2e/management const name = dockerfile.replace('docker/Dockerfile.', ''); return `${IMAGE_PREFIX}/${name}`; } function baseService(opts: { /** * Which Dockerfile bakes this image: `docker/Dockerfile.firewall` -> * `celilo-e2e/firewall`. The yaml carries `image:` ONLY. `cele2e * build-infra` is the one builder (it owns the network phase); a suite * run verifies the tag exists instead of building, so a missing image * fails loudly rather than silently resolving FROM from docker.io. * See tests/composes-reference-baked-images.test.ts. */ dockerfile?: string; image?: string; networks: Record; cap_add?: string[]; sysctls?: Record; volumes?: string[]; command?: string; depends_on?: string[]; environment?: Record; tmpfs?: string[]; privileged?: boolean; /** * Docker `security_opt` entries, e.g. `seccomp=unconfined`. Needed by the * management box so the hook jail's bubblewrap can build a namespace inside * a container — see the measured ladder at its call site. */ security_opt?: string[]; dns?: string[]; /** Host devices to expose, e.g. `/dev/net/tun` for a userspace WireGuard tunnel. */ devices?: string[]; }) { // The Dockerfile name drives the tag but must NOT reach the yaml: compose // rejects unknown service keys. The emitted service carries `image:` only. const { dockerfile, ...rest } = opts; const service: Record = { ...rest, restart: 'unless-stopped' }; if (dockerfile && !opts.image) { service.image = imageTag(dockerfile); } return service; } // --------------------------------------------------------------------------- // Shared infrastructure project name. Used by both the shared infra // manager and per-test compose files (to reference external networks). // --------------------------------------------------------------------------- /** * The baked images a run needs — every service's `celilo-e2e/*` tag, paired * with the tag `baseService` assigned it. * * Used to answer "is everything the compose references present?" before the * run. The yaml carries `image:` only (never `build:`), so this is the full * set: a missing tag is a loud failure naming `cele2e build-infra`, never a * silent suite-time build resolving FROM from docker.io (tasks.md 5b.2). * * Pure and parsed back out of the emitted YAML rather than computed alongside * it, so it reports what the compose file actually says. A service that gains * a reference is covered without anyone remembering to add it here. */ export function referencedImages(yaml: string): string[] { const compose = parseYaml(yaml) as { services?: Record; }; const images = new Set(); for (const service of Object.values(compose.services ?? {})) { if (service.image) images.add(service.image); } return [...images].sort(); } export const SHARED_PROJECT_NAME = 'celilo-e2e-shared'; /** Docker network names as created by the shared infrastructure project */ export const SHARED_NETWORKS = { 'internet-external': `${SHARED_PROJECT_NAME}_internet-external`, } as const; // --------------------------------------------------------------------------- // Topology: services south of fw-ext (firewalls, ISP routers) // --------------------------------------------------------------------------- /** * Default topology: fw-isp (greenwave) + fw-main (iptables) * Two-layer NAT: internet → fw-isp → fw-main → target */ /** * celilo's own control-plane network, used only when `managementZone` is * 'secure-mgmt'. Deliberately NOT one of the segmented data-plane tiers: it is * where celilo-mgr itself lives, mirroring the production topology the suite * previously could not express. */ const SECURE_MGMT_GATEWAY = ZONE_GATEWAYS['secure-mgmt']; const SECURE_MGMT_MANAGEMENT_IP = zoneIp('secure-mgmt', 100); /** * Zones deliberately LEFT OUT of the no-NAT class, each mapped to the reason it * is excluded. The no-NAT set is computed as every declared zone (ZONE_SUBNETS) * minus these, so a zone added to the Zone type joins the class by default and * leaving one out is a deliberate edit to this map, never an omission. A test * over this map is the recurrence gate for celilo#1263: the suite once * reproduced that bug class by listing these zones by hand, and a hand-written * list rots the moment a zone is added. */ export const NO_NAT_EXCLUDED_ZONES: Partial> = { internal: 'A LAN device’s DNS query must arrive SNAT’d to a zone-gateway address so it matches the passthrough view and gets the natIp answer, not the protected one (ISS-0156).', }; /** * The zones fw-main routes between WITHOUT NAT, so the dmz-resident resolver * sees each client's real zone (source-based split-horizon, ISS-0156). * * Computed, never listed: every declared zone minus * {@link NO_NAT_EXCLUDED_ZONES}. THE COST OF EXCLUDING `internal`, recorded so * the next reader can connect it (celilo#1263): traffic in the other direction — * from a no-NAT zone to an internal host, e.g. the control plane on secure-mgmt * reaching fw-isp at 10.226.1.1 — matches no RETURN row and falls through to the * blanket MASQUERADE. Traffic to every other declared zone keeps its source; * traffic to internal does not. Whether that asymmetry is tolerable is a * control-plane question, not a fixture typo. */ export function noNatZones(): Zone[] { return (Object.keys(ZONE_SUBNETS) as Zone[]).filter((zone) => !(zone in NO_NAT_EXCLUDED_ZONES)); } const PROTECTED_SUBNETS = noNatZones() .map((zone) => ZONE_SUBNETS[zone]) .join(' '); /** * The zone legs fw-main actually holds, per topology — the SINGLE source of truth * for "which networks is the customer firewall wired to". * * This exists because `deployFirewall`'s `opts.zones` was doing two jobs at once: * saying which zones get `provided_networks` written to system config, AND * standing in as a claim about the box's hardware. It was only ever the first. * The compose file below is what decides the second, and roughly twenty call * sites passed `['dmz']` while the generator wired four legs regardless — so * every one of them under-declared, silently. * * Both the compose services and `deployFirewall` read this, so a test cannot * under-declare by omission and a new leg cannot be added in one place only. * `packages/e2e/tests/firewall-zone-legs.test.ts` holds them to each other. */ export type FirewallLeg = Zone | 'external'; export function firewallZoneLegs(topology: TopologyPreset): FirewallLeg[] { const segmented: FirewallLeg[] = ['internal', 'dmz', 'app', 'secure']; return topology === 'direct-internet' ? [...segmented, 'external'] : segmented; } /** * The compose `networks:` block for fw-main, derived from {@link firewallZoneLegs} * so the wiring and the declaration cannot drift. `external` is the public leg and * lives on the `isp-external` docker network rather than a zone-named one. */ function firewallNetworks(topology: TopologyPreset): Record { const nets: Record = {}; for (const zone of firewallZoneLegs(topology)) { if (zone === 'external') { nets['isp-external'] = { ipv4_address: externalWanIp() }; } else { nets[zone] = { ipv4_address: ZONE_GATEWAYS[zone] }; } } return nets; } function defaultTopologyServices(config: NetworkConfig): Record { return { 'fw-main': baseService({ dockerfile: 'docker/Dockerfile.firewall', networks: firewallNetworks('default'), cap_add: ['NET_ADMIN', 'SYS_ADMIN'], // A firewall may also terminate a VPN (the admin tunnel the `wireguard` // module owns). wireguard-go needs the tun device; the kernel module is // deliberately not relied on — see Dockerfile.firewall. devices: ['/dev/net/tun'], sysctls: ROUTER_SYSCTLS, volumes: ['ssh-keys:/ssh-keys:ro'], environment: { PROTECTED_SUBNETS }, }), 'fw-isp': baseService({ dockerfile: 'docker/Dockerfile.greenwave-sim', networks: { internal: { ipv4_address: greenwaveRouterIp() }, 'isp-external': { ipv4_address: '203.0.113.100' }, }, cap_add: ['NET_ADMIN', 'SYS_ADMIN'], // A firewall may also terminate a VPN (the admin tunnel the `wireguard` // module owns). wireguard-go needs the tun device; the kernel module is // deliberately not relied on — see Dockerfile.firewall. devices: ['/dev/net/tun'], sysctls: ROUTER_SYSCTLS, volumes: ['ssh-keys:/ssh-keys:ro'], // Which real ISP router this sim stands in for. The protocol is identical // across the two; only the TR-181 vendor extension prefix differs. environment: { ROUTER_VENDOR_PREFIX: config.routerVendorPrefix ?? 'X_GWS_', // The simulated router serves DHCP on `internal` by default, which is // what an operator's router does. A suite testing celilo's OWN DHCP // turns it off, because two servers on one broadcast domain race. ROUTER_DHCP: config.routerDhcp === false ? 'off' : 'on', }, }), }; } /** * Direct internet topology: fw-main has external interface, no fw-isp * Single-layer NAT: internet → fw-main → target * Management routes default through fw-main instead of fw-isp */ function directInternetTopologyServices(): Record { return { 'fw-main': baseService({ dockerfile: 'docker/Dockerfile.firewall', networks: firewallNetworks('direct-internet'), cap_add: ['NET_ADMIN', 'SYS_ADMIN'], // A firewall may also terminate a VPN (the admin tunnel the `wireguard` // module owns). wireguard-go needs the tun device; the kernel module is // deliberately not relied on — see Dockerfile.firewall. devices: ['/dev/net/tun'], sysctls: ROUTER_SYSCTLS, volumes: ['ssh-keys:/ssh-keys:ro'], environment: { PROTECTED_SUBNETS }, }), // No fw-isp — fw-main handles external routing directly }; } const TOPOLOGY_SERVICES: Record< TopologyPreset, (config: NetworkConfig) => Record > = { default: defaultTopologyServices, 'direct-internet': directInternetTopologyServices, }; // --------------------------------------------------------------------------- // Shared infrastructure compose (DNS, Pebble, registry, etc.) // --------------------------------------------------------------------------- /** * Generate the compose YAML for shared infrastructure that runs once * per test suite. Contains: DNS hierarchy, Pebble ACME, registry, * isitup prober, celilo-website, npm-registry simulators. * Owns the internet-external network. * * fw-ext and comcast-resolver are per-test because they route to * per-test containers (fw-isp/fw-main) and need those to exist first. * * The real-internet network is per-test (no shared service uses it). */ export function generateSharedInfraYaml(): string { const networks: Record = { 'internet-external': networkDef('100.64.0.0/24', '100.64.0.250'), }; const volumes: Record = {}; const services: Record = {}; services['root-dns'] = baseService({ dockerfile: 'docker/Dockerfile.authoritative-dns', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.ROOT_DNS } }, cap_add: ['NET_ADMIN'], volumes: [ './config/dns/knot-root.conf:/config/knot.conf:ro', './config/dns/root.zone:/config/root.zone:ro', ], }); services['tld-dns'] = baseService({ dockerfile: 'docker/Dockerfile.authoritative-dns', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.TLD_DNS } }, cap_add: ['NET_ADMIN'], volumes: [ './config/dns/knot-tld.conf:/config/knot.conf:ro', './config/dns/com.zone:/config/com.zone:ro', './config/dns/org.zone:/config/org.zone:ro', './config/dns/net.zone:/config/net.zone:ro', ], }); services['namecheap-dns'] = baseService({ dockerfile: 'docker/Dockerfile.namecheap-dns', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.NAMECHEAP_DNS } }, cap_add: ['NET_ADMIN'], // Per-domain DDNS passwords for the cross-domain test. Fallback // `test123` still applies to any domain not listed. environment: { DDNS_PASSWORDS: '{"iamtheinternet.org":"test123","celilo.computer":"test456","example.net":"test789"}', }, volumes: [ './config/dns/knot-namecheap.conf:/config/knot.conf:ro', // Zone files are mounted READ-ONLY under /seed; namecheap-startup.sh // copies them into the writable /config that Knot + the DDNS simulator // mutate at runtime. This keeps DDNS/Knot zone rewrites container-local // instead of polluting the version-controlled fixtures (see // namecheap-startup.sh for the full rationale). './config/dns/iamtheinternet.org.zone:/seed/iamtheinternet.org.zone:ro', './config/dns/celilo.computer.zone:/seed/celilo.computer.zone:ro', './config/dns/example.net.zone:/seed/example.net.zone:ro', './config/dns/park-your-domain.com.zone:/seed/park-your-domain.com.zone:ro', './config/dns/tangohost.com.zone:/seed/tangohost.com.zone:ro', ], }); // apt-cache simulator removed: target images no longer route apt // through a caching proxy. Tests apt direct upstream now, which // eliminates the cold-start race that bit every fresh-VM debug // cycle. See CLAUDE.md history for details. services.letsencrypt = baseService({ dockerfile: 'docker/Dockerfile.pebble', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.PEBBLE } }, cap_add: ['NET_ADMIN'], command: '-config /config/pebble-config.json -dnsserver 203.0.113.1:53', volumes: [ './config/pebble/pebble-config.json:/config/pebble-config.json:ro', './config/pebble/pebble-tls.crt:/config/pebble-tls.crt:ro', './config/pebble/pebble-tls.key:/config/pebble-tls.key:ro', ], }); // Registry stand-in. Build context is the @celilo/e2e package root — // Dockerfile.registry COPYs from `registry-server/` which lives inside // the package (bundled at build time from packages/registry-server, // shipped in the npm tarball). See packages/e2e/src/registry-bundle.ts. services.registry = baseService({ dockerfile: 'docker/Dockerfile.registry', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.REGISTRY } }, cap_add: ['NET_ADMIN'], volumes: getRegistryVolumes(), }); // OFF-FLEET recursive resolver — the rig's 1.1.1.1, and the vantage point // celilo's `public_dns` check resolves from. // // A peer of comcast-resolver, not a replacement: the fleet is configured to // use THAT one (`system init` writes dns.primary=203.0.113.1), and celilo // refuses to accept a resolver it already asks as evidence about the public // internet — a check that used the fleet's own resolver would pass whatever // the internet sees, which is the celilo#626 blindness one layer up. So the // simulation needs two independent public resolvers, exactly as the real // world has your ISP's and Cloudflare's. // // Same image as comcast-resolver (identical unbound.conf → identical // authoritative answers) with a different startup: this one sits ON // internet-external with the authoritative servers, so it needs no route // through fw-ext to reach them. services['public-resolver'] = baseService({ dockerfile: 'docker/Dockerfile.resolver', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.PUBLIC_RESOLVER } }, cap_add: ['NET_ADMIN'], command: '/public-resolver-startup.sh', }); // IP echo service — mimics api.ipify.org. Reports the source address a // request appears to come from, which for the customer fleet is the // firewall's external address after SNAT. This is where the `public_dns` // check's EXPECTATION comes from; comparing public DNS against the // registrar's own response would be self-agreement. services['ip-echo'] = baseService({ dockerfile: 'docker/Dockerfile.ip-echo', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.IP_ECHO } }, cap_add: ['NET_ADMIN'], }); // External website prober simulator — mimics isitup.org's /api.json endpoint. // Sits on internet-external so probe requests exercise the real public-facing // path (target domain → DNS → ISP → firewall → Caddy). Uses comcast-resolver // at 203.0.113.1 as DNS so it can recursively resolve the simulated public // zones (e.g. iamtheinternet.org). services.isitup = baseService({ dockerfile: 'docker/Dockerfile.isitup', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.ISITUP } }, cap_add: ['NET_ADMIN'], dns: ['203.0.113.1'], }); // celilo.computer static site simulator — serves install.sh and the docs // over HTTPS with a Pebble-issued cert. Exists so the e2e harness can // exercise `curl -fsSL https://celilo.computer/install.sh | bash` end-to-end // without reaching the real internet. Static content is staged into the // image at build time from modules/celilo-website/site/dist/. services['celilo-website'] = baseService({ dockerfile: 'docker/Dockerfile.celilo-website-sim', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.WEBSITE } }, cap_add: ['NET_ADMIN'], dns: ['203.0.113.1'], }); // npm-compat registry simulator — serves @celilo/* tarballs to install.sh's // `bun add -g` step. Tarballs are staged at build time by // packages/e2e/scripts/pack-celilo-packages.ts (cli, capabilities, // cli-display, e2e, event-bus). Reachable at http://npm-registry.lab from // any container that uses the internal resolver. // OFF-FLEET web host (external_web). A PUBLIC PEER on internet-external, not // a fleet member and not behind the customer firewall: celilo reaches it by // SSH egress the way it would reach any host on the real internet, and public // DNS resolves tangohost.com straight to this address. It runs its own Apache // and its own pre-installed TLS — celilo governs none of it. services['cpanel-host'] = baseService({ dockerfile: 'docker/Dockerfile.cpanel-host-sim', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.CPANEL_HOST } }, cap_add: ['NET_ADMIN'], }); services['npm-registry'] = baseService({ dockerfile: 'docker/Dockerfile.npm-registry-sim', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.NPM_REGISTRY } }, cap_add: ['NET_ADMIN'], }); // apt-repo simulator — serves the celilo + celilo-bootstrap .debs as a // plain apt repository (mirrors apt.celilo.computer). Reachable at // http://apt.celilo.lab. Debs are staged at build time by // packages/e2e/scripts/stage-apt-repo.ts; the Packages index is generated // inside the image (Dockerfile.apt-repo-sim). Drives the bootstrap-apt // test's `apt install celilo-bootstrap`. services['apt-repo'] = baseService({ dockerfile: 'docker/Dockerfile.apt-repo-sim', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.APT_REPO } }, cap_add: ['NET_ADMIN'], }); // MinIO S3-compatible object storage — the backup/restore target that stands // in for AWS S3. Reachable at http://minio.lab from any container using // the internal/public resolver. celilo's S3 provider (forcePathStyle + custom // endpoint) drives it unchanged. Inbound-only like the other HTTP sims, so no // dns/route wiring; the celilo-backups bucket is pre-created at startup. // Drives the migration restore e2e (push via `backup --storage`, pull via // `backup pull`). services.minio = baseService({ dockerfile: 'docker/Dockerfile.minio', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.MINIO } }, cap_add: ['NET_ADMIN'], }); const compose = { networks, volumes, services }; return stringify(compose, { lineWidth: 120 }); } // --------------------------------------------------------------------------- // Per-test compose generation // --------------------------------------------------------------------------- /** * The Proxmox API simulator, on `secure-mgmt` with celilo-mgr (D6). * * Defined once and emitted from BOTH compose blocks. Adding a simulator to * only one of them is the standard way to end up with something that works in * exactly half the suites, and the half it fails in is whichever one nobody * ran first. * * It holds the host Docker socket because creating an LXC has to produce a * real container (D2). The containment for that is in the provisioner: every * container it creates carries the `celilo-e2e-` prefix and it refuses to * touch one that does not. */ function proxmoxSimService() { return baseService({ dockerfile: 'docker/Dockerfile.proxmox-sim', networks: { 'secure-mgmt': { ipv4_address: PROXMOX_SIM_IP } }, volumes: ['/var/run/docker.sock:/var/run/docker.sock'], // Compose substitutes this at up-time from `-p`, so the generator does not // have to know the project name (it is not a property of the YAML). environment: { CELILO_E2E_PROJECT: '${COMPOSE_PROJECT_NAME}' }, cap_add: ['NET_ADMIN'], }); } /** * Generate the compose YAML for a single test's containers. * References the shared infrastructure's isp-external network as external. * Creates its own internal, dmz, app, secure networks. */ export function generateTestComposeYaml(config: NetworkConfig, celiloRoot?: string): string { const mgmtOnOwnNetwork = config.managementZone === 'secure-mgmt'; // The control-plane network exists if celilo-mgr lives there OR if any // machine does — a module declaring `zone: secure-mgmt` needs somewhere to // land whether or not the management box shares the network (#436). // The proxmox simulator lives here too, so enabling it must bring the network // up even in a topology where celilo-mgr sits on the internal LAN. const needsSecureMgmt = mgmtOnOwnNetwork || (config.secureMgmtMachines ?? []).length > 0 || config.proxmoxSim === true; const networks: Record = { internal: zoneNetworkDef('internal'), dmz: zoneNetworkDef('dmz'), app: zoneNetworkDef('app'), secure: zoneNetworkDef('secure'), // Only when the control plane is in use (celilo-mgr, a secure-mgmt // machine, or the proxmox simulator). With the default managementZone of // `secure-mgmt` this is now present in every generated compose. ...(needsSecureMgmt ? { 'secure-mgmt': zoneNetworkDef('secure-mgmt') } : {}), 'isp-external': networkDef('203.0.113.0/24', '203.0.113.250'), // internet-external is owned by shared infra; real-internet is per-test // (it just bridges fw-ext ↔ comcast-resolver, no shared service uses it). // real-internet uses Docker's default gateway (.1) because fw-ext's // startup script hardcodes `ip route add default via 172.30.0.1`. 'internet-external': { external: true, name: SHARED_NETWORKS['internet-external'], }, 'real-internet': { driver: 'bridge', ipam: { config: [{ subnet: '172.30.0.0/24' }] }, }, }; const volumes: Record = { 'ssh-keys': { driver: 'local' }, }; const services: Record = {}; // --- Management (always present) --- const managementDefaultGw = config.topology === 'direct-internet' ? ZONE_GATEWAYS.internal : greenwaveRouterIp(); // Pick the management image based on variant. Both branches use // `image:` (not `build:`) so docker compose uses the existing tag — // critical for two reasons. First, `cele2e build-infra` runs // install.sh and `docker commit`s the result onto :latest (Phase 2 // bake step). If compose rebuilt from Dockerfile.management it would // OVERWRITE the bake commit, destroying the post-install.sh image. // Confirmed bitten on 2026-05-06: caddy-custom-hostname test failed // with `celilo: command not found` because the compose rebuild had // wiped install.sh's output. Second, no compose anywhere may carry a // `build:` section: a suite-time build resolves FROM from docker.io // and fetches over the network (tasks.md 5b.2). Every generated // compose is image-references-only; the gate is // tests/composes-reference-baked-images.test.ts. // // :vanilla — bun + unzip + bunfig, no celilo. Used by the // install-sh regression test. // :latest — vanilla + post-install.sh (the bake step's // docker commit), used by every other test. // // build-infra produces both tags; tests just reference them. // A management-class box. Emitted once for the primary `management` box and, // when a secondary variant is configured (migration restore e2e), again for // a fresh `celilo-mgr-2`. Both use existing image tags (never `build:` — // see the bake-commit note above) and identical networking/env so the second // box reaches the apt/npm/minio sims over internet egress exactly like the // first. container-manager's celilo-init + routing-verify target the // `management` container by name, so `celilo-mgr-2` comes up fresh and // un-init'd — the empty target restore needs. const emitManagement = ( variant: 'default' | 'vanilla' | undefined, ip: string, zone: 'internal' | 'secure-mgmt' = 'internal', ) => baseService({ image: variant === 'vanilla' ? 'celilo-e2e/management:vanilla' : 'celilo-e2e/management:latest', networks: { [zone]: { ipv4_address: ip } }, // SYS_ADMIN is for the hook jail, NET_ADMIN for the routing the box does // anyway. See the security_opt block below for why both are here. cap_add: ['NET_ADMIN', 'SYS_ADMIN'], // The hook jail runs each hook under bubblewrap, which has to build a // user namespace and then mount inside it. Docker denies that four // separate ways, and every one of these is load-bearing — measured on a // colima Ubuntu 24.04.1 VM (kernel 6.8.0-50, // kernel.apparmor_restrict_unprivileged_userns=1), by removing one at a // time from the working set: // // drop seccomp=unconfined -> bwrap: pivot_root: Operation not permitted // drop apparmor=unconfined -> bwrap: Failed to make / slave: Permission denied // drop systempaths=unconfined -> bwrap: Can't mount proc on /newroot/proc // drop SYS_ADMIN -> bwrap: setting up uid map: Permission denied // // Deliberately NOT `privileged: true`, which design D8 measured as // working and task 4.1 asked to improve on. It would also grant every // capability and every host device; this set grants four named things. // // `systempaths=unconfined` is the one D8's ladder was missing: it unmasks // the /proc paths Docker hides, which is why "all three" of the earlier // options still failed on the proc mount. // // NOTE this makes the container a WEAK place to test the AppArmor // profile: apparmor=unconfined means the restriction the profile exists // to satisfy is not in force here. That gate is the celilo-builder probe // on a real kernel, never this container. security_opt: ['seccomp=unconfined', 'apparmor=unconfined', 'systempaths=unconfined'], // ISS-0157: `module import`'s `bun install` hard-links packages from the // bun cache into node_modules, which on overlayfs forces a copy-up + fsync // per file. On a slow/contended builder disk that storm blows past the // 120s import timeout (intermittently). Put the celilo data dir (the // node_modules install target) on tmpfs (RAM): bun then copies cache→tmpfs // (cross-fs, no overlay copy-up) at RAM speed, immune to host disk. NOT // /root/.bun itself — that holds the baked `celilo` CLI (`bun add -g`). // Tmpfs the bun *cache* subdir too: the copy-up + fsync storm is the // overlay upper layer under /root/.bun/install/cache. RAM-backing it (cold // → re-fetch over the fast sim/net) keeps the CLI intact. Ephemeral e2e. // `:exec` on the celilo data dir, because compose's default tmpfs options // include noexec and terraform EXECUTES the provider plugin it unpacks // into `/generated/terraform/.terraform/providers/...`. Without it // the deploy dies with // fork/exec .../terraform-provider-proxmox_v3.0.2-rc07: permission denied // on a file that is `-rwxr-xr-x` and owned by the root running it — which // reads as a file-permission bug and is a mount-option one. The tmpfs // itself stays: it is what keeps `module import` inside its timeout // (ISS-0157), and that reason is unaffected by allowing exec. tmpfs: ['/root/.local/share/celilo:exec', '/root/.bun/install/cache'], volumes: [ ...(celiloRoot ? [`${celiloRoot}:/celilo`] : []), 'ssh-keys:/ssh-keys', ...(config.managementVolumes ?? []), ], environment: { // On its own control-plane network the only router in reach is fw-main's // leg there — both for the default route and for the segmented zones. DEFAULT_GATEWAY: zone === 'secure-mgmt' ? SECURE_MGMT_GATEWAY : managementDefaultGw, // BOTH branches always set the hop, never let the image's baked value // through: the bake `docker commit`s the management container, so a // bake from the secure-mgmt default baked FW_MAIN_HOP=10.226.120.1 // into :latest, and every internal-topology suite then inherited a // nexthop that is not on-link from the internal LAN. The mgmt box // exited 2 at the segmented-zone routes in management-routes.sh and // crash-looped (celilo#1351). Compose env overrides image env, so an // always-present FW_MAIN_HOP immunizes every topology against any // baked value, current and future. FW_MAIN_HOP: zone === 'secure-mgmt' ? SECURE_MGMT_GATEWAY : ZONE_GATEWAYS.internal, CELILO_REGISTRY_URL: 'http://e2e-registry.lab', // The hook jail is REQUIRED in the e2e, not `auto` (peba, 2026-09-08). // ce-rez7 made an unset policy resolve to `off`, which silently turned // hook-jail-trespass into a test of nothing: its stage 2 (no policy set) // and its stage 3 control (explicit `off`) became the same experiment. // celilo#1329. `required` beats `auto` here because a missing bubblewrap // backend is then a hard failure instead of a silent drop to unjailed, so // it cannot rot the same way twice. A per-command `CELILO_HOOK_JAIL=off` // prefix still overrides this, which is how the deliberate unjailed // controls in both jail suites keep working. CELILO_HOOK_JAIL: 'required', // `cele2e run --source-cli` sets this, and the image's shim reads it to // run the mounted workspace instead of the CLI install.sh installed. // Off by default: the installed CLI is the artifact under test, and // running the source over the bind mount roughly doubles a celilo // command's start-up (see the shim's comment in bin/e2e-bake-management). ...(process.env.CELILO_E2E_SOURCE_CLI ? { CELILO_E2E_SOURCE_CLI: process.env.CELILO_E2E_SOURCE_CLI } : {}), }, }); services.management = mgmtOnOwnNetwork ? emitManagement(config.managementVariant, SECURE_MGMT_MANAGEMENT_IP, 'secure-mgmt') : emitManagement(config.managementVariant, zoneIp('internal', 100)); if (config.secondaryManagementVariant) { services['celilo-mgr-2'] = emitManagement( config.secondaryManagementVariant, zoneIp('internal', 101), ); } // --- Internal split-horizon DNS resolver --- // Resolves the test domain to caddy's internal DMZ IP so management can // reach it via a direct route rather than through the external interface // (hairpin NAT would be needed otherwise). Management already has // `internalResolverIp()` as its primary resolver in /etc/resolv.conf. const caddyIp = config.dmzMachines.find((m) => m.name === 'caddy')?.ip ?? config.dmzMachines[0]?.ip ?? zoneIp('dmz', 10); services['dns-int'] = baseService({ dockerfile: 'docker/Dockerfile.internal-resolver', networks: { internal: { ipv4_address: internalResolverIp() } }, cap_add: ['NET_ADMIN'], environment: { CADDY_IP: caddyIp, DOMAIN: config.domain ?? 'iamtheinternet.org', DEFAULT_GATEWAY: managementDefaultGw, }, }); // --- Topology-specific infrastructure (south of fw-ext) --- const topologyFn = TOPOLOGY_SERVICES[config.topology] || TOPOLOGY_SERVICES.default; Object.assign(services, topologyFn(config)); // Give fw-main a leg on the control-plane network when celilo-mgr lives there, // so the management box is routed like any other segmented network rather than // being stranded. Applied as a post-step so the default topology's generated // compose is untouched. if (needsSecureMgmt) { const fwMain = services['fw-main'] as { networks?: Record } | undefined; if (fwMain?.networks) { fwMain.networks['secure-mgmt'] = { ipv4_address: SECURE_MGMT_GATEWAY }; } } // --- Routing infrastructure (bridges per-test to shared infra) --- services['fw-ext'] = baseService({ dockerfile: 'docker/Dockerfile.router', networks: { 'isp-external': { ipv4_address: '203.0.113.101' }, 'internet-external': { ipv4_address: '100.64.0.1' }, 'real-internet': { ipv4_address: '172.30.0.5' }, }, cap_add: ['NET_ADMIN', 'SYS_ADMIN'], sysctls: ROUTER_SYSCTLS, }); services['comcast-resolver'] = baseService({ dockerfile: 'docker/Dockerfile.resolver', networks: { 'isp-external': { ipv4_address: '203.0.113.1' }, 'real-internet': { ipv4_address: '172.30.0.4' }, }, cap_add: ['NET_ADMIN'], }); // --- Dynamic test machines --- if (config.proxmoxSim) { services['proxmox-sim'] = proxmoxSimService(); } const allMachines = getAllMachines(config); for (const machine of allMachines) { const zoneNetwork = machine.zone; const gateway = ZONE_GATEWAYS[machine.zone]; const needsDocker = machine.zone === 'app' || machine.docker === true; const dockerfile = machine.dockerfile || (needsDocker ? 'docker/Dockerfile.target-machine-docker' : 'docker/Dockerfile.target-machine'); const machineVolumes = ['ssh-keys:/ssh-keys:ro']; // Mount the host image cache into Docker-capable machines so the // docker-image-preload service can load them at startup without pulling. if (needsDocker) { machineVolumes.push('./docker-image-cache:/docker-image-cache:ro'); } services[machine.name] = baseService({ dockerfile, networks: { [zoneNetwork]: { ipv4_address: machine.ip } }, privileged: true, volumes: machineVolumes, environment: { GATEWAY: gateway }, tmpfs: ['/run', '/run/lock', '/tmp'], }); } // --- Optional DHCP client --- if (config.dhcpClient) { services['dhcp-client'] = baseService({ dockerfile: 'docker/Dockerfile.dhcp-client', networks: { internal: { ipv4_address: zoneIp('internal', 210) } }, cap_add: ['NET_ADMIN'], }); } // --- Optional real signal-cli daemon --- // On `internal`, matching the signal module's declared zone — it is a local // service celilo talks to, not an internet-facing one, so it does not belong // on internet-external. Unlinked: the JSON-RPC surface needs no account. if (config.signalCli) { services['signal-cli'] = baseService({ dockerfile: 'docker/Dockerfile.signal-cli', networks: { internal: { ipv4_address: zoneIp('internal', 90) } }, cap_add: ['NET_ADMIN'], }); } // --- Optional signal-cli release host --- // On internet-external because that is what it models: an artifact the // module downloads FROM the internet at deploy time. if (config.signalRelease) { services['signal-release'] = baseService({ dockerfile: 'docker/Dockerfile.signal-release', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.SIGNAL_RELEASE } }, cap_add: ['NET_ADMIN'], }); } // --- Optional signal-cli simulator --- // Same network placement as the real daemon so a test can swap one for the // other without touching the module's configured endpoint. if (config.signalSim) { services['signal-sim'] = baseService({ dockerfile: 'docker/Dockerfile.signal-sim', networks: { internal: { ipv4_address: zoneIp('internal', 91) } }, cap_add: ['NET_ADMIN'], environment: { SIGNAL_ACCOUNT: '+15551234567', SIGNAL_KNOWN_RECIPIENTS: '+15550001,+15550002', // Mirrors the flag the module's systemd unit must pass. Without it the // simulator refuses `receive`, exactly as the real daemon does — which // is the point: a test may only read replies from a daemon started the // way celilo has to deploy it. SIGNAL_RECEIVE_MODE: 'manual', }, }); } // --- Observer vantages (passive spies; ISS-0117) --- // Each is the SAME image at a different network location with a routing profile that // mirrors a real device there — that routing profile is what makes it a faithful seat // to assert from, rather than the all-VLAN management trap. for (const spec of normalizeObservers(config.observers ?? [])) { const p = observerPlacement(spec.vantage); services[p.service] = baseService({ dockerfile: 'docker/Dockerfile.observer', networks: { [p.network]: { ipv4_address: p.ip } }, cap_add: ['NET_ADMIN'], // A vantage may dial a VPN and probe from INSIDE the tunnel — the only way // to assert VPN reach by real signal instead of by reading a rule string. devices: ['/dev/net/tun'], environment: observerEnv(p), }); } const compose = { networks, volumes, services }; return stringify(compose, { lineWidth: 120 }); } // --------------------------------------------------------------------------- // Legacy: monolithic compose (kept for standalone/fallback use) // --------------------------------------------------------------------------- export function generateComposeYaml(config: NetworkConfig, celiloRoot = '..'): string { const networks: Record = { internal: zoneNetworkDef('internal'), dmz: zoneNetworkDef('dmz'), app: zoneNetworkDef('app'), secure: zoneNetworkDef('secure'), // The proxmox simulator lives on secure-mgmt, so enabling it has to bring // the network with it — a service on a network the file never declares is // a compose error at `up`, not a missing feature at deploy. ...(config.proxmoxSim ? { 'secure-mgmt': zoneNetworkDef('secure-mgmt') } : {}), 'isp-external': networkDef('203.0.113.0/24', '203.0.113.250'), 'internet-external': networkDef('100.64.0.0/24', '100.64.0.250'), 'real-internet': { driver: 'bridge', ipam: { config: [{ subnet: '172.30.0.0/24' }] }, }, }; const volumes: Record = { 'ssh-keys': { driver: 'local' }, }; const services: Record = {}; const managementDefaultGw = config.topology === 'direct-internet' ? ZONE_GATEWAYS.internal : greenwaveRouterIp(); services.management = baseService({ dockerfile: 'docker/Dockerfile.management', networks: { internal: { ipv4_address: zoneIp('internal', 100) } }, cap_add: ['NET_ADMIN'], volumes: [ ...(celiloRoot ? [`${celiloRoot}:/celilo`] : []), 'ssh-keys:/ssh-keys', ...(config.managementVolumes ?? []), ], environment: { DEFAULT_GATEWAY: managementDefaultGw, CELILO_REGISTRY_URL: 'http://e2e-registry.lab', // The hook jail is REQUIRED in the e2e, not `auto` (peba, 2026-09-08). // ce-rez7 made an unset policy resolve to `off`, which silently turned // hook-jail-trespass into a test of nothing: its stage 2 (no policy set) // and its stage 3 control (explicit `off`) became the same experiment. // celilo#1329. `required` beats `auto` here because a missing bubblewrap // backend is then a hard failure instead of a silent drop to unjailed, so // it cannot rot the same way twice. A per-command `CELILO_HOOK_JAIL=off` // prefix still overrides this, which is how the deliberate unjailed // controls in both jail suites keep working. CELILO_HOOK_JAIL: 'required', }, }); const topologyFn = TOPOLOGY_SERVICES[config.topology] || TOPOLOGY_SERVICES.default; Object.assign(services, topologyFn(config)); const legacyCaddyIp = config.dmzMachines.find((m) => m.name === 'caddy')?.ip ?? config.dmzMachines[0]?.ip ?? zoneIp('dmz', 10); services['dns-int'] = baseService({ dockerfile: 'docker/Dockerfile.internal-resolver', networks: { internal: { ipv4_address: internalResolverIp() } }, cap_add: ['NET_ADMIN'], environment: { CADDY_IP: legacyCaddyIp, DOMAIN: config.domain ?? 'iamtheinternet.org', DEFAULT_GATEWAY: managementDefaultGw, }, }); services['fw-ext'] = baseService({ dockerfile: 'docker/Dockerfile.router', networks: { 'isp-external': { ipv4_address: '203.0.113.101' }, 'internet-external': { ipv4_address: '100.64.0.1' }, 'real-internet': { ipv4_address: '172.30.0.5' }, }, cap_add: ['NET_ADMIN', 'SYS_ADMIN'], sysctls: ROUTER_SYSCTLS, }); services['comcast-resolver'] = baseService({ dockerfile: 'docker/Dockerfile.resolver', networks: { 'isp-external': { ipv4_address: '203.0.113.1' }, 'real-internet': { ipv4_address: '172.30.0.4' }, }, cap_add: ['NET_ADMIN'], }); services['root-dns'] = baseService({ dockerfile: 'docker/Dockerfile.authoritative-dns', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.ROOT_DNS } }, cap_add: ['NET_ADMIN'], volumes: [ './config/dns/knot-root.conf:/config/knot.conf:ro', './config/dns/root.zone:/config/root.zone:ro', ], }); services['tld-dns'] = baseService({ dockerfile: 'docker/Dockerfile.authoritative-dns', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.TLD_DNS } }, cap_add: ['NET_ADMIN'], volumes: [ './config/dns/knot-tld.conf:/config/knot.conf:ro', './config/dns/com.zone:/config/com.zone:ro', './config/dns/org.zone:/config/org.zone:ro', './config/dns/net.zone:/config/net.zone:ro', ], }); services['namecheap-dns'] = baseService({ dockerfile: 'docker/Dockerfile.namecheap-dns', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.NAMECHEAP_DNS } }, cap_add: ['NET_ADMIN'], // Per-domain DDNS passwords for the cross-domain test. Fallback // `test123` still applies to any domain not listed. environment: { DDNS_PASSWORDS: '{"iamtheinternet.org":"test123","celilo.computer":"test456","example.net":"test789"}', }, volumes: [ './config/dns/knot-namecheap.conf:/config/knot.conf:ro', // Zone files are mounted READ-ONLY under /seed; namecheap-startup.sh // copies them into the writable /config that Knot + the DDNS simulator // mutate at runtime. This keeps DDNS/Knot zone rewrites container-local // instead of polluting the version-controlled fixtures (see // namecheap-startup.sh for the full rationale). './config/dns/iamtheinternet.org.zone:/seed/iamtheinternet.org.zone:ro', './config/dns/celilo.computer.zone:/seed/celilo.computer.zone:ro', './config/dns/example.net.zone:/seed/example.net.zone:ro', './config/dns/park-your-domain.com.zone:/seed/park-your-domain.com.zone:ro', './config/dns/tangohost.com.zone:/seed/tangohost.com.zone:ro', ], }); // apt-cache simulator removed: target images no longer route apt // through a caching proxy. Tests apt direct upstream now, which // eliminates the cold-start race that bit every fresh-VM debug // cycle. See CLAUDE.md history for details. services.letsencrypt = baseService({ dockerfile: 'docker/Dockerfile.pebble', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.PEBBLE } }, cap_add: ['NET_ADMIN'], command: '-config /config/pebble-config.json -dnsserver 203.0.113.1:53', volumes: [ './config/pebble/pebble-config.json:/config/pebble-config.json:ro', './config/pebble/pebble-tls.crt:/config/pebble-tls.crt:ro', './config/pebble/pebble-tls.key:/config/pebble-tls.key:ro', ], }); // Registry stand-in. Build context is the @celilo/e2e package root — // Dockerfile.registry COPYs from `registry-server/` which lives inside // the package (bundled at build time from packages/registry-server, // shipped in the npm tarball). See packages/e2e/src/registry-bundle.ts. services.registry = baseService({ dockerfile: 'docker/Dockerfile.registry', networks: { 'internet-external': { ipv4_address: SIMULATOR_IPS.REGISTRY } }, cap_add: ['NET_ADMIN'], volumes: getRegistryVolumes(), }); if (config.proxmoxSim) { services['proxmox-sim'] = proxmoxSimService(); } const allMachines = getAllMachines(config); for (const machine of allMachines) { const zoneNetwork = machine.zone; const gateway = ZONE_GATEWAYS[machine.zone]; const needsDocker = machine.zone === 'app' || machine.docker === true; const dockerfile = machine.dockerfile || (needsDocker ? 'docker/Dockerfile.target-machine-docker' : 'docker/Dockerfile.target-machine'); const machineVolumes = ['ssh-keys:/ssh-keys:ro']; if (needsDocker) { machineVolumes.push('./docker-image-cache:/docker-image-cache:ro'); } services[machine.name] = baseService({ dockerfile, networks: { [zoneNetwork]: { ipv4_address: machine.ip } }, privileged: true, volumes: machineVolumes, environment: { GATEWAY: gateway }, tmpfs: ['/run', '/run/lock', '/tmp'], }); } if (config.dhcpClient) { services['dhcp-client'] = baseService({ dockerfile: 'docker/Dockerfile.dhcp-client', networks: { internal: { ipv4_address: zoneIp('internal', 210) } }, cap_add: ['NET_ADMIN'], }); } const compose = { networks, volumes, services }; return stringify(compose, { lineWidth: 120 }); } export function getAllMachines(config: NetworkConfig): MachineSpec[] { return [ ...config.dmzMachines, ...config.appMachines, ...config.secureMachines, ...config.internalMachines, ...(config.secureMgmtMachines ?? []), ]; }