# Celilo E2E Test Environment A Docker-based simulated internet for end-to-end testing of Celilo's full deployment flow: importing modules, configuring firewalls, registering DNS, deploying services via Ansible, and obtaining TLS certificates via ACME. ![Network Diagram](docs/network-diagram.svg) ## Why This Exists Unit and integration tests validate individual components, but can't test the real interactions between SSH, Ansible, iptables, DNS resolution, and ACME certificate issuance. The E2E environment simulates a complete internet with real routing, a real DNS hierarchy, and service simulators that behave like Namecheap, the ISP router, and Let's Encrypt. Celilo runs inside this simulated network and deploys to Docker containers as if they were real machines. From Celilo's perspective, it's deploying to a real home lab with a real ISP and real internet services. ## Network Architecture The network consists of 7 Docker bridge networks simulating a realistic home lab topology: | Network | Subnet | Purpose | |---------|--------|---------| | **internal** | 10.226.1.0/24 | Home LAN (management, firewalls) | | **dmz** | 10.226.10.0/24 | Public-facing services | | **app** | 10.226.20.0/24 | Internal applications | | **secure** | 10.226.30.0/24 | Sensitive services | | **isp-external** | 203.0.113.0/24 | ISP network (between home and internet) | | **internet-external** | 100.64.0.0/24 | Simulated internet (DNS, ACME, etc.) | | **real-internet** | 172.30.0.0/24 | Bridge to actual internet (for apt, pip, etc.) | All networks except `real-internet` have Docker IP masquerade **disabled** — traffic flows through explicit iptables routing, not Docker's NAT. The `real-internet` network has masquerade enabled to provide actual internet access for package downloads. ### Why these addresses (#539) Every simulated **private** zone lives inside `SIM_PRIVATE_SUPERNET` (`10.226.0.0/16` — mnemonic `0xE2` = "e2"), declared once in `src/types.ts` and derived from there by the compose generator, the zone classifier and the harness's `system init`. Two reasons, both load-bearing: 1. **A leaked stack must not be able to collide with a real fleet.** The sim used to number its zones exactly like a production celilo deployment; on celilo's own forgejo-builder, which lives in the real dmz, a leaked bridge claimed the builder's own /24 and blackholed every containerized CI job's route to the forge. Teardown cannot be made reliable enough to prevent that (a SIGKILL runs no handler), so the sim moved instead. 2. **Nothing may be magic about production's octets.** A suite that only passes on one site's address plan is asserting an accident, not celilo's behaviour. Don't hardcode an octet: use `zoneIp(zone, host)` / `ZONE_SUBNETS` / `ZONE_GATEWAYS`. `src/address-plan.test.ts` fails if a zone leaves the supernet, or if any retired fleet prefix reappears anywhere in the suite. ## Machines ### Fixed Infrastructure (always present) | Machine | Role | Networks | IPs | |---------|------|----------|-----| | **management** | Celilo CLI, Ansible, SSH | internal | 10.226.1.100 | | **fw-main** | iptables firewall (managed by Celilo) | internal, dmz, app, secure | 10.226.1.254, 10.226.10.1, 10.226.20.1, 10.226.30.1 | | **fw-isp** | Greenwave router simulator | internal, isp-external | 10.226.1.1, 203.0.113.100 | | **fw-ext** | Edge router + transparent HTTPS proxy (Squid) | isp-external, internet-external, real-internet | 203.0.113.101, 100.64.0.1, 172.30.0.5 | | **comcast-resolver** | Unbound recursive DNS resolver | isp-external, real-internet | 203.0.113.1, 172.30.0.4 | | **root-dns** | Knot authoritative DNS (root zone) | internet-external | 100.64.0.53 | | **tld-dns** | Knot authoritative DNS (.com, .org TLDs) | internet-external | 100.64.0.54 | | **namecheap-dns** | Knot DNS + DDNS API simulator | internet-external | 100.64.0.55 | | **letsencrypt** | Pebble ACME test server | internet-external | 100.64.0.100 | | **apt-cache** | apt-cacher-ng package proxy | internet-external, real-internet | 100.64.0.2, 172.30.0.2 | ### Dynamic Test Machines Added per-test via the `network()` builder API. Each is an Ubuntu 22.04 container running systemd, with SSH server, placed on the appropriate zone network. | Example | Network | IP | Purpose | |---------|---------|-----|---------| | caddy | dmz | 10.226.10.10 | Caddy reverse proxy deployment | | idp | app | 10.226.20.100 | Identity provider deployment | | db | secure | 10.226.30.50 | Database deployment | ## Routing Traffic flows through explicit routing chains, mimicking a real network: | From | To | Path | |------|----|------| | management | dmz/app/secure | via fw-main (10.226.1.254) | | management | internet | via fw-isp (10.226.1.1) → fw-ext | | dmz machines | internet | via fw-main → fw-isp → fw-ext | | Pebble | caddy (HTTP-01 challenge) | via fw-ext → fw-isp DNAT → fw-main DNAT → caddy | All firewall/router containers run `ip_forward=1` and `MASQUERADE` on their outbound interfaces. ## DNS The DNS system is a hybrid of simulated and real resolution: **Simulated domains** (handled by the E2E DNS hierarchy): - `iamtheinternet.org` — stub zone to namecheap-dns (100.64.0.55) - `park-your-domain.com` — stub zone to namecheap-dns (Namecheap DDNS API) - `acme-v02.api.letsencrypt.org` — local-data in Unbound pointing to Pebble (100.64.0.100) **Real domains** (forwarded to actual DNS): - Everything else (e.g., `dl.cloudsmith.io`, `deb.debian.org`) is forwarded by Unbound to `8.8.8.8` and `1.1.1.1` via the `real-internet` bridge. This means Celilo's modules install real packages from the internet while all domain-specific operations (DDNS, ACME) go through the simulation. ## Transparent HTTPS Proxy A Squid proxy with SSL bumping runs on fw-ext, transparently intercepting outbound HTTPS traffic from the simulated network. This lets target machines `curl https://dl.cloudsmith.io` (to install Caddy from its apt repo) without explicit proxy configuration. **How it works:** 1. iptables REDIRECT rules on fw-ext's ISP interface capture port 80/443 traffic 2. Exception: traffic to `100.64.0.0/24` (simulated services) passes through directly 3. Squid does SSL bump (MITM) with a pre-shared CA trusted by all machines 4. Squid resolves DNS via real public DNS and fetches from the real internet ## Simulators ### Namecheap Dynamic DNS Runs on `namecheap-dns` (100.64.0.55). A Bun HTTP server on port 8080 that implements the Namecheap DDNS API: ``` GET /update?host=&domain=&password=&ip= ``` When called, it updates the Knot DNS zone file and signals a zone reload. DNS propagates through the full chain (namecheap-dns → tld-dns → root-dns → comcast-resolver). ### Greenwave Router Runs on `fw-isp` (10.226.1.1). A Bun HTTPS server implementing the C4000XG REST API subset: - `POST /cgi/cgi_action` — Login/logout - `GET /cgi/cgi_get` — Read config (public IP, port mappings) - `POST /cgi/cgi_set` — Add/remove port forwarding rules Port forwarding rules are applied via actual iptables commands, making NAT functional in the simulated network. The management HTTPS interface binds to the internal interface only (10.226.1.1), not the external interface. ### Pebble (Let's Encrypt) Uses the official Pebble ACME test server, wrapped in a custom Dockerfile that adds routing. Pebble uses the comcast-resolver for DNS and validates HTTP-01 challenges through the full routing chain. The Pebble TLS certificate includes `acme-v02.api.letsencrypt.org` as a SAN, so Caddy's default ACME configuration works with zero changes (DNS resolves the real Let's Encrypt hostname to Pebble). fw-ext automatically fetches Pebble's runtime ACME root CA from its management API (`https://pebble:15000/roots/0`) at startup, so `curl` from fw-ext can verify Caddy's ACME-issued certificates. ## Manual Usage ### Starting the Environment ```bash cd e2e # Infrastructure only (no target machines) ./bin/e2e-up # With a caddy machine in the DMZ ./bin/e2e-up --caddy # Full stack (caddy + idp + db) ./bin/e2e-up --full-stack # Custom machine spec ./bin/e2e-up --custom '{"dmz":{"caddy":"10.226.10.10","web":"10.226.10.20"}}' ``` After startup, you're dropped into the management machine shell with tab completion for `celilo` commands (aliased as `c`). ### Accessing Machines ```bash # Reconnect to management (default) ./bin/e2e-shell # Shell into any container ./bin/e2e-shell caddy ./bin/e2e-shell fw-main ./bin/e2e-shell fw-ext ./bin/e2e-shell namecheap-dns ./bin/e2e-shell letsencrypt ``` Management uses `zsh` (with celilo aliases). All other containers use `bash`. ### Running the Manual Test Script From the management shell (`./bin/e2e-shell` or after `./bin/e2e-up`): ```bash cd /celilo/modules # System init c system init --accept-defaults \ network.dmz.subnet=10.226.10.0/24 \ network.app.subnet=10.226.20.0/24 \ network.secure.subnet=10.226.30.0/24 \ network.internal.subnet=10.226.1.0/24 \ primary_domain=iamtheinternet.org \ admin.email=admin@iamtheinternet.org \ dns.primary=203.0.113.1 \ dns.fallback=1.0.0.1,8.8.8.8 # Import all modules c module import namecheap c module import greenwave c module import iptables c module import caddy # Add machines (--ssh-user root needed for non-interactive) c machine add 10.226.1.1 --ssh-user root --earmark greenwave c machine add 10.226.1.254 --ssh-user root --earmark iptables c machine add 10.226.10.10 --ssh-user root # Pre-configure secrets and non-derivable config c module config set namecheap domains '["iamtheinternet.org"]' c module secret set namecheap ddns_passwords '{"iamtheinternet.org":"test123"}' c module config set greenwave router_ip 10.226.1.1 c module secret set greenwave router_username admin c module secret set greenwave router_password admin c module config set iptables nat_ip 10.226.1.253 c module config set caddy hostname www c module config set caddy acme_ca https://acme-v02.api.letsencrypt.org/dir # Deploy (iptables auto-derives config from earmarked machine) c module deploy namecheap --no-interactive c module deploy greenwave --no-interactive c module deploy iptables c module deploy caddy --no-interactive ``` ### Verifying the Deployment From fw-ext (the "internet" side): ```bash ./bin/e2e-shell fw-ext curl -s https://www.iamtheinternet.org # Expected: "Caddy reverse proxy is running" ``` ### Checking Status ```bash ./bin/e2e-status # Shows containers, DNS, connectivity ``` ### Tearing Down ```bash ./bin/e2e-down # Stop and remove everything ./bin/e2e-down --keep # Stop but keep volumes (faster restart) ``` ## Writing Automated Tests Tests use Vitest and the `NetworkBuilder` API to start isolated networks, run Celilo commands, and verify results. ### Test Structure ```typescript import { afterAll, describe, expect, it } from 'vitest'; import { CADDY_DEPLOYMENT } from '../src/fixtures'; import type { NetworkHandle } from '../src/types'; describe('my deployment test', () => { let net: NetworkHandle; afterAll(async () => { await net?.stop(); // Always clean up }); it('deploys and verifies', async () => { // 1. Start the network with desired machines net = await CADDY_DEPLOYMENT().start(); // 2. Add machines, import modules, configure await net.celilo('machine add 10.226.10.10 --ssh-user root'); await net.celilo('module import /celilo/modules/caddy'); await net.celilo('module config set caddy hostname www'); // 3. Deploy const result = await net.celilo('module deploy caddy --no-interactive'); expect(result.exitCode).toBe(0); // 4. Verify from any container const curl = await net.exec('fw-ext', 'curl -s https://www.iamtheinternet.org'); expect(curl.stdout).toContain('Caddy'); }, 300_000); // Per-test timeout }); ``` ### NetworkHandle API | Method | Description | |--------|-------------| | `celilo(cmd, timeout?)` | Run a celilo CLI command on the management machine | | `exec(container, cmd, timeout?)` | Execute a command in any container | | `dig(name)` | Resolve a DNS name from management | | `waitFor(check, timeout, label)` | Poll until a condition is true | | `debug(container?)` | Pause the test and shell into a container (default `management`); exit resumes ([details](#mid-test-pause)) | | `socksProxy(opts?)` | Spawn a SOCKS5 proxy attached to the test network ([details](#browser-driven-tests)) | | `browser(opts)` | Launch a host-side browser routed into the network ([details](#browser-driven-tests)) | | `stop()` | Tear down the entire network | ### Mid-test pause `net.debug()` freezes a running suite at the line that calls it and hands you a bash shell inside the named container (default `management`). Type `exit` and the test resumes. In a non-interactive run it instead prints `Resume: touch ` and waits, so you can probe the stack from another terminal and resume by touching the file. This shows live state at the exact failing moment, which `--keep` cannot: a kept stack is only available after the suite finished, once later stages have already mutated it. The pause holds the machine-global run lock for up to 24 hours, so a forgotten debug session blocks every other session's run. Resume promptly. See `docs/RUNNING_CELE2E_TESTS.md` for the full write-up. ### Browser-driven tests `net.browser()` returns a Playwright `Browser` whose traffic enters the simulated network through a containerized SOCKS5 proxy, so a host-side browser sees the deployed app exactly as a real user would (DNS, TLS, ingress firewall, all in the loop). **Bring your own playwright.** `@celilo/e2e` has zero playwright dependency — the consuming module imports its own `chromium` (or `firefox`/`webkit`) and passes it in. This keeps version alignment between the playwright runtime and the cached browser binaries entirely under the consumer's control. ```typescript import { chromium } from 'playwright-core'; // or: import { chromium } from '@playwright/test'; const handle = await net.browser({ vantage: 'isp-external', chromium }); const browser = handle.browser as import('playwright-core').Browser; const ctx = await browser.newContext({ ignoreHTTPSErrors: true }); const page = await ctx.newPage(); await page.goto('https://www.iamtheinternet.org/'); // ... assertions ... await handle.close(); // tears down browser + proxy ``` **Vantage** controls which simulated zone the proxy attaches to: - `'isp-external'` (default) — residential-user view via comcast-resolver. Exercises the full public ingress path (DNS publication, fw-ext DNAT, ACME, Caddy host routing). - `'internal'` — operator-on-LAN view. Uses the internal split-horizon resolver to reach caddy directly via the internal NAT IP. Useful for debugging. For manual operator sessions (no browser, just SOCKS), use `net.socksProxy(opts)` directly. The published port is logged so you can configure Firefox/Chrome to use `socks5://127.0.0.1:` with remote DNS. **Consumer-side setup.** Each consuming module is responsible for: 1. **Picking a playwright version.** If your module already uses `@playwright/test`, add `playwright-core` as a devDependency at the same major.minor — both should track the same chromium build, otherwise the playwright runtime asks for a chromium build the cache doesn't have. 2. **Installing the chromium binary.** Once per host (or after a playwright bump): ```bash bunx playwright install chromium ``` The proxy container itself is built from `docker/Dockerfile.socks-proxy` in this package — eagerly by `e2e-build`, or lazily on first `net.browser()` call. No setup needed for that piece. ### Fixtures Pre-built network configurations in `src/fixtures.ts`: ```typescript CADDY_DEPLOYMENT() // caddy machine in DMZ FULL_STACK() // caddy (dmz) + idp (app) + db (secure) INFRASTRUCTURE_ONLY() // no dynamic machines ``` ### Custom Network Configurations ```typescript import { network } from '../src/network-builder'; const net = await network() .dmz({ caddy: '10.226.10.10', web: '10.226.10.20' }) .app({ api: '10.226.20.100' }) .secure({ db: '10.226.30.50' }) .start(); ``` ### Running Tests The repo's own tests live in `e2e/tests/` at the repo root. They resolve `@celilo/e2e` through the root workspace install, so there is nothing to install in `e2e/` itself. ```bash # From the repo root, after `bun run setup`: cele2e run # Docker-backed suite, e.g. `cele2e run machine-add` cele2e run --all # every module suite + the top-level e2e/tests/ ``` See `docs/RUNNING_CELE2E_TESTS.md` for the full operator guide (run lock, build-infra triggers, `--keep` debugging). ## Key Design Decisions **Systemd on target machines:** Target machines run systemd as PID 1 (privileged mode) so Ansible's `systemd` module works naturally. Network setup runs as a oneshot systemd service at boot. **No test parallelism:** Fixed network CIDRs mean only one test network can run at a time. Tests run sequentially via Vitest's `singleFork` config. **Bind-mounted source:** The celilo source tree is bind-mounted at `/celilo` on the management container, which is where module sources are read from (`celilo module import /celilo/modules/`). The CLI itself is NOT read from there. Tests run the `celilo` baked into the management image by `cele2e build-infra` — the artifact install.sh produces — so **a change under `apps/celilo/` or `packages/` needs a `build-infra` before it reaches a test.** `cele2e doctor` compares a fingerprint stamped on the image against the working tree and warns when they differ, and `cele2e run --source-cli` runs the mounted source instead for iteration. Running the source roughly doubles each celilo command's start-up (measured `celilo --version`: 0.14s installed against 0.31s from the mount on virtiofs, 0.16s against 0.56s on sshfs). **Docker image caching:** Images are built once and cached. Only config/simulator changes require rebuilds. A run skips `docker compose build` entirely when every image it needs already exists — that check costs one `docker images` listing, against about 0.7s per service for a build that rebuilds nothing. **ACME via DNS interception:** Instead of overriding Caddy's ACME URL, `acme-v02.api.letsencrypt.org` resolves to Pebble (100.64.0.100) in the simulated DNS. The caddy module's `acme_ca` variable allows pointing to Pebble's `/dir` endpoint (vs Let's Encrypt's `/directory`). ## Directory Structure ``` e2e/ bin/ e2e-up # Start interactive network e2e-down # Tear down network e2e-shell # Shell into containers e2e-status # Show network status config/ dns/ # Knot zone files and configs pebble/ # Pebble ACME config and TLS certs proxy/ # Squid transparent proxy config resolver/ # Unbound recursive resolver config routing/ # Per-container routing scripts ssh/ # SSH key generation docker/ Dockerfile.* # Container images docs/ network-diagram.svg # Network topology diagram simulators/ greenwave/ # C4000XG router REST API simulator namecheap-ddns/ # Namecheap DDNS API simulator src/ container-manager.ts # Docker compose orchestration docker-compose-generator.ts # Generates compose YAML fixtures.ts # Pre-built network configs network-builder.ts # Fluent API for network setup types.ts # Shared type definitions tests/ caddy-deploy.test.ts # Full caddy deployment with HTTPS smoke.test.ts # Basic connectivity verification ... ``` ## Suite health policy (e2e-confidence) The suite must never be tolerated as ambient-red — a regression hiding in chronic red is how the install-sh DNS-pollution bug survived for weeks. Two rules: - **No ambient red.** A flaky or failing test is either *fixed* or *quarantined with a tracking issue*, never left standing-red. Quarantine by adding a `cele2e-ci-unsafe` marker comment to the test file (with the issue id); the nightly runs `--ci-safe`, which skips quarantined tests, so it stays green on the supported set. `--complete` / `--all` run everything including quarantined. - **Randomized-order lane.** Order-dependent bugs (a test polluting shared infra for a *later* test) pass in fixed order and solo. The weekly `cele2e-shuffle` workflow runs `--ci-safe --shuffle` to flush them out. The shuffle is seeded and the seed is logged; replay a failing order exactly with: ```bash cele2e run --ci-safe --shuffle --seed= ``` Supporting guards (so a future bug is caught at the cause, not 15 tests later): the shared-DNS reset verifies its own postcondition, the namecheap-ddns sim refuses overwrites of protected seeded records, the runner names any test that leaves shared DNS polluted, and readiness waits attach live-state diagnostics on timeout instead of speculating.