# Product integration testing on the DooD runners

Guidelines for writing (and debugging) a split product's slow-tier integration
suite against the org's self-hosted CI. These are hard-won: the failures below
_look_ like license, networking, or engine bugs but are almost always
**runner/harness setup** — mounts, ports, hosts, image provenance. Chasing them
as product bugs wastes time.

The reference implementations are the `norsk-ctl-product-playout` and
`norsk-ctl-product-commentary` harnesses; the shared helpers live in
`@norskvideo/ctl-test-harness`.

## Runner topology (docker-outside-of-docker)

The CI runner is itself a **container**. The daemon (`norsk-ctl`) and the product
backend are spawned **in-process inside that container**. Studio + the Norsk
media engine launch as **host siblings** (the runner shares the host docker
socket), with their ports **published on the host**.

Consequence — two address classes the harness must not confuse:

| Reaching…                            | Address                                                                                               |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| the daemon, the product backend      | `localhost:<port>` (in-process)                                                                       |
| studio / media host-published ports  | `${NORSK_TEST_HOST}:<port>`                                                                           |
| a launched container by compose name | `<instance>-<service>-1` on its **instance network** (`<instance>_default`); studio also on norsk-net |

`NORSK_TEST_HOST` is `host.docker.internal` in CI and unset (→ `localhost`)
locally. Set it in the workflow env. The shared `studio-state` fetchers
(`fetchComponents`, `fetchComponentState`, …) already honour it; **any local
support code that hits a studio/media port must route through a `TEST_HOST`
helper** (`process.env.NORSK_TEST_HOST ?? "localhost"`), never a hardcoded
`localhost`. Daemon/backend fetches stay on `localhost`.

### A studio host port binds loopback unless you ask otherwise

norsk-ctl gives the **`studio`** service's compose ports (`STUDIO_HOST_PORT`) an
explicit `127.0.0.1` bind at launch — docker publishes an unbound port on every
interface and its rules sit ahead of ufw. No other service's ports are touched.

A loopback bind is **unreachable via `host.docker.internal`**. So in the DooD
runner, one of these must hold:

- `NORSK_TEST_NET=direct` — the preferred answer, and what the canonical
  `integration.yml` / `smoke.yml` / `build-docs.yml` topology step already sets.
  Instances are reached by compose name at the **container** port, so no host
  publish is involved at all. Studio is on `norsk-net`; **media is on its
  instance network `<instance>_default` only** (Docker Desktop delivers published
  UDP to either IP of a dual-homed container and breaks SRT replies, see
  norsk-ctl `docs/product-template-format.md`, Networking). The harness's media helpers
  (`mediaHttpBase`, `srtEgressUrl`, `mediaDirectUrl`) join the runner to that
  network for you, and `cleanupDaemon` leaves it before deleting the instance.
- the launch passes `--publish-debug-ports`, restoring the old binding. The
  shared `runProductSmoke` and demo drivers do this for you when the host they
  fetch from is not loopback, and only after probing that the resolved ctl
  advertises the flag (the CLI is `.strict()`, and a product pins a released ctl
  independently of the harness). A product's own launch code must do both.

A harness reaching a studio port at `${NORSK_TEST_HOST}` while neither holds
will see every studio gate time out with connection refused.

## The host docker daemon is shared between jobs

The x64 runners are sibling containers on ONE host docker daemon, so anything
named globally is shared by every concurrent job, whichever repo it belongs to.
On 2026-10-02 a reuters tier and a commentary tier each `compose up`ed
`norsk-proxy` over the other's, and reuters' teardown removed it mid-test.

- `startDaemon` (and so every harness built on it) names the daemon's proxy and
  cpu-monitor sidecar after its port (`norsk-proxy-<port>`,
  `norsk-ctl-cpu-monitor-<port>`). Don't override them back to the defaults.
- An image your suite builds must carry a **per-run tag** (e.g.
  `<repo>:ci-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT`), not `:dev` — a concurrent run
  can replace a fixed tag between your build and your `product add`. `docker rmi`
  it at teardown, or the host fills up.
- The names above carry the store dir as well as the port: two repos' slugs can
  hash into the same port band, so a port-only name is not unique.
- Still box-global and NOT namespaced: `norsk-net`, and a registered image
  product's control-plane container NAME `norsk-product-<name>`. Its host port
  is already docker-assigned (`127.0.0.1::4321`, read back by `docker inspect`);
  older text here and in the reap steps that names a fixed 14321 is stale. Two
  concurrent runs of the SAME product that both `product add --image` still
  collide on the name.
- The daemon's port and its nginx proxy's host port are assigned by the system
  when the ctl under test advertises the `ephemeral-ports` daemon capability
  (`norsk-ctl capabilities`, or `daemonCapabilities` in `/api/health`). The
  harness then starts the daemon with `NORSK_CTL_PORT=0` and `init --proxy-port
  0`: the kernel picks the daemon's port, docker picks the proxy's host port
  (nginx listens on 443 inside the container), and the daemon publishes both as
  `<store>/daemon.port` and `<store>/proxy.port` (also `/api/proxy/status`).
  Neither can collide. `runProductSmoke`, `runDemo`, `demo check --export-only`
  and the dev-loop do this on their own. A `ProductHarness` you construct does
  it only with `ephemeralPorts: true`, because your harness most likely keeps
  its own `demoPorts(slug)` copy: switch every daemon/proxy port you read to
  `harness.ports` (or `session.proxyBase`) first, then turn it on.
- Against a ctl without the capability (the released `latest` until it catches
  up) both stay on the slug's band (`daemon`, `daemon+3`), exactly as before,
  and can still collide.
- Still banded either way: the studio host port a launch publishes
  (`daemon+2`, and `daemon+3+n` for further launches), and the dev backend's
  port (bound inside the runner container, so not shared). In `direct` reach
  mode with `--internal-only` the studio port is not published at all.
- A test daemon runs WITHOUT the node MoQ relay (`NORSK_CTL_MOQ_RELAY_ENABLE=0`):
  the daemon never removes its relay, so a test one would leave it behind. A
  suite that needs it (MoQ auth end to end) passes `moqRelay: { auth:
  "required" }` to its `ProductHarness`: the private daemon gets a relay named
  per run (`norsk-moq-relay-<slug>-<hash>`) on three free host ports, with
  `moqRelayAuth` set through `config set` before `serve`, and teardown removes
  the relay and its `-certs` volume. `harness.moqRelay` names the container and
  `statusUrl` (`/relay/info`) for the test's evidence.

### One suite per host

Names can be made unique; CPU cannot. A runner container is pinned to its own
cores, but the media containers its suite launches are host siblings and run
anywhere, so two suites on one host starve each other's engines. On 2026-10-15/16
commentary's timing tests (ears, realtime drift, soak preview) failed in 10 of the
12 test2 runs that overlapped another suite, most often a second commentary run on
the other runner. The canonical `integration.yml` therefore runs its suite under
an `flock` on `/srv/ci-scratch/locks/norsk-ctl-integration.lock`:
`/srv/ci-scratch` is bind-mounted at that path into every jankings runner on a
host, so the file is shared by that host's runners and by no other host's. A
runner without the mount runs unserialised and says so in the log; a suite that
has waited an hour runs anyway with a warning. A hand-owned `integration.yml`
does not take the lock until it converges onto the canonical.

### Contention verdicts: a starved host is not a red

The lock stops two suites starving each other, but not everything else on the
host. On test2, commentary has failed with browser main threads starved for
6-80% of a test window, the engine 50 s behind realtime, a 0.55 s switch taking
5-20 s, and jobs killed mid-run by SIGTERM. Those failures are the host's, not
the product's. A product can opt in to having the tier say so.

**Opting in.** Add `scripts/integration/contention.json` (and a
`@norskvideo/ctl-test-harness` new enough to ship `ctl-contention`). `{}` takes
every default; any key below overrides one, and a misspelt key is an error. With
the file present, the canonical workflow runs `bun run test:integration` through
`conventions/contention-suite.sh`, which runs it under the harness's
`ctl-contention run`:

- A sampler reads host pressure once a second for the whole suite: Linux PSI
  (`/proc/pressure/{cpu,memory,io}`, host-wide, falling back to the cgroup v2
  root's `*.pressure`), the load average and `MemAvailable`. The timeline is
  `pressure.jsonl` in the job's `integration-attempt-N` artifact.
- Each failing test's window is read off bun's own `(fail) ... [duration]` line
  as it streams past, so the product's `test:integration` script needs no
  reporter flag.
- When the suite fails and **every** failure overlapped contention, the run is
  **uncertain**: the job writes a `::warning::` and a job summary (the verdict,
  each failure with its numbers, pressure per minute), plus a
  `::notice title=ctl-contention-uncertain::` marker, then cancels its own run.
  A cancelled run is not reported to the CI board, and bot-prs merges only a
  success. One failure on a quiet stretch keeps the run red, as before.
- On a bot PR, bot-prs reads that marker off the cancelled job's annotations
  and treats the run as **not yet judged**: it re-runs it once a backoff has
  passed since the cancel (30 minutes, doubling per attempt), up to 4 attempts,
  then reports it as "still contended after N attempts: needs a human or a
  quieter runner", distinct from a failure. Repo variables
  `BOT_PRS_CONTENDED_ATTEMPTS` and `BOT_PRS_CONTENDED_BACKOFF_MINUTES` override
  the budget. A run cancelled for any other reason (a person, a concurrency
  group) is treated exactly as before.
- If the host is starved early in the suite, or for a long stretch anywhere, the
  suite is stopped there rather than run for 40 minutes to an uncertain end.
- An uncertain attempt is rerun in the same job (`rerun.max`, default 1), after
  waiting up to `rerun.maxWaitSec` for the host to go quiet.

**When nothing changes.** No `contention.json`: the suite runs exactly as it did.
No readable PSI (an old kernel, `psi=0`, a runner that cannot read it): the
summary says "pressure unavailable" and every verdict is the suite's own. A
harness without `ctl-contention`, or a config it cannot read: a warning, then the
suite runs as before. Every job prints a "Probe whether this runner can read
host pressure (PSI)" step whether or not the repo has opted in, so a runner's
first log answers whether the signal exists there.

**Defaults** (`classify`, judged on the failure's window widened by `marginSec`):

| Key                        | Default | Why                                                                                                                    |
| -------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------- |
| `classify.cpuSomeMean`     | 0.25    | cpu `some` bounds any one thread's wait from above; passing runs saw at most ~5% browser starvation, so 25% is 5x that |
| `classify.cpuSomePeak`     | 0.60    | over any `peakWindowSec` (10 s): a stall long enough to break a seconds-scale assertion that a long test's mean hides  |
| `classify.memSomeMean`     | 0.10    | reclaim as a constant cost of running                                                                                  |
| `classify.memFullPeak`     | 0.05    | half a second in any 10 in which nothing ran at all                                                                    |
| `classify.minCoverage`     | 0.5     | a window the sampler barely saw is not judged (and so stays red)                                                       |
| `classify.killWindowSec`   | 120     | a suite killed from outside is judged on its last two minutes                                                          |
| `abort.cpuSome`            | 0.70    | mean over the rule's window; with `memSome` 0.30 and `memFull` 0.10                                                    |
| `abort.earlyWindowSec`     | 60      | starved for a minute within the first `earlyWithinSec` (300): stop                                                     |
| `abort.sustainedWindowSec` | 300     | starved for five minutes anywhere: stop                                                                                |
| `rerun.max`                | 1       | reruns after an uncertain attempt; 0 turns them off                                                                    |

Calling a real failure uncertain costs a rerun; calling a starved one red costs a
human an investigation of nothing. The bars are set where the first is unlikely,
and they are config so test2's recorded timelines can move them.

**From a test.** `hostContendedDuring({ fromMs, toMs })` from
`@norskvideo/ctl-test-harness/contention` answers for any window of the running
suite (it reads the timeline the wrapper names in `NORSK_CONTENTION_TIMELINE`),
with the numbers. Outside the wrapper it answers "unavailable", never
"contended".

**Exit statuses** of `ctl-contention run`: 0 pass; the suite's own status on a
real failure; 75 (EX_TEMPFAIL) when uncertain, which the workflow believes only
alongside a `verdict.json` that says so.

**Caveats.** A `ctl-candidate` dispatch that is cancelled is still scored red by
release-collect (the gate is advisory). bot-prs re-runs an uncertain cancel
only while the annotation is readable with the job token (`checks: read`); if
it cannot read it, the cancel counts as red, as every cancel did before.

## Reaching an image product's control plane

`product add --image` loopback-publishes the control plane on a
docker-assigned host port (`127.0.0.1::4321`). On a DooD runner that loopback is the HOST's, so a
test that probes `localhost:<published port>` reaches nothing. With
`NORSK_TEST_NET=direct`, dial the container's own address
(`docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}'`)
on 4321 — the same thing the daemon does under
`NORSK_CTL_PRODUCT_REACH=container-address`.

## Host-shared working directory (or studio EACCESes on boot)

Studio bind-mounts its instance working directory as `${workdir}:/data` and runs
as a **non-root** auto-detected user. If the workdir is the daemon default
(`~/norsk-runtime`, container-only), the sibling mount materialises **root-owned**
and studio dies at boot with `EACCES: mkdir '/data/moq-certs'` — the workflow
never starts, the SRT graph never builds, and every downstream test times out
looking like a networking/license fault.

Fix: point the daemon at a **host-shared** workdir under the test store dir.
Pre-write `config.yaml` with `defaultWorkingDirectory` under `makeStoreDir(...)`
(which roots in the gitignored, host-shared `test-temp/`):

```ts
writeFileSync(
  join(storeDir, "config.yaml"),
  `networkMode: docker\ndefaultWorkingDirectory: ${JSON.stringify(join(storeDir, "norsk-runtime"))}\n`,
);
```

Docker still leaves **root-owned** read-only bind-mount target dirs inside that
workdir, which the non-root runner can't `rm`. So `teardown()` must nuke the
store dir from a throwaway **root** container, or `afterAll` (and the runner's
own checkout cleanup) blocks:

```ts
spawnSync("docker", ["run", "--rm", "--user", "0:0", "-v", `${dirname(storeDir)}:/base`,
  "--entrypoint", "sh", STUDIO_IMAGE, "-c", `rm -rf /base/${basename(storeDir)}`]);
```

## Reaching served media / HLS output

Studio reports playback URLs (HLS multivariant, WHIP endpoints) with the host it
was **configured with** — loopback locally, which the in-container test can't
reach. Two cases:

- **Proxied harness** — the URL is on the daemon's `publicHost` (default
  `:443`). Route the fetch through `applyTestHost(url)` (swaps a loopback host
  for `NORSK_TEST_HOST`). `assertMultivariantHasRenditions` does this by default.
- **Proxy-less harness** — the advertised `/instance/<id>/media/...` route is
  served **only by the daemon's nginx proxy**; with no proxy, nothing answers on
  `:443`. Reach the media container **directly** instead: `mediaDirectUrl(url)`
  rewrites it to `http://<id>-media-1:8080/<native-path>` (the same bypass
  `whip-driver` uses; the media route is unauthenticated) and joins the runner to
  the instance network `<id>_default`, where that name resolves. Pass it via
  `assertMultivariantHasRenditions`'s `resolveFetchUrl` hook. Anything else that
  addresses `<id>-media-1` by hand (a sibling container, a browser in the runner)
  must be on `<id>_default` too: `currentInstanceNetworks().join(id)` for the
  runner, `docker network connect <id>_default <container>` for a sibling, and
  leave or remove it before the instance is deleted, or `compose down` cannot
  remove the network.

## Capture servers (studio pushes → the runner)

When a test stands up an HLS/SCTE-35 capture server (or an SRT sink) that
**media pushes to**, media sits on its instance network only and can't reach a
containerised runner via the host LAN IP. Hand it the runner's address **on the
instance network**: `currentInstanceNetworks().addressOn(instanceId)` joins that
network and returns the runner's IP on it, or `null` when the runner is not in a
container (or the join failed). The harness ships the primitive, not the policy:
each product picks its own fallback for the `null` case, usually the address
media already reaches the host by.

```ts
import { currentInstanceNetworks } from "@norskvideo/ctl-test-harness/container-net";

function captureHost(instanceId: string): string {
  // hostLanIp is the product's own off-container fallback.
  return currentInstanceNetworks().addressOn(instanceId) ?? hostLanIp();
}
```

The runner leaves every network it joined during `cleanupDaemon`, before the
instance is deleted.

## Build the product image in the integration job

A `kind: dev` product still launches an instance whose compose references the
product's **image** (e.g. the runtime-screen `funke-backend` service on
`<product>:dev`). The `build-image` job builds it on a _different_ runner, so the
integration job must build it too — otherwise `docker compose up` tries to
**pull** the local-only tag and dies with `pull access denied` before any test
runs. Add `bun run build:image` inside the `nix develop .#build` block (docker is
reachable there); it also builds the dist the dev backend serves `/configure`
from.

## Stale node_modules on reused workspaces

The convention workflows now check out `clean: true` with test-temp routed to
`$RUNNER_TEMP`, but a hand-owned workflow on `clean: false` keeps
`node_modules` **between runs**, and `bun install --frozen-lockfile`
does **not** reliably relink a workspace-hoisted package whose version moved. A
stale `@norskvideo/ctl-test-harness@0.1.2` once survived a bump to `0.1.3`, so a
newly-added export was `SyntaxError: export 'X' not found` and the suite crashed
in <1s. The bun **global cache is integrity-checked** (it can't serve a wrong
version); only the persisted `node_modules` dir goes stale. Guard the install:

```yaml
rm -rf node_modules/@norskvideo/ctl-test-harness
bun install --frozen-lockfile
```

When a product pins the shared harness in **both** `package.json` **and**
`tests/package.json`, bump **both** — a lone root bump can leave the tests
workspace resolving the old version.

## GPU

Instances launch with `--hardware nvidia` and encode on **nvenc** on the runners
(the nvenc log-grep + no-software-fallback assertions pass). Note the
`/usr/bin/nvidia-smi` **binary** may be absent in the media container even when
nvenc works — assert on the encoder logs, not on `nvidia-smi` exit status.

## License

The org secret `NORSK_LICENSE_V2` + a modern engine accept the license (a green
`probe` run proves it). V2 licenses need a **post-2026-07-17** engine, and SDK
clients that create media nodes (e.g. a WHIP driver) must answer the license
**handshake** via `createLicenseChallengeResponder` or the engine aborts node
creation with `10 ABORTED: License handshake required`. The `license is marked
NON-PRODUCTION` warning is expected in CI, not an error. The media engine's
`hugepages / rte_eal_init failed` line is **benign noise** — do not chase it.

## The workflow: `integration.yml` is a convention

`.github/workflows/integration.yml` is single-sourced in this dev-kit
(`conventions/integration.yml`) and drift-gated like `smoke.yml`, with the same
two per-repo lines: the `product:` dashboard key and the matrix `runner:` label
array. Everything else a repo used to hand-edit into the workflow now lives in
three optional hooks the workflow calls if they are executable (and one opt-in
file, `scripts/integration/contention.json`, described under "Contention
verdicts" below):

| Hook                                   | When                              | What goes there                                                                           |
| -------------------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------- |
| `scripts/integration/prepull-extra.sh` | before the suite, outside nix     | `docker pull` of images the suite launches beyond media + studio (a driver, a sink)       |
| `scripts/integration/prepare.sh`       | before `bun run test:integration` | builds the suite needs on disk — a frontend dist the daemon fetches, an image it launches |
| `scripts/integration/verify.sh`        | after the suite                   | post-suite checks; absent, a repo with `tests/demo.spec.ts` gets `bun run demo -- check`  |

The hooks run in the **same workflow step** as the suite, so a hook that writes
to `$GITHUB_ENV` changes nothing the suite sees — GitHub applies that file only
to later steps. Commentary's enforced-path opt-out was dead for its whole life
that way. Hand the suite a value by having both sides derive it (a tiny script
the hook runs and the test imports), not through `$GITHUB_ENV`.

`bun run test:integration` is the product's claim of what its tier is, and
two things believe it: the repo's nightly bump PR runs it on a new ctl pin and
merges only when it is green, and ctl's release gate sends a `ctl-candidate`
dispatch that runs this workflow and shows the result on the CI board. The
gate is advisory — the result does not hold ctl's `latest` — so the bump PR
is what keeps a ctl that breaks this product out of it.

**Converging a repo.** The gate only takes over a copy whose first line is the
canonical's marker (`# ctl-dev-kit convention: integration`); a copy without it
is hand-owned and ungated, which is the state every repo started in. To
converge: move the repo's bespoke steps into the hooks above, replace the
workflow with the canonical (product and runner filled in), and prove it with
`gh workflow run integration.yml --ref <branch>` before merging — the tier
only fires on `main` and on dispatch, so a PR's checks never run it. Once the
marker is in, the nightly `bump.yml` keeps the copy current and `check:drift` names any
hand edit.

**Why the pipe is shaped the way it is.** The suite runs inside a shell
function piped into `tee` so the log survives as an artifact, and the step's
first line is `set -euo pipefail`. GitHub's default `run:` shell is `bash -e
{0}` with no pipefail, so without it the step exits with tee's status and a
dead suite reads green; a `set -euo pipefail` inside the nested `bash -c` body
cannot reach the pipeline one shell up. Both turnkeys ran a void tier for weeks
that way. `tee-pipefail.test.ts` guards every convention workflow for it.

The workflow also uploads `test-results/` beside the log, and reaps the
instances the harness recorded in `$NORSK_LAUNCHED_MANIFEST` after a cancelled
or failed run, so one killed job cannot poison the next with a held port.