# `@norskvideo/norsk-auto-manager` The opinionated orchestration layer above Norsk Manager. Manager exposes a mechanical API — start node, terminate node, register job, report events — without policy. AutoManager owns policy: where jobs run, how nodes are sized and packed, how failures recover, when hot spares are provisioned. This README is the conceptual guide. The TSDoc comments on each public symbol are the authoritative API reference; run `npx tsc --declaration` or read `lib/src/automanager.d.ts` for the full signatures. The class lives in `src/automanager.ts` with three collaborators it builds over its own fleet view and delegates to: `src/hotSpares.ts` (the spare reconciler, warmth and reservation), `src/migrationSaga.ts` (job migration) and `src/nodeServices.ts` (node-service placement). "Where is job J" — its node, instance key, placement of record, in-flight gate and migration claim — is one record per job in `src/jobBinding.ts`. For the design rationale, see `.ai/steve/tasks/37-auto-manager-design.md` in the repo root. --- ## Quick start ```ts import { AutoManager, PlacementPool } from "@norskvideo/norsk-auto-manager"; // A pool is an ORDERED list of tiers. Placement walks them cheapest-first // and takes the first that can place or provision — the order is the cost // cascade. A single-cloud pool is a one-tier pool. const production: PlacementPool = { name: "production", tiers: [ { name: "reserved", kind: "aws", region: "eu-west-1", packingStrategy: "binpack", scaleOut: "elastic", launchMode: "reserved", maxNodes: 4, // spill to the next tier past the reservation candidateInstanceTypes: [ // Capacity ≡ vCPUs: an m5.xlarge has 4, so it advertises 4. { instanceType: "m5.xlarge", totalCapacity: 4, totalCores: 4, capabilities: [] }, ], }, { name: "on-demand", kind: "aws", region: "eu-west-1", packingStrategy: "binpack", scaleOut: "elastic", candidateInstanceTypes: [ { instanceType: "m5.xlarge", totalCapacity: 4, totalCores: 4, capabilities: [] }, ], }, ], }; const [autoManager, runPromise] = await AutoManager.run({ url: "manager.example.com:6789", pendingWindow: 60, removeStoppedNodesAfter: 3600, placementPools: [production], // Recommended callbacks for ops tooling: onPlacementDecision: (info) => console.log("placement", info), onBundleBroken: (id, reason) => console.warn("bundle broken", id, reason), onConnectionStateChange: (state) => console.log("manager", state), }); // AutoManager runs in the background. `runPromise` resolves when the // event-stream loop exits (e.g. after `autoManager.close()`). ``` The `AutoManager.run` call validates the settings up front (throws `AutoSettingsValidationError` on shape/typo problems) and connects to Manager. Once the event stream is established, AutoManager hydrates its mirror of Manager's state (jobs, nodes, bundles, inventories) from the initial snapshot — replaying each through the ordinary `onJobUpdated` / `onNodeStarted` / `onBundleUpdated` callbacks, as if fresh events — and only then starts running placement decisions and the reconcilers. ### Tiers Each `PlacementTier` is one homogeneous capacity segment: a `kind` (`"aws" | "oci" | "local-servers"`), an optional `region`, a `packingStrategy` (`"binpack" | "spread"`), and `scaleOut` (`"elastic"` provisions from `candidateInstanceTypes`; `"fixed"` is pre-registered cluster nodes, at most one such tier per pool). Optional per-tier knobs: `launchMode` (`"on-demand" | "spot" | "reserved"`), `reliability` (`"durable" | "interruptible"`, defaulting to interruptible for spot), `maxNodes` (a tier at its cap cannot provision, so the cascade spills to the next tier), `rootVolume` (aws only), `placementLeadMs` (how far ahead of a scheduled `startDateTime` to start placing; 5 min for cloud tiers, 0 for cluster) and `provisionTimeoutMs` (how long a provision may stay prospective before its jobs are re-placed; default `max(placementLeadMs, 5 min) × 2`). Provisioned nodes are tagged with the tier name; untagged cluster nodes belong to the pool's fixed tier. --- ## Bundles A **bundle** is the unit of work you submit. It runs a **primary** copy (replica index 0) and — only when it carries a `resiliencePolicy` — one **backup** (replica index 1). The copy count is derived (`replicaCount`), never set by the operator; there is no N-way redundancy. Each copy contains one or more **jobs** — the atomic placeable units. ```ts import { Bundle } from "@norskvideo/norsk-auto-manager"; const bundle: Bundle = { bundleId: "my-channel-1", pool: "production", // the ONE pool the primary is placed into resiliencePolicy: { // present ⇒ a backup is expanded too pool: "oci-fra", // optional: where the backup goes (default: the bundle's pool) sameNode: "forbid", sameAz: "forbid", sameCloud: "allow", }, jobs: [ { jobName: "encode", requirements: { requiredCapacity: 2, // vCPUs consumed requiredCores: 2, // integer core count (omit/0 = no pin) requiredCapabilities: [], // hardware/software requirements requiredHostPorts: ["1935/tcp"], onRuntimeFailure: "preferSpare", // see "Runtime failure policy" below }, }, ], }; await autoManager.createBundle(bundle); ``` Submit a bundle by calling `autoManager.createBundle(bundle)`. Manager expands it into one runtime Job per replica, AutoManager runs placement for each, and the workflow begins. `updateBundle` always sends `state: "live"` (a spec update is a revival by definition); `deleteBundle` starts the daemon's stop-then-delete saga. A job with `startMode: "manual"` is expanded but stays dormant until `startBundleJob(bundleId, replicaIndex, jobName)` activates it; the default is `"auto"`. `stopBundleJob` flips it back to dormant for future placement (it does not stop a running worker today). ### Where a copy is placed The primary is always placed from the bundle's single `pool`, walking that pool's tiers in order. The backup is placed from `resiliencePolicy.pool` when one is set, else from the bundle's pool. That is the cross-cloud DR pattern: primary in AWS, backup in OCI. The resilience flags (`sameNode` / `sameAz` / `sameCloud`, each `"allow" | "forbid"`) are **best-effort, never hard**: a `forbid` keeps the backup out of the primary's failure domain whenever any compliant node fits; if none does, the backup is placed anyway and `onResilienceDegraded` reports which flags were violated (and whether the primary sits on interruptible capacity with no durable backup). ### Intra-replica rules Within a replica, jobs can be co-located or kept apart (`coLocateWith`, `separateFrom` on `BundleJobSpec` — hard rules). The first mover of a co-located group sizes its node for the whole group. `intraReplicaPlacement.sameNodeDefault: true` co-locates every job that declares neither rule, as a hard rule; `intraReplicaPlacement.sameAz` is `"hard" | "soft" | "off"` (default soft). `exclusive: true` gives a job a node of its own (co-located peers excepted); `singleUse: true` (requires `exclusive`) terminates that node when the job ends rather than leaving it for reuse — provisioned cloud nodes only, never registered cluster nodes. `backupParameterOverrides` supplies parameter values that differ on the backup replica. ### Spares at launch `onLaunch` (bundle-level, default `"avoidSpare"`) says whether a bundle's FIRST placement may take an unconsumed hot spare; `"allowSpare"` lifts the penalty without adding a preference. `overrideReservation: true` lets the bundle take a spare that is *exclusively* reserved for other templates — the operator's explicit "yes, take it". --- ## Runtime failure policy Every job declares `onRuntimeFailure: "preferSpare" | "avoidSpare" | "drop"` — three behaviours, not a ranking: | Policy | On failure | Recovery placement | |---------------|---------------------|-----------------------------------------------------| | `preferSpare` | re-place | an unconsumed spare ranks first among equals (below warmth) | | `avoidSpare` | re-place | an unconsumed spare ranks below every ordinary node | | `drop` | do not re-place | n/a — the runtime is terminal | The **restart ceiling** applies to the two re-placing policies. It is a *rate*: `maxRestartsPerHour` (default 6; per job via `requirements.maxRestartsPerHour`, per policy via `failurePolicies.{preferSpare,avoidSpare}.maxRestartsPerHour`, else the flat `AutoSettings.maxRestartsPerHour`) counted over a trailing hour. The history clears when a node provisioned for the job starts, and when the bundle is updated or deleted. Consecutive recovery attempts for one job are spaced at least `MIN_RECOVERY_INTERVAL_MS` (5 s) apart. `drop` has no ceiling entry: it never restarts. What "ultimately lost" means depends on `requirements.essential` (absent ⇒ true): an essential runtime past its ceiling — or dropped, or rejected by the worker as `JOB_REJECTED_INVALID_SPEC` — ends the production: AutoManager writes the terminal verdict to the daemon, marks the bundle `broken`, fires `onBundleBroken`, and stops placing every replica of it. A non-essential runtime takes itself out alone (`onJobDropped`) and the rest of the production carries on. An invalid spec is terminal on the *first* rejection, with the worker's own words as the reason — no node would do better, so nothing is retried. The broken mark clears when the bundle is updated or deleted. ```ts failurePolicies: { preferSpare: { maxRestartsPerHour: 6 }, avoidSpare: { maxRestartsPerHour: 6 }, } ``` Spare preference is not configurable here — it follows from the policy. --- ## How placement ranks a node Placement is pure (`place` / `placeWithTrace` in `src/placement.ts`). For each tier in order, every node is run through a hard filter chain — target-node pin, pool, tier, recovery exclusion, reachable, cordoned, exclusive reservation, capabilities, capacity, cores, host ports, GPU fit, exclusivity, intra-replica rules, hard intra-replica AZ — and the survivors are ranked by a **lexicographic key** (`PlacementKey`, `comparePlacementKeys`). There are no weights: each component is decided only if every component above it tied. | # | Component | Wins | |---|-------------------------|-----------------------------------------------------------------| | 1 | `resilienceViolations` | fewer `forbid` domains violated (backups; best-effort) | | 2 | `prospective` | a node that has booted over one still launching | | 3 | `avoidedSpare` | NOT an unconsumed spare, when the policy in force says avoid (recovery: the job's `onRuntimeFailure`; launch: the bundle's `onLaunch`) | | 4 | `warmth` | more of the job's template images already held | | 5 | `reservationMatch` | a still-empty spare reserved for this job's template | | 6 | `preferredSpare` | an unconsumed spare, when the recovering job says `preferSpare` | | 7 | `packing` | fuller node under `binpack`, emptier under `spread` | | 8 | `sameAz` | shares an AZ with a same-replica peer (soft intra-replica AZ) | | 9 | node id | lexicographic, for determinism | If no node survives the filter and the tier is elastic and under `maxNodes`, a node is provisioned from the first fitting `candidateInstanceTypes` shape (a pinned `targetNode` never provisions). Otherwise the next tier is tried; when all are exhausted the result is `noCapacityInAnyPool`. `onPlacementDecision` carries the full trace (per-node filter reasons plus every survivor's key); `placeDryRun` runs the same engine without acting. Only an *unconsumed* spare (tagged `spare`, carrying no job) counts as reserve for components 3, 5 and 6 — a spare that took a job is an ordinary node from then on. ### Capacity, GPU and port contracts - `NodeInventory.utilizationCap` is a **fraction (0, 1]** of `totalCapacity`; free capacity is `total × cap − reserved`. A producer that sends anything else (an absolute, 0, negative, NaN) is read as 1 at the wire boundary (`fromPbNodeInventory`), so a node can never advertise more than its declared capacity. - `gpus` is per device. A job asks for `requiredGpuCapacity` (fractional, optionally `requiredGpuModel`) and placement picks a device (`PlacementResult.gpuIndex`); that device's `reservedCapacity` is then derived from the jobs placed on it, so two jobs cannot be double-booked onto one GPU. Whole-GPU exclusive use is the `norsk.io/nvidia-gpu` capability instead. - `requiredHostPorts` (`"PORT/proto"` tokens) are exclusive per node: placement refuses a node where another job already claimed one. Declared ports only; the worker's bind failure stays authoritative. --- ## Hot spares A **hot spare** is a warm-running node that holds no jobs but is ready to take one immediately. Configure per pool: ```ts hotSpares: [ { pool: "production", targetCount: 2, spec: { instanceType: "m5.xlarge", region: "eu-west-1" }, warmFor: ["xyz"], // pre-pull these templates' images onto the spare reservedFor: ["xyz"], // these templates score it first … exclusive: true, // … and nobody else may take it (default false) }, ], ``` `warmFor` and `reservedFor` are independent: warmth is observed (what the node advertises holding, scored against `templateImages` / `setTemplateImages`), `warmFor` only says which pulls to start. `reservedFor` with `exclusive: false` is a preference; with `exclusive: true` it is a guarantee, overridable per bundle or migration with `overrideReservation`. `updateHotSpareNode` edits one live spare's `SpareSpec`; `spareSpec(nodeId)` reads it back. `updateHotSpareSpecs(list)` replaces the whole live `hotSpares` list **without a reconnect** (an embedder used to reload settings for this, which tore the AutoManager down and aborted every in-flight migration). The list is validated as at boot — one entry per pool, since the pool is the key — and rejected whole with `AutoSettingsValidationError`; otherwise the reconciler converges on it at once: a removed entry's unconsumed spares are reaped as orphans, an added one is provisioned, a lowered target terminates the excess coldest first, and a changed `warmFor` / `reservedFor` / `exclusive` is written to the live spares' tags (only the fields that changed). It returns what changed, keyed by pool (`HotSpareSpecsDiff`). The **reconciler** runs after hydration, on node lifecycle events and on a periodic tick (`hotSpareReconcileIntervalMs`, default 60 s): - Under target → fires `onSpareUnderflow(pool, effective, target)` and provisions the shortfall from the pool's first elastic tier (tagged `spare=true`). A failed provision backs off `SPARE_PROVISION_RETRY_BACKOFF_MS` (60 s) before the next attempt. - Over target → terminates the excess, **coldest first** (warmth against the entry's `warmFor`), `lastActivity` as the tie-break. - A spare that picks up a job is no longer a spare; the underflow path provisions a replacement. For pools with no elastic tier the reconciler cannot auto-scale — it fires `onSpareUnderflow` and leaves the hardware to the operator. --- ## Drain / cordon Take a node out of service gracefully: ```ts autoManager.cordonNode(nodeId); // no new placements land here autoManager.uncordonNode(nodeId); // reverse const state = await autoManager.drainNode(nodeId, { timeoutMs: 10 * 60_000, // default DEFAULT_DRAIN_TIMEOUT_MS (10 min) force: false, // true: terminate at the timeout regardless strategyFor: (jobId) => "handover", // per-job migration strategy (optional) }); autoManager.drainStatus(nodeId); // → DrainState | undefined autoManager.evictNode(nodeId); // cordon + terminate (existing jobs // recover per their onRuntimeFailure) autoManager.isCordoned(nodeId); // → boolean autoManager.terminateNode(nodeId); // terminate without cordoning ``` `drainNode` cordons the node, starts a migration (the same saga as `migrateJob`, placement elsewhere) for every job bound to it, and terminates the node once it is empty — through the same path as `terminateNode`, so `onNodeTerminated` fires once. A job the saga refuses (not running, not AutoManager-placed), one placement finds nowhere for, or one whose migration aborts is `stuck` and stays where it is until the timeout, when the drain ends `partial` (nothing terminated, node still cordoned, `jobs` naming the stuck ones) — or, with `force`, the node is terminated with them on it and they recover per their `onRuntimeFailure`. The call resolves once the migrations are started; progress arrives on `onDrainProgress(state)` and is readable via `drainStatus`. `DrainState` is `{ nodeId, phase, started, timeoutMs, force, jobs: [{ jobId, status, migrationId?, reason? }], reason? }` with `phase` one of `migrating` | `waiting` | `terminating` | `done` | `partial` | `failed` and each job's `status` one of `migrating` | `migrated` | `moved` | `stuck` | `terminated`. One drain per node (a second call returns the running one); `uncordonNode` cancels a drain that has not issued its terminate (its migrations in flight run to their end); `close()` cancels the timers. `evictNode` is the heavy-handed variant: cordon then terminate at once. The daemon's `NodeStopping` / `NodeStopped` for the node unbinds every job it hosted and drives the standard recovery path, which re-places them onto other nodes per each job's `onRuntimeFailure` (`drop` jobs are dropped). ### Spot interruption / rebalance When a provider says it is taking a node back, run it down ahead of the deadline instead of finding out when the node dies: ```ts const state = await autoManager.handleNodeInterruption(nodeId, { kind: "spot-interruption", // | "rebalance" deadline: new Date(Date.now() + 120_000), // optional: when the provider reclaims it strategyFor: (jobId) => "handover", // optional, as drainNode }); autoManager.nodeInterruption(nodeId); // → NodeInterruptionState | undefined ``` This is `drainNode` with `force: true` and a timeout cut to fit: `deadline − now − SPOT_INTERRUPTION_DRAIN_MARGIN_MS` (30 s), or with no deadline the kind's default — `DEFAULT_SPOT_INTERRUPTION_DRAIN_MS` (90 s, EC2's two-minute warning less the margin) for a spot interruption, `DEFAULT_REBALANCE_DRAIN_MS` (5 min) for a rebalance recommendation. Jobs that migrate in time land elsewhere; whatever is still on the node at the timeout goes down with it and is recovered per its `onRuntimeFailure` — and **none of it counts against the restart ceiling**: a job lost with a reclaimed node was not a job that failed (`restartCount` stays put; N reclaims never break a bundle). A notice for a node already draining (the interruption that follows a rebalance) keeps that drain and tightens its timeout; `uncordonNode` withdraws the notice with the drain. `onNodeInterrupted(state)` fires once per call with `{ nodeId, kind, deadline?, noticedAt, drain: DrainState }`. Who calls it: the event loop, on the daemon's `nodeInterruptionNotice`. On EC2 the norsk-ctl worker polls the instance metadata service (`spot/instance-action`, `events/recommendations/rebalance`; IMDS is reachable only from the instance) and reports the notice on its stream; the daemon relays it, tags the node `interruption=` and audits it. The SDK kinds `spotInterruption` / `rebalanceRecommendation` become `spot-interruption` / `rebalance` here. The embedder (norsk-mgr) may also call it directly for a notice from any other source. --- ## Migration Move a running job from one node to another (design: norsk-ctl `docs/internal/design/job-migration.md`): ```ts // Placement chooses the target (the job's current node is never a // candidate); or pin one with `targetNodeId`. Resolves once the target // is placed and its StartJob dispatched; rejects if placement finds // nowhere for it. const migrationId = await autoManager.migrateJob(jobId, { strategy: "stopThenStart", // default; or "overlap" | "handover" targetNodeId: "node-B", // optional hard pin overrideReservation: true, // optional: may take an exclusively reserved spare }); // Track progress via callbacks: // onMigrationPhase(state) — fires on every phase transition // onMigrationCompleted(id) — the job now lives on the target // onMigrationAborted(id, reason) — rolled back, or lost past the point of no return // Or poll: autoManager.migrationById(migrationId); autoManager.migrationForJob(jobId); autoManager.migrations(); // Operator escape hatch — a clean rollback only until the source has // been told to stop; after that it rejects and the saga fails forward. await autoManager.abortMigration(migrationId, "operator changed mind"); ``` Strategies: - `stopThenStart` — the target is created without starting (`deferStart`), the source stopped, the prepared target started. Never two running at once; a gap of one container start. No job cooperation. - `overlap` — the target started, and once running the source stopped. Both run for a moment. No job cooperation. - `handover` — **cooperative**: the target signals ready (with an opaque payload), the source stops output at that point and reports where, the target applies from there and confirms; only then is the source stopped. The job owns the payload's meaning. The target goes through the same `placeAndProvision` path as any other job — requirements, host ports, resilience, hot spares, provision-a-node — with the source node excluded and an explicit target as a pin. A migration passes no `onRuntimeFailure` to placement: the reserve is protected by reservation there, not by the spare penalty. During the overlap the job is reserved on both nodes. Each phase has a deadline (`migration` settings); expiry before the point of no return aborts, after it the saga fails forward. A target the worker rejects as an invalid spec aborts the migration; the source keeps running. No concurrent migrations per source jobId. Instances are addressed by an **instance key**: the target wears `migrationTargetRole(slot, id)` (`primary@3f9c1e2a`) while being told the slot (`primary`) in `NORSK_RUNTIME_ROLE`; at `done` the job's binding, placement of record and active key move to the target, so a second migration addresses the right instance. For workflows that cannot migrate, fall back to `evictNode` (which is a frame drop). --- ## Metrics ```ts const m = autoManager.metrics(); // { // startedAt: Date, // uptimeMs: number, // placementsAttempted / Succeeded / Failed / DispatchFailed, // jobsRejected, // bundlesBroken, // jobsDropped, // pools: [{ poolName, nodeCount, liveSpares, spareTarget, usedCapacityFraction }], // brokenBundleIds: BundleId[], // } ``` `placementsSucceeded` counts a dispatched wire call; a decision whose `startJob` / `createAwsNode` / `createOciNode` then fails counts under `placementsDispatchFailed` instead. Counters are cumulative since `startedAt`; per-pool gauges are computed at call time. Poll on your own cadence; AutoManager doesn't push. --- ## Validation Two checks worth running before submitting bundles: ```ts // Static shape: pool refs, job-name uniqueness, placement rule // consistency, exclusive/single-use groups, capability satisfiability. const { errors, warnings } = autoManager.validateBundle(bundle); // errors → submission must be rejected // warnings → capacity misses against a cluster pool, where workers may // join later; surface them, but allow submission // Static + placement viability: "would each replica place right now?" const result = autoManager.validateBundleAdmission(bundle); // → { kind: "ok" } // | { kind: "validationErrors"; errors: ValidationError[] } // | { kind: "noCapacity"; unplaceable: { replicaIndex, jobName, reason }[] } // One replica/job through the engine, with the trace, without acting: autoManager.placeDryRun(bundle, replicaIndex, jobName); ``` `AutoSettings` itself is validated by `AutoManager.run` — bad settings throw `AutoSettingsValidationError` carrying the structured error list (e.g. `duplicateTier`, an unknown `hotSpares[].pool`, a `nodeServices[].lifecycle` other than `"eager"`, or — `experimentalNotEnabled` — a non-empty `nodeServices` without `experimental: { nodeServices: true }`: what exists is a working eager placement and none of the rest, so nobody configures a node service by accident). --- ## Callbacks reference All optional. None block AutoManager's event loop on error — exceptions are caught and forwarded to `onError`. ### Connection / lifecycle - `onConnectionStateChange(state)` — `"idle" | "connecting" | "ready" | "transientFailure" | "shutdown"`. - `onError(err)` — catch-all; back-compat surface. - `signal` (an `AbortSignal`, not a callback) — aborting before connect rejects `run()`; aborting a running instance tears its connection down. ### Job lifecycle - `onJobUpdated(jobWithHistory)` — Manager pushed a job change (also replayed for every job in the initial snapshot at hydration). - `onJobInfo(info)` — informational worker message (launch progress, container events); no placement state changes. - `onJobDeleted(jobId)` — job removed from the cache. - `onJobRejected(info)` — worker rejected a placement (typed, with the inventory it sent). A single rejection does not mark the node. - `onJobDropped(jobId)` — a failed job was dropped rather than restarted: `onRuntimeFailure: drop`, a non-essential runtime past its ceiling, or an invalid spec. ### Node lifecycle - `onNodeStarting / onNodeStarted / onNodeStopping / onNodeStopped` — the daemon's node events. `onNodeStopping` is the prompt "worker dropped" signal (a Ctrl-C'd cluster worker fires it first; Stopped follows after the daemon's keepalive timeout). - `onNodeTerminated(nodeId)` — THIS AutoManager issued a terminate for the node, from any of its own paths (`terminateNode` / `evictNode` / a `drainNode` that emptied its node, spare excess and orphan reap, single-use release, startup-grace expiry, the stopped-node sweep). Exactly once per node, at issue time, not on the daemon's confirmation. A node terminated by anyone else surfaces only as `onNodeStopping` / `onNodeStopped`. - `onNodeUnhealthy(nodeId)` — `rejectionEscalationCount` (default 3) rejections inside `rejectionBackoffMs` (default 5 s); the node is excluded from placement until a job reaches running on it or `unhealthyCooldownMs` (default 10 min) elapses. - `onNodeHealthy(nodeId)` — that exclusion cleared. ### Bundle lifecycle - `onBundleUpdated(bundle)` / `onBundleDeleted(bundleId)`. Hydration replays every bundle in the initial snapshot through `onBundleUpdated`, as it replays jobs and nodes. - `onBundleBroken(bundleId, reason)` — restart ceiling tripped, or a runtime ended terminally (`drop`, invalid spec). ### Placement - `onPlacementDecision(info)` — fires every time placement runs, with full trace. - `onPlacementFailed(info)` — typed failure callback (also surfaces via `onError`). - `onResilienceDegraded(info)` — a backup was best-effort placed inside a forbidden failure domain, or the primary is on interruptible capacity with no durable backup. - `onSpareUnderflow(pool, current, target)` — hot-spare reconciler is short. - `onImagesPrefetched(nodeId, results)` — one prefetch batch settled on a node; an `ok: false` entry means that spare stays cold for that ref. ### Migration - `onMigrationPhase / onMigrationCompleted / onMigrationAborted`. ### Drain - `onDrainProgress(state)` — every phase transition of a node drain, and every change to a job's status within it (`drainStatus(nodeId)` reads the same snapshot). - `onNodeInterrupted(state)` — once per `handleNodeInterruption` call, after the node is cordoned and its drain started or tightened (`nodeInterruption(nodeId)` reads the same snapshot). ### Audit - `onAuditAppended(entry)` — pass-through of the daemon's audit-log append. --- ## Operator-side config overrides (Manager) Independent of this package but worth mentioning: Manager supports an external JSON overrides file at `/var/data/manager-overrides.json` (or `NORSK_MANAGER_OVERRIDES_FILE`) for the cloud / region / AZ / instance- type fields of provider configs. Lets operators tune those without rebuilding the Manager container. See `.ai/steve/tasks/37-auto-manager-design.md` §D2 for the shape. --- ## Lifecycle ```ts autoManager.close(); // cancels every timer, stops the event loop, aborts // in-flight migrations, closes the gRPC channel, // resolves runPromise autoManager.dispose(); // alias for close() ``` Idempotent. An instance is never reconnected — dispose it and run a fresh one — so `close()` is the only place an in-flight migration can be wound down. Before the point of no return that is the clean rollback (abort to both instance keys, the target stopped; the source keeps running). Past it there is no rollback: the migration is marked lost (`onMigrationAborted` plus an `onError`), the target is left carrying the job, and the next instance rebuilds the placement from history. A `migrateJob` still parked on its propose rejects. The channel stays open just long enough for those aborts to be sent — `CLOSE_ABORT_GRACE_MS` (5 s) at most; with nothing in flight it closes immediately. After `close()`, `runPromise` settles, and an initial snapshot arriving late is not hydrated. --- ## Worker-side dependencies Several capabilities depend on worker-side work that lives outside this package (in norsk-ctl, by deployment convention): - **Hardware inventory & capacity scores** — landed. Workers advertise cores, GPUs, NICs, capabilities and held images (`norsk-ctl/backend/src/worker-mode/inventory.ts`). Without an inventory a node is invisible to placement. - **Worker-side enforcement** — still missing. Workers do not refuse incoming starts that exceed their advertised capacity, so AutoManager's derived reservation is the only capacity backstop. - **Migration mechanics** — landed. The create/start split (`defer_start` → `job_prepared`, `start_prepared_job`) and, for the `handover` strategy, the job socket that translates the `NodeControllerMigration*` messages into job-level signals and back. - **NodeService capability advertisement** — a service's `provides[]` reaching sibling jobs' view of the node is worker-side work; without it AutoManager places the service but no consumer can see what it offers. See the `.ai/steve/tasks/37-auto-manager-design.md` §"Worker-side work" section for the full norsk-ctl contract. --- ## Changed 2026-09-16 Behaviour changes an integrator would notice, from the review follow-ups: - Placement ranks by a lexicographic key (table above) instead of an additive score; "avoid-spare outranks warmth" is now true by construction. - `resiliencePolicy.pool` is real: a backup is placed from it when set. - `intraReplicaPlacement.sameNodeDefault: true` now co-locates the rule-less jobs of a replica as a hard rule. - `RecoveryContext` carries `onRuntimeFailure` only; the old `preferSpare` / `avoidSpare` booleans are gone. - GPU capacity is reserved per device from placed jobs' `gpuIndex`; two jobs can no longer share one GPU's capacity unknowingly. - `NodeInventory.utilizationCap` outside (0, 1] is read as 1 at the wire boundary (a worker that sent its vCPU count used to advertise `cpu²`). - `onNodeTerminated` fires once per AM-issued terminate, at issue time; third-party terminations surface only as Stopping / Stopped. - Hydration replays every bundle through `onBundleUpdated`. - A node's `NodeStopped` alone (no prior Stopping) unbinds and recovers its jobs. - `close()` aborts in-flight migrations, holding the channel open up to `CLOSE_ABORT_GRACE_MS` for the aborts to be sent. - `BundleJobSpec.startMode` round-trips through the wire; a bundle that predates the field reads as `"auto"`. - `JOB_REJECTED_INVALID_SPEC` ends the job on the first rejection with the worker's detail as the reason; no re-place, no phantom retry after a drop or a ceiling trip. A broken bundle gets no retry. - A failed spare provision leaves no phantom node and backs off 60 s. - `nodeServices[].lifecycle: "lazy"` is refused by settings validation. - Settings/validation codes renamed: `duplicateTier` (was `duplicatePool`), `emptyPoolName` (was `emptyPoolList`). - Removed exports: `freeCapability`, `findCapability`. - The retired wire surface is gone from the SDK: `onJobConfigUpdated` (and the `jobConfigUpdated` event), the `physicalNodeConnected` / `nodeServiceUpdated` / `nodeServiceDeleted` events and the `inlineTar` product-template source no longer exist. - `handleNodeInterruption(nodeId, notice)`: a provider reclaim (spot interruption / rebalance) is a forced drain sized to the deadline, and the jobs it loses do not count against the restart ceiling.