# Pack: async jobs, messaging, realtime

Load when: the touch list exhibits BullMQ (`new Queue(`, `new Worker(`), Redis pub/sub or Streams (`publish(`, `xReadGroup(`), WebSockets / socket.io (`ws.send(`, `socket.emit(`), SSE (`text/event-stream`), realtime channel templates, or outbound webhook senders.

## bullmq-job-payload-shape — HIGH · serialized-shape
**Contract:** BullMQ producer's `queue.add(name, data)` payload and the worker's `job.data` reads agree on a JSON shape with zero schema enforcement — and Redis holds in-flight jobs serialized in the OLD shape across any rolling deploy.
**Detect:** `queue.add(`, `job.data`, `new Queue(`, `new Worker(`, `**/queues/**`, `**/workers/**`, `flow.add(`
**Ships green, breaks:** Deploy new worker first → it dequeues jobs already sitting in `wait`/`delayed`/`prioritized` (plus every retry of an old `failed` job, which replays the ORIGINAL payload, possibly days later) with the old shape → `undefined` field reads, or worse: no crash, wrong values written. Deploy producer first → old workers process new-shape jobs. TypeScript can't help; the queue boundary erases types.
**Safe change:** Add fields optional-first, worker tolerates both shapes; deploy worker before producer; only remove old-shape handling after the queue AND the `failed` set drain (retries replay old payloads); consider a `v` field in `data` and branch on it.

## bullmq-job-name-routing — HIGH · rendezvous-string
**Contract:** The first argument of `queue.add('email-welcome', …)` and the worker's `switch (job.name)` meet on a bare string literal; BullMQ workers receive EVERY job on the queue regardless of name — there is no named-processor registration (unlike Bull v3's `process('name', fn)`).
**Detect:** `queue.add('`, `job.name`, `switch (job.name)`, `case '`
**Ships green, breaks:** Rename either side and the compiler sees two unrelated literals. Worst variant: a `switch` without a throwing `default` — the processor falls through, returns `undefined`, and BullMQ marks the job **completed successfully**. No error, no failed job, no retry; the work silently never happens.
**Safe change:** Centralize job names in one shared const/enum imported by both sides; make the worker's `default` branch throw `UnrecoverableError` so unknown names surface in `failed`; on rename, worker accepts old+new names during the deploy window.

## bullmq-repeatable-scheduler-identity — HIGH · rendezvous-string
**Contract:** A recurring job's identity is a string: legacy repeatables (`{ repeat: {…} }`, deprecated since BullMQ 5.16.0) derive it by **hashing the repeat options** (name, cron `pattern`, `every`, `tz`), while Job Schedulers use the explicit `schedulerId` passed to `upsertJobScheduler(schedulerId, repeatOpts, template)`.
**Detect:** `upsertJobScheduler(`, `removeJobScheduler(`, `repeat:`, `pattern:`, `every:`, `removeRepeatable(`, `removeRepeatableByKey(`, `getRepeatableJobs(`, `getJobSchedulers(`
**Ships green, breaks:** Legacy API: change the cron string (or tz, or job name) → the hash changes → a brand-new repeatable is registered while the OLD one keeps firing forever; `removeRepeatable(name, opts)` only works with the EXACT original options (else list with `getRepeatableJobs()` and use `removeRepeatableByKey`). Job Schedulers: re-upserting the SAME `schedulerId` updates in place — but changing the `schedulerId` string recreates the legacy problem. Registration is restart-time-only, so schedulers removed from code persist in Redis indefinitely.
**Safe change:** Use `upsertJobScheduler` with a stable, never-derived `schedulerId`; change cadence by re-upserting the same id, never by renaming; on any rename, explicitly `removeJobScheduler(oldId)`; audit live state with `getJobSchedulers()` + `getRepeatableJobs()` after deploy.

## bullmq-lock-stall-redelivery — CRITICAL · lifecycle-protocol
**Contract:** The worker promises to finish (or keep renewing the lock on) a job within `lockDuration` (default 30000 ms, auto-renewed at `lockDuration/2` — renewal runs on the event loop); the queue promises not to hand the job to anyone else until then.
**Detect:** `lockDuration`, `lockRenewTime`, `stalledInterval`, `maxStalledCount`, `on('stalled'`, `concurrency:`
**Ships green, breaks:** Any CPU-bound section, sync I/O, or GC pause longer than the renewal window blocks lock renewal → the stalled-checker (every `stalledInterval`, default 30000 ms) moves the job back to `wait` → a second worker starts it **while the first is still running it**: emails sent twice, payments charged twice. No test catches this — it needs production-sized payloads. After `maxStalledCount` (default 1) the job fails with `job stalled more than allowable limit`.
**Safe change:** Size `lockDuration` above worst-case runtime when adding slow steps; keep the processor's event loop breathing (chunk CPU work, `await` regularly) or move heavy jobs to sandboxed processors (separate process = renewal survives CPU blocks); make side effects idempotent (dedup key per `job.id`); alert on the `stalled` event — it means double-processing happened.

## bullmq-retry-retention-dlq — HIGH · lifecycle-protocol
**Contract:** `attempts`/`backoff` on the producer's job options promise the handler will be re-invoked with the same payload on failure; the handler must be idempotent to that promise; the `failed` set is the implicit dead-letter queue, and Redis retains completed/failed jobs per `removeOnComplete`/`removeOnFail`.
**Detect:** `attempts:`, `backoff:`, `removeOnComplete`, `removeOnFail`, `UnrecoverableError`, `on('failed'`, `defaultJobOptions`
**Ships green, breaks:** (1) Someone adds `attempts: 5` in producer options — the handler, written for exactly-once, now re-executes partial side effects up to 5×. (2) BullMQ's default keeps **all** completed and failed jobs forever — no `removeOnComplete`/`removeOnFail` means Redis grows until OOM weeks later (auto-removal is lazy: it runs only when a subsequent job finishes, so an idle queue never cleans up). (3) Exhausted retries land in `failed` — the de-facto DLQ nothing monitors; work is lost invisibly. Wrapping `UnrecoverableError` in a plain `Error` during a refactor silently re-enables retries.
**Safe change:** Set `defaultJobOptions: { removeOnComplete: {age, count}, removeOnFail: {age} }` on every queue; treat any `attempts > 1` change as an idempotency review of the handler; alert on `failed` depth (`getFailedCount()`); preserve `UnrecoverableError` semantics through error-wrapping refactors.

## bullmq-queue-name-prefix-rendezvous — HIGH · rendezvous-string
**Contract:** Producer `new Queue(name, { prefix, connection })` and worker `new Worker(name, fn, { prefix, connection })` rendezvous on the Redis key `{prefix}:{name}:*` (default prefix `bull`) — same name, same prefix, same Redis logical DB, or they never meet.
**Detect:** `new Queue(`, `new Worker(`, `prefix:`, `QueueEvents(`, `db:` in redis connection opts, `{` hash-tag in prefix
**Ships green, breaks:** Any mismatch — name typo, one side adding `prefix: 'myapp'`, different `db` index, different Redis instance after a config refactor — and jobs pile up in `wait` forever while the worker idles at zero errors on both sides. Redis Cluster: the prefix must be a hash tag (`prefix: '{myqueue}'`) so all keys land in one slot; producer with `{tag}` + worker without = two disjoint key spaces.
**Safe change:** Define queue name + prefix once in a shared module; never inline the string; after any connection/prefix refactor, run a canary job end-to-end; on cluster, hash-tag the prefix identically everywhere including dashboards (bull-board with a different prefix shows an empty queue).

## redis-pubsub-fire-and-forget — HIGH · rendezvous-string
**Contract:** `PUBLISH channel` and `SUBSCRIBE channel` / `PSUBSCRIBE pattern` meet on a literal channel string at one instant in time — Redis pub/sub has zero buffering, zero persistence, zero delivery guarantee, and `PUBLISH` returns a receiver-count nobody checks.
**Detect:** `.publish(`, `.subscribe(`, `.pSubscribe(`, `.psubscribe(`, `.sSubscribe(`, `duplicate()` on redis clients
**Ships green, breaks:** No subscriber connected (worker restarting, deploy gap) = message silently gone — `publish()` resolves fine, returning `0`, unchecked. Renaming a channel on one side, or editing a `PSUBSCRIBE` glob, silently partitions publisher from subscriber. Redis 7 cluster: plain `SUBSCRIBE` broadcasts cluster-wide but sharded `SSUBSCRIBE`/`SPUBLISH` route by slot — mixing `publish` with `sSubscribe` never meets. node-redis requires a dedicated (`.duplicate()`) connection in subscriber mode.
**Safe change:** Treat pub/sub as lossy signaling only — anything that must be processed goes to Streams or BullMQ; centralize channel names/patterns in shared constants; pair `publish`↔`subscribe` and `sPublish`↔`sSubscribe`, never cross.

## redis-streams-group-offset-identity — CRITICAL · rendezvous-string
**Contract:** A consumer group's NAME is the durable owner of its offset (`last-delivered-id`) and its pending-entries list (PEL); consumers within it are named too, and un-acked messages belong to the consumer name that read them.
**Detect:** `xGroupCreate(`, `XGROUP CREATE`, `xReadGroup(`, `xAck(`, `xAutoClaim(`, `xClaim(`, `XGROUP DELCONSUMER`, `MKSTREAM`
**Ships green, breaks:** Rename the group in code → `XGROUP CREATE` mints a fresh group starting wherever the code says: `$` = silently SKIP everything undelivered (data loss), `0` = REPROCESS the entire retained stream (duplicate side effects) — and the old group's PEL messages are stranded forever. `XGROUP DELCONSUMER` **discards** that consumer's pending entries outright. `XADD` with `MAXLEN` trimming can delete messages a lagging group never consumed.
**Safe change:** Group names are permanent identifiers — never rename; migrate: create new group, drain old PEL via `XAUTOCLAIM` + ack, then `XGROUP DESTROY`; claim a dead consumer's PEL before any `DELCONSUMER`; monitor `XPENDING` depth and `XINFO GROUPS` lag; size `MAXLEN` against worst-case consumer downtime.

## websocket-envelope-mixed-clients — HIGH · serialized-shape
**Contract:** Server and a MIXED-VERSION population of long-lived clients agree on the message envelope — the `{type, payload}` discriminator values and payload fields — with no per-message negotiation and no redeploy of clients: tabs stay open for days, mobile apps for weeks.
**Detect:** `JSON.parse(event.data)`, `ws.send(JSON.stringify(`, `switch (msg.type)`, `socket.emit('`, `socket.on('`, `**/ws/**`, `**/realtime/**`
**Ships green, breaks:** Server renames a `type` value or restructures `payload` → every already-connected old client either throws or — the silent case — hits a `switch` with no matched case and drops the message: the UI just stops updating, no error anywhere. socket.io event names are the same bare-literal rendezvous — a renamed event is silently never delivered. TypeScript types verify nothing about deployed clients.
**Safe change:** Message types are append-only — add new `type` values, never rename/repurpose; clients MUST ignore unknown types explicitly (with a metric, not a crash); payload fields additive-only until a protocol-version bump negotiated at connect; keep the server accepting old client messages for the longest realistic client lifetime, not the deploy window.

## realtime-channel-template-tenant-scope — CRITICAL · trust-invariant
**Contract:** The channel-name template (`chat:{tenantId}:{sessionId}` — Durable-Object room names, Redis channels, socket.io rooms alike) is built independently by the publisher, the subscriber, AND the subscribe-authorization check; the tenant segment embedded in the string IS the isolation boundary.
**Detect:** `` `chat:${ ``, `` `${tenantId}:` ``, `.join(':')`, `split(':')`, `io.to(`, `socket.join(`, `idFromName(`, room/channel builder functions
**Ships green, breaks:** Two directions, both silent. (1) **Availability** — the publisher's template changes while subscribers build the old string: clients subscribe to a channel the server never publishes to; connects succeed, auth passes, zero messages, zero errors. (2) **Isolation** — drop or misorder the tenant segment, or interpolate an unvalidated value: a `sessionId`/`tenantId` containing `:` shifts segments so the string parses as a different tenant's channel — segment injection through the delimiter. `PSUBSCRIBE chat:*` grants cross-tenant reads in one line.
**Safe change:** One shared `buildChannelName()`/`parseChannelName()` module used by publish, subscribe, AND authz — never three inline template literals; validate segments against `[^:]+` before interpolation; authz parses the ACTUAL requested channel string, never trusts client-supplied segments; template changes are a versioned migration (dual-publish during rollout).

## socketio-multinode-adapter — HIGH · config-elsewhere
**Contract:** `io.to(room).emit(…)` reaching ALL connected clients depends on every node running the same adapter (`@socket.io/redis-adapter` / `@socket.io/redis-streams-adapter`) against the same Redis — and HTTP long-polling fallback depends on load-balancer sticky sessions declared in infra config the app repo never sees.
**Detect:** `createAdapter(`, `@socket.io/redis-adapter`, `io.adapter(`, `io.to(`, `io.in(`, `transports:`, `allowEIO3`, LB config (`ip_hash`, `sessionAffinity`, ALB stickiness)
**Ships green, breaks:** Works perfectly with one node in dev. Scale to two nodes without the adapter: broadcasts only reach sockets on the emitting node — a random ~half of users silently miss every message. Sticky sessions missing: the polling handshake round-robins to a node that doesn't know the session → `400 Session ID unknown` → silent reconnect loops, presenting as "flaky realtime". A REST endpoint emitting via a bare `io` instance from a different process (no adapter) publishes to nobody.
**Safe change:** Wire the adapter before the second node exists; verify with a 2-node compose setup emitting cross-node; declare the sticky-session requirement next to the socket.io code or force `transports: ['websocket']` deliberately; treat `allowEIO3` removal as a client-population question, not a cleanup.

## outbound-webhook-contract — CRITICAL · serialized-shape
**Contract:** The org's emitted webhooks — payload shape, event-type names, signature header name + algorithm, and the shared secret — are consumed by EXTERNAL subscribers you cannot redeploy, coordinate with, or even enumerate.
**Detect:** `createHmac(`, `timingSafeEqual(`, signature header literals (`X-*-Signature`, `X-Hub-Signature-256`), `event_type`, webhook dispatch tables, `WEBHOOK_SECRET`
**Ships green, breaks:** Everything here ships green because the other side is off-repo. Rename an event type: subscribers filter on the literal string and silently ignore the new one — their integrations just stop firing. Rotate the secret in one step: every delivery fails verification the instant the env var flips; if the sender retries-then-drops, events are permanently lost. Canonicalization drift is subtler: signing re-serialized JSON instead of the exact raw bytes sent (key order, whitespace, unicode escaping) makes verification fail only for some payloads.
**Safe change:** Event types and payload fields are append-only; breaking payload changes = a new version delivered per-subscriber, old version supported indefinitely; rotate secrets with an overlap window — sign with BOTH (Stripe/Svix `v1=sig1,v1=sig2` style) until subscribers confirm; sign the exact raw body bytes; include a signed timestamp; never repurpose the header name or algorithm prefix.

## sse-resume-and-event-names — MED · lifecycle-protocol
**Contract:** Server SSE frames and browser `EventSource` agree on two things with no static link: the `id:` field ↔ `Last-Event-ID` resume protocol across auto-reconnects, and the `event:` field name ↔ `addEventListener(name)` dispatch.
**Detect:** `text/event-stream`, `Last-Event-ID`, `new EventSource(`, `res.write('id:`, `res.write('event:`, `onmessage`, `X-Accel-Buffering`
**Ships green, breaks:** (1) `EventSource` auto-reconnects invisibly and sends `Last-Event-ID`; a server that never assigned `id:` — or ignores the header — silently drops every event from the gap while the client believes it is live. (2) Adding an `event: order-update` field to previously-unnamed frames reroutes dispatch: `onmessage` fires ONLY for unnamed frames, so existing clients silently stop receiving everything. Infra trap: a buffering reverse proxy (nginx default) delays events unboundedly until `X-Accel-Buffering: no` is set.
**Safe change:** Assign monotonic `id:` from day one and implement replay from `Last-Event-ID` (or send an explicit `snapshot` event on connect); never add/rename the `event:` field on an existing stream; keep event names in a shared constant when the client is in-repo; verify proxy buffering headers whenever the route moves.