# Design: Durable / Checkpointed Agent Runs Status: Phase 1 + Phase 2 implemented (flagged, off by default); Phase 0 shipped; internal per-step checkpointing (Option A core) still recommended-not-built. Owner: core / run-manager + deploy Related code: `packages/core/src/agent/run-manager.ts`, `packages/core/src/agent/durable-background.ts`, `packages/core/src/agent/production-agent.ts`, `packages/core/src/server/agent-chat-plugin.ts`, `packages/core/src/deploy/build.ts`, `packages/core/src/deploy/workspace-deploy.ts`, `packages/core/src/agent/engine/builder-engine.ts` > **Final architecture, in one line.** Durable background runs are a > **host-agnostic core** — the foreground turn fires an HMAC-signed _self > dispatch_ to a sibling worker route, the worker runs the full multi-step loop > and persists every event to SQL, and the browser streams those events through > the existing cross-isolate SQL-poll reconnect path — with the **Netlify 15-min > `-background` function as a per-host optimization** layered on top. The > portable baseline works on any host that can re-invoke itself (it just > server-chains continuations when the host budget is short); the Netlify layer > simply lets one invocation run ~15 min so a long turn finishes in **one** > chunk instead of many re-hydrating ones. Nothing about the baseline is > Netlify-specific; see [Final layered architecture](#final-layered-architecture). ## Problem Hosted agent runs are bounded by a ~40s soft timeout enforced in `run-manager.ts` (`DEFAULT_HOSTED_RUN_SOFT_TIMEOUT_MS = 40_000`, also the `HOSTED_SOFT_TIMEOUT_CEILING_MS`). That budget is deliberate and correct: it sits just under a stack of upstream walls that the framework does not control. When the soft timeout fires, the run-manager aborts the current chunk, persists the partial turn, writes a terminal event, and emits an `auto_continue` event (`reason: "run_timeout"`) so the client transparently resumes the turn in a fresh chunk. This works well for a single long model call that just needs more wall-clock time. It does **not** work well for long _multi-step_ operations. A turn that performs many sequential side effects — for example an agent appending many dashboard panels through many separate write calls, or any "do N independent mutations in a loop" workflow — fails in a characteristic way: - **Continuation thrash / re-hydration.** Each `auto_continue` chunk starts the model over from the rebuilt context. If the work isn't expressed as resumable progress, the model frequently re-reasons about and re-issues steps it already attempted in the previous chunk instead of advancing. Successive chunks burn their entire 40s budget re-deciding rather than completing new steps. - **Partial or zero net progress.** Because each chunk can be cut mid-step and the next chunk may redo earlier steps, the run can churn for many chunks while the _persisted_ end state barely moves — or, when nothing reaches a committed state before each cutoff, moves not at all. - **Silent "looked-done" failure.** A tool call that returned a success marker (✓) in an aborted chunk does not guarantee its effect was committed and survived the cutoff. The model can reasonably believe a step succeeded, report the whole task complete, and leave nothing (or only some rows) actually persisted. The user is told it worked; the data says otherwise. Net effect: long multi-step runs can spin indefinitely, never finish, and terminate with an untruthful "done" state. ## Goals 1. Long multi-step runs **complete reliably** — they make monotonic forward progress across continuation chunks rather than re-doing work. 2. The user always gets a **truthful terminal state**: either "completed, here is concrete proof (N of N persisted, ids …)", or an honest "did not finish, here is what was committed (M of N) and what remains" — never a false success. 3. No change to the upstream walls and no raising of the 40s soft timeout (see Guardrail). ## Non-goals - Raising or removing the soft timeout. It is correct; see Guardrail. - Changing the gateway, serverless function limits, or model call timeout. - Replacing `auto_continue`. Both approaches below build on it. ## Guardrail: the 40s soft timeout is correct and must not be raised `DEFAULT_HOSTED_RUN_SOFT_TIMEOUT_MS` / `HOSTED_SOFT_TIMEOUT_CEILING_MS` = `40_000` is intentional headroom under the upstream hard walls. Raising it does not buy more time — it just converts a graceful hand-off into a hard kill. The walls, in order: 1. **Builder model gateway foreground cap — ~45s.** Hosted foreground calls keep a 45s cap in `packages/core/src/agent/engine/builder-engine.ts`. Local/non-hosted and proven background-function calls use the longer background-style cap because they are not constrained by the synchronous function wall. 2. **Serverless function kill — ~60–65s.** The hosting function is terminated shortly after; the heartbeat then reaps the run row as `stale_run`. 40s leaves ~5s under the gateway wall to abort, persist the partial turn, write the terminal event, and emit a clean `auto_continue` so the client resumes. A larger value (production saw per-template overrides like `240_000`) pushes the cutoff past both walls, so `auto_continue` never fires and the run dies as `builder_gateway_timeout` / `stale_run` instead. The ceiling clamp in `resolveRunSoftTimeoutMs` exists precisely to defeat that footgun. **Do not raise it. Fix durability above the timeout, not by moving the timeout.** ## Approach options Both options keep the 40s budget and build on the existing `auto_continue` mechanism. They differ in _where the long work lives_. ### Option A — Checkpointed / idempotent continuation Keep the work inside the normal run/`auto_continue` loop, but make each continuation chunk **resume from committed progress instead of restarting**. Mechanism: - **Persist a progress record** for the operation (a checkpoint): the planned unit of work (the N items / steps), and which units are already committed. This lives in SQL so it survives chunk boundaries and function recycling, the same way run rows do. - **Idempotent steps.** Each step keys off a stable identity so re-issuing a completed step is a no-op (upsert by natural key, or "skip if checkpoint says done"). Re-hydration after `auto_continue` then can't double-apply or thrash — a redone step costs a cheap check, not a duplicate write. - **Resume, don't replan.** On `auto_continue`, the next chunk reads the checkpoint, skips committed units, and continues with the remainder. Progress is monotonic: every chunk that does anything moves the committed count up. - **Truthful terminal state from the checkpoint.** "Done" means the checkpoint shows N of N committed. If the run is cut for good (e.g. it exhausts a continuation budget), the checkpoint still reports M of N committed and the exact remainder — so the terminal message is honest by construction. Interaction with the walls and the 40s budget: - Fully respects the 40s soft timeout and the gateway/function walls — it never needs a single chunk to outlast them. It just makes the _sequence_ of chunks productive. - Works hand-in-glove with `auto_continue`: today a continuation can redo work; with a checkpoint, a continuation can only advance. Tradeoffs: - Pro: smallest change to the runtime model; no new infrastructure; the user keeps watching one live turn; degrades gracefully (even a half-finished run is truthful and re-runnable). - Pro: directly kills re-hydration thrash, the actual failure mode. - Con: still bounded by however many continuation chunks the client/turn budget allows. A truly enormous job (thousands of steps) can still run out of chunks — but it now ends _truthfully partial and resumable_, not silently empty. - Con: requires per-operation work to define the unit of progress and make steps idempotent. Best paid down once at the primitive/action layer (see Tie-in) so individual agents don't have to. ### Option B — Out-of-band durable background execution Hand a long run to a **queued background job** that executes beyond the function/gateway lifetime and reports progress back into the run/event stream. Mechanism: - The foreground turn **enqueues** a durable job (the full operation + its inputs) and returns immediately with "started, tracking as job X". The user-facing run does not try to do the work itself within 40s. - A durable worker (outside the per-request serverless function lifetime — e.g. the core run-manager / agent-teams background infrastructure the framework already mandates for background agents) runs the job to completion, free of the 45s gateway cap and the ~60s function kill on the _original_ request. - The worker **streams progress** (committed counts, ids, errors) back so the UI and the agent can observe and the final state is truthful. Interaction with the walls and the 40s budget: - Sidesteps the gateway/function walls for the _long_ work by moving it off the request path. The walls still apply to each individual model call the worker makes, so the worker itself should checkpoint internally (i.e. Option B is strongest when it contains Option A). - `auto_continue` becomes a lightweight "is the job still running / what's its progress" poll on the foreground turn rather than the vehicle for the work. Tradeoffs: - Pro: removes the hard ceiling on total operation length — genuinely large jobs can finish. - Pro: the foreground turn stays responsive and cheap; the user can leave and come back. - Con: more infrastructure and lifecycle complexity (job queue, durable worker, progress fan-in, failure/retry semantics, surfacing job state in the UI and to the agent). - Con: changes the UX from "one live turn" to "fire-and-track"; needs clear status surfacing so it doesn't become its own kind of silent failure. ## Recommendation Build **Option A first**, then layer **Option B** for the genuinely unbounded cases. Option A delivers the most reliability per unit of effort: it directly removes re-hydration thrash and silent looked-done failure for the common case (tens of steps), needs no new infrastructure, and makes terminal state truthful by construction. Option B is the right ceiling-remover but is a larger build and is most valuable _on top of_ a checkpointed core (the durable worker should itself checkpoint). ### Phased plan > Status: **Phase 0 shipped. Phase 1 + Phase 2 implemented** (the host-agnostic > durable-background worker and the Netlify 15-min `-background` optimization), > default-on for deployed Netlify apps with an explicit opt-out via > `AGENT_CHAT_DURABLE_BACKGROUND=false`. Internal per-step > checkpointing (the Option A core of Phase 1) is recommended-not-yet-built; the > worker today gets its durability from the long single invocation plus > server-chained continuations rather than per-step idempotent checkpoints. See > [Final layered architecture](#final-layered-architecture). 1. **Phase 0 — Stop hitting the ceiling so often (near-term, cheapest).** Land the mitigations in the Tie-in below (one-call atomic primitives, self-documenting actions, loud termination, proof-of-done verification). These don't fix the ceiling but sharply cut how often multi-step loops are even attempted, and make the failures that remain _loud and truthful_ instead of silent. Capture the agent-facing half as the `reliable-mutations` skill. _(Shipped.)_ 2. **Phase 1 — Checkpointed continuation (Option A).** Add a SQL-backed progress checkpoint for long operations and make their steps idempotent/resumable so each `auto_continue` chunk advances committed progress instead of replanning. Drive terminal state ("N of N", or "M of N + remainder") from the checkpoint. This is the primary reliability win. _(The durable-background worker + host-agnostic self-dispatch/SQL-event/reconnect baseline is implemented; per-step idempotent checkpointing is still recommended-not-built.)_ 3. **Phase 2 — Durable background execution (Option B).** For operations that can exceed any reasonable number of continuation chunks, enqueue them onto the core background infrastructure, have the durable worker run them to completion (checkpointing internally per Phase 1), and stream truthful progress back to the foreground run and UI. _(Implemented: host-agnostic worker baseline + Netlify 15-min `-background` per-host optimization, flagged off by default.)_ ## Tie-in: cheaper near-term mitigations reduce, but do not replace, the fix The following reduce _how often_ the 40s ceiling is hit and make the remaining failures honest. They are valuable and should ship first (Phase 0), but the **actual fix is durable/checkpointed runs** (Phases 1–2): - **One-call atomic primitives.** Where an action can accept the whole batch (e.g. "set all panels" / "append many in one call"), a single call commits atomically inside one chunk instead of looping N writes that race the budget. - **Self-documenting actions.** Action descriptions that steer agents toward the atomic/batch call and away from per-item loops. - **Loud termination.** On a time-budget cutoff, fail loud with what was and wasn't committed — never report success on an aborted chunk. - **Proof-of-done verification.** After a write, re-read the end state and report concrete proof (counts/ids) rather than trusting a tool ✓. The agent-facing rules for these live in the `reliable-mutations` skill (`.agents/skills/reliable-mutations/SKILL.md`). They lower the blast radius; checkpointed and durable runs remove the ceiling itself. --- # Final layered architecture Status: implemented (default-on for deployed Netlify, explicit opt-out) Owner: core / run-manager + deploy Durable background runs ship as **two layers**. Deployed Netlify apps enable the path by default and can opt out with an explicit falsy `AGENT_CHAT_DURABLE_BACKGROUND` value. Other hosted platforms retain explicit opt-in. When off, the agent-chat run path and deploy output are byte-for-byte the pre-existing synchronous behavior. ### Layer 1 (portable baseline) — host-agnostic durable execution This is the actual durability mechanism and it is **not tied to any host**: - **Self-dispatch worker.** The foreground POST claims the run slot, inserts the run row, and `fireInternalDispatch`es (`server/self-dispatch.ts`) an HMAC-signed request to a sibling worker route, `AGENT_CHAT_PROCESS_RUN_PATH = /_agent-native/agent-chat/_process-run` (`agent/durable-background.ts`). The worker re-enters the same agent-chat handler set as the background worker and runs the full multi-step `runAgentLoop` (`agent/production-agent.ts`). - **SQL event log as the transport.** The worker persists every event to `agent_run_events` (`run-store.ts`, idempotent on `(run_id, seq)`). The browser streams those events through the existing `subscribeToRun` → cross-isolate SQL-poll path (`run-manager.ts subscribeFromSQL`), so the client needs no change and reconnect/leave-and-return already works (`GET /runs/active`, `GET /runs/:id/events?after=N`). - **Idempotent claim + auth.** The worker claims the run with a conditional update (`claimBackgroundRun`) so a duplicate delivery no-ops, and verifies the HMAC token (`prepareProcessRunRequest`) exactly like the agent-teams / A2A / webhook processors. - **Continuation by self-chaining, not by the browser.** If a worker chunk hits its soft-timeout unfinished, it emits `auto_continue` and **re-fires another background dispatch** (mode `continue`) instead of bouncing back to the client. On a host with a short invocation budget this just produces more chunks; the run still completes, server-driven. This is what makes the baseline portable: it degrades to chained self-dispatch on any host that can re-invoke itself. Nothing in Layer 1 assumes Netlify, a 15-minute budget, or a particular preset. It is the host-agnostic Option B worker with a SQL fan-in. ### Layer 2 (per-host optimization) — Netlify 15-min `-background` function On Netlify, Layer 1's worker invocation can be made to run for up to **15 minutes** in a single shot, eliminating almost all re-hydration (the costly part). This is purely an optimization of _where_ Layer 1's worker runs: - **Deploy emit (build-time gated).** When the flag is set at build time, the deploy emits a **second** Netlify function whose name ends in `-background`, re-exporting the **same** `main.mjs` handler bundle, with a `config.path` of the process-run route. Netlify invokes any function whose deployed name ends in `-background` asynchronously (202 immediately, up to 15-min budget). Single template: `emitSingleTemplateNetlifyBackgroundFunction` (`deploy/build.ts`). Workspace: `emitNetlifyBackgroundFunction` (`deploy/workspace-deploy.ts`), per app with a base-path-scoped `config.path`. **When the flag is not set at build time, neither emit runs and the single-function output is byte-identical to today** (see [Build-time gate](#build-time-gate-byte-identical-when-off)). - **Raised soft-timeout on that invocation only.** When the worker is running inside the background function it calls `startRun(..., { backgroundFunction: true })`, which lifts `resolveRunSoftTimeoutMs`'s hosted ceiling from 40s to `BACKGROUND_SOFT_TIMEOUT_CEILING_MS` (~13 min, ~2 min under Netlify's 15-min hard kill) for that invocation. The interactive/foreground 40s clamp is unchanged (see Guardrail). If Layer 2 is absent (or its routing resolves to the synchronous function), Layer 1 still works — the run just completes via more, shorter, server-chained chunks. **No regression, only a missed optimization.** ### Other hosts Any host that can re-invoke itself runs Layer 1 unchanged (chained self-dispatch). A host-specific Layer 2 would mirror the Netlify pattern: emit or configure a long-lived async worker invocation and pass `backgroundFunction: true` to `startRun` on that path. No such layer is built for non-Netlify hosts yet; they get the portable baseline. ## Build-time gate (byte-identical when off) The safety-critical claim is that with `AGENT_CHAT_DURABLE_BACKGROUND` **unset at build time**, the emitted Netlify deploy output is unchanged. How it is guaranteed: - The only code that produces the `-background` artifact is the two emit functions above, and each is reached **only** through a call site guarded by `isDurableBackgroundDeployEnabled()` (single template: `deploy/build.ts`, inside the `preset === "netlify"` block; workspace: `deploy/workspace-deploy.ts`, inside `copyNetlifyFunctionIntoWorkspace`). `isDurableBackgroundDeployEnabled()` returns `false` for an unset or non-truthy flag, so when the flag is absent the emit functions are **never invoked** — no second function directory, no extra entry file, no extra `config.path` route is written. The deploy walks exactly the same code path it does today and emits exactly one function per app. - The gate reads the flag at **build time** (in the deploy process env), so a build produced without the flag can never contain the artifact regardless of runtime env. - This is covered by tests that run the real deploy path with the flag unset and assert that no `*-agent-background` directory exists, and with the flag set and assert that exactly one does (`deploy/workspace-deploy.spec.ts`, `deploy/build.spec.ts`). --- # Netlify Layer 2: concrete implementation notes This section details the Netlify-specific Layer 2 optimization (the deploy emit and the 15-min budget), reusing the framework's existing background-run machinery and Netlify's **background functions** (async, up to **15 minutes**). It is deliberately not a from-scratch system: ~90% of the required plumbing already ships for Agent Teams, A2A, and integration webhooks — and the host-agnostic Layer 1 above is what actually carries the run. ## Why a 15-min background invocation changes the math The whole `auto_continue` / 40s soft-timeout dance exists to stay under the **serverless function wall** (~60–65s synchronous on Netlify), not under any model limit. Evidence: - `builder-engine.ts` allows the long background gateway cap in local/non-hosted runs; the run loop has no inherent reason to stop at 40s. - `run-manager.ts:58,68` `DEFAULT_HOSTED_RUN_SOFT_TIMEOUT_MS` / `HOSTED_SOFT_TIMEOUT_CEILING_MS = 40_000` are pinned just under the function wall, and `templates/brain/netlify.toml` sets `[functions."*"] timeout = 75`. Netlify **background functions** (any function whose deployed name ends in `-background`) are invoked asynchronously: the HTTP POST returns `202 Accepted` immediately and the function runs detached for up to **15 minutes**. Inside that function there is no ~60s wall, so: - The agent loop can run for minutes in a single invocation with **few or no `auto_continue` continuations**. - The foreground hosted per-model-call gateway cap still applies per call — see [Per-model-call gateway cap](#per-model-call-gateway-cap) — but the _run_ is no longer chopped into 40s chunks. This is exactly the Layer 1 worker with a concrete long-lived host: Netlify is the durable worker, reached through the existing self-dispatch primitive — the same primitive that, absent this layer, just chains shorter invocations. ## What already exists and is reused verbatim The Agent Teams background processor is the template. The chat path can reuse nearly all of it: | Capability | Existing code | Reuse for chat | | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Fire a fresh function invocation with its own budget | `server/self-dispatch.ts:122` `fireInternalDispatch()` (HMAC-signed POST, 250ms settle race) | Point it at the `-background` function path instead of the same-function processor route | | Processor route pattern (claim → run full loop → persist → finalize) | `agent-teams.ts` `processAgentTeamRun()` + route at `agent-chat-plugin.ts:6001` (`AGENT_TEAM_PROCESS_RUN_PATH = /_agent-native/agent-teams/_process-run`) | Add a sibling `/_agent-native/agent-chat/_process-run` that runs the **chat** loop | | HMAC processor auth | `integrations/internal-token.ts:61,72` `signInternalToken` / `verifyInternalToken` (5-min TTL, task-id-bound, timing-safe); gated by `A2A_SECRET` | Same — sign with `runId` | | Atomic single-claim (no double-processing) | `agent-teams-run-queue.ts:177` `claimAgentTeamRun()` (`UPDATE … WHERE status='queued' OR (status='running' AND updated_at < stuckCutoff)`) | Same shape; the chat already has an equivalent with `updateRunStatusIfRunning()` (`run-store.ts:392`, `UPDATE … WHERE id=? AND status='running'`) and `tryClaimRunSlot(threadId)` | | The actual multi-step agent loop | `runAgentLoop()` (imported into both `production-agent.ts` and `agent-teams.ts:1790`) | Identical — call it from the background processor | | Event persistence (SQL, ordered, idempotent) | `run-store.ts`: `insertRunEvent(runId, seq, json)` with `ON CONFLICT (run_id, seq) DO NOTHING` (`:452,466`); `getRunEventsSince(runId, fromSeq)` (`:471`) | Unchanged — background worker writes the same `agent_run_events` rows | | Client reconnect / replay by cursor | `agent-chat-plugin.ts:7103` `GET /runs/:id/events?after=N` → `subscribeToRun(runId, after)`; cross-isolate SQL polling path `run-manager.ts:750` `subscribeFromSQL()` (polls `getRunEventsSince` every 500ms) | **This is the key reuse**: the background worker is a different isolate, so the client already falls through to the SQL-polling subscription. Reconnect already works across isolates. | | Discover an in-flight run on reload | `agent-chat-plugin.ts:7130` `GET /runs/active?threadId=X` (returns `runId`, `turnId`, `status`, `heartbeatAt`, `lastProgressAt`, `serverNow`) | Unchanged | | Heartbeat + stale reaper | `run-manager.ts:340` (1.5s heartbeat), `run-store.ts:19` `RUN_STALE_MS = 15_000`, `reapIfStale` / `reapAllStaleRuns` | Reused; see [Failure handling](#failure-handling-and-loud-terminal-state) for the 15s-vs-15min tension | | Stuck-dispatch re-fire / reconcile | `agent-teams.ts:689` refire, `reconcileAgentTeamRunsForOwner` (`:797`) | Optional reuse for chat reconcile | The single genuinely **new** infrastructure piece is **deploy-time function splitting**: today the Nitro `netlify` preset emits exactly one function (`.netlify/functions-internal/server/` → `server.mjs` re-exporting `main.mjs`, patched in `workspace-deploy.ts:670` `patchNetlifyFunctionEntry`; routed via the `config.path` array at `:742`). We must additionally emit a second function whose name ends in `-background` that re-exports the **same** `main.mjs` handler. ## The dispatch flow (designed) ``` ┌── Browser ─────────────────────────────────────────────────────────────┐ │ 1. POST /_agent-native/agent-chat { message, threadId, turnId } │ └────────────────────────────────────────────────────────────────────────┘ │ (foreground function: the normal interactive handler) ▼ ┌── agent-chat handler (production-agent.ts) ────────────────────────────┐ │ 2. tryClaimRunSlot(threadId) → 409 if a run is already active │ │ 3. runId = generateRunId(); insertRun(runId, threadId, turnId) │ │ (status='running', heartbeat set) — run-store.ts:233 │ │ 4. IF durable-background enabled (hosted + flag + A2A_SECRET): │ │ fireInternalDispatch({ │ │ path: BACKGROUND_FUNCTION_INVOKE_PATH, // 202, detached │ │ taskId: runId, body: { threadId, turnId, message, … } }) │ │ return SSE stream = subscribeToRun(runId, 0) // immediately │ │ ELSE (local / flag off): │ │ startRun(runId, …) inline (today's behavior, unchanged) │ └────────────────────────────────────────────────────────────────────────┘ │ 202 leaves the box (250ms settle race) ▼ ┌── Netlify background function (…-background, up to 15 min) ───────────┐ │ 5. Route: POST /_agent-native/agent-chat/_process-run │ │ 6. verifyInternalToken(runId, bearer) // internal-token.ts:72 │ │ 7. atomically claim runId (updateRunStatusIfRunning-style guard) │ │ 8. raise gateway cap for this invocation (see §gateway-cap) │ │ 9. startRun(runId, …, { softTimeoutMs: 0 OR ~13min }) │ │ → runAgentLoop(...) runs the FULL multi-step turn │ │ → emitRunEvent(...) persists every event to agent_run_events (SQL) │ │ 10. on finish: updateRunStatusIfRunning(runId,'completed'|'errored') │ │ + terminal event ('done'/'error') persisted │ └────────────────────────────────────────────────────────────────────────┘ ▲ writes SQL events │ reads SQL events (cross-isolate) ┌── Browser (same SSE response from step 4, OR reconnect) ──────────────┐ │ 11. The SSE stream is subscribeToRun(runId,0). Because the producer is │ │ a *different* isolate, it serves via subscribeFromSQL() — │ │ run-manager.ts:750 — polling getRunEventsSince every 500ms. │ │ 12. On disconnect/reload: GET /runs/active?threadId → runId+lastSeq, │ │ then GET /runs/:id/events?after=lastSeq resumes the same stream. │ └────────────────────────────────────────────────────────────────────────┘ ``` ### Endpoints / constants to add - `BACKGROUND_FUNCTION_INVOKE_PATH` — the Netlify async path, `/.netlify/functions/-agent-background` (Netlify maps `…-background` to async/202). The dispatch URL is built with `resolveSelfDispatchBaseUrl(event)` (`self-dispatch.ts:59`) + that path. (Alternatively, give the background function a `config.path` like `/_agent-native/_bg/*` so dispatch stays a clean framework path and Netlify still routes it to the `-background` function and invokes it async.) - `AGENT_CHAT_PROCESS_RUN_PATH = "/_agent-native/agent-chat/_process-run"` — the handler the background function actually runs (sibling to `AGENT_TEAM_PROCESS_RUN_PATH`). It is reached _through_ the background function, so it inherits the 15-min budget. - Feature flag: `AGENT_CHAT_DURABLE_BACKGROUND` (env or per-app config), defaulting **off**, gated additionally on `isHostedRuntime()` and `hasConfiguredA2ASecret()`. Local dev keeps the inline path so SSE stays a single live stream and no second function is needed. ## Where the soft-timeout / auto_continue logic changes Today (`run-manager.ts:344`) every hosted run gets a 40s soft timeout that emits `auto_continue` and the client re-POSTs. In the background-function path: - **The background `startRun` gets `softTimeoutMs` ≈ 13 min** (a margin under Netlify's 15-min hard kill), not 40s. To allow this, `resolveRunSoftTimeoutMs` must learn a **background context**: the `HOSTED_SOFT_TIMEOUT_CEILING_MS = 40_000` clamp (`run-manager.ts:170`) currently _defeats_ any larger value on hosted. Add a `backgroundFunction: true` option that raises the ceiling to ~`780_000` (13 min) for that one invocation. **Do not** change the default hosted ceiling — the 40s clamp stays correct for the interactive/foreground path; the Guardrail above still holds for non-background runs. - **`auto_continue` becomes the rare exception, not the rule.** Most turns finish inside 13 min with zero continuations, killing the re-hydration thrash described in the Problem section. If a turn _does_ exceed 13 min, the existing mechanism still works: emit `auto_continue` and re-fire **another background dispatch** (mode `continue`) exactly as `agent-teams.ts:1886` does — i.e. the continuation chains background invocations instead of bouncing back to the browser. The client never has to drive continuation. - **Internally, the worker should still checkpoint (Option A).** 15 min is large but not infinite; combining background execution with idempotent/checkpointed steps means even a continued run advances monotonically. Option B is strongest containing Option A. ## How the client UX changes (SSE → SSE-over-SQL + reconnect) Minimal client change, because the reconnect machinery already exists: - **The response shape is unchanged.** Step 4 still returns an SSE `ReadableStream` from `subscribeToRun(runId, 0)`. The client keeps reading SSE exactly as today. The only difference is that the events are produced in another isolate and arrive via the 500ms SQL-poll path (`subscribeFromSQL`, `run-manager.ts:750`) instead of the in-memory fast path. Latency goes from ~instant to ≤500ms per event — acceptable for chat. - **Reconnect/leave-and-return becomes a first-class, reliable flow.** Because the producer is detached on Netlify, closing the tab no longer kills the run. On return, the client calls `GET /runs/active?threadId` (`:7130`) → gets `runId` + status, then `GET /runs/:id/events?after=` (`:7103`) to replay from the cursor. This already works; we are just making it the primary UX. Recommend the client persist `lastSeq` per thread so reconnect resumes precisely. - **Optional: drop the long-held foreground SSE entirely and poll.** Instead of holding step-4's SSE open against the 75s foreground-function `timeout` (`netlify.toml`), the foreground POST can return `{ runId, turnId }` (202-style JSON) and the client immediately opens `GET /runs/:id/events?after=0`. This avoids tying up a foreground function for the run's lifetime. Either works; the JSON-then-poll variant is cleaner on Netlify because the interactive function returns in well under 75s. ## Event-driven viewer access revocation **Revised proposal, 2026-10-13. Implementation and independent proof remain pending.** The inspected run subscriptions authorize at admission without reauthorization in their delivery loops. This section defines the revised target, not deployed behavior. The [tenancy ADR](organization-team-tenancy.md#proposed-stream-policy-amendment-2026-10-13) defines linked-conversation access. The revision uses access-change events instead of frequent authorization renewal for each subscription. It does not establish immediate revocation or a maximum stale-access window. A guaranteed bound remains a separate policy decision. ```text Committed access change -> recoverable notification to affected stream hosts -> invalidate the decision and pause protected delivery -> recheck current viewer authority and applicable grants -> resume if allowed, or close the viewer if denied ``` 1. **Scope:** The contract covers authorized viewer subscriptions, including non-team and standalone runs under their applicable policies. Supported public access retains its existing rights. A missing linked conversation denies access. This contract creates no new public live-stream grant. 2. **Access changes:** Relevant changes include organization and group/team membership, deletion, sharing and applicable account/session or capability revocation. Publication must cover every relevant mutation path, not only the Teams UI. A local event emitter is not cross-host delivery. 3. **Server invalidation:** Each affected host invalidates its cached decision and pauses protected delivery. A current-authority recheck evaluates the original viewer and complete applicable policy. An event prompts a recheck, not unconditional denial. Another valid access path permits continuation. An unavailable authority cannot approve resumed delivery. 4. **Delivery gate:** A synchronous local gate precedes each protected write at the final application-controlled writer. This gate does not query membership for every event. Known invalidation blocks live, SQL-polled, replayed and buffered content until a current recheck succeeds. A newer invalidation must prevent an older in-flight approval from reopening delivery. Only demonstrably content-free heartbeats can bypass the gate. Bytes already handed to transport cannot be recalled. The gate cannot detect a revocation that its host missed. 5. **Viewer and producer:** Confirmed denial closes only the viewer subscription. Late approvals cannot revive a closed subscription. The producer continues. Client notifications support UI updates, not security enforcement. Ignored client notifications cannot preserve server delivery rights after invalidation. 6. **Admission and recovery:** New connections and reconnects require current admission before protected replay. Authorization snapshots, event registration and change-log positions need consistent ordering to prevent missed revocation. Catch-up after restart or reconnect must complete before protected delivery. The existing client recovery requirements remain applicable. ### Publication, freshness and outage design gates Access mutations need a durable change record coupled to successful commit. Commit followed by best-effort publication alone can lose a revocation. The transport must reach every affected stream host, not one competing consumer. Replay, duplicates, gaps, restarts and scale-out require a demonstrated recovery mechanism. Recipient routing must preserve access boundaries and avoid disclosing protected policy details. The existing event bus is process-local. Durable sync replay is precedent, not proof of distributed stream revocation. Shared reconciliation of an authoritative change log can reduce repeated per-stream membership reads. This reconciliation can itself use polling. Its feasibility depends on the actual serverless, stream-host and gateway topology. No shared watcher, broker or outbox implementation is selected here. A connected feed or caught-up cursor does not prove current authority. A committed change can remain unpublished while the host appears healthy. A guaranteed stale-access bound needs an authoritative freshness deadline and a demonstrated backstop at the delivery gate. Admission token expiry does not automatically close an admitted stream. The 15-minute chat token is not a universal viewer lease, and other stream families have different admission mechanisms. Known invalidation pauses protected delivery even when a recheck fails or times out. The pause does not approve access or assert producer failure. The design must separately select outage behavior for missed events or uncertain feed freshness. The maximum stale-access window, any deadline-triggered closure and the response to feed uncertainty remain unresolved. No implementation can claim a guaranteed bound until those choices and their authority mechanisms have proof. ### Superseded lease policy, 2026-10-10 through 2026-10-12 The previous proposal selected a ten-second maximum authorization age from authoritative check start. Option A kept an idle connection open after expiry without protected delivery rights. The 2026-10-11 fault option retained an existing approval only until its original deadline, then closed the faulted viewer unless a fresh check approved continuation. No valid approval meant immediate closure. These lease-specific choices are superseded, not runtime guarantees. The revised proposal does not retain their deadline or fault grace. The 2026-10-11 bounded recovery decision and the 2026-10-12 closure/reset, recovery limits and baseline-driven validation decisions remain applicable. ### Viewer closure and recovery The closure contract accepted on 2026-10-12 reuses the shared-client subscription error envelope, without a policy-specific reason or producer-terminal event. Confirmed denial is non-recoverable; backend fault is recoverable within the finite episode. Fresh rejected opens retain the existing HTTP classification: 401/403/404 stops automatic retries; 408/429/5xx can retry within the budget. Bare EOF is a transient interruption requiring fresh authorization, not proof of denial or producer termination. The existing client does not consistently consume these distinctions; this is a target contract, not shipped behavior. A healthy reset requires fresh authoritative admission and forward protected cursor progress observed by the client. Opening, heartbeats and old replay cannot reset an episode. A verified producer terminal result ends recovery normally. An idle recovered connection retains episode history; exhaustion prevents later automatic reconnects without closing a healthy idle connection. Keep accounting inside the shared Core client, with no app-level counters, timers, configuration or persisted state. These accepted closure/reset requirements still need implementation and compatibility proof. Accepted recovery limits are 250 ms initial exponential backoff, a 15-second backoff cap, 55 seconds total elapsed per episode including waits and in-flight attempts, and 12 total attempts including rejected opens and immediate-close cycles. The earlier ceiling wins; opening alone never resets the episode and background subscriptions have no exemption. Bound each attempt by remaining episode time. Honor Retry-After only within the remaining wait and attempt budget; otherwise stop rather than retry early. These limits do not authorize delivery after known invalidation or define a freshness deadline. ### Current authority and supported admission Credential freshness must bypass both request-event memoization and credential provider caches. Core's pinned Better Auth 1.7.6 uses a five-minute signed cookie cache; a library probe still accepts that cookie after session-row deletion unless cache bypass is requested. Token signatures and streaming POST owner seeds likewise do not establish current membership or session authority. Resolver-specific revocation and external-policy contracts remain necessary. Preserve the supported admission families. Explicit-thread POST prepares the thread and requires editor access; POST without an explicit thread can start with the run ID as its effective thread ID. Neither creates a new standalone GET grant. Linked run-events GET checks the current linked conversation and denies a missing target. Public token GET returns a validated unbound transcript and limited run summaries, not a live subscription. A purpose-bound POST JWT is a producer handoff, not renewal of its original viewer credential. Custom session and anonymous-owner APIs keep their existing signatures and fallback order. Their owners must identify the governing credential, refresh and revocation semantics before they can support current-authority checks. Repeated callback invocation or a returned owner string alone is not that proof. Static auth-disabled/anonymous authority must not acquire an invented revocable session; applicable revocable resource policy still applies. A current access decision includes the applicable credential, linked binding, organization and team membership, and alternative valid grants. Its snapshot and event position need consistent ordering with admission and invalidation. Current sequential checks do not establish that ordering. External federation can consult remote authority and change local membership. No shared revision or ordering primitive is established. Uncertain authority cannot approve delivery. A fresh remote call and SQL transaction alone do not prove one coherent decision. The revised design must establish the necessary authority mechanism without changing permissions or assuming an existing lease API. ### Earlier authority candidates, 2026-10-12 These candidates record earlier exploration under the superseded lease proposal. They are not selected mechanisms or implementation prerequisites in their exact form. The event-driven design still needs current authority and consistent ordering. It does not mandate encrypted proof handoff, remote epochs or a new credential store. The candidate descriptions retain their original assumptions. **Original proof across hosts.** Capture the successful credential family and the inputs its authority actually validates, not just its returned identity. The purpose-token receiver needs confidential original-proof presentation bound to the destination audience, original viewer, applicable request scope and expiry. A client-carried authenticated encrypted envelope is a candidate, not an existing API. Its key distribution, bounded lifetime, replay protection and callback-input coverage need demonstrated mechanisms before selection. A signed owner claim, an in-process token-mint record or encryption without replay protection is insufficient. Do not default to a shared secret escrow: no reviewed store or atomic redemption facility is established, and this would add persisted sensitive bookkeeping. Retain usable proof only for the subscription; close releases it, and reconnect presents newly captured proof. Producer context remains separate. **Current callback authority.** Each credential family needs a documented provider check that bypasses request and provider caches. Classify its authority as revocable with a demonstrated current-proof validator, deliberately static, or not yet characterized. Preserve callback signatures, original request inputs and fallback order. A callback's `null` retains its fallback meaning; it is not automatically a confirmed denial. Keep approval, confirmed denial, absence, unreadable authority and timeout distinct through the complete decision. Static identity still undergoes current resource-policy checks. An uncharacterized provider is a readiness dependency, not a permanently faulting replacement for a previously supported mode. **One coherent policy decision.** Wholly SQL-governed authority uses a primary read-only repeatable-read transaction with its first snapshot acquired at or after check start. Pass that transaction through every applicable reader. For mixed remote/SQL authority, a conditional version-bracket design reads remote grant and epoch, acquires the complete SQL snapshot, then reads remote authority again. It works only if reads are linearizable, the monotonic epoch changes on every relevant mutation including revoke/regrant, and any projected membership in the SQL snapshot is demonstrably tied to that epoch. Both remote reads must grant at the same epoch. The current membership validator can mutate local rows; an unfenced validator write or lagging projection cannot satisfy this contract. No inspected provider establishes these guarantees. Ordinary repeated Boolean checks are rejected. The missing provider/fence capability blocks selection; uncertainty cannot approve or silently convert remote authority into local policy. ### Separate delivery safety requirements Finite buffering, backpressure and lossless replay remain delivery requirements. They are separate from the choice of revocation transport or freshness deadline. The following source/writer design remains a candidate, not a selected migration. **Bounded lossless source and writer.** Current SQL subscriptions call the event reader without its optional row limit, then parse and encode complete event strings. Row-count batching is useful but cannot bound one oversized row. Memory replay also iterates complete shared events. A candidate source interface must expose immutable event metadata and bounded payload reads before driver return or viewer encoding; a range-readable payload with SQL references is one candidate, not evidence of an existing storage migration. Legacy inline events need a demonstrated lossless bounded-read path too. Reject fetching a whole row and slicing it afterward. For each source and supported host, record finite engineering values for source chunk bytes, rows per batch, source-buffer bytes, encoder carry, queued encoded bytes, maximum final-write chunk and adapter high-water mark. Include temporary copies and one in-flight chunk in the summed viewer allocation bound; account shared producer storage separately. Pull only when capacity exists and gate each protected final write synchronously. Pressure closes that viewer recoverably, without pausing the shared producer or another viewer. Oversized logical events retain their identity and bytes. Bounded wire frames require compatible parsing and lossless reassembly; a protocol that demands an unbounded intermediate value is not a solution. Exact geometry and adapter primitives remain selection gates, not the superseded performance ceilings. The server can observe a complete frame handed to its writable, not receipt by the client. The client alone advances its resume cursor after a complete logical event. On interruption, discard uncommitted partial assembly and replay from the client's previous complete cursor after fresh admission. Do not add an acknowledgment protocol merely to mislabel server handoff as client observation. On 2026-10-11, the user approved the viewer-only heading **Live updates unavailable**, reusing the existing inline connection alert and Reconnect control where recovery is permitted. This must not report that the agent run failed or disclose the access-policy reason. Confirmed denial still stops automatic retries; a manual reconnect must validate access again. Implementation must update the English source and configured locale translations together. ### Verification and existing-path observations Before implementation, resolve commit-coupled publication, recoverable fan-out, admission ordering, current authority and outage behavior. Independently review the proof plan. A guaranteed freshness bound needs a separate accepted requirement and authoritative backstop. Closure/recovery and baseline-driven validation remain accepted. Finite lossless source/queue designs remain separate delivery obligations. Existing-path measurements inform feasibility, not event-driven revocation compliance. Implementation establishes justified query-count baselines and focused proof. Independent integrated verification must cover safety, finite queues, query regression, faults, buffering, client behavior and synchronized hosts before release. Characterize latency, CPU/IO, connections and memory before selecting additional ceilings. Policy approval alone is not runtime completion or readiness. Independent discovery now connects current credential-backed Core admission and SQL subscriptions in one local fixture with 32 distinct credential/team/share scopes across four organizations. Twenty-second mixed windows and a separate 20.187-second delivery window exercise actual SQL SSE; the latter delivers 2,560 matching events with advancing cursors while fresh grant/deny checks run. Events are synthetic persisted inserts, not an agent producer. This is neither the mounted route nor a coherent lease. Separated-client, forced-GC server measurements compare 0/16/32/64 idle-socket cohorts. Matched controls show about 2.40/2.74–2.77/3.24–3.26 MB aggregate held heap differences. They include server transport/database caches and do not attribute owned memory per viewer. PostgreSQL process-tree CPU and cluster IO are observed separately from Node RSS, but are not phase-specific load ceilings. Idle authorization checks are zero in the fixture; SQL event polling remains. These observations are not production budgets or connection-capacity limits. Actual shared-client probes also expose compatibility defects: recovery retries HTTP denial despite non-retryable runtime classification; an unknown generic subscription error becomes producer-terminal failure; bare EOF without run authority becomes `run.failed/stream_ended`. Passing current-behavior assertions does not implement the accepted closure/reset contract or bounded recovery. The simulated Node writer gate still proves only a transport seam. No integrated event-driven revocation latency, authorization delivery-pause distribution or total protected queue bound is measured. Local overlapping workers are not independent hosts. Independent-host proof needs distinct execution nodes against one shared isolated PostgreSQL fixture with a synchronized barrier and observability. The validation strategy accepted on 2026-10-12 uses security/correctness invariants and measured query-count regression budgets. Characterize latency, PostgreSQL CPU/IO and server memory before selecting ceilings; the earlier full numeric proposal is superseded, not approved. Bounded source/materialization and queues still require concrete, finite design limits and lossless cursor/replay semantics. Independent integrated proof, including synchronized hosts, remains mandatory. Proof must cover committed-change publication, missed events, duplicate or reordered events, host restarts and admission races. It must demonstrate paused delivery on known invalidation across live, buffered, SQL-polled and replayed paths. Retained alternative grants and continued producer operation need proof. Feed outages need evidence against the selected policy, not an assumed deadline. Readiness requires concrete publication, authority, ordering, outage and delivery designs plus the reviewed proof plan. Existing fixture results do not establish the revised mechanism or authorize implementation now. ## Per-model-call gateway cap Builder gateway calls now use a runtime-aware cap. Hosted foreground calls keep the 45s cap so the synchronous function can still checkpoint before its hard wall. Local/non-hosted runs use a longer local cap, and proven Netlify background-function runs may use a background cap below the 15-minute function wall. The run-manager's 13-minute background soft-timeout should fire before the gateway cap in normal durable background operation. Options, in scope-order: 1. **Foreground hosted:** keep the 45s cap. The synchronous function wall is the constraint, and a larger cap would turn graceful checkpointing into a hard platform kill. 2. **Local/non-hosted:** allow longer calls by default. Local development should not inherit a serverless wall it does not have. 3. **Durable background:** allow longer calls only when the runtime proves it is inside the emitted background function. Keep the cap below 15 minutes and above the 13-minute run soft-timeout so background checkpointing still owns logical-turn continuation. **Recommendation:** preserve this split. Do not raise the foreground cap; use durable background for long-running tool-input generation. ## Idempotency / dedup Already strong; make the new claim match: - **Run claim.** The foreground inserts the run row (`insertRun`, `run-store.ts:233`) _before_ dispatching. The background processor must claim it with a conditional update (mirror `updateRunStatusIfRunning`, `run-store.ts:392`, or add a `claimRunForProcessing(runId)` that flips a `processing` marker only from the unclaimed state). A duplicate Netlify delivery (background functions can in theory be retried) then no-ops on the second claim, exactly like `claimAgentTeamRun` returning `null` (`agent-teams-run-queue.ts:177`). - **Event dedup.** `insertRunEvent` is already idempotent on `(run_id, seq)` via `ON CONFLICT … DO NOTHING` (`run-store.ts:452,466`). Re-emitting an event with the same seq is a safe no-op, so a retried/overlapping producer cannot duplicate the stream. - **Dispatch token.** `fireInternalDispatch` signs `runId` (`self-dispatch.ts:131`); the processor verifies it (`internal-token.ts:72`, 5-min TTL). A stale or forged dispatch is rejected. ## Failure handling and loud terminal state - **15s heartbeat vs 15-min runs — the one real conflict.** The stale reaper (`RUN_STALE_MS = 15_000`, `run-store.ts:19`; `reapIfStale`/`reapAllStaleRuns`) marks a run `errored` after 15s without a heartbeat. The background worker _does_ run `startRun`'s 1.5s heartbeat (`run-manager.ts:340`), so as long as the worker is alive the row stays fresh and the reaper leaves it alone — this is fine. The risk is a worker that is _slow to start_ (Netlify cold-start of the background function) leaving a freshly-inserted `running` row unheartbeaten for >15s, which the reaper would falsely kill. Mitigation: the foreground insert sets `heartbeat_at = now` (`insertRun`), and the background claim should bump the heartbeat immediately on entry; if cold starts can exceed 15s, widen `RUN_STALE_MS` for rows known to be background-dispatched (e.g. a `dispatch_mode` column or a separate, larger stale window for background runs). - **Worker dies mid-run (crash / 15-min kill).** Heartbeat stops, the reaper flips the row to `errored` and appends a synthetic terminal event (`reapIfStale`), so the client's SQL-poll subscription sees a terminal event and stops — no infinite spinner. This is the **loud terminal** contract from Phase 0 already wired. - **Dispatch never lands (202 lost).** Reuse the Agent-Teams reconcile pattern (`reconcileAgentTeamRunsForOwner`, `agent-teams.ts:797`): if a `running` row has no heartbeat after a grace window, re-fire the dispatch once, then fail loud. Optional for the first slice. - **Truthful terminal state.** Combined with the `reliable-mutations` Phase-0 proof-of-done discipline, a completed background run reports concrete proof; a killed one reports `errored` with what was committed — never a false success. ## Phased implementation plan (smallest working slice first) > Status: \*\*Slices 0–1 implemented and the Slice-3 background-aware stale window > > - background→background continuation chaining are implemented\*\*, default-on > for deployed Netlify apps with `AGENT_CHAT_DURABLE_BACKGROUND=false` as the > opt-out. The host-agnostic baseline > (Layer 1) carries the run on any host; the Netlify `-background` emit (Layer 2) > is the deploy-time optimization. Slice 2's richer reconnect-first client UX and > the internal per-step checkpointing (Option A) remain follow-ups; Slice 4 > (raising the per-call gateway cap) is intentionally out of scope (see > [Per-model-call gateway cap](#per-model-call-gateway-cap)). **Slice 0 — prove async dispatch on Netlify (no chat yet).** Emit one extra `-background` function in the deploy build that re-exports the existing `main.mjs` handler (extend `patchNetlifyFunctionEntry`, `workspace-deploy.ts:670`, and the single-template Netlify output in `deploy/build.ts:1935`). Add `AGENT_CHAT_PROCESS_RUN_PATH` returning a stub that just writes a run event. Dispatch to it from a temporary test route via `fireInternalDispatch`. **Done when:** a POST returns 202 immediately and the stub writes an event to `agent_run_events` from the background function, visible via `GET /runs/:id/events`. This de-risks the only genuinely new infra. **Slice 1 — route the real chat loop through the background function.** On deployed Netlify it is enabled by default; `AGENT_CHAT_DURABLE_BACKGROUND=false` opts out. Other hosted platforms require explicit opt-in, and all hosts require `A2A_SECRET`: the foreground handler inserts the run row, dispatches, returns the SSE stream from `subscribeToRun`. The background `_process-run` claims the run and calls the same `startRun` + `runAgentLoop` the inline path uses, with `softTimeoutMs ≈ 13min` (new `backgroundFunction` option in `resolveRunSoftTimeoutMs`). Builder gateway calls use the background cap only when the worker proves it is inside the emitted background function. **Done when:** a long multi-step turn that thrashes today completes in one background invocation with zero `auto_continue`, events streaming to the client via the SQL-poll path, terminal `done` persisted. **Slice 2 — reconnect/leave-and-return UX.** Make the client persist `lastSeq` per thread and, on load, use `/runs/active` + `/runs/:id/events?after=lastSeq` as the primary resume path. Optionally switch the foreground POST to return `{ runId, turnId }` JSON instead of holding SSE, then poll. **Done when:** closing/reopening the tab mid-run resumes the live stream with no lost or duplicated events. **Slice 3 — robustness.** Background-aware stale window (cold-start tolerance), reconcile/re-fire for lost dispatches, and background→background `auto_continue` chaining for the rare > 13-min turn (mirror `agent-teams.ts:1886`). Internal checkpointing (Option A) > for monotonic progress across any continuation. **Slice 4 — foreground remains capped.** Do not raise the hosted foreground per-call gateway cap. Any future tuning should stay in the local/background timeout regime unless the synchronous platform wall changes. ## Open risks / unknowns 1. **Netlify background-function invocation contract.** Confirm the exact trigger: name-suffix `-background` invoked at `/.netlify/functions/` returning 202, and whether a `config.path` route can also mark a function background. The deploy currently emits a single function — splitting it is the main new work and must not regress the existing `config.path` routing (`workspace-deploy.ts:742`). 2. **Cold-start vs 15s stale reaper** (detailed above) — the highest-likelihood false-failure; needs the background-aware stale window or an on-entry heartbeat bump. 3. **Hosted gateway >45s** — unknown whether upstream allows it; gates Slice 4. 4. **Two functions sharing one bundle** — both re-export the same `main.mjs`, so `includedFiles: ["**"]` (`workspace-deploy.ts:747`) must cover the background function dir too; verify bundle size and that env (`A2A_SECRET`, DB URL) is present in the background function's environment. 5. **Cost / concurrency** — background functions are billed and concurrency- limited differently; long runs hold a slot for minutes. Out of scope to solve, but flag for capacity planning.