# Competitive Gap #2: Provider circuit breaker + automatic model failover > **Verdict: ADAPT, DO NOT VENDOR.** Borrow the concept (per-model circuit breaker + cooldown + half-open recovery + failover down a precedence chain) from pi-maestro-flow, but implement it inside OUR deterministic VM / disk-journal / role-routing / Foundation fail-closed architecture. Never copy their `EndpointCircuitBreaker` class or config surface wholesale (per #137 adapt-don't-merge policy). --- ## Problem (what we lack, with evidence from the competitive analysis) Today the workflow engine has **task-level retry resilience**, not **provider-failure resilience**. Concretely: 1. `retry()` and `gate()` (`src/workflow.ts:2530`, `src/workflow.ts:2542`) retry a **thunk** — they re-run the same agent() call, hitting the **same model**, on the **same provider**. If a provider is 429-ing or its endpoint is down, retrying into it just burns the retry budget and fails. 2. The agent() retry loop (`src/workflow.ts:2010` `for (let attempt = 1; attempt <= maxAttempts; attempt++)`) wraps each attempt in `runWithAbortableTimeout` and calls `agentRunner.run(prompt, { model: modelSpec, ... })` with a **fixed `modelSpec`** resolved once at the top of agent(). There is no per-attempt model re-resolution and no circuit state consulted before the call. A provider outage therefore exhausts `agentRetries` (default `MAX_AGENT_RETRIES = 3`, `src/config.ts`) and the run fails — or, for `PROVIDER_USAGE_LIMIT`, checkpoints and pauses for a bounded resume that re-hits the same provider. 3. The only provider-aware logic is **classification**, not resilience: `classifyProviderLimit()` (`src/errors.ts`) detects usage/quota/429 text and `classifyWorkflowError()` maps it to `usage_exhausted` (pause+resume) vs `transport_transient` (retry) vs `permanent` (terminal). There is no notion of "this provider/model is currently tripped — route around it to the next model in the precedence chain." 4. Model resolution (`resolveAgentModelSpec` in `src/agent.ts:252`, `resolveRoleModel`/`resolveTierModel` in `src/model-tier-config.ts`) is **static and deterministic**: it picks exactly one model spec from the role map and never consults runtime health. `enforceModelPolicy`/`enforceRoutingPolicy` only enforce the security-model isolation + API-billing opt-in — they have no failure-awareness hook. **The gap:** when a provider in our fleet degrades (429 quota, endpoint timeout, network blip, overloaded 5xx), every in-flight and subsequent agent() call against that provider fails or pauses individually, with no coordinated "this route is hot — cool it down and failover to the next healthy model in the chain" behavior. A worker-tier fan-out of 16 agents against a 429-ing `litellm-ny2/local-qwen27` produces 16 independent failures instead of 1 coordinated failover. The peer (pi-maestro-flow / maestro-flow) closes exactly this with an `EndpointCircuitBreaker` + half-open recovery sweep + healthy-endpoint-first fallback. --- ## Peer reference (pi-maestro-flow approach, with source URL) **Sources:** - pi-maestro-flow README (npm/GitHub): — section "API Retry and Model Failover": *"Circuit breaker — model calls are protected by a circuit breaker that trips on repeated failures and automatically recovers after a cooldown period. Failover routing — `/model-failover` configures automatic failover to backup models when the primary model is unavailable. `/model-failover status` shows live circuit breaker state."* - The implementing commit (upstream maestro-flow monorepo, not the Pi package): — *"feat: Add circuit breaker functionality and improve endpoint recovery logic: Introduced `EndpointCircuitBreaker` class to manage endpoint states and handle failures. Implemented half-open recovery sweep for tripped endpoints in `recoverTrippedEndpoints`. Enhanced fallback mechanism to prioritize healthy endpoints based on configured order. Added timeout configuration for pre-flight probes to improve responsiveness."* (`src/agents/api-explore/circuit-breaker.ts`, `src/agents/api-explore/runner.ts`, `src/agents/api-explore/circuit-breaker.test.ts`). ### Key mechanisms (the CONCEPT — not their code) 1. **Per-endpoint breaker state machine.** Each endpoint (provider+model route) holds one of `closed` (healthy, calls flow) / `open` (tripped, calls rejected immediately) / `half-open` (one trial call allowed to probe recovery). 2. **Trip-on-consecutive-failures.** A configurable threshold of consecutive failures flips `closed → open`. Non-consecutive (interleaved) successes reset the counter. 3. **Cooldown + half-open probe.** After a cooldown interval an `open` endpoint transitions to `half-open` and a single probe call is permitted; success → `closed`, failure → back to `open` (cooldown restarts, typically with exponential backoff). 4. **Exponential backoff on repeated trips.** Each successive open→half-open→open cycle grows the cooldown (exponential, capped). 5. **Healthy-endpoint-first fallback.** The fallback/failover mechanism walks the configured precedence order and **skips tripped (`open`) endpoints**, dispatching to the first healthy one. This is the automatic failover: primary down → next in chain up. 6. **Pre-flight probe timeout.** A short timeout on the probe so a hung endpoint doesn't block the recovery sweep. 7. **Observability surface.** `/model-failover status` renders live breaker health (state per endpoint, trip count, cooldown remaining). ### What we borrow conceptually - Per-model circuit breaker state machine (closed/open/half-open). - Trip on consecutive failures; cooldown with exponential backoff; half-open single-probe recovery. - Failover that walks a precedence chain and skips tripped models. - A status surface for observability. ### What we reject (do not vendor) - Their `EndpointCircuitBreaker` class and its config shape — we implement against OUR `ModelRoleConfig` / role-routing / tier model, not their endpoint model. - Their `/model-failover` and `/api-manager` command surfaces — ours integrates into `/workflows-models` + the monitor pane, not a new command family. - Any non-deterministic time source or background sweep thread that conflicts with our deterministic VM. **Our breaker is driven by the agent() call path and resume journal, not a wall-clock sweep.** Cooldown comparisons use a monotonic clock injected at the boundary, never `Date.now()`/`setTimeout` inside the workflow VM. - Vendoring their code or config files. Per #137: adapt the concept, write our own implementation. --- ## Our adaptation (how it fits OUR deterministic VM / disk journal / checkpoint() / herdr / role-based routing / Foundation fail-closed) ### Where the breaker lives: per **model spec** (provider/model), keyed on the resolved route We key the breaker on the **fully-resolved model spec string** (e.g. `litellm-ny2/local-qwen27`, `openai-codex/gpt-5.6-luna`, `meridian/claude-opus-5`), which is exactly what `resolveAgentModelSpec` / `resolveRoleModel` / `resolveTierModel` already produce and what `agentRunner.run({ model: modelSpec })` consumes. This is coarser than a raw HTTP endpoint (one provider may back several models) and finer than a tier — it matches the unit at which we already enforce policy (`enforceModelPolicy`/`enforceRoutingPolicy`) and at which the SDK actually routes a request. A single shared `CircuitBreakerRegistry` lives next to the model-resolution layer (new module `src/provider-circuit-breaker.ts`), keyed by model spec. It is **process-scoped state** (like the existing `ensureDiskModelContext` cache in `src/agent.ts:392`), not persisted to the run journal — circuit state is runtime health, not determinism input. Resume/journal replay **ignores** breaker state (a replayed cached result never hits the provider); a live re-run after a miss consults the breaker fresh. ### What trips it (failure classification reuse) We reuse the existing taxonomy in `src/errors.ts`, extended minimally: - **Trip-worthy (transient provider failure):** `transport_transient` errors that are **provider-attributable** — specifically `AGENT_TIMEOUT` (provider/endpoint hang), `AGENT_EXECUTION_ERROR` where the error text matches a provider-transport signature (network, ECONNRESET, fetch failed, 5xx overloaded), and `PROVIDER_USAGE_LIMIT` (429/quota — these currently *pause*; with the breaker they ALSO trip the model so failover can proceed to a healthy model rather than waiting out the refill). - **NOT trip-worthy:** `SCHEMA_NONCOMPLIANCE`, `AGENT_EMPTY_OUTPUT`, `CONTEXT_WINDOW_EXCEEDED`, `SCRIPT_VALIDATION_ERROR`, `TOKEN_BUDGET_EXHAUSTED`, `WORKFLOW_ABORTED`, `HARNESS_NOT_WIRED`. These are task/config/operator errors — retrying against a different model is wrong (a schema failure on Opus will fail on GLM too). These keep their current `permanent` classification and do not touch the breaker. A new helper `isProviderAttributable(error: WorkflowError): boolean` in `src/errors.ts` codifies this so the breaker and the existing `classifyWorkflowError` stay in agreement. ### Cooldown + exponential backoff (deterministic-friendly) - **Threshold to trip:** `PROVIDER_CIRCUIT_TRIP_THRESHOLD` consecutive trip-worthy failures against the same model spec (default `3`, matching `MAX_AGENT_RETRIES` so a normal retry budget that fully fails trips the breaker exactly once, not three times). - **Base cooldown:** `PROVIDER_CIRCUIT_BASE_COOLDOWN_MS` (default `30_000` ms = 30s, the same order as `SCRIPT_TIMEOUT_MS`). - **Exponential backoff:** cooldown × `2^(tripCount-1)`, capped at `PROVIDER_CIRCUIT_MAX_COOLDOWN_MS` (default `15 * 60_000` = 15 min, matching the order of `DEFAULT_WORKFLOW_TIMEOUT_MS` fractions). A model that keeps failing the half-open probe backs off 30s → 60s → 120s → … → 15min ceiling. - **Half-open probe:** after the cooldown elapses, the next agent() call against the model is allowed (one trial). Success → `closed` (reset tripCount). Failure → `open` again with the next backoff step. - **Clock injection:** the breaker reads `now()` from an injected `Clock` (default `() => Date.now()` at the host boundary in `src/provider-circuit-breaker.ts`). The workflow VM itself never calls `Date.now()` (existing invariant). Tests inject a deterministic fake clock. Cooldown expiry is checked **lazily** at call time (is `now >= trippedAt + cooldown`?), not via `setTimeout` — there is no background sweep thread, preserving determinism. ### How failover picks the next model (interplay with role-based routing) The breaker intervenes at **model resolution time**, inside a new wrapper around the existing resolution, NOT by editing `resolveAgentModelSpec`'s contract. The flow: 1. The workflow `agent()` resolves the **preferred** model spec exactly as today (`resolveAgentModelSpec` → role/tier/model → `enforceRoutingPolicy`). 2. Before the call, it asks the breaker: `breaker.isAvailable(preferredSpec)`. If `closed` or `half-open` → use it (half-open counts as the probe). 3. If `open` (tripped), the breaker returns a **failover chain** for that spec, derived from the role config: the ordered list of alternate models for the **same role** (e.g. for `worker`: `litellm-ny2/local-qwen27` → `litellm-ny2/oc-qwen35-397b` → `openai-codex/gpt-5.6-luna`), each filtered by `breaker.isAvailable`. The first available spec wins. 4. The chosen failover spec is **re-run through `enforceRoutingPolicy`** (security-model isolation + API-billing opt-in) — failover never bypasses policy. A security role that fails over still lands on a security model; an API-billed model is never selected without `googleBillingOptIn`. 5. If **no** model in the chain is available, the call **fails closed** with an actionable error (`PROVIDER_ALL_ROUTES_TRIPPED`), never silently degrades to an unrelated model. This is the Foundation fail-closed posture applied to routing. 6. On a successful call, the breaker records a success for the **actually-used** spec (closes a half-open probe / resets consecutive-failure count). On a trip-worthy failure, it records a failure for the **actually-used** spec (not the preferred one — failover models trip independently). The failover chain is **operator-configured**, not heuristic. It extends `ModelRoleConfig` (`src/model-tier-config.ts:124`) with an optional `failover?: string[]` per role (and per `RoleRoute`), declared in `~/.pi/workflows/model-tiers.json`. If absent, the breaker trips but there is nowhere to fail over to → the call fails closed (fail-closed default). This keeps failover **opt-in and explicit**, matching how we already treat API billing. ### Interplay with security-model isolation + API-billing opt-in - `isSecurityModel()` / `isApiBilledModel()` (`src/model-tier-config.ts:353-360`) gate every failover candidate. A security role's failover chain may only list security models; a non-security role may never fail over onto `SECURITY_MODEL`. The existing `enforceRoutingPolicy` runs on the resolved failover spec, so a misconfigured chain throws `SCHEMA_NONCOMPLIANCE` rather than silently crossing the boundary. - API-billed models (`google-ai-studio/*`) remain **never** a failover target unless `googleBillingOptIn` is set; the breaker's `isAvailable` checks the policy, not just the circuit state. ### Interplay with checkpoint() / disk journal / resume - **Journal replay is breaker-blind.** `runWorkflow`'s resume logic (`src/workflow.ts:1827` `cached !== undefined && ... hashMatches`) returns the cached result without calling the provider, so the breaker is never consulted on a replay. Determinism of resume is unchanged. - **A live miss consults the breaker.** When `state.firstMiss` advances and the call runs live (`src/workflow.ts:1948` `limiter(async () => { ... })`), the new resolution wrapper runs before `agentRunner.run`. - **Failover changes the model → changes the call hash.** `hashAgentCall` (`src/workflow.ts:2852`) already folds `model` into the hash. A failover to a different spec therefore produces a different hash, so a resumed run does not replay a failover result as if it were the preferred model's result. The journal records the **actually-used** model (`onAgentJournal { model: displayModel }`), and `displayModel` is updated via the existing `onModelResolved`/`onModelFallback` callbacks (`src/workflow.ts:2074-2080`). - **PROVIDER_USAGE_LIMIT + breaker:** today a usage limit checkpoints the run. With the breaker, the tripped model is also marked `open`, so the bounded `resume()` (issue #136 §3) re-resolves and **failovers to a healthy model** instead of re-hitting the same exhausted provider. The pause/checkout semantics are unchanged; only the post-resume route changes. This is strictly better than the current "pause and replay into the same wall." ### Interplay with herdr herdr (`src/herdr-reporter.ts`, `src/conductor-finalization.ts`) reports per-agent model + outcome. The breaker's trip/failover events flow through the existing `onAgentStart`/`onAgentEnd` payloads (extend `WorkflowAgentTelemetryConfig` and the `onAgentEnd` event with an optional `failoverFrom?: string` + `circuitState?: "closed"|"open"|"half-open"`). No new event bus — we enrich the existing one. herdr's run summary gains a "provider health" line listing tripped models + cooldown remaining. ### Observability (TUI / monitor pane) - The workflow monitor pane (`src/workflow-monitor-pane.ts`) and task panel (`src/task-panel.ts`) already render per-agent `model` + `error` + `errorCode`. We add: - A per-agent `⟳ failover: → ` indicator when `failoverFrom` is set (one line, dimmed). - A run-level "Provider health" summary in `buildMonitorRunViews` / `renderWorkflowMonitorTree`: list of tripped models with cooldown remaining, sourced from the shared `CircuitBreakerRegistry`. - `/workflows-models status` (or a new `--health` flag on the existing command in `src/workflows-models-command.ts`) prints the breaker registry: each known model spec, its state, consecutive failures, cooldown remaining, last trip reason. This is our equivalent of pi-maestro-flow's `/model-failover status`, reusing our existing command surface rather than adding a new command family. ### Concrete API / signature sketches ```ts // src/provider-circuit-breaker.ts (NEW) export type CircuitState = "closed" | "open" | "half-open"; export interface CircuitEntry { state: CircuitState; consecutiveFailures: number; tripCount: number; // for exponential backoff trippedAt?: number; // host-clock ms when last tripped open cooldownUntil?: number; // host-clock ms when half-open probe is allowed lastReason?: string; // error code/text that tripped } export type Clock = () => number; // injected; default Date.now at host boundary only export class CircuitBreakerRegistry { isAvailable(modelSpec: string, now: Clock): { available: boolean; state: CircuitState; entry: CircuitEntry }; recordSuccess(modelSpec: string, now: Clock): void; // closes half-open / resets failures recordFailure(modelSpec: string, error: WorkflowError, now: Clock): void; // trips if threshold reached snapshot(now: Clock): Array<{ modelSpec: string; entry: CircuitEntry }>; } // src/errors.ts (EXTEND) export function isProviderAttributable(error: WorkflowError): boolean; // true for AGENT_TIMEOUT, AGENT_EXECUTION_ERROR with provider-transport signature, PROVIDER_USAGE_LIMIT // false for SCHEMA_*, EMPTY_OUTPUT, CONTEXT_WINDOW_EXCEEDED, TOKEN_BUDGET_EXHAUSTED, ABORTED, HARNESS_NOT_WIRED // src/model-tier-config.ts (EXTEND — config shape only, opt-in) export interface RoleDefinition { default: string; routes?: Record; failover?: string[]; // NEW: ordered alternate model specs for this role } export interface RoleRoute { model: string; note?: string; failover?: string[]; // NEW: route-scoped failover chain } // src/workflow.ts (the agent() retry loop — wrap resolution + record outcome) // pseudo, inside limiter(async () => { ... for (attempt...) ... }): // const preferred = modelSpec; // already resolved // const used = breaker.resolveWithFailover(preferred, role, config, clock); // // used.spec is preferred (closed/half-open) OR first available in failover chain // // used.failoverFrom is set iff used.spec !== preferred // enforceRoutingPolicy(used.spec, role, googleBillingOptIn); // fail-closed policy // ... agentRunner.run(prompt, { model: used.spec, onModelResolved, onModelFallback, ... }) // on success: breaker.recordSuccess(used.spec, clock) // on trip-worthy catch: breaker.recordFailure(used.spec, workflowError, clock) ``` --- ## Scope (in-scope for the first PR) 1. **New module `src/provider-circuit-breaker.ts`**: `CircuitState`, `CircuitEntry`, `CircuitBreakerRegistry` with `isAvailable` / `recordSuccess` / `recordFailure` / `snapshot`, clock-injected, no background threads, no `setTimeout`/`Date.now` inside. 2. **`src/errors.ts`**: add `isProviderAttributable(error)` codifying which errors are trip-worthy (reuse `classifyWorkflowError` taxonomy + provider-transport text signatures). 3. **`src/model-tier-config.ts`**: extend `RoleDefinition` / `RoleRoute` with optional `failover?: string[]`; extend `resolveRoleMap` validation to accept it; extend `modelTierConfigWarnings` to warn on a failover chain that references an unknown model spec or crosses the security boundary. 4. **`src/agent.ts`**: expose a shared `CircuitBreakerRegistry` singleton (alongside `ensureDiskModelContext`), injected into `WorkflowAgent` via `WorkflowAgentOptions`. 5. **`src/workflow.ts`**: in the agent() retry loop, consult the breaker before `agentRunner.run`, resolve failover through the role config, re-enforce `enforceRoutingPolicy` on the failover spec, record success/failure on the **actually-used** spec, and surface `failoverFrom`/`circuitState` through `onAgentStart`/`onAgentEnd`/`onAgentJournal`. Update `hashAgentCall` note: the actually-used model is already hashed. 6. **`src/workflow-monitor-pane.ts` + `src/task-panel.ts`**: render per-agent failover indicator + run-level provider-health summary. 7. **`src/workflows-models-command.ts`**: add a `status`/`--health` view printing the breaker snapshot. 8. **Tests** (see Test plan): unit tests for the breaker state machine, integration test for a failover through the agent() loop using a fake registry + fake clock, and a fail-closed test when the whole chain is tripped. 9. **Doc update**: `docs/architecture.md` section on provider resilience; this plan doc. --- ## Non-goals (what NOT to build; per #137 adapt-don't-merge — do not vendor their implementation) - **No vendoring** of maestro-flow's `EndpointCircuitBreaker`, `circuit-breaker.ts`, `runner.ts`, or config files. We write our own against our role-routing model. - **No new `/model-failover` or `/api-manager` commands.** Reuse `/workflows-models` surface only. - **No background sweep thread / `setTimeout`-driven recovery loop.** Recovery is lazy (checked at call time) to preserve determinism and avoid a non-deterministic timer inside the VM. - **No cross-process / persistent breaker state.** The registry is process-scoped; a restart starts with all circuits closed. Persistent health state is a non-goal (it would couple determinism to runtime health). - **No automatic model-family escalation heuristics** (e.g. "small failed → try big"). Failover chains are **operator-declared per role**; the engine never invents a target. An empty chain fails closed. - **No breaker for non-provider errors.** Schema/empty/context-window/budget/abort errors do not trip — those are task errors, not provider health. - **No changes to the resume/journal hash contract.** Failover changes the live model (and thus the hash), which the existing `hashAgentCall` already handles; we do not introduce a new hash dimension. - **No opt-out escape hatch in the first PR.** The breaker is always-on but failover is opt-in (empty chain = fail closed = today's behavior). A future PR may add `disableCircuitBreaker` if an operator needs to force a tripped model. --- ## File-touch list (exact files/symbols that change, with the role of each) | File | Symbols | Role | |---|---|---| | `src/provider-circuit-breaker.ts` | `CircuitState`, `CircuitEntry`, `Clock`, `CircuitBreakerRegistry` (all NEW) | New module: the breaker state machine + registry, clock-injected, lazy recovery. | | `src/errors.ts` | `isProviderAttributable` (NEW); reuse `classifyWorkflowError`, `WorkflowErrorCode` | Codify trip-worthy vs non-trip-worthy errors; keep agreement with existing taxonomy. | | `src/model-tier-config.ts` | `RoleDefinition`, `RoleRoute` (extend with `failover?`), `resolveRoleMap`, `modelTierConfigWarnings` | Opt-in failover chain in config; validate + warn on unknown/cross-boundary specs. | | `src/agent.ts` | shared `CircuitBreakerRegistry` singleton accessor (NEW), `WorkflowAgentOptions`, `WorkflowAgent.run` call path | Expose the registry to the workflow layer; carry breaker through the provider call. | | `src/workflow.ts` | `runWorkflow` agent() loop (`limiter(async () => {...})`, `for attempt...`), `enforceRoutingPolicy` call site, `onAgentStart`/`onAgentEnd`/`onAgentJournal` payloads, `WorkflowAgentTelemetryConfig` | Consult breaker before `agentRunner.run`, resolve failover, re-enforce policy, record outcome, surface telemetry. | | `src/workflow-monitor-pane.ts` | `buildMonitorRunViews`, `renderWorkflowMonitorTree` | Render per-agent failover indicator + run-level provider-health summary. | | `src/task-panel.ts` | per-agent render (`shortModel`, error line) | Render `failoverFrom` arrow when set. | | `src/workflows-models-command.ts` | `registerWorkflowModelsCommand` handler | Add `status`/`--health` view printing breaker snapshot. | | `test/...` (new) | breaker unit tests, failover integration test, fail-closed test | See Test plan. | | `docs/architecture.md` | provider-resilience section | Document the breaker + failover contract. | | `docs/plans/competitive-gap-2-circuit-breaker.md` | this file | The plan. | --- ## Test plan (tautological-test warning: expected values from an independent oracle, not recomputed the same way) **Tautology guard:** every expected value below is hand-computed from the state-machine spec in this doc (the oracle), never by calling the breaker implementation under test. The fake clock advances deterministically; we assert the **spec-derived** expected state, not a value produced by re-running the breaker. 1. **State machine unit tests** (`test/provider-circuit-breaker.test.ts`) — oracle = the closed/open/half-open rules + backoff table (30s,60s,120s,…,15min cap) hand-derived from this doc: - `closed` → `open` after exactly `TRIP_THRESHOLD` consecutive trip-worthy failures; an interleaved success resets the counter (assert counter=0, state=closed). - `open` rejects availability until `cooldownUntil`; at `cooldownUntil` it flips to `half-open` (assert `isAvailable` flips exactly at the hand-computed `trippedAt + base*2^(n-1)`). - `half-open` + success → `closed`, `tripCount` reset to 0 (assert, not recompute). - `half-open` + failure → `open` with `tripCount+1`, next cooldown = hand-computed backoff step, capped at `MAX_COOLDOWN_MS`. - `recordSuccess` on a `closed` entry is a no-op (no spurious state change). 2. **Failover resolution test** (`test/workflow-failover.test.ts`) — oracle = the operator-declared failover chain order, filtered by `isAvailable` and `enforceRoutingPolicy`: - Preferred model `open` → first available in `failover[]` is selected; assert the **exact** spec string, not "any available". - Whole chain `open` → `PROVIDER_ALL_ROUTES_TRIPPED` thrown (fail-closed); assert the error code and that **no** `agentRunner.run` call happened (spy/mock the runner). - Failover spec re-checked by `enforceRoutingPolicy`: a chain that lists a `SECURITY_MODEL` for a non-security role throws `SCHEMA_NONCOMPLIANCE` before any provider call (assert the thrown code). - API-billed model in a chain without `googleBillingOptIn` → throws (assert), never selected. 3. **Journal-replay breaker-blindness test** — oracle = the cached result: a resume with a matching hash returns the cached result and **never** calls `breaker.isAvailable` (assert the spy call count is 0). Determinism of resume is unchanged. 4. **Failover changes the live hash test** — oracle = `hashAgentCall`'s documented model-folding: a failover to a different spec yields a hash **different** from the preferred-spec hash (assert `hashA !== hashB` where both are computed by the real `hashAgentCall` with the two model strings — this tests that the hash *differs*, using the hash function as the oracle for "hash includes model", not as a tautology for the breaker). 5. **`isProviderAttributable` test** — oracle = the table in this doc: feed representative `WorkflowError` instances of each code and assert `true`/`false` matches the documented classification exactly (hand-written table, not derived from the function). 6. **Determinism test** — two runs with the same fake clock + same inputs produce the same failover decisions (assert identical `used.spec` sequences); advancing the fake clock past a cooldown changes the decision (assert the flip), proving the only non-determinism is the injected host clock, never an internal timer. Full gate: `npm run check && npm run build && npm run test:unit -- provider-circuit-breaker workflow-failover && npm test`. --- ## Risks + guardrails (fail-closed, budget bounds, determinism, no silent shipping) - **Fail-closed by default.** An empty/unconfigured `failover` chain never falls back to an arbitrary model; the call throws `PROVIDER_ALL_ROUTES_TRIPPED` with an actionable message. No silent degrade. This matches the Foundation fail-closed posture and the existing `enforceModelPolicy` "must never be a default/fallback" rule for API-billed models. - **Security-model isolation preserved.** Every failover candidate is re-run through `enforceRoutingPolicy`; a security role cannot escape to a non-security model and a non-security role cannot land on `SECURITY_MODEL`. The breaker never bypasses policy. - **API-billing opt-in preserved.** `googleBillingOptIn` is consulted for every failover candidate; an API-billed model is never auto-selected without the visible opt-in. No silent cost. - **Budget bounds preserved.** Failover does not bypass `tokenBudget` / phase `budget` / `agentTimeoutMs`. A failover attempt is still an agent() attempt counted against `maxAttempts` and `shared.spent`; we do **not** reset the retry budget on failover (a trip-worthy failure consumes the attempt). `budget.remaining()` governs skipping optional rounds exactly as today. - **Determinism preserved.** The breaker is process-scoped runtime state, **not** journal input. Resume replays cached results breaker-blind. Live re-runs consult a clock-injected breaker; two runs with the same injected clock + inputs make identical decisions. The workflow VM contains no `Date.now()`/`setTimeout`/`Math.random` (existing invariant upheld). Only the host boundary reads the real clock. - **No silent shipping.** Every failover is surfaced via `onAgentStart`/`onAgentEnd` `failoverFrom` + the monitor pane indicator + the `/workflows-models --health` view. herdr reports provider health. A trip is never a silent route change. - **Trip-worthy scope is narrow and codified.** Only provider-attributable transient errors trip; schema/empty/context/budget/abort errors do not, so a broken prompt doesn't trip a healthy provider. `isProviderAttributable` is the single source of truth, shared with `classifyWorkflowError`. - **No runaway backoff.** Cooldown is capped at `PROVIDER_CIRCUIT_MAX_COOLDOWN_MS`; a permanently-down model settles to `open` at the cap rather than growing unboundedly. Combined with the `DEFAULT_MAX_RESUME_ATTEMPTS` cap on usage-limit resume, a degraded provider cannot produce an infinite loop. - **Resume hash soundness.** Failover changes the live model → different hash → a resumed run does not replay a failover result as the preferred model's. The journal records the actually-used model. No stale-results-on-resume risk. - **Per the cross-repo discipline:** if implementation surfaces a gap in the Pi SDK's provider error surfacing (e.g. the SDK does not expose a typed provider error we need to classify trip-worthy reliably), file an issue on the SDK repo rather than editing across repos in this PR.