/** * Advance through a declared provider chain when a member cannot serve. * * The sibling of `withProviderRetry`, one level out: retry asks "will the SAME * member succeed if I ask again?", this asks "will ANOTHER member succeed if I * ask it instead?". They compose in exactly one order — * `fallback(retry(m0), retry(m1), …)` — and that order is the policy, not an * implementation detail: * * - the primary is the operator's choice and gets its retries first, so a * throttle or a 5xx only reaches this decorator once the inner budget is * spent; * - a failure the inner loop refuses to retry (`auth`, `not_found`) arrives * here immediately, which is what "fall over at once on a bad credential" * means — retrying a wrong key just spends the turn; * - a server-directed `Retry-After` is honoured by the inner loop before any * error escapes, so a transient wait is a wait and not a swap. * * None of those three behaviours is implemented here. They fall out of the * nesting, which is why `query()` builds the composition rather than letting a * host assemble it in whichever order it happens to pick. */ import type { LLMProvider } from '../types/provider/index.js'; import type { Logger } from '../utils/logger.js'; /** * One member of the chain: a constructed provider, and the model to ask it for. * * `model` overrides {@link ChatCompletionParams.model} for this member's * requests and nothing else. Every shipped driver reads that field * (`params.model`, or `params.model || `), so a member * needs no reconstruction to be asked for a different model — which is what * keeps this decorator ignorant of credentials, base URLs and registries. A * chain is an ordered list of (provider, model); building one is the host's * job, walking it is this file's. * * Absent means "whatever the request already asked for", which is the right * default for a member declared without a model: the registry default was * resolved into the request before it got here. */ export interface ProviderChainMember { readonly provider: LLMProvider; readonly model?: string; } /** * The member serving from now on. * * `index` is a position in the chain the host declared, so a reader can name * the member without holding the chain: "member 2 of 4" is the sentence an * operator writes in an incident note. */ export interface ServingMember { readonly index: number; readonly providerId: string; /** Absent for a member declared without one — see {@link ProviderChainMember.model}. */ readonly model?: string; } export interface WithProviderFallbackOptions { /** Additional live admission policy before selecting another provider. */ readonly canFallback?: () => boolean; readonly log?: Logger; /** * Called once per swap, with the member that serves from here on. * * A callback is enough to describe the WHOLE truth, not a sample of it, * and that is a property of the cursor rather than of this option: the * chain never rewinds, so "who is serving" is exactly "the head, plus * every swap so far". A listener that starts at member 0 and applies each * call is never behind. * * It exists beside the in-band `fallback` chunk rather than instead of it * because the two have different observers and neither covers the other's * case. The chunk reaches whoever is iterating the stream, at the moment * of the swap — that is the operator. This reaches a party that has to * know AFTER the request is over and may never have iterated the stream at * all — that is the turn record. Two things follow that the chunk alone * cannot give it: * * - the cursor outlives the request, so a swap on the turn at step 3 * still describes steps 4..N, which emit no further chunk; * - a side call that aggregates the stream through `collectChatCompletion()` — the * compaction verifier and the forced-final summary both do — drops the * `fallback` chunk on the floor, so a swap inside one is invisible to * every chunk consumer. (The advisory executor calls its OWN advisor's * provider, not the turn's, so it is not one of these.) * * ## Fired when the replacement is ASKED, not when the cursor moves * * The two are not the same instant and the difference is observable. The * cursor moves inside the catch; the notice chunk is then yielded, and the * replacement request is only issued when the consumer comes back for * another chunk. A consumer that stops there — a Stop, a `break`, a host * that abandons the iterator — leaves a chain that selected a member and * never asked it. * * Announcing at cursor-move would report that member as serving, and a * ledger saying a provider served a turn it was never sent is the exact * defect this callback exists to end, reintroduced one layer down. So the * announcement sits at the top of the loop, immediately before the * replacement's `chatStream` — the earliest moment at which the member is * actually being asked. * * ## One stream at a time * * `cursor` is shared by every concurrent `chatStream` on this wrapper, so * two overlapping calls can advance it under one another: one call's * failure moves the cursor while the other is still being served by the * head, and a listener would hear about a member that answered nothing for * that call. Nothing here serializes or refuses concurrency — the * property held before this option existed and is not introduced by it. * `query()` issues its main turn and its side calls in sequence, which is * what makes the reading exact there. */ readonly onSwap?: (to: ServingMember) => void; } /** * Wrap an ordered chain so a member that cannot serve is replaced in place. * * ## The cursor's lifetime IS the scope * * Once this decorator advances, every later request in its life goes to the new * member; the chain never rewinds. That is deliberate and it is how "the * primary is restored at each new user message" is implemented — by NOT * implementing it. `query()` builds one of these per call and a host's call is * its turn, so the cursor cannot outlive the turn because the object cannot. * There is no reset to forget to call, and no way for a rate limit at 14:00 to * leave an operator on a cheaper model at 17:00. * * Rewinding within the turn would be the alternative, and it is worse: the * member that just failed would be re-asked on the next iteration of the same * turn, which is a retry wearing a chain's clothes and which the inner * decorator already declined to do. * * ## Each member is tried at most once per turn, and the whole chain is walked * * A chain of N members yields up to N attempts, not one. An operator who * declares four members and is served only by the second on a bad day has been * given three decorative entries — that is the declared-but-undriven defect * this file exists to remove, and stopping after one step would reintroduce it * at position 2 instead of position 1. When the last member fails, its error is * thrown untouched and ordinary error handling takes over. * * ## Once output is out, there is no fallback * * Inherited from retry, for the same reason and not a weaker one: a stream that * has emitted bytes cannot be restarted without duplicating them, and the * consumer has already appended them to a message it is rendering. A mid-stream * failure after output is surfaced, never swapped. See {@link isOutputChunk} — * the definition of "output" is where this property is actually won or lost. * * ## Capabilities are the head's * * The getters below are transparent, so a turn negotiates tools, vision and * documents ONCE against `members[0]` and keeps that answer after a swap. This * is a real limitation and it is why the host is expected to refuse a chain * whose members disagree before ever building one (`@namzu/cli` does, and only * runs a mismatched chain when the operator has said so explicitly). Taking the * intersection here instead would cost the primary a capability on every turn to * guard against a failure that happens rarely. Reasoning effort is the deliberate * exception: the SAME field is replayed unchanged after a swap, so only a common * level can be truthfully offered before the request. */ export declare function withProviderFallback(members: readonly ProviderChainMember[], options?: WithProviderFallbackOptions): LLMProvider; //# sourceMappingURL=fallback.d.ts.map