/** * Amortised re-anchoring: rewriting history when the rewrite is already paid for. * * THE ECONOMICS, because every decision here follows from them. Anthropic bills * a cache read at 0.1x the base input rate and a cache write at 1.25x. So * rewriting a cached prefix is not a small cost -- it converts 50,000 tokens * billed as 5,000 into 50,000 billed at the write rate. Compression that busts * the cache can cost several times what it saves, which is why V1 refuses to * touch anything at or before the last breakpoint. * * WHAT THAT REFUSAL GETS WRONG. It assumes the provider's cache holds the * CLIENT'S prefix. That is only true when something other than us put it there. * Once we are in the path and rewriting deterministically, the cache holds OUR * prefix -- so re-deriving the same bytes next turn HITS, on a prefix that is a * fraction of the size. The write is paid once and read back cheaply for the * rest of the conversation. That is the amortisation. * * THREE SITUATIONS, AND THEY GET DIFFERENT ANSWERS: * * we anchored this conversation before rewrite again, identically. The * cache holds our bytes; reproducing * them is a hit. Stopping now would * be the miss. * * we have never seen it, and it is small rewrite. The prefix is written * either way on a first turn, and a * compressed one is cheaper to write. * * we have never seen it, and it is large leave it alone. We joined * mid-conversation, so the provider * probably holds the client's * original; rewriting spends a write * on a large prefix to buy back an * unknown number of remaining turns. * * ONCE ANCHORED, ALWAYS ANCHORED -- for a given prefix. This is the part that * is easy to get backwards: the decision cannot be "rewrite only when it is * free", because the turn after a free rewrite is precisely when NOT rewriting * would miss. What is remembered is therefore not just the prefix but whether * we anchored it. * * THE PRECONDITION IS DETERMINISM. Re-anchoring only works if next turn * re-derives byte-identical bytes for the same history. Two things had to * change for that to hold: cached content is compressed WITHOUT the question * (`strategy.ts` -- the question changes every turn), and the spill sink is * content-addressed (`proxy/server.ts` -- a random path per call changes the * output). Neither is optional; without them this makes billing worse. * * NO PER-REQUEST STATE ON A SHARED OBJECT. HeadRoom's #3486 is one shared * `ContentRouter` keeping request state on `self`, so concurrent requests * cross-contaminate. What is kept here is per CONVERSATION, is only ever a hash * plus a boolean, is passed in explicitly, and is consulted for nothing but * this one question. */ import { type ProviderRequest, type Position } from './frontier.js'; /** * How many conversations are remembered. * * Bounded because this lives in a long-running proxy and an unbounded map keyed * by conversation is a slow leak. Least-recently-used eviction: a conversation * untouched across a thousand others is over. */ export declare const MAX_TRACKED = 1000; /** * Above this many characters of cached prefix, a conversation we have never * seen is left alone. * * The trade is a one-time write against an unknown number of remaining turns. * Small prefixes are cheap to be wrong about; a large one on a conversation * that ends next turn is the single most expensive mistake available here, and * it is worse than compressing nothing. Roughly five thousand tokens. */ /** * How many messages a conversation may already have and still count as one we * are seeing from its start. * * THIS REPLACED A PREFIX-SIZE TEST, and the replacement is the whole point. * The old rule anchored only when the cached prefix was under 20,000 * characters, as a stand-in for "did we see this conversation from the * beginning". Against Claude Code that stand-in is always false: its system * prompt and tool schema alone exceed 20,000 characters on the very first * request, so every conversation was classified as joined-mid-conversation, * the proxy never anchored, v1 kept respecting a frontier that sits at the * newest turn, and NOTHING was ever compressed -- measured at 0 bytes removed * across 44 real requests in two campaigns. * * Message count asks the question directly. A proxy running before its client * sees the conversation at one or two messages however heavy they are; a proxy * attached to a session already in flight sees many, and still declines -- * which is the protection the old limit was there to provide. */ export declare const COLD_MESSAGE_LIMIT = 4; /** * Identity of a conversation, stable across its turns. * * The system prompt plus the first text block: both are fixed for the life of a * conversation and differ between conversations. NOT the whole prefix, which * grows every turn -- keying on that would make each turn look like a new * conversation, report "never seen" forever, and re-anchor on every request, * which is the behaviour this exists to avoid. */ export declare function conversationKey(request: ProviderRequest): string; /** What we last did for one conversation. */ export interface AnchorRecord { /** Digest of the CLIENT's prefix as it arrived, so a change is detectable. */ readonly prefixDigest: string; /** * Length of the prefix that digest covers. * * WITH IT, EXTENSION IS PROVABLE. Re-digesting exactly this many * characters of the new prefix and comparing answers "is what I saw last * time still an exact prefix of what I see now" -- which separates a * conversation that grew from one whose history was edited. A sample * comparison cannot: samples are spot checks at doubling offsets, so an * edit that lands between two of them passes, and a test pinning that * exact case is what caught the guess. */ readonly prefixLength?: number; /** * Digests sampled across that prefix, at doubling offsets. * * THIS IS WHAT SEPARATES AN EDIT FROM A COLLISION. Two different sessions * whose system prompt and opening message are identical -- a user who * starts two conversations with "fix the tests" -- share a key. Without * this, the second one looks like the first with its history rewritten, * and the answer to that is "anchor, the miss already happened", which * would spend a 1.25x write on a large prefix the provider is holding in * its original form. With it, an unrelated conversation is recognised as * unrelated and falls back to the size rule for a first sighting. */ readonly samples: readonly string[]; /** Did we rewrite that prefix? */ readonly anchored: boolean; /** * Where this conversation's cache breakpoint sat when we last saw it. * * THE ONE FACT THAT SAYS WHAT IS NOT YET CACHED. A client puts its * cache_control marker on the LAST message of every request -- verified * across three consecutive captured turns, which reported breakpoints at * message 1 of 2, 3 of 4 and 5 of 6 -- so "after the breakpoint in THIS * request" is always empty and a frontier policy reading it compresses * nothing at all. * * Between the breakpoint we saw last turn and the one in this request * lies everything the conversation has grown since, which the provider * has not cached yet and is about to cache now. Rewriting THAT costs no * invalidation, shrinks the 1.25x write happening this turn, and shrinks * every 0.1x read after it. */ readonly breakpoint?: Position | null; /** * Where compression STARTED for this conversation, and never moves after. * * A FROZEN BOUNDARY IS WHAT MAKES THE PREFIX REPRODUCIBLE. `breakpoint` * advances every turn as the client moves its cache marker, and using it * as the compression floor means a span compressed on turn N falls BELOW * the floor on turn N+1 and would be sent uncompressed -- the provider * then holds our bytes and receives the client's, which is a miss on * everything from that point on. * * Measured before this existed: 20 of 418 message-turns changed across a * 30-turn replay, which invalidated a modelled 21% effective-token saving * outright, because every one of those changes is a cache miss rather * than the 0.1x read the model assumed. * * Frozen, the rule is simply: everything after this point is compressed, * every turn, by a transform that depends only on the content. Same * input, same output, cache hit. */ readonly compressFrom?: Position | null; /** * The knowledge block we last put in this prefix, if any. * * Kept here because it IS part of the prefix, and the prefix has to * arrive byte-identical every turn. Re-selecting findings each turn * would rewrite it each turn and convert a 0.1x read into a 1.25x write * on everything -- so the block is chosen once and replayed verbatim * until a turn on which rewriting is already free. Null means nothing * was injected and nothing may start being injected without a rewrite. */ readonly knowledge?: string | null; } export interface AnchorStore { seen(key: string): AnchorRecord | undefined; remember(key: string, record: AnchorRecord): void; } /** An in-memory store, bounded and explicit. */ export declare function anchorStore(max?: number): AnchorStore; export type AnchorReason = 'already-anchored' | 'first-turn' | 'client-invalidated' | 'joined-mid-conversation' /** The same conversation, one or more turns longer. The common case. */ | 'extended' | 'left-alone'; export interface AnchorDecision { /** Rewrite history on this turn? */ readonly reanchor: boolean; readonly reason: AnchorReason; /** The conversation this decision was made for. */ readonly key: string; /** Record this once the request is built, so the next turn can compare. */ readonly record: AnchorRecord; } /** * Decides whether history may be rewritten on this turn. * * CONSERVATIVE WHERE IT IS UNCERTAIN, consistent where it is not. The one case * that spends money for nothing -- rewriting a large prefix the provider has * cached in its original form -- is the one case that answers no. */ export declare function anchorDecision(request: ProviderRequest, store: AnchorStore, coldMessageLimit?: number): AnchorDecision; //# sourceMappingURL=anchor.d.ts.map