/** * Resolves durable `catalog-tombstone::` records left * behind by {@link import('../catalog/removal.ts').removeCatalogEntry} * (WFT-17/18, Codex review on PR #958) — a tombstone exists only in the * brief window between its delete-and-tombstone commit and the SAME call's * own resolution (restore or finalize), normally sub-millisecond. A * tombstone that outlives that window is an ORPHAN: the process that * created it crashed (or otherwise failed) before resolving it. This * module makes that recoverable by ANY process, not just the one that * crashed — the exact gap the tombstone design closes (see * `core/catalog/removal.ts`'s module doc for the full mechanism). * * Deliberately lives in `core/engine/`, not alongside the tombstone * primitives in `core/catalog/`: resolving an orphan needs a FRESH * reference count, and the only reference signals meaningful for a * crashed peer's tombstone are the durable ones — * `nonterminal-revision-count.ts`'s `wf:`-prefix scan (both the * non-terminal AND terminal-but-unpurged buckets, WFT-17/WFT-21), * `pinned-schedule-revision-count.ts`'s `schedule:`-prefix scan (WFT-20), * and `retained-recovery-record-count.ts`'s `wf-teardown-deadletter:`-prefix * scan (WFT-21) — in-process signals like * `registeredDefinitions`/`inFlightStarts` are per-engine-instance and say * nothing about what a DIFFERENT, crashed process was doing. Before WFT-21 * this sweep checked ONLY `nonTerminalRuns`, silently ignoring a pinned * schedule or a terminal/dead-lettered reference — a pre-existing gap * fixed alongside this batch's own `retainedRecoveryRecords` wiring, since * both are the same class of durable-reference check. * `core/catalog/**` never imports from `core/engine/**` — a directional * boundary `check-import-cycles.ts` enforces — so this orchestration has to * live on the engine side of that line. * * Two call sites: `catalog-readiness.ts`'s boot-time sweep (every * orphaned tombstone in the whole store, run once per engine instance * before recovery or any new start can observe stale state) and * `catalog-removal.ts`'s `removeWorkflowRevision` (a single, targeted * check for the EXACT `(name, revision)` key it is about to act on, since * a long-lived engine could observe a peer crash mid-lifetime, after its * own boot sweep already ran). * * @module core/engine/catalog-tombstone-recovery */ import { type Storage } from '../../storage/interface.ts'; /** * Sweep every `catalog-tombstone:` record in `storage` and resolve each * one. Bounded by the number of orphaned tombstones actually present — * normally zero, so this is a cheap prefix scan that yields nothing to * iterate; only non-trivial after a real crash left orphans behind. Called * once per engine instance, at catalog-boot time * (`catalog-readiness.ts`), before recovery's own preflight or any fresh * start can observe a revision this sweep would otherwise still be * resolving. * * A malformed tombstone KEY (wrong shape) still fails the whole sweep * closed outright — that can only mean actual storage corruption or a * foreign write into this namespace, never an ordinary decode gap, so * there is no safe per-record default to isolate it behind. * * Everything past the key-shape check IS isolated per tombstone (WFT-21, * Codex review, item 8): before this fix, either `decodeCatalogEntryRecord` * (the tombstone's own manifest bytes are corrupt) or * `resolveCatalogTombstone` (a reference-count scan hit an unrelated * undecodable record ELSEWHERE in the store) rejecting for ANY one * tombstone propagated out of this whole function uncaught. Its only * caller, `ensureWorkflowCatalogReady()`, has no surrounding try/catch, so * that rejection meant `internals.catalogRestored` never became `true` — * blocking `start`/`resume`/`fork`/recovery entirely, on every future call, * from a single undecodable record anywhere in the store, until an * operator repaired it. Each tombstone's resolution is now caught * independently, mirroring this codebase's other "one bad record fails * only its own unit, siblings continue" precedent * (`DynamicWorkflowSourceUnavailableError`'s recovery classification — * `documentation/reference/api-errors.md`): * * - A `decodeCatalogEntryRecord` failure means the tombstone's own bytes * cannot be trusted as a real entry at all, so this tombstone is left * completely untouched — neither restored (would risk reinstating * corrupt bytes as a live catalog entry) nor finalized (would durably * confirm a removal with no evidence the bytes were ever valid). The * revision stays deleted-but-unresolved until an operator repairs the * record. * - A `resolveCatalogTombstone` failure means the tombstone's OWN bytes * are already known valid (the decode above already succeeded) but its * fresh reference count could not be computed. The conservative default * under that uncertainty is the same one `resolveCatalogTombstone` * itself uses for a nonzero count: restore the entry, keeping the * revision installed rather than risk finalizing a removal the evidence * could not actually prove safe. * * Either isolated failure invokes the optional `onIsolatedFailure` callback * with the affected `(name, revision)` and the caught error — a bounded, * low-cardinality diagnostic (at most one call per orphaned tombstone, not * per scanned record) — before continuing to the next tombstone in the * scan. `catalog-readiness.ts` wires this to * `engine.dispatchEvent(new CleanupWarningEvent(...))`, the same * background-failure event class `termination/cleanup.ts` already uses for * this exact "caught, reported, moved on" shape. */ export declare function resolveOrphanedCatalogTombstones(storage: Storage, onIsolatedFailure?: (name: string, revision: string, error: unknown) => void): Promise; /** * Targeted check for ONE `(name, revision)` tombstone — used by * `removeWorkflowRevision` immediately before it acts, so a peer crash * that happened after this engine's own boot-time sweep already ran (a * long-lived engine) does not leave a stale tombstone permanently blocking * every future removal attempt on that exact key (`removeCatalogEntry`'s * own CAS requires the tombstone key absent). A no-op when no tombstone is * present for this exact key — the overwhelmingly common case, costing one * `storage.get`. */ export declare function resolveCatalogTombstoneIfPresent(storage: Storage, name: string, revision: string): Promise;