/** * A consumer sink: the destination adapter a consume loop writes through. * * Without a sink, the user owns the three hard parts of a durable indexer — * checkpoint persistence, rows+cursor atomicity, and reorg rollback — and * the reasoning that makes them safe lives in doc comments. A sink owns all * three: the loop loads its committed cursor, hands the handler a * transaction, commits rows AND cursor atomically, and rolls back reorgs * without any user code. Crucially, a sink also closes the silent-skip * hazard where omitting `onReorg` meant reorgs were ignored forever: with a * sink attached, rollback is unconditional. * * The interface is dependency-free on purpose: implementations with real * database drivers live behind subpath exports (`@secondlayer/sdk/sinks/*`) * so the root entry ships no DB dependency. * * ## The contract * * Binding on every implementation; the conformance kit * (`@secondlayer/sdk/sinks/testing`) probes each one. Violations are SILENT * in production — they surface as gaps or duplicates weeks later, never as * errors at the violation site — which is why they are spelled out here and * mechanically tested. * * What the loop guarantees the sink: * * 1. **Init-before-first-page.** `loadCursor` is called exactly once, before * the first fetch — even when the caller passed an explicit `fromCursor`. * It is the sink's init hook: create checkpoint storage, validate * rollback preconditions. * 2. **Deterministic replay.** Re-reading from the same cursor yields the * same rows in the same order. `cursor` is therefore a valid idempotency * key: an append-only sink can dedup on it with no contract change. * 3. **At-least-once delivery.** A batch whose commit outcome was lost (crash * after commit, before the loop observed it) is re-committed on restart * with the SAME cursor and the same rows. * * What the sink must guarantee — `commitBatch`: * * 4. **Rows+cursor atomicity.** The handler's writes and `cursor` commit in * ONE transaction. Committing them separately is the classic torn-batch * bug: a crash between the two either re-delivers (duplicates) or skips * (gap) depending on the order. * 5. **Abort on throw.** A throw from `write` aborts the WHOLE transaction — * neither rows nor cursor land — so a crashed batch is simply re-read. * 6. **The lent transaction is the real one.** `write(tx)` receives the same * live transaction the cursor commits in; every handler write must go * through it. Writes outside `tx` escape atomicity AND rollback. * 7. **Replay safety.** Committing the same cursor twice must not corrupt * state (per #3 the rows are identical; upsert/dedup make it exact). * * What the sink must guarantee — `rollback`: * * 8. **Delete+rewind atomicity.** Undoing rows at/above the fork and * committing `rewindCursor` happen in ONE transaction. A crash between * them resumes above the fork and the deleted range is never re-read — * the silent-gap bug. * 9. **Inclusive fork point.** Undo AT OR ABOVE `forkPointHeight` (`>=`) — * the new canonical chain re-supplies the fork block itself. * 10. **Idempotent.** The loop may re-apply the same rollback after a crash * (reorgs are deduped in memory only); a second application must be a * harmless no-op. * 11. **Scope by height, not cursor.** On a page reporting several forks the * loop calls `rollback` once per fork: each call carries its own * `forkPointHeight` but ALL carry the same `rewindCursor` (the lowest * fork point). Derive the undo range from `forkPointHeight` only; a * sink that derives it from `rewindCursor` under-deletes silently. * * Error semantics and concurrency: * * 12. **Throw, don't swallow; no internal retry.** The loop owns retry * policy. A swallowed commit error advances the loop past unwritten * data; an internal retry can double-apply around a partial failure. * 13. **Single writer per checkpoint id.** The loop assumes one live * consumer per checkpoint identity. A sink that can detect a second * writer (e.g. a lock) must fail loudly — never block or interleave. */ interface ConsumerSink { /** Phantom marker carrying `Tx` in a directly-inferable position, so the * consume loops' `TTx` type parameter resolves from the `sink` option * (never set at runtime). */ readonly _tx?: Tx; /** Static capabilities, read by the loop before the first fetch to fail * fast on impossible pairings. */ readonly capabilities?: { /** This sink cannot undo committed rows (append-only store, ClickHouse, * parquet, …), so `rollback` is unimplementable and following the * unfinalized tip would corrupt it on the first fork. The loop throws * loudly unless consuming with `finalizedOnly: true` — in that mode * reorgs never reach the sink and `rollback` may simply throw. */ finalizedOnly?: boolean; }; /** The committed checkpoint, or `null` on first run. Called once, before * the first page. Implementations may create their checkpoint storage * here and SHOULD validate their rollback preconditions (e.g. that every * declared table carries the height column). */ loadCursor(): Promise; /** * Apply one batch atomically: open a transaction, run `write(tx)` (the * user's inserts), and commit the rows AND `cursor` together. A throw * from `write` must abort the whole transaction — leaving neither rows * nor cursor, so a crashed batch is simply re-read on restart. */ commitBatch(cursor: string, write: (tx: Tx) => Promise | void): Promise; /** * Roll the projection back to the fork: delete everything AT OR ABOVE * `forkPointHeight` (inclusive `>=` — the new chain re-supplies the fork * block) and commit `rewindCursor` in the SAME transaction. Deleting * without the rewound cursor is the classic silent-gap bug: a crash * between the two writes resumes above the fork and the deleted range is * never re-read. * * On a multi-fork page this is called once PER fork, every call with the * same `rewindCursor` (the lowest fork point) but its own * `forkPointHeight` — scope the undo by `forkPointHeight` only (contract * invariant #11), and expect re-application after a crash (#10). * * `rewindCursor` is `null` only for a fork at genesis: clear the * checkpoint so the next run starts from the first event. */ rollback(forkPointHeight: number, rewindCursor: string | null): Promise; } /** Default checkpoint table name, shared by every SQL-backed sink so two * sinks on the same database land in one table keyed by `id`. */ declare const DEFAULT_CHECKPOINT_TABLE = "sl_consumer_checkpoints"; /** * The driver surface a database adapter implements — the ~7 methods that * actually differ between SQL stores. Everything above them (the * commit/rollback call sequences, the guards, identifier validation, the * checkpoint schema) is policy owned by {@link createSink}, so a driver * cannot get the ordering wrong: it never sees a cursor write outside the * transaction that carries the rows, and never sees a delete without the * rewound cursor riding along. * * Identifier args (`table`, `column`) arrive pre-validated against * `[A-Za-z_][A-Za-z0-9_]*` — safe to interpolate after quoting (see * {@link quoteIdent}). Values (cursor, height, id) must still be bound as * parameters. */ interface SinkDriver { /** Run `fn` inside ONE transaction: begin, run, commit — a throw from * `fn` must roll the whole transaction back and re-throw. */ transact(fn: (tx: Tx) => Promise): Promise; /** Create the checkpoint store if missing: `(id primary key, cursor)`. * Must be safe under concurrent first start (two consumers racing the * same CREATE). */ ensureCheckpointStore(): Promise; /** The committed cursor for this sink's `id`, or `null` if none. */ readCursor(): Promise; /** Upsert this sink's cursor row inside `tx`. */ writeCursor(tx: Tx, cursor: string): Promise; /** Delete this sink's cursor row inside `tx`: a rewind to pre-genesis * (fork at block 0). Optional for drivers written before it existed; * without it a genesis rewind throws instead of writing a bad cursor. */ clearCursor?(tx: Tx): Promise; /** Delete rows with `height column >= height` from `table`, inside `tx`. */ deleteAtOrAbove(tx: Tx, table: string, height: number): Promise; /** Whether `table` exists AND carries `column` — the first-use check that * keeps reorg rollback from being a silent no-op. */ hasColumn(table: string, column: string): Promise; /** Optional: assert this consumer is the only live writer for its `id`, * inside `tx`. Must FAIL LOUDLY (throw) when another writer holds it — * never block or interleave (contract invariant #13). */ acquireLock?(tx: Tx): Promise; } /** Context handed to {@link CreateSinkOptions.onRollback}. `forkPointHeight` * is the undo range (inclusive `>=`); `rewindCursor` is what the checkpoint * will become after this transaction — `null` only for a fork at genesis. */ type SinkRollbackContext = { forkPointHeight: number; rewindCursor: string | null; }; /** Options for {@link createSink} — the policy inputs, driver-agnostic. * Concrete sinks re-expose these with schema-typed `tables`/`height`. */ interface CreateSinkOptions { /** Error-message prefix naming the concrete sink (e.g. `"kyselySink"`), * so a thrown guard points at the thing the user actually constructed. */ label: string; /** Checkpoint identity AND concurrency key. */ id: string; /** Rollback scope: on a reorg, rows at/above the fork point are deleted * from exactly these tables. Fact tables only — a fold (balances, * counters) does not belong here; invert it in `onRollback`. */ tables: readonly string[]; /** The block-height stamp column, present on every declared table. */ height: string; /** Checkpoint table name. Default {@link DEFAULT_CHECKPOINT_TABLE}. */ checkpointTable?: string; /** Forwarded to {@link ConsumerSink.capabilities}. */ capabilities?: ConsumerSink["capabilities"]; /** Runs inside the rollback transaction after the lock, before any * `DELETE … height >= fork`. Doomed fact rows are still visible. * Remaining facts are `height < forkPointHeight`. A throw aborts the * whole rewind (no delete, no cursor write). Called on every * `rollback`, including re-application after a crash — derive undo * from the doomed rows so a second pass is a no-op. */ onRollback?: (tx: Tx, ctx: SinkRollbackContext) => Promise | void; } /** Throw unless `name` is a bare SQL identifier (letters, digits, * underscores; no leading digit) — the guard that makes interpolating * table/column names into DDL and DELETE statements safe. */ declare function assertSqlIdentifier(name: string, label: string): void; /** Double-quote a (pre-validated) identifier for interpolation. */ declare function quoteIdent(name: string): string; /** * Build a {@link ConsumerSink} from a {@link SinkDriver}: the portable ~90 * lines every SQL sink otherwise re-implements — and re-risks. Owns the two * transaction sequences (begin → lock → write rows → write cursor; * begin → lock → onRollback? → delete `>=` fork → write rewound cursor), the * empty-`tables` guard, identifier validation, and the first-use height-column * check. * * A driver implemented against this base cannot violate contract invariants * #4, #5, #8, #9, or #11 (see {@link ConsumerSink}) without breaking * `transact` itself — which is exactly what the conformance kit probes. */ declare function createSink(driver: SinkDriver, options: CreateSinkOptions): ConsumerSink; export { quoteIdent, createSink, assertSqlIdentifier, SinkRollbackContext, SinkDriver, DEFAULT_CHECKPOINT_TABLE, CreateSinkOptions };