/** * A consumer sink: the destination adapter a consume loop writes through. * * Without a sink, the user owns the three hard parts of a durable indexer — * checkpoint persistence, rows+cursor atomicity, and reorg rollback — and * the reasoning that makes them safe lives in doc comments. A sink owns all * three: the loop loads its committed cursor, hands the handler a * transaction, commits rows AND cursor atomically, and rolls back reorgs * without any user code. Crucially, a sink also closes the silent-skip * hazard where omitting `onReorg` meant reorgs were ignored forever: with a * sink attached, rollback is unconditional. * * The interface is dependency-free on purpose: implementations with real * database drivers live behind subpath exports (`@secondlayer/sdk/sinks/*`) * so the root entry ships no DB dependency. * * ## The contract * * Binding on every implementation; the conformance kit * (`@secondlayer/sdk/sinks/testing`) probes each one. Violations are SILENT * in production — they surface as gaps or duplicates weeks later, never as * errors at the violation site — which is why they are spelled out here and * mechanically tested. * * What the loop guarantees the sink: * * 1. **Init-before-first-page.** `loadCursor` is called exactly once, before * the first fetch — even when the caller passed an explicit `fromCursor`. * It is the sink's init hook: create checkpoint storage, validate * rollback preconditions. * 2. **Deterministic replay.** Re-reading from the same cursor yields the * same rows in the same order. `cursor` is therefore a valid idempotency * key: an append-only sink can dedup on it with no contract change. * 3. **At-least-once delivery.** A batch whose commit outcome was lost (crash * after commit, before the loop observed it) is re-committed on restart * with the SAME cursor and the same rows. * * What the sink must guarantee — `commitBatch`: * * 4. **Rows+cursor atomicity.** The handler's writes and `cursor` commit in * ONE transaction. Committing them separately is the classic torn-batch * bug: a crash between the two either re-delivers (duplicates) or skips * (gap) depending on the order. * 5. **Abort on throw.** A throw from `write` aborts the WHOLE transaction — * neither rows nor cursor land — so a crashed batch is simply re-read. * 6. **The lent transaction is the real one.** `write(tx)` receives the same * live transaction the cursor commits in; every handler write must go * through it. Writes outside `tx` escape atomicity AND rollback. * 7. **Replay safety.** Committing the same cursor twice must not corrupt * state (per #3 the rows are identical; upsert/dedup make it exact). * * What the sink must guarantee — `rollback`: * * 8. **Delete+rewind atomicity.** Undoing rows at/above the fork and * committing `rewindCursor` happen in ONE transaction. A crash between * them resumes above the fork and the deleted range is never re-read — * the silent-gap bug. * 9. **Inclusive fork point.** Undo AT OR ABOVE `forkPointHeight` (`>=`) — * the new canonical chain re-supplies the fork block itself. * 10. **Idempotent.** The loop may re-apply the same rollback after a crash * (reorgs are deduped in memory only); a second application must be a * harmless no-op. * 11. **Scope by height, not cursor.** On a page reporting several forks the * loop calls `rollback` once per fork: each call carries its own * `forkPointHeight` but ALL carry the same `rewindCursor` (the lowest * fork point). Derive the undo range from `forkPointHeight` only; a * sink that derives it from `rewindCursor` under-deletes silently. * * Error semantics and concurrency: * * 12. **Throw, don't swallow; no internal retry.** The loop owns retry * policy. A swallowed commit error advances the loop past unwritten * data; an internal retry can double-apply around a partial failure. * 13. **Single writer per checkpoint id.** The loop assumes one live * consumer per checkpoint identity. A sink that can detect a second * writer (e.g. a lock) must fail loudly — never block or interleave. */ interface ConsumerSink { /** Phantom marker carrying `Tx` in a directly-inferable position, so the * consume loops' `TTx` type parameter resolves from the `sink` option * (never set at runtime). */ readonly _tx?: Tx; /** Static capabilities, read by the loop before the first fetch to fail * fast on impossible pairings. */ readonly capabilities?: { /** This sink cannot undo committed rows (append-only store, ClickHouse, * parquet, …), so `rollback` is unimplementable and following the * unfinalized tip would corrupt it on the first fork. The loop throws * loudly unless consuming with `finalizedOnly: true` — in that mode * reorgs never reach the sink and `rollback` may simply throw. */ finalizedOnly?: boolean; }; /** The committed checkpoint, or `null` on first run. Called once, before * the first page. Implementations may create their checkpoint storage * here and SHOULD validate their rollback preconditions (e.g. that every * declared table carries the height column). */ loadCursor(): Promise; /** * Apply one batch atomically: open a transaction, run `write(tx)` (the * user's inserts), and commit the rows AND `cursor` together. A throw * from `write` must abort the whole transaction — leaving neither rows * nor cursor, so a crashed batch is simply re-read on restart. */ commitBatch(cursor: string, write: (tx: Tx) => Promise | void): Promise; /** * Roll the projection back to the fork: delete everything AT OR ABOVE * `forkPointHeight` (inclusive `>=` — the new chain re-supplies the fork * block) and commit `rewindCursor` in the SAME transaction. Deleting * without the rewound cursor is the classic silent-gap bug: a crash * between the two writes resumes above the fork and the deleted range is * never re-read. * * On a multi-fork page this is called once PER fork, every call with the * same `rewindCursor` (the lowest fork point) but its own * `forkPointHeight` — scope the undo by `forkPointHeight` only (contract * invariant #11), and expect re-application after a crash (#10). * * `rewindCursor` is `null` only for a fork at genesis: clear the * checkpoint so the next run starts from the first event. */ rollback(forkPointHeight: number, rewindCursor: string | null): Promise; } /** Context handed to {@link CreateSinkOptions.onRollback}. `forkPointHeight` * is the undo range (inclusive `>=`); `rewindCursor` is what the checkpoint * will become after this transaction — `null` only for a fork at genesis. */ type SinkRollbackContext = { forkPointHeight: number; rewindCursor: string | null; }; import { Kysely, Transaction } from "kysely"; /** * Column present on EVERY declared table — `keyof` a union of row types is * the intersection of their keys, so a column missing from any one table is * a compile error at the `height:` option. */ type CommonColumn< DB, T extends keyof DB > = keyof DB[T] & string; interface KyselySinkOptions< DB, T extends keyof DB & string > { /** Checkpoint identity AND concurrency key: the cursor row's primary key, * and the input to `pg_try_advisory_xact_lock(hashtextextended(id, 0))`. * The lock is transaction-scoped: it stops two replicas from * INTERLEAVING inside a commit, but is released at each commit boundary * — it is NOT a fence, so two replicas alternating batches both pass. * Run exactly one consumer per id; the lock turns the racy overlap loud, * it does not make a second replica safe. */ id: string; /** Rollback scope. On a reorg, rows AT OR ABOVE the fork point are deleted * from exactly these tables (inclusive `>=` — the new chain re-supplies * the fork block). Fact tables only: a fold (balances, counters) does * not belong here — invert it in `onRollback`. A table you write but * don't declare keeps orphaned rows forever. */ tables: readonly T[]; /** The block-height stamp column, present on every declared table (compile * and first-use checked). Height-stamp rollback is correct for * append-only projections (one row per event/call). */ height: CommonColumn; /** Checkpoint table name. Default `sl_consumer_checkpoints` (created on * first use). */ checkpointTable?: string; /** Same transaction as the fact-table delete, before it. Doomed rows are * still visible; remaining facts are `height < forkPointHeight`. A throw * aborts the rewind. Derive undo from the doomed rows so a re-applied * rollback is a no-op. */ onRollback?: (tx: Transaction, ctx: SinkRollbackContext) => Promise | void; } /** * A {@link ConsumerSink} over a Kysely Postgres database — keyed on the * QUERY BUILDER (not the driver) so the `DB` schema generic flows through: * `tables` is `(keyof DB)[]`, `height` must exist on every declared table, * and `ctx.tx` inside `onBatch` is a fully typed `Transaction`. * * What it owns (so your indexer doesn't): * - the checkpoint table + cursor load on start; * - rows AND cursor committed in ONE transaction per batch — a handler * throw aborts both, so a crashed batch is re-read on restart with no * gaps and no double-writes; * - reorg rollback: `onRollback` (doomed facts still visible), then delete * `>= fork_point_height` from every declared table, then the rewound * cursor — one transaction (the crash-between-the-two-writes gap is * unrepresentable); * - a per-id advisory lock, so two replicas of the same consumer can't * interleave commits. * * Structurally it is a 7-method {@link SinkDriver} on the shared * `createSink` base (`@secondlayer/sdk/sinks/core`), which owns the * transaction sequences and guards — this file only knows how to speak * Postgres-flavored Kysely. * * ```ts * const sink = kyselySink(db, { id: "sales", tables: ["sales"], height: "block_height" }); * await index.contractCalls.consume({ * contractId: MARKETPLACE, functionName: "purchase-asset", fromHeight: 0, * sink, * onBatch: (calls, _env, ctx) => * ctx.tx.insertInto("sales").values(calls.map(toSale)).execute(), * }); * ``` */ declare function kyselySink< DB, T extends keyof DB & string >(db: Kysely, options: KyselySinkOptions): ConsumerSink>; export { kyselySink, SinkRollbackContext, KyselySinkOptions };