/** * A consumer sink: the destination adapter a consume loop writes through. * * Without a sink, the user owns the three hard parts of a durable indexer — * checkpoint persistence, rows+cursor atomicity, and reorg rollback — and * the reasoning that makes them safe lives in doc comments. A sink owns all * three: the loop loads its committed cursor, hands the handler a * transaction, commits rows AND cursor atomically, and rolls back reorgs * without any user code. Crucially, a sink also closes the silent-skip * hazard where omitting `onReorg` meant reorgs were ignored forever: with a * sink attached, rollback is unconditional. * * The interface is dependency-free on purpose: implementations with real * database drivers live behind subpath exports (`@secondlayer/sdk/sinks/*`) * so the root entry ships no DB dependency. * * ## The contract * * Binding on every implementation; the conformance kit * (`@secondlayer/sdk/sinks/testing`) probes each one. Violations are SILENT * in production — they surface as gaps or duplicates weeks later, never as * errors at the violation site — which is why they are spelled out here and * mechanically tested. * * What the loop guarantees the sink: * * 1. **Init-before-first-page.** `loadCursor` is called exactly once, before * the first fetch — even when the caller passed an explicit `fromCursor`. * It is the sink's init hook: create checkpoint storage, validate * rollback preconditions. * 2. **Deterministic replay.** Re-reading from the same cursor yields the * same rows in the same order. `cursor` is therefore a valid idempotency * key: an append-only sink can dedup on it with no contract change. * 3. **At-least-once delivery.** A batch whose commit outcome was lost (crash * after commit, before the loop observed it) is re-committed on restart * with the SAME cursor and the same rows. * * What the sink must guarantee — `commitBatch`: * * 4. **Rows+cursor atomicity.** The handler's writes and `cursor` commit in * ONE transaction. Committing them separately is the classic torn-batch * bug: a crash between the two either re-delivers (duplicates) or skips * (gap) depending on the order. * 5. **Abort on throw.** A throw from `write` aborts the WHOLE transaction — * neither rows nor cursor land — so a crashed batch is simply re-read. * 6. **The lent transaction is the real one.** `write(tx)` receives the same * live transaction the cursor commits in; every handler write must go * through it. Writes outside `tx` escape atomicity AND rollback. * 7. **Replay safety.** Committing the same cursor twice must not corrupt * state (per #3 the rows are identical; upsert/dedup make it exact). * * What the sink must guarantee — `rollback`: * * 8. **Delete+rewind atomicity.** Undoing rows at/above the fork and * committing `rewindCursor` happen in ONE transaction. A crash between * them resumes above the fork and the deleted range is never re-read — * the silent-gap bug. * 9. **Inclusive fork point.** Undo AT OR ABOVE `forkPointHeight` (`>=`) — * the new canonical chain re-supplies the fork block itself. * 10. **Idempotent.** The loop may re-apply the same rollback after a crash * (reorgs are deduped in memory only); a second application must be a * harmless no-op. * 11. **Scope by height, not cursor.** On a page reporting several forks the * loop calls `rollback` once per fork: each call carries its own * `forkPointHeight` but ALL carry the same `rewindCursor` (the lowest * fork point). Derive the undo range from `forkPointHeight` only; a * sink that derives it from `rewindCursor` under-deletes silently. * * Error semantics and concurrency: * * 12. **Throw, don't swallow; no internal retry.** The loop owns retry * policy. A swallowed commit error advances the loop past unwritten * data; an internal retry can double-apply around a partial failure. * 13. **Single writer per checkpoint id.** The loop assumes one live * consumer per checkpoint identity. A sink that can detect a second * writer (e.g. a lock) must fail loudly — never block or interleave. */ interface ConsumerSink { /** Phantom marker carrying `Tx` in a directly-inferable position, so the * consume loops' `TTx` type parameter resolves from the `sink` option * (never set at runtime). */ readonly _tx?: Tx; /** Static capabilities, read by the loop before the first fetch to fail * fast on impossible pairings. */ readonly capabilities?: { /** This sink cannot undo committed rows (append-only store, ClickHouse, * parquet, …), so `rollback` is unimplementable and following the * unfinalized tip would corrupt it on the first fork. The loop throws * loudly unless consuming with `finalizedOnly: true` — in that mode * reorgs never reach the sink and `rollback` may simply throw. */ finalizedOnly?: boolean; }; /** The committed checkpoint, or `null` on first run. Called once, before * the first page. Implementations may create their checkpoint storage * here and SHOULD validate their rollback preconditions (e.g. that every * declared table carries the height column). */ loadCursor(): Promise; /** * Apply one batch atomically: open a transaction, run `write(tx)` (the * user's inserts), and commit the rows AND `cursor` together. A throw * from `write` must abort the whole transaction — leaving neither rows * nor cursor, so a crashed batch is simply re-read on restart. */ commitBatch(cursor: string, write: (tx: Tx) => Promise | void): Promise; /** * Roll the projection back to the fork: delete everything AT OR ABOVE * `forkPointHeight` (inclusive `>=` — the new chain re-supplies the fork * block) and commit `rewindCursor` in the SAME transaction. Deleting * without the rewound cursor is the classic silent-gap bug: a crash * between the two writes resumes above the fork and the deleted range is * never re-read. * * On a multi-fork page this is called once PER fork, every call with the * same `rewindCursor` (the lowest fork point) but its own * `forkPointHeight` — scope the undo by `forkPointHeight` only (contract * invariant #11), and expect re-application after a crash (#10). * * `rewindCursor` is `null` only for a fork at genesis: clear the * checkpoint so the next run starts from the first event. */ rollback(forkPointHeight: number, rewindCursor: string | null): Promise; } /** Context handed to {@link CreateSinkOptions.onRollback}. `forkPointHeight` * is the undo range (inclusive `>=`); `rewindCursor` is what the checkpoint * will become after this transaction — `null` only for a fork at genesis. */ type SinkRollbackContext = { forkPointHeight: number; rewindCursor: string | null; }; import { Table } from "drizzle-orm"; /** * Structural slice of a drizzle database. Postgres databases expose * `execute`; sqlite databases expose `run`/`all` (sync drivers return * plainly, async drivers return promises — `await` absorbs both). */ interface DrizzleDatabaseLike { transaction(fn: (tx: any) => Promise): Promise; execute?(query: unknown): Promise; run?(query: unknown): unknown; all?(query: unknown): unknown; } /** The transaction type `db.transaction` lends — what `ctx.tx` is typed as * inside `onBatch` (e.g. `PgTransaction<..., TSchema>`). */ type DrizzleTx = TDb extends { transaction(fn: (tx: infer Tx) => any): any; } ? Tx : never; /** Column key present on EVERY declared table — `keyof` a union of column * maps is the intersection, so a key missing from any one table is a * compile error at the `height:` option (same trick as kyselySink). */ type CommonColumnKey = keyof T["_"]["columns"] & string; interface DrizzleSinkOptions< T extends Table, TTx = unknown > { /** Checkpoint identity AND (on Postgres) advisory-lock key. */ id: string; /** Rollback scope: the drizzle table objects the handler writes. Fact * tables only — invert folds (balances) in `onRollback`. */ tables: readonly T[]; /** The block-height stamp column KEY (the TypeScript property, e.g. * `blockHeight` for `blockHeight: integer("block_height")`) — resolved * to each table's real SQL column name. Must exist on every declared * table; append-only projections only. */ height: CommonColumnKey; /** Checkpoint table name. Default `sl_consumer_checkpoints`. */ checkpointTable?: string; /** Same transaction as the fact-table delete, before it. Doomed rows are * still visible; a throw aborts the rewind. */ onRollback?: (tx: TTx, ctx: SinkRollbackContext) => Promise | void; } /** * A {@link ConsumerSink} over a drizzle database — Postgres or SQLite * dialect, detected structurally. `tables` are your schema objects, `height` * is compile-checked against every declared table, and `ctx.tx` inside * `onBatch` is the fully typed drizzle transaction for YOUR database. * * Dialect notes: * - Postgres (node-postgres, postgres-js, …): a per-id advisory lock makes * a second replica fail loudly (contract invariant #13). * - SQLite (bun:sqlite, better-sqlite3): commits run under a manual * `BEGIN IMMEDIATE` rather than `db.transaction()` — sync drivers would * otherwise COMMIT at the handler's first `await`, silently breaking * rows+cursor atomicity. `ctx.tx` is the database itself inside the open * transaction. Remote/libsql drivers are NOT supported for commits (their * statement-per-request model can't hold this transaction open). * * ```ts * const sink = drizzleSink(db, { id: "sales", tables: [sales], height: "blockHeight" }); * await index.contractCalls.consume({ * contractId: MARKETPLACE, functionName: "purchase-asset", fromHeight: 0, * sink, * onBatch: (calls, _env, ctx) => * ctx.tx.insert(sales).values(calls.map(toSale)).onConflictDoNothing(), * }); * ``` */ declare function drizzleSink< TDb extends DrizzleDatabaseLike, T extends Table >(db: TDb, options: DrizzleSinkOptions>): ConsumerSink>; export { drizzleSink, SinkRollbackContext, DrizzleTx, DrizzleSinkOptions };