/** * Streaming scan builder with filter + aggregate support. * * `Collection.scan()` now returns a `ScanBuilder` that * implements `AsyncIterable` (for existing `for await … of` * consumers) AND exposes chainable `.where()` / `.filter()` clauses * plus a `.aggregate(spec)` async terminal that reduces the scan * stream through the same reducer protocol as `Query.aggregate()` *. * * **Memory model:** O(reducers), not O(records). The aggregate * terminal initializes one state per reducer, iterates through the * scan one record at a time via `for await`, applies every reducer's * `step` per record, and never collects the stream into an array. * This is what makes `scan().aggregate()` suitable for collections * that don't fit in memory — the bound is a code-level invariant * visible in the function body, not a runtime assertion. * * **Paginated iteration:** the builder holds a `pageProvider` * closure that maps `(cursor, limit) → Promise`, plumbed by * `Collection.scan()` to `collection.listPage(...)`. The page * iterator walks cursors forward until exhaustion, same as the * previous async-generator `scan()` did. * * **Backward compatibility:** existing `for await (const rec of * collection.scan()) { … }` code continues to work because * `ScanBuilder` implements `[Symbol.asyncIterator]`. The previous * signature returned an `AsyncIterableIterator` (which has both * `[Symbol.asyncIterator]` and `.next()`). We verified at grep time * that no call sites use `.next()` on the scan result directly, so * the narrowed interface is safe. * * **Immutability:** each `.where()` / `.filter()` call returns a * fresh builder sharing the same page provider and page size. This * lets a base scan be reused for multiple parallel aggregations: * * ```ts * const scan = invoices.scan() * const [open, paid] = await Promise.all([ * scan.where('status', '==', 'open').aggregate({ n: count() }), * scan.where('status', '==', 'paid').aggregate({ n: count() }), * ]) * ``` * * Note that each aggregation pays a full scan — there's no shared * iteration across the two. Multi-way aggregation in a single pass * is out of scope; consumers who need it should build a compound spec * and run a single `.aggregate({ openN, paidN })` at the DSL level. * * **Out of scope for (tracked separately):** * - `scan().aggregate().live()` — unbounded scan + change-stream * reconciliation is a design problem, not just a code one * - `scan().groupBy().aggregate()` — high-cardinality grouping on * huge collections would re-introduce the O(groups) memory * problem that aggregate fixes * - Parallel scan across pages — race-safe page cursor contracts * are not in the adapter API yet * - `scan().join(...)` — tracked under (streaming join) */ import type { QueryField } from '../types.js'; import type { ReducerBuilder } from '../../with-lookup/aggregate/reducers.js'; import type { Clause, Operator } from './predicate.js'; import type { AggregateSpec, AggregateResult } from '../../with-lookup/aggregate/aggregation.js'; import type { JoinContext, JoinLeg } from './join.js'; import type { ViaPipeline } from '../via/pipeline.js'; /** * Page provider — the Collection-shaped hook the builder calls to * walk cursors forward. Kept as a structural interface so tests can * wire up a synthetic provider without pulling in the full * Collection class. Collection's `listPage` matches this shape * exactly. */ export interface ScanPageProvider { listPage(opts: { cursor?: string; limit?: number; }): Promise<{ items: T[]; nextCursor: string | null; }>; } /** * Chainable streaming scan. Implements `AsyncIterable` for * drop-in use with `for await … of`; adds `.where()` / `.filter()` * chainable clauses and a `.aggregate(spec)` async terminal. * * The builder is immutable per operation — each chained call * returns a fresh `ScanBuilder` sharing the same page provider and * page size. The original builder is never mutated, so it's safe * to reuse across multiple parallel consumers. */ export declare class ScanBuilder implements AsyncIterable { private readonly pageProvider; private readonly pageSize; private readonly clauses; /** * Zero-or-more join legs to apply per record as the stream flows. * Each leg attaches the resolved right-side record (or null) under * its alias. — streaming joins. * * Joins are evaluated AFTER clauses, so a `where()` filtered-out * record never triggers a right-side lookup. This is the same * ordering as `Query.toArray()` (clauses first, joins after) and * keeps the streaming path from doing wasted work. */ private readonly joins; /** * Join resolution context. Required for `.join()` to translate a * field name into a target collection + ref mode and to resolve * the right-side `JoinableSource`. Optional because tests * construct ScanBuilder directly with synthetic page providers * that don't know about ref() — calling `.join()` without a * context throws with an actionable error. */ private readonly joinContext; /** * The backing collection's compiled Via pipeline (money now; more Via * features later). When it declares a result decode, yielded records * are decoded (e.g. money: stored scaled-int → canonical decimal) so * `scan()` agrees with `get()`/`list()`/`query().toArray()`. Decoded * with `'raw'` (canonical decimal, no locale-formatted virtuals) since * the scan stream carries no locale context, mirroring `Query.toArray()`. */ private readonly via; constructor(pageProvider: ScanPageProvider, pageSize?: number, clauses?: readonly Clause[], joins?: readonly JoinLeg[], joinContext?: JoinContext, via?: ViaPipeline); /** * Decode this scan's Via-covered fields on a record (e.g. money: stored * scaled-int → canonical decimal). No-op when the Via pipeline declares * no result decode. See {@link via}. */ private decodeVia; /** * Add a field comparison. Runs per record as the scan stream * flows through, so non-matching records are dropped before they * reach `.aggregate()` or the iteration consumer. Multiple * `.where()` calls are AND-combined — same semantics as * `Query.where()`. * * Clauses cannot use the secondary-index fast path here because * the scan sources records from the adapter's paginator, not from * the in-memory cache where indexes live. Index-accelerated scans * are a future optimization — the current implementation * evaluates clauses per record in O(1) per clause. * * Consults the Via pipeline's posture before building a clause (#629 * Task 8): a field whose posture is `queryable: 'none'` throws * `FieldNotQueryableError` here, at the call site — same gate as * `Query.where()`. */ where(field: QueryField, op: Operator, value: unknown): ScanBuilder; /** * Escape hatch: add an arbitrary predicate function. Same * non-serializable caveat as `Query.filter()` — filter clauses * don't round-trip through `toPlan()`. Prefer `.where()` when * possible. */ filter(fn: (record: T) => boolean): ScanBuilder; /** * Resolve a `ref()`-declared foreign key per record as the scan * stream flows, attaching the right-side record (or null) under * `opts.as`. — streaming joins over `scan()`. * * ```ts * for await (const inv of invoices.scan().join('clientId', { as: 'client' })) { * await processInvoice(inv) // inv.client is attached * } * * // Or terminate with .aggregate() for streaming joined aggregation * const { total } = await invoices.scan() * .where('status', '==', 'open') * .join('clientId', { as: 'client' }) * .aggregate({ total: sum('amount') }) * ``` * * **The key difference from eager `.join()`:** the LEFT * side streams page-by-page from the adapter and is never * materialized. Memory ceiling on the left is O(pageSize), not * O(rowCount). This is what makes streaming joins suitable for * collections that exceed the eager join's 50_000-row ceiling. * * **Right-side strategy** is auto-selected per leg: * - **Indexed** — right source exposes `lookupById`, so each * left row costs O(1). This is the common path for * Collection right sides, which back `lookupById` with a Map * lookup over the in-memory cache. The right collection must * be in eager mode (the same constraint as eager join's * `querySourceForJoin` from ). * - **Hash** — right source has only `snapshot()`. Build a * `Map` once at iteration start, probe per left * row. Same correctness, same per-row cost as the indexed * path; the difference is the upfront cost of materializing * the right side once. * * Both strategies hold the right side in memory for the duration * of the iteration. The "streaming" property applies to the LEFT * side only — true left-and-right streaming joins (where neither * side fits in memory) require a sort-merge join planner that's * out of scope for. * * **Ref-mode semantics** match eager `.join()` exactly: * - `strict` → throws `DanglingReferenceError` mid-stream * when a left record points at a non-existent right id. * The throw aborts the async iterator — consumers should * wrap the `for await` in try/catch if they want to recover. * - `warn` → attaches `null` and emits a one-shot warning * per unique dangling pair (deduped via the same warn * channel as eager join). * - `cascade` → attaches `null` silently. A delete-time mode; * dangling refs at read time are mid-flight or pre-existing * orphans, not a DSL error. * * Left records with null/undefined FK values attach `null` * regardless of mode — same "no reference at all" policy as * eager join and write-time `enforceRefsOnPut`. * * **Multi-FK chaining** is supported via repeated `.join()` * calls: each leg resolves an independent ref. Each leg * independently picks its right-side strategy and applies its * own ref mode. * * **Joins are NOT applied** to a `.aggregate()` terminal that * doesn't reference joined fields — wait, that's not quite * right. The streaming path actually DOES apply joins before * `.aggregate()` because the join attaches a field that the * spec might reference. Unlike `Query.aggregate()` (which skips * joins entirely as a projection-only short-circuit), the * streaming aggregation can't know whether the spec touches a * joined field, so it always applies joins. Consumers who want * unjoined streaming aggregation should leave `.join()` off the * chain — the chain is composable for a reason. * * constraint #1 — every JoinLeg carries `partitionScope: * 'all'` plumbed through but never read by. Same seam as * eager join. */ join(field: QueryField, opts: { as: As; }): ScanBuilder, S, M>; /** * Iterate the scan as an async iterable. Walks the page * provider's cursors forward until exhaustion, applying every * clause per record — only matching records are yielded. * * Backward-compatible with the previous async-generator `scan()` * return type for `for await … of` consumers. */ [Symbol.asyncIterator](): AsyncIterator; /** * Per-leg right-side resolution state. Built once at iteration * start and reused for every left record. Two strategies: * * - `lookupById`: present when the right source exposes the * hook directly (typical Collection right side). Per-row * cost is O(1). * - `hashByPrimaryKey`: built from `snapshot()` when no * lookupById. Per-row cost is O(1) after the upfront O(N) * materialization. Same as eager join's hash strategy. * * `warnedKeys` is the per-leg dedup set for ref-mode 'warn'. We * key on `field→target:refId` so the same dangling pair only * warns once per iteration. The dedup is per-iteration, not * per-process — a long-running scan that re-iterates would warn * again, which is the desired behavior (the data may have * changed between iterations). */ private buildJoinResolvers; /** * Resolve a single join leg for one left record and return the * left record with the joined field attached under * `leg.as`. Pure function over `(left, resolver)`; never * mutates the input. * * Ref-mode dispatch matches eager `applyJoins` from : * - null/undefined FK → attach null silently (always allowed) * - dangling FK + strict → throw `DanglingReferenceError` * - dangling FK + warn → attach null, warn-once per pair * - dangling FK + cascade → attach null silently */ private applyOneJoinStreaming; /** * Reduce the scan stream through a named set of reducers and * return the final aggregated shape. * * Memory is O(reducers): one mutable state slot per spec key. * Records flow through the pipeline one at a time via * `for await` and are discarded after their `step()` is applied * — never collected into an array. This is the distinguishing * property from `Query.aggregate()`, which materializes the full * match set first. * * Reuses the same reducer protocol as `Query.aggregate()`, * so `count()`, `sum(field)`, `avg(field)`, `min(field)`, * `max(field)` all work unchanged. The `{ seed }` parameter * plumbing from constraint #2 is honored transparently — the * factories ignore it in and the scan executor never * touches the per-reducer state construction. * * **Returns a Promise**, unlike `Query.aggregate().run()` which * is synchronous. The scan is inherently async because it walks * adapter pages, so the terminal has to be too. Consumers * destructure with await: * * ```ts * const { total, n } = await invoices.scan() * .where('year', '==', 2025) * .aggregate({ total: sum('amount'), n: count() }) * ``` * * **No `.live()` in.** `scan().aggregate().live()` would * require reconciling an unbounded streaming iteration with a * change-stream subscription — a design problem, not just a code * one. Consumers with huge collections and live needs should * narrow with `.where()` enough to fit in the 50k `query()` * limit and use `query().aggregate().live()` instead. * * Consults the Via pipeline's posture before reducing (#629 Task 8 review * fix wave 1): a reducer over a field whose posture is `queryable: 'none'` * throws `FieldNotQueryableError` here, metadata-only — via * `ViaPipeline.refuseUnqueryableReducers`, NOT the full `wrapReducers` * (which would also activate money's exact-reducer rewrite, a path this * method has never run and must not start running as a side effect of * this gate). */ aggregate(spec: Spec): Promise>; aggregate(build: (b: ReducerBuilder) => Spec): Promise>; /** * Evaluate the clause list against a single record. Linear in * the clause count; short-circuits on first false. Clauses on a * scan are always re-evaluated per record — no index-accelerated * path, because the stream sources records from the adapter * paginator, not from the in-memory cache where indexes live. */ private recordMatches; }