/** * Derived tokenization, part 2: emit the SCANNER for an alphabet. * * Shape, and why each piece is what it is (all measured on the css grammar): * * - Literal and keyword terminals become a DATA TRIE walked by a fixed-size * driver. The trie walk is also the macro-time SEARCH for a distinguishing * position set: it walks until the node is uniquely accepting, so it does not * depend on a fixed discriminator. A fixed one does not work — `(len,c1,c2)` * collides on `@counter-style`/`@color-profile` and * `@font-feature-values`/`@font-palette-values`, and widening to the last * char still leaves 11 buckets for 13 keys. * Emitting the trie as CODE (nested switches) was measured at 25,223 B for 13 * css at-keywords; as DATA it is 2,057 B for the same set and 1.4x the speed * of today's slice-plus-char-chain rather than 2.5x. Data also means the * driver's size is independent of how many keys the grammar has. * * - Regex terminals stay sticky regexes, consulted ONLY when the position's * first char is in their first set. Ungated, a 42-regex core alphabet scans * benchmark.css at 14.5 MB/s; first-char gated, 30.4 MB/s. * * - The candidate set consulted is LOCAL to the decision point (a packed table * of id lists). The id space is global. Global maximal munch over the whole * alphabet produces seven tokens for a 123 KB stylesheet. * * - `tight` is set AT SCAN TIME: 1 when no trivia preceded the token. This is * what carries `noTrivia` adjacency (262 sites across the four dialects, 50 in * css) rather than the hand-spelled `boundary` classes, which are not even * self-consistent today: the 26 css `keywords(boundary:)` sites use THREE * different spellings and 16 of them omit `€-￿` and/or `\\`. * * - Whitespace is a token in `MODE_SELECTOR` (the descendant combinator is * significant there) and skipped otherwise. Mode is part of the memo key. */ import type { Alphabet } from './token-alphabet.ts'; export type ScannerEmission = { /** Module-scope declarations for the artifact prelude. */ decls: string[]; /** Name of the cursor entry point. */ nextFn: string; /** Register a candidate set, returning its table index. */ candidateIndexOf: (ids: readonly number[]) => number; /** True when the alphabet produced a usable scanner at all. */ usable: boolean; }; /** * Emit the scanner for `alphabet`. `ns` namespaces every hoisted name exactly * like `_re`/`_fx`/`_pf` so two fused pieces never collide. */ export declare function emitScanner(alphabet: Alphabet, ns: string): ScannerEmission; /** Patch the candidate table placeholders once every decision point registered. */ export declare function finalizeScanner(emission: ScannerEmission): string[]; //# sourceMappingURL=token-scanner.d.ts.map