/** * @copyright Sister Software * @license AGPL-3.0 * @author Teffen Ellis, et al. * * Per-word BIO tag-consistency repair (#727 + the fr.country / admin-token fragmentation class). * * The model emits per-PIECE BIO labels. On rows where an admin token is adjacent to a non-latin * (byte-fallback) locality, carries diacritics, or is all-caps/reordered, the per-piece labels * can DISAGREE within a single word — `VERMONT` → `VER`[B-locality] + `MONT`[B-region], `Lozère` * → `Loz`[locality] + `ère`[B-region] — and the span decoder reads that as a tag-change * mid-token, fracturing the admin component. The model already KNOWS the word's tag (99.7% of * normal rows are unanimous); the defect is the lack of a word-level consistency constraint. * * Fix (DeepSeek-Pro consult `contested-frag`, 2026-06-19): a SentencePiece word — a `▁`-started * piece + its non-`▁` continuations — must carry ONE tag. The tag is chosen by a * CONFIDENCE-WEIGHTED vote, NOT first-piece-wins: sum each piece's softmax mass per TAG TYPE * (B-X * * - I-X collapsed; `O` included) across the word, argmax the type, and force `B-` then * `I-` (or all `O`). The near-certain `ère`→region then pulls the whole word to region, * healing both the fragment and the `Loz` bleed. Operates ONLY within a `▁`-delimited word, * so cross-word multi-token names ("Saint Paul" → two words) are untouched and the decoder's * existing cross-word merge still joins them. * * Safety: a word whose pieces already agree IN TYPE is left byte-identical — enforced structurally * (the vote never runs on a type-consistent word; single-piece words are trivially consistent). * Until 2026-07-15 this held only when the vote happened to agree with the decoder, which let the * heal RE-DECODE consistent words from local type-mass and override viterbi (`▁Broadway` * B-street→O; all-street `Gamle` →locality) — the mechanism behind the 2026-06-19 street * regression below. The vote includes `O`, so a disagreeing word can still resolve to all-`O`. * * Gate outcome (2026-06-19, fr-admin-split-gate + per-locale-f1, MAILWOMAN_WORD_CONSISTENCY=1): NOT * a clean win — net-regressed street −12.6 on the adversarial golden — so it shipped DEFAULT-OFF, * with a confidence-gated variant hypothesized as the path to a clean win. * * Re-diagnosis (2026-07-15): the regression was NOT vote noise — it was two defects in this module. * (1) The heal re-decoded words whose pieces already AGREED whenever the local type-mass preferred * another type, overriding viterbi (`▁Broadway` B-street→O; all-street `Gamle`→locality). Fixed * structurally: the vote now only runs on words whose pieces disagree in type. (2) Punctuation * continuation pieces joined the preceding word's vote group (`Ave` + `,`), and their `O` mass * manufactured fake disagreements that killed real spans — the `WordConsistencyOpts.splitOnPunctuation` * gate. With both fixed (+ `skipByteFallbackWords`), the heal is a clean win with NO confidence floor: * golden us street 82.0→82.2, fr macro 42.2→51.5, adversarial flat; parity house_number .767→.808, * postcode →1.000, street .543→.573; error-analysis 2pp gate PASS. Ships ON at the pipeline call * sites via `WORD_CONSISTENCY_SHIP_DEFAULT` (core/pipeline/types.ts). A `minMeanConfidence` floor * was measured NET-NEGATIVE on the parity corpus (fragment rows are low-confidence but heal * correctly) — it exists as an opt, unused by the ship default. */ export interface WordConsistencyOpts { /** * Skip the heal when the vote's mean p(bestType) across the word is below this floor. The ungated variant's failure * mode (the 2026-06-19 gate) was amplifying noise on rows where the per-piece confidence is itself unreliable — a * low-confidence vote is exactly that signature. `0` (default) never skips. */ minMeanConfidence?: number; /** * Skip healing any word containing a raw byte-fallback piece (`<0xNN>`). On byte-soup words the * confidence-weighted-vote premise ("the surviving pieces are trustworthy") breaks — see the module docstring's gate * outcome. Default false. */ skipByteFallbackWords?: boolean; /** * Treat a pure-punctuation piece (no letters/digits) as a word-group separator, like whitespace. Punctuation is never * word-content for tag purposes, but SentencePiece can emit it as a continuation piece that joins the preceding * word's group — `Ave` + `,` — where its `O` label manufactures a fake intra-word disagreement and the vote kills the * real span (the 2026-07-15 golden `street_suffix`→O class). Also keeps the halves of a slash compound (`12/345` = * unit 12 + house number 345) voting independently. Default false. */ splitOnPunctuation?: boolean; } /** * Interpret the `MAILWOMAN_WORD_CONSISTENCY` env string as a heal setting. `"1"` = the original ungated vote; `"gated"` * = the #727 gated preset (slash grouping + byte-fallback skip, no confidence floor); `"gated:"` adds a * `minMeanConfidence` floor (e.g. `"gated:0.5"`). Anything else (unset included) = off. */ export declare function parseWordConsistencyEnv(value: string | undefined): boolean | WordConsistencyOpts; export interface WordConsistencyResult { /** * A new per-piece label-index array, word-consistent (input is not mutated). */ labelIndices: number[]; /** * PieceIndex → mean p(chosen type) across the word, for pieces in a word that was HEALED. */ healedConfidence: Map; /** * Count of words whose labels were rewritten (0 = byte-identical to the input). */ healedWords: number; } /** * Rewrite per-piece label indices so every `▁`-delimited word carries one tag, chosen by a confidence-weighted vote * over the post-prior `emissions`. See the module docstring. * * @param pieces SentencePiece pieces (the `▁`-marked surface is the word-boundary signal). * @param emissions Per-piece × per-label scores AFTER all priors/masks (the distribution the argmax would see). * Softmaxed per piece for the vote so each piece's confidence carries its weight. * @param labels The BIO label vocabulary (index ↔ label). * @param labelIndices The current per-piece decision (viterbi path or argmax). Not mutated. * @param opts Optional gates on the heal (confidence floor, byte-fallback skip, slash grouping) — the #727-tracked * "confidence-gated variant". Omitted = the original ungated behavior, byte-identical. */ export declare function enforceWordConsistency(pieces: ReadonlyArray<{ piece: string; }>, emissions: ReadonlyArray>, labels: readonly string[], labelIndices: readonly number[], opts?: WordConsistencyOpts): WordConsistencyResult; //# sourceMappingURL=word-consistency.d.ts.map