/** * Near-duplicate detection for issue titles (issue #580). * * The failure this exists to stop: #579 restated #427 almost word for word, * three days apart, and nothing caught it. Nothing caught it because **no * dedupe check had ever existed in code** — the only one was an instruction in * `loop-mode.md` telling the loop to eyeball `renaiss-shipflow issues list * --json`, whose default `--limit 30` (commands/issues.ts) returns a * newest-first slice that structurally cannot contain a 150-issue-old * duplicate. An instruction that reads a truncated window is not a weak check; * it is a check that reports "no duplicates" on a window that never held one. * * ## Why this is NOT a bare Dice threshold (review of PR #586) * * The first cut scored Dice over token sets and refused at ≥0.70, calibrated by * counting flagged pairs among the repo's *already-filed, already-deduped* open * titles. That measurement bounds nothing about the **next** filing, which is * the only thing this guard gates. Measured directly — mutate one content word * of every open title into another real word, then ask whether the guard would * refuse that filing — bare Dice at 0.70 blocked **99.9%** of one-word variants * and **96.1%** of two-word variants, and all 10 hand-built sibling pairs. * * That is not a tuning problem, it is arithmetic: for an n-token title, a * one-word substitution scores `1 − 1/n` (0.857 at n=7) while the true * duplicate #579/#427 scores 0.875. No threshold separates 0.857 from 0.875, * so any Dice cutoff that catches the duplicate refuses nearly every legitimate * sibling. A guard that blocks legitimate filings is worse than the duplicate * it prevents. * * So refusal now requires **containment, not resemblance**: every word of the * shorter title must appear in the longer one. A substitution — the shape of a * genuinely different sibling issue — always breaks containment, while a * restatement that repeats the original and adds detail (exactly #579 over * #427) does not. Three hard gates run first (area, type, discriminators), and * Dice survives only as a length-ratio sanity floor and the reported score. * * The cost is deliberate and documented, and it runs in BOTH directions: * * | Direction | What happens | Consequence for a filer | * |---|---|---| * | **False negative** | a paraphrase in different words (#404 vs #569, ~0.38) is not caught | keyword-search before filing anyway — see `loop-mode.md` | * | **False negative** | a citation of some OTHER issue (`… (see #123)`) still discriminates | only the self-citation is excused — see `citations` below | * | **False positive** | a **strict superset** inside the Dice floor is refused **100%** of the time | a narrower issue that extends an open title is refused — file it with `--allow-duplicate` | * * The **self-citation** row of that table used to be a total bypass (issue * #587): `normalizeTitle` strips `#`, so `#427` became the digit token `427`, * `discriminators` made it must-match-exactly, and a restatement that CITED the * issue it restates scored `{427}` against `{}` — gate 3 rejected the pair * before containment ever ran and a genuine duplicate filed clean. Citing the * original is the first thing a filer does when restating it, so the guard was * weakest exactly where duplicates are most likely. `citations` + the `exclude` * argument of `discriminators` fix that narrowly: see both doc blocks. * * The false-positive row is the one readers miss. Containment asks only * "is every word of the shorter title in the longer one" — so a new issue that * repeats an open title verbatim and ADDS detail always refuses, however much * detail it adds: the median title here is 10 tokens, and the Dice floor of * 0.70 still admits up to 8 added content words. That is not a regression (the * bare-Dice first cut refused the same shape 100% too, and far more besides), * but it is the cost of choosing containment, and `--allow-duplicate` is the * documented way through it. * * Pure and dependency-free — like `issue-order.ts` and `escalation-format.ts`, * every rule here is unit-testable in isolation, and both the corpus lock and * the mutation sweep run against the repo's live open titles rather than being * asserted. */ /** Normalize apostrophe variants, then expand negative contractions. Exported * for the sweep, which mutates real titles into contracted forms. */ export declare function expandContractions(s: string): string; /** Dice floor. With containment mandatory this is no longer a resemblance * cutoff — the two token sets already nest — it is a **length-ratio** sanity * check: `2m/(m+M) ≥ 0.7` means the longer title is at most ~86% longer than * the shorter, so a 4-word title cannot be swallowed by a 12-word one that * happens to repeat it. Still the number reported as `score`. */ export declare const DUPLICATE_THRESHOLD = 0.7; /** How many open issues the pre-flight fetches. Emphatically NOT `ghIssueList`'s * default of 30: that window is the specific thing that hid #427 behind 118 * newer issues. A repo with more open issues than this needs a bigger number, * never a smaller one — and `duplicatePreflight` warns when the window comes * back full, so a truncated scan is visible rather than silent. * * The comparison window is `ghIssueList(repo, "open", DUPLICATE_SCAN_LIMIT)` * — closed issues and merged PRs are never scored, so a restatement of a * closed issue always files clean (defensible; a refile is often deliberate). * `findDuplicateCandidates` does not filter by state; `duplicatePreflight` * does, by passing only that open list. */ export declare const DUPLICATE_SCAN_LIMIT = 1000; /** Shortest token set that may be judged a duplicate **by containment**. Two * content words nest inside almost anything; refusing on that little evidence * is how a guard starts blocking unrelated filings. Below this, containment is * tightened to EQUALITY — nothing can nest, so there is no room for the short * title to be swallowed, and an exactly-repeated short title is still caught. * * This is also what gives non-Latin titles protection. Scripts without word * spacing (`fix(cli): 登录失败`) tokenize to a single blob, so they always take * the equality path: an identical title is refused, a different one is not. * That is exact-title protection, not near-duplicate protection — stated * plainly rather than claimed as more. */ export declare const MIN_TOKENS_FOR_MATCH = 3; /** A title reduced to the things scoring looks at. */ export interface NormalizedTitle { /** Conventional-commit type (`fix(cli):` → `"fix"`), or null when the title * carries no recognized prefix. A feature request and a bug report on the * same subject are two issues. */ type: string | null; /** Conventional-commit scope (`fix(cli):` → `"cli"`), or null when the title * carries no recognized prefix/scope. Null means "ungated", not "no area". */ area: string | null; /** Lowercased `\p{L}\p{N}`-based terms, stopwords and single ASCII letters * removed, in title order (duplicates retained; set semantics are applied by * `similarity`). */ tokens: string[]; /** Digit strings that appear in the raw title **only** as a `#N` / * `#N-suffix` citation — never as a plain number (issue #587). Stored by * numeric identity (issue #596 C2): `… (#427 regression)`, `… (#0427)` and * `… (#427-regression)` all yield `{"427"}`. `retry 427 fails, see #427` * (or `see #0427`) yields `{}` because `427` also occurs bare, and * `owner/repo#427` yields `{}` because a cross-repo reference is not a * citation of an issue in THIS repo. * * Consumed only by `findDuplicateCandidates`, and only for the ONE number * it is currently scoring against — see the `exclude` note there. The * tokens themselves stay in `tokens`: a cited number still counts in * containment and in public `similarity()`. Gate 5 Dice drops * `citationExclude` (issue #588). */ citations: Set; /** Tokens that originated as a `#N` / `#N-suffix` citation, after numeric * identity (`#0427` → `"427"`, `#0427-regression` → `"427-regression"`). * Gate 3 excludes these — not just the bare number — so `(#427-regression)` * is forgiven the same way `(#427 regression)` is (issue #596 C1). They * remain in `tokens`; containment and public `similarity()` are unchanged. * Gate 5 Dice drops the same set (issue #588). */ citationTokens: Set; } /** A near-duplicate of the title being filed. */ export interface DuplicateCandidate { number: number; title: string; /** Dice coefficient in [0,1], rounded to 3 decimals for stable display/JSON. */ score: number; } /** The minimum shape `findDuplicateCandidates` needs from an open issue — a * structural subset of `GhIssue`, so this module never imports `gh.ts`. */ export interface TitledIssue { number: number; title: string; } export interface DuplicateSearchOptions { /** Defaults to `DUPLICATE_THRESHOLD`. */ threshold?: number; /** Issue number to exclude (e.g. when re-checking an existing issue). */ excludeNumber?: number; /** Max candidates returned (default 5) — a refusal message listing twenty * near-misses is a refusal nobody reads. */ limit?: number; } /** * Split a title into `{type, area, tokens, citations, citationTokens}`: * lowercase, strip a leading conventional-commit prefix (capturing its type * and scope), tokenize on Unicode letters/numbers, drop stopwords and single * ASCII letters. * * Hyphenated terms survive whole (`post-review` stays one token) — they are the * highest-signal words in this corpus, and splitting them would let "review" * alone pull unrelated titles together. * * Tokenization is Unicode-aware: an ASCII-only class produced **no tokens** for * a title like `fix(cli): 登录失败`, which made `findDuplicateCandidates` return * early and scored even an identical open title at 0 — duplicate protection was * off entirely for non-Latin reports. * * Digit-bearing tokens are always kept, including single digits. The old * `length > 1` filter erased `1`, `2`, `5` — so `epic 96 slice 1` and * `… slice 2`, and `pr approve exits 5` and `… exits 7`, normalized to * IDENTICAL sets. Those are precisely the titles this repo files. */ export declare function normalizeTitle(title: string): NormalizedTitle; /** * Tokens whose presence flips WHICH bug a title describes, so they must match * exactly rather than merely overlap: digit-bearing tokens (`5`, `30`, `427`, * `v2`) and negations (`not`, `never`, `without`). * * Contracted negations reach this set because `normalizeTitle` expands them * first (`CONTRACTIONS`): `isn't`/`isnt` → `not`, `cannot`/`can't` → `not`. * Without that step they arrive as `isn` — or, for `can't`, as nothing at all — * and the gate silently does not apply. * * Containment cannot police these on its own. `cache is not populated` is a * strict superset of `cache is populated`, and `retry 1 fails on windows` is a * strict superset of `retry 1 fails` — the first pair is two opposite bugs, the * second is one bug stated twice. Comparing the discriminator sets separates * them; comparing token counts does not. * * `exclude` (optional, default none) drops tokens from the RESULT only — the * caller still sees them in `tokens`, so containment is unaffected. Gate 5 * Dice uses the same set independently (issue #588). * `findDuplicateCandidates` passes at most ONE number through it: the candidate * issue's own number, and only when that number is a pure `#N` self-citation on * at least one of the two sides (issue #587). The SAME set goes to both calls — * excluding per side is not monotone and can unmatch a pair gate 3 already * matched. Every other digit token, on either side, discriminates as before. */ export declare function discriminators(tokens: string[], exclude?: Set): Set; /** * Dice coefficient over the two titles' token SETS: `2·|A∩B| / (|A|+|B|)`. * * Set-based, not multiset: a word repeated in one title shouldn't inflate (or * deflate) the overlap. Returns 0 when either side normalizes to nothing. * Ignores `type`/`area` and containment — those gates are * `findDuplicateCandidates`' decision, so this stays a pure similarity measure. * * On its own this number does **not** decide duplication (see the module note): * a one-word substitution scores `1 − 1/n`, indistinguishable from a real * restatement. Public contract is raw tokens, including citations. Gate 5 * reports Dice on the same sets minus `citationExclude` (issue #588). */ export declare function similarity(a: string, b: string): number; /** * Does the shorter title's every word appear in the longer one? * * This is the gate that replaced the bare threshold. A genuinely different * sibling issue is written by SUBSTITUTING a word (`slice 1` → `slice 2`, * `--limit 30` → `--limit 1000`, `pr create` → `issue edit`), which always * leaves a word on the short side that the long side lacks. A restatement * repeats the original and adds detail, which does not. */ export declare function contains(shorter: Set, longer: Set): boolean; /** * Near-verbatim restatements of `title` among the issues the caller passed, * best match first. Does **not** filter by state — `openIssues` is whatever * the caller handed over. `duplicatePreflight` is what restricts the window * to `ghIssueList(repo, "open", DUPLICATE_SCAN_LIMIT)`: closed issues and * merged PRs never reach this function, so a restatement of a closed issue * always files clean (defensible; a refile is often deliberate). * * A candidate must clear ALL of these — resemblance alone is never enough: * * | # | Gate | Rejects | * |---|---|---| * | 1 | **area** — both scopes declared and equal | `fix(cli): X` vs `fix(server): X` | * | 2 | **type** — both types declared and equal | `feat(cli): X` vs `fix(cli): X` | * | 3 | **discriminators** — digit/negation sets equal | `slice 1` vs `slice 2`; `is` vs `is not` | * | | …minus a **self-citation** of `#M` on either side (#587) | nothing — this is the one number gate 3 forgives | * | 4 | **containment** — shorter ⊆ longer; below MIN_TOKENS, equality | any one-word substitution | * | | …minus a **self-citation** of `#M` on the equality branch (#766 C3) | nothing — sets that differ only by `citationExclude` still match | * | 5 | **Dice ≥ threshold** — length-ratio floor; minus `citationExclude` (#588) | a 4-word title inside a 12-word one | * * Gates 1 and 2 are decisive only when BOTH sides declare the field; when * either is unprefixed the gate cannot decide and degrades to the remaining * checks (which is also how this behaves in a repo that doesn't use * conventional-commit titles — see the caveat in #580). * * The gate-3 exception (issue #587) is scoped to the candidate being scored: * while scoring against `#M`, the digit token `M` — and any citation-origin * `M-…` token (issue #596 C1) — is dropped from **both** sides' discriminator * sets **iff** at least one side cites `#M` purely (that side carries the * number nowhere except in a `#M` / `#M-suffix`, including zero-padded * `#0M`). Citing `#427`, `#0427` or `#427-regression` therefore stops * discriminating against #427 and against #427 ONLY — against every other * open issue the token still gates in full. * * Both sides, not the citing side alone: a per-side drop can remove `M` from * one set while it survives in the other, turning a pair gate 3 already matched * into a miss. Dropping the same token from both keeps equal sets equal, so the * exception can only ever move a pair from "excused" to "scored", never the * reverse. The token also stays in `tokens`, so containment must still hold * on the raw sets. Gate 5 Dice drops `citationExclude` (issue #588) — a * 1-token title plus `(#N)` no longer sinks under 0.7. Extra non-citation * parenthetical words stay in the sets. * * Gate 4 below MIN_TOKENS_FOR_MATCH is equality, not containment — that is * what stops a 2-word title being swallowed by a 3-word superset at Dice 0.8. * A self-citation is the one extra token that equality still accepts (issue * #766 C3): `sameSetIgnoring(..., citationExclude)` on that branch only. * The size≥3 path still calls `contains` on the raw sets. Stripping cited * tokens into `contains` is the #594 B2 / #587 one-way regression. */ export declare function findDuplicateCandidates(title: string, openIssues: TitledIssue[], opts?: DuplicateSearchOptions): DuplicateCandidate[]; //# sourceMappingURL=issue-similarity.d.ts.map