/** * corpus-v2.ts — SYNCED COPY of test/bench/recall-harness/corpus-v2.ts * (the flair recall-eval harness's eval instrument v2). * * flair-bench ships standalone (npx-able, no monorepo checkout required at * runtime), so it cannot import the harness's corpus at runtime from * test/bench/ — that directory is dev-only and excluded from the published * package. This file is a build-time copy kept faithful by * test/corpus-sync.test.ts, which deep-equals this file's exported CORPUS * and QUERIES against the live harness source on every `bun test` run * inside the monorepo checkout — any drift fails CI loudly. Regenerate with * `bun run sync:corpus` (packages/flair-bench/scripts/sync-corpus.mjs) * whenever the harness corpus changes. * * Below this header, content is byte-for-byte the harness's corpus-v2.ts — * see that file for the full design rationale (near-duplicate clusters, * cross-cluster lexical traps, durability/recency stress pairs). * * ORIGINAL HEADER (test/bench/recall-harness/corpus-v2.ts): * * * corpus-v2.ts — eval instrument v2: a larger, harder synthetic corpus + * ground-truth query set for the isolated recall-eval harness * (test/bench/recall-harness/run.ts, --corpus v2). * * WHY THIS EXISTS (vs. corpus.ts / v1): v1 is 87 records / 30 queries. A * flair#504 Phase 2 prefix A/B on v1 measured a small regression (p@3 * 0.967→0.933) that is, at N=30, just 1-2 queries shifting rank — not enough * queries for the instrument to tell "ship" from "park" apart from noise. v2 * exists to have enough queries per "kind" (see QueryKind below) that a * kind-level delta is a real signal, not a coin flip. v2 does NOT replace v1 * — v1's frozen numbers (README.md) stay reproducible verbatim by never * touching corpus.ts; v2 is additive and becomes the standing gate for * future embedding/scoring changes (larger N, same design principles). * * DESIGN — same three axes as v1, just wider: * 1. NEAR-DUPLICATE / ADJACENT DENSITY — every cluster below holds one or * two pairs of records about the SAME narrow sub-topic from different * angles (see each record's "Near-dup facet of X" note). A "hard" query * targets the specific facet, with the sibling as the plausible * rank-2/3 confusion. * 2. CROSS-CLUSTER LEXICAL "TRAP" PAIRS — three genuinely different-domain * cluster pairs deliberately share one pivotal, ambiguous term used in * a different sense in each domain, with high token overlap: * - DBSTORE (database internals) ↔ FINOPS (finance operations): "transaction" * - GITWF (git branching workflows) ↔ HORTIC (gardening/horticulture): "branch" * - MUSICPROD (music production) ↔ SPORTAN (sports analytics): "score" * Every record in a trap-pair cluster uses its cluster's sense of the * shared term naturally and prominently (not shoehorned in for the * trap alone) — the trap only works if the token overlap is real. * 3. VARIED DURABILITY + createdAt (recency) — 17 of the 30 clusters carry * a deliberate "stress pair": one older, standard/persistent-durability * CORRECT record vs. a same-topic, fresh (~2-4 day) permanent-durability * DISTRACTOR sibling — the same composite-vs-raw discriminator * mechanism v1 uses (see corpus.ts's header and flair#623). The * remaining records in every cluster (stress or not) still spread * naturally across all four durability levels and a wide age range * (~4-100 days), not just the engineered stress pairs. * * 30 topic clusters, genuinely different domains (distributed systems, * medical scheduling, logistics, cooking, astronomy, legal contracts, * fitness, team process, home renovation, marine biology, typography, * database internals, finance ops, git workflow, horticulture, music * production, sports analytics, cloud infra, winemaking, ornithology, * carpentry, meteorology, chess, photography, aviation, insurance, * archaeology, perfumery, ceramics, linguistics), 9 records each = 270 * records. * * ALL content below is authored synthetic prose — no real memories, no real * project/people/host/key names, nothing copied from ~/ops or any * production corpus. Every domain fact is invented-but-plausible, written * fresh for this file. * * GROUND TRUTH: identical convention to v1 (see corpus.ts's header) — every * query was written by hand against ONE specific record, then checked that * no other record in the corpus answers the query's literal question as * directly. See QueryKind below for what each kind tests. * * ageDays is relative to seed time (computed fresh by run.ts on every seed), * matching v1. */ export type Durability = "permanent" | "persistent" | "standard" | "ephemeral"; export interface CorpusRecord { /** Unique marker, e.g. "DISTSYS::1". Also used to derive the record's id. */ marker: string; cluster: string; text: string; durability: Durability; /** Age in days at seed time (createdAt = now - ageDays). */ ageDays: number; /** Free-form note on why this record's durability/age was chosen. */ note?: string; } export type QueryKind = "stress" | "trap" | "hard" | "clean"; export interface GroundTruthQuery { q: string; expectMarker: string; kind: QueryKind; /** What this query is specifically testing, and against which distractor. */ note: string; } export declare const CORPUS: CorpusRecord[]; export declare const QUERIES: GroundTruthQuery[];