/** * Semantic AST-chunk retrieval vs whole-file reads — real-API measurement of * whether Feature #3 is worth building. * * The claim (cocoindex-code / oh-my-pi): replacing "grep then read whole files" * with "retrieve only the relevant AST chunks" cuts ~70% of the tokens an agent * spends locating code, with no loss of answer quality. We test that directly on * OUR OWN repo against a live model. * * For a set of natural-language questions about real files in this repo, we build * three context strategies and ask the model the same question with each: * * BASELINE (whole-file): deliver the FULL text of the top files a lexical * grep would surface — this is what the agent reads today (read + grep). * * SEMANTIC (AST chunks): parse every file into top-level declarations * (functions / classes / interfaces / consts), rank chunks with a real BM25 * retriever, and deliver only the top-k chunks. No embedding model needed; a * learned embedding retriever would land between BM25 and ORACLE. * * ORACLE (upper bound): deliver only the hand-labelled answer chunk(s) — the * best case any retriever could achieve. * * We measure, per question and strategy: INPUT tokens delivered (the headline * cost) and whether the model's answer was correct (deterministic keyword grade). * The verdict: does SEMANTIC reach BASELINE-level correctness at a fraction of * the input tokens? * * Usage: * npx tsx src/core/semantic-search-benchmark.ts * * Env overrides: * GG_SS_PROVIDER / GG_SS_MODEL (default openai / gpt-5.5) * GG_SS_TOPK (chunks/files delivered, default 3) */ export declare function grade(answer: string, mustInclude: string[]): boolean; //# sourceMappingURL=semantic-search-benchmark.d.ts.map