/** * Match eval log — JSONL ranking-decision writer (T3 of the Round 3 plan). * * Why this module exists: * The bootstrap evaluator (`evaluateSolutionMatcher`) measures matcher * quality against a labeled fixture, but production traffic is open-ended. * T2 hoisted query normalization out of the per-solution loop, which is * fast, but it also hid the "what did we actually rank, and why?" signal * from offline review. This module appends a single JSONL line per matcher * call capturing the normalized query, the top candidates with their * matched terms, and which ones the caller ultimately surfaced. * * The target consumer is offline analysis: a reviewer can tail or grep * the file to spot systematic recall misses or spurious matches without * instrumenting production. * * Privacy posture (T3 security review fix): * The raw user prompt is NEVER written to disk. Instead, we store a * short SHA-256 prefix (`rawQueryHash`) plus character length * (`rawQueryLen`). This keeps dedup and "was the prompt substantial" * signals available for offline analysis while eliminating the PII / * API-key / credential leakage risk of persisting raw prompts in * `~/.forgen/state/match-eval-log.jsonl`. The `normalizedQuery` array * already carries the matching-signal payload and is safe to persist * because it only contains short tag tokens (never the full prompt). * * Operational principles: * 1. **Off the critical path.** Never throw; never block. A failed write * is silently swallowed — the hook must continue to return its * solutions even if the log is misconfigured, read-only, or full. * 2. **Bounded record size.** Candidates are capped at 5 (the matcher's * own top-5 cap). `normalizedQuery` is capped at 64 terms. Each * candidate's `matchedTerms` is capped at 16. Worst-case record ≈ * 2KB, which stays under Linux PIPE_BUF=4096 for safe concurrent * appends on local filesystems. * 3. **Symlink defense.** `fs.openSync` with `O_NOFOLLOW` refuses to * follow a symlink at the log path. Without this guard, an attacker * with write access to `~/.forgen/state/` could redirect appends to * `~/.ssh/authorized_keys`, `~/.bashrc`, or other sensitive files. * 4. **File-lock for concurrency.** Uses `withFileLockSync` to serialize * concurrent writers. macOS PIPE_BUF=512 is smaller than the worst- * case record size so POSIX atomic append alone isn't enough. * 5. **Opt-out via env, fail-closed on invalid config.** * `FORGEN_MATCH_EVAL_LOG=off|disabled|0|false|no` disables entirely. * `FORGEN_MATCH_EVAL_LOG_SAMPLE=` samples. An invalid * sample value (NaN, out of range, whitespace) falls back to 0 * (skip) rather than 1 (log everything) — fail-closed for privacy. * 6. **File size cap.** `readMatchEvalLog` refuses to parse files * larger than 50 MB to prevent OOM in the offline analyzer. Callers * are responsible for rotating the log externally. */ /** Environment variable controlling log enable/disable. */ export declare const MATCH_EVAL_LOG_ENV = "FORGEN_MATCH_EVAL_LOG"; /** Environment variable controlling sample rate (0.0 – 1.0). */ export declare const MATCH_EVAL_LOG_SAMPLE_ENV = "FORGEN_MATCH_EVAL_LOG_SAMPLE"; /** * Single ranking decision captured at matcher call time. * * Rationale for each field: * - `source`: distinguishes the hook path (`solution-injector`) from the * MCP path (`solution-reader.searchSolutions`). They have different * query shapes and the log should support filtering by origin. * - `rawQueryHash`: first 16 hex chars of SHA-256 over the user prompt. * Enables dedup ("this query shape recurred") without persisting the * prompt text. NOT cryptographically reversible — only useful for * grouping identical queries in offline analysis. * - `rawQueryLen`: character count of the original prompt. A rough * "was this a substantial query?" signal that helps triage. * - `normalizedQuery`: the output of `defaultNormalizer.normalizeTerms` * over `extractTags(rawQuery)`. This is what actually drove matching, * so it's the most important piece for debugging ranking surprises. * Only short tag tokens — safe to persist. * - `candidates`: top-N ranked solutions with relevance and matched * terms. Bounded by `MAX_CANDIDATES_LOGGED`. * - `rankedTopN`: the names of the top-N solutions the CALLER RECEIVED * from the matcher at the time of logging. This is the pre-filter top * (hook path) or post-`limit` top (MCP path). Caller-side budget / * experiment / disjoint filtering happens AFTER logging and is not * captured here — by design, this field records what the matcher * returned, not what the hook ultimately injected. * - `ts`: ISO 8601 timestamp. Always set by the logger, never by the * caller — prevents clock injection from polluting the log. */ export interface MatchEvalLogRecord { source: 'hook' | 'mcp'; rawQueryHash: string; rawQueryLen: number; normalizedQuery: string[]; candidates: Array<{ name: string; relevance: number; matchedTerms: string[]; }>; rankedTopN: string[]; ts: string; } /** * Caller payload. `ts` and `rawQueryHash`/`rawQueryLen` are derived by * the logger from the caller-supplied `rawQuery`. `rawQuery` itself is * consumed in-process only and never written to disk. */ export interface MatchEvalLogInput { source: 'hook' | 'mcp'; /** Raw user prompt. Hashed + length-captured, never persisted. */ rawQuery: string; normalizedQuery: string[]; candidates: Array<{ name: string; relevance: number; matchedTerms: string[]; }>; /** * Top-N by relevance that the matcher returned to the caller at log * time. See `MatchEvalLogRecord.rankedTopN` for semantics — this is * NOT the post-filter "actually injected" set. */ rankedTopN: string[]; } /** * Append a single ranking decision to the match-eval-log JSONL file. * * Fail-open: any error is caught and debug-logged. Callers can invoke * this without guarding — the logger will never bubble an exception into * the hook critical path. */ export declare function logMatchDecision(input: MatchEvalLogInput): void; /** * Read all records from the match-eval-log file. Intended for tests and * offline analysis tools; NOT for hot-path use. * * Malformed lines (non-JSON, missing required fields, wrong shape) are * silently skipped — preserves the debug value of the rest of the file * if one entry gets corrupted by a partial write or tool error. * * DoS guard: refuses to read files larger than `MAX_LOG_FILE_SIZE_BYTES` * to prevent OOM when a long-running log grows unbounded. Returns [] in * that case and debug-logs the skip. */ export declare function readMatchEvalLog(): MatchEvalLogRecord[];