/** * Advisory (never-fatal) lint over an eval suite's `expectations` prose (issue #145 * follow-up). An adopter reported that WEAK expectations — pure "mentions X" / * "includes Y" presence checks with no discriminating or negative counterpart — * pass even for a hallucinated or right-for-the-wrong-reason answer, giving false * confidence in a green run. This module flags that pattern so authors can * strengthen the eval; it never blocks a run or changes an exit code. * * Deliberately conservative: this is a heuristic over free-text prose, not a * grammar. When in doubt, do NOT warn — a false "your eval is weak" nag is worse * than a missed one, since the grader (an LLM judging the transcript) is the real * backstop for eval quality. */ import { type EvalEntry } from './eval-inputs.js'; /** One advisory finding: `evalId` names the offending eval, `message` is the full advisory text. */ export interface EvalLintWarning { evalId: string; message: string; } /** * Lint a parsed eval suite's `expectations` for the weak-presence-only pattern. * Flags an eval only when ALL of these hold (conservative — minimize false * positives): * (a) it has at least one expectation; * (b) EVERY expectation string is presence-only (see {@link isPresenceOnlyExpectation}); * (c) the eval declares no `toolExpectations` — a `mustRun`/`mustNotRun`/`sequence` * assertion is itself discriminating (it can fail a transcript that never * invoked the right tool), so its presence already covers the concern this * lint exists to catch. * * Pure and side-effect free — callers decide how/where to surface the warnings * (this never throws and never affects exit code). */ export declare function lintEvalExpectations(evals: EvalEntry[]): EvalLintWarning[]; /** * Advisory (never-fatal) lint: flag a `toolExpectations` entry that names an * executable which looks like a TYPO of one of the skill's `declaredExecutables` * (issue #145 adopter follow-up). A name that never matches a real tool is a quiet * footgun — `mustRun`/`mustSucceed`/`sequence` then fail for the wrong reason, and * `mustNotRun` passes vacuously (the tool "never ran" only because the name is * wrong). This is zero-token static plumbing, run before any spend. * * Conservative, like {@link lintEvalExpectations}: it only fires when there is a * SPECIFIC declared name the reference is probably a typo of (so it can suggest * "did you mean X?"). A reference that matches a declared name exactly, or that is * not close to any declared name (a deliberate built-in/system tool reference), is * never flagged. When the skill declares no executables there is nothing to compare * against, so it returns no warnings. Pure + side-effect free. */ export declare function lintToolExpectationExecutables(evals: EvalEntry[], declaredExecutableNames: string[]): EvalLintWarning[]; //# sourceMappingURL=eval-lint.d.ts.map