/** * Soft-404 Detector * * Detects pages that return HTTP 200 but whose content is actually a * "page not found" placeholder. These are surprisingly common — many * CMS and wiki platforms prefer 200+template over a real 404: * * - Wikipedia / Wikisource: "Wikipedia does not have an article with * this exact name", "does not have a text with this exact name" * - GitHub: "This is not the web page you are looking for" * - Many CMSes: a themed "Page not found" template * - E-commerce sites: "Product not available" with full site chrome * - Custom 404 handlers that serve 200 for SEO-adjacent reasons * * A status-code-only check passes these pages, but their content is * worthless for the caller. This detector adds a second signal based * on a curated list of phrases that reliably indicate a soft-404. * * ## Design choices * * - **Phrase list, not regex**: phrases are faster to reason about, * easier to extend, and less prone to catastrophic false-positives * than regex patterns. * - **Case-insensitive substring match**: matches the convention used * by the rest of the verification engine (`excludesText` behavior). * - **Title-aware**: the title often carries the strongest signal * ("Page not found — Example.com") but isn't in the markdown body, * so we scan both. * - **Conservative**: phrases are chosen to minimize false positives * on legitimate content. A product description about "page not * found errors" won't trip the detector because the phrase is * embedded in other text and the phrase "Page not found" alone is * intentionally NOT on the list — we require more specific markers. * * ## Not in scope * * This detector doesn't try to detect: * - Paywalled or auth-gated content (use `containsText` for the * specific paywall text, or check `excludesText: 'sign in'`) * - Rate limits (handled by the standard `excludesText: 'rate limit * exceeded'` built-in check) * - JavaScript-rendered content that didn't load (handled by * `minLength` built-in check) * - Generic 404 text in documentation about HTTP (false-positive * risk — we avoid matching the literal phrase "404 Not Found" * unless it also appears in the page title) */ /** * Structured signal returned by `detectSoft404`. Callers that want * the full signal (which phrases matched, where) use this; callers * that only need a pass/fail use `isLikelySoft404`, which is a thin * wrapper around `.detected`. */ export interface Soft404Signal { /** True if at least one phrase matched in title or body. */ detected: boolean; /** Phrases that matched in the page title, if any. */ matchedTitlePhrases: string[]; /** Phrases that matched in the page body, if any. */ matchedBodyPhrases: string[]; } /** * Detects whether a page is a soft-404 based on title and body * content, returning a structured signal with the specific phrases * that triggered the detection. This is the primary detection API; * use `isLikelySoft404` when you only need a boolean. * * Both arguments are optional — at least one should be provided for * a meaningful check. If both are empty/missing, returns a signal * with `detected: false` and empty phrase arrays. * * @param body Page markdown or text content. Case is ignored. * @param title Page title. Case is ignored. Only matched against * TITLE_PHRASES, which is stricter than BODY_PHRASES. */ export declare function detectSoft404(body?: string | null, title?: string | null): Soft404Signal; /** * Returns true if the page looks like a soft-404. Thin wrapper over * `detectSoft404` for callers that only need a boolean. * * @param body Page markdown or text content. Case is ignored. * @param title Page title. Case is ignored. Only matched against * TITLE_PHRASES, which is stricter than BODY_PHRASES. */ export declare function isLikelySoft404(body?: string | null, title?: string | null): boolean; /** * Exposed for tests and for callers that want to extend the detector * with their own phrases. Mutating these lists affects subsequent * `isLikelySoft404` calls — callers that need isolation should pass * their own phrase lists through a future extension point. */ export declare const _internalForTests: { BODY_PHRASES: readonly string[]; TITLE_PHRASES: readonly string[]; }; //# sourceMappingURL=soft-404-detector.d.ts.map