/** * Soft-Auth Detector * * Detects pages that return HTTP 200 but whose content is actually a * paywall, login gate, or "you need to sign in to access this" placeholder. * These are common on: * * - Academic publishers: JSTOR, Elsevier, Springer, Wiley, Sage * - News paywalls: NYT, WSJ, FT, The Economist, Bloomberg, The Atlantic * - Subscription services: Medium (member-only posts), Substack paid tiers * - B2B SaaS: Salesforce community posts, Slack archive, Atlassian wikis * - Enterprise docs: "employees only" intranets exposed to crawlers * - Government portals: "login required" for specific case files * * A status-code-only check passes these pages — and a soft-404 check * passes them too, because the actual "please sign in" text differs from * "page not found" phrasing. But the user landing on them hits a wall. * * For callers using Unbrowser to validate AI-generated citations, this * is a distinct failure mode worth catching separately: the URL exists, * the page resolves, but the content the AI claims is behind a gate. * A verification layer that accepts soft-auth pages will surface links * that frustrate users on click-through. * * ## Design choices * * Same approach as soft-404-detector: curated phrase lists, case- * insensitive substring match, title-aware. Phrases are chosen to be * specific enough that legitimate content about authentication (e.g. * a blog post titled "How to implement OAuth" or a Wikipedia article * about subscription business models) doesn't trip the detector. * * Conservative by design: we'd rather miss some paywalled pages than * false-positive on legitimate content that discusses logins. The * caller can always add domain-specific checks on top. * * ## Distinction from soft-404 * * Soft-404: "this page doesn't exist." The content is a not-found template. * Soft-auth: "this page exists but you can't see it." The content is a * sign-in prompt, subscription offer, or "upgrade to access" call-to-action. * * The two signals are independent and can both be present (a dead link * that redirects to a generic "members only" landing page can trigger * both). Callers typically want to reject both, but the signals are * separate so a policy like "accept paywalls but reject soft-404s" is * expressible. */ /** * Structured signal returned by `detectSoftAuth`. Mirrors the shape * of Soft404Signal so callers can treat the two uniformly. */ export interface SoftAuthSignal { /** True if at least one phrase matched in title or body. */ detected: boolean; /** Phrases that matched in the page title, if any. */ matchedTitlePhrases: string[]; /** Phrases that matched in the page body, if any. */ matchedBodyPhrases: string[]; } /** * Detects whether a page is gated behind authentication or a paywall * based on title and body content. Returns a structured signal with * the specific phrases that triggered detection. Use `isLikelySoftAuth` * when you only need a boolean. * * Both arguments are optional — at least one should be provided for * a meaningful check. If both are empty/missing, returns a signal * with `detected: false` and empty phrase arrays. * * @param body Page markdown or text content. Case is ignored. * @param title Page title. Case is ignored. Matched against a * stricter TITLE_PHRASES list than the body. */ export declare function detectSoftAuth(body?: string | null, title?: string | null): SoftAuthSignal; /** * Returns true if the page looks like a paywall or login gate. * Thin wrapper over `detectSoftAuth` for boolean-only callers. */ export declare function isLikelySoftAuth(body?: string | null, title?: string | null): boolean; /** * Exposed for tests and for callers that want to extend the detector * with their own phrases. */ export declare const _internalForTests: { BODY_PHRASES: readonly string[]; TITLE_PHRASES: readonly string[]; }; //# sourceMappingURL=soft-auth-detector.d.ts.map