/** * Pass-1 extraction worker pool (change: optimize-parallel-extraction-pool). * * Per-file tree-sitter parsing and fact extraction is the dominant cost of * `openlore analyze`, and it is embarrassingly parallel: no file's extraction reads * another file's state. This module runs that pass on a fixed-size pool of * `worker_threads` while preserving the one property the rest of the pipeline * depends on — **input order**. * * Determinism rules (the reason this module exists rather than a bare `Promise.all`): * * 1. **Index-keyed slots, never completion order.** Each file is dispatched with its * input index and its result is written to `outcomes[index]`. Completion order is * free to vary with core contention; the array the caller merges from is always in * input order, so every downstream pass sees byte-identical input to the serial lane. * 2. **No extraction logic lives here.** Workers call the same `dispatchFileExtract` * the serial lane calls (via {@link ./extraction-worker.ts}), and the serial lane * stays the reference implementation — the caller passes it in as `serialExtract`, * which is also the fallback executor. * 3. **Fail-soft, never fail-different.** A worker that dies mid-file leaves its slot * empty and the file is re-extracted on the main thread. A worker that cannot start, * or fails its startup health probe, disables the lane wholesale. An extractor that * *throws inside* a worker is reported as an error for that file — exactly what the * serial lane would record — and is NOT retried, so a deterministic parse failure * costs the same in both lanes. * 4. **An unproven silence is never trusted.** The extractors return an empty result * (rather than throwing) when a grammar is unavailable, so a worker whose grammar for * some language failed to load in ITS thread would report every file of that language * as containing nothing — and the merge would read that as "no symbols here". A startup * probe cannot close this (it proves one grammar; the other ~20 load lazily per thread), * so the pool re-checks an empty result on the main thread until that worker has PROVEN * it can extract that language. See {@link runPooled}. * 5. **Nothing here writes to stdout.** `openlore mcp` speaks JSON-RPC over stdout, and * `build()` runs inside it. Workers keep their stdout off the parent's, relay their * logging as messages, and the lane's own disclosure is returned on the build result * for the CLI to render — never logged from the builder. * * File contents are sent to workers rather than re-read from disk. Deliberate: `build`'s * input content is authoritative and is not always what's on disk — HTML pages arrive as * inline-script-blanked text, and the incremental path passes in-memory content — so a * worker-side re-read would change extracted facts, not just I/O. */ /** One Pass-1 input record — the same shape `CallGraphBuilder.build` receives. */ export interface ExtractionFile { path: string; content: string; language: string; } /** * The outcome for one file. `ok` carries the extractor's return value (`undefined` for a * language with no extractor); `error` carries whatever the extractor threw, so the caller * can record the identical parse-health failure it records on the serial lane. */ export type ExtractOutcome = { status: 'ok'; value: T | undefined; } | { status: 'error'; error: unknown; }; /** Why the serial lane ran instead of the pool. */ export type SerialLaneReason = 'disabled-by-env' | 'too-few-files' | 'insufficient-cores' | 'pool-saturated' | 'worker-entry-unresolved' | 'pool-unavailable'; /** What lane Pass 1 actually ran on, and what (if anything) degraded. */ export interface ExtractionLaneDisclosure { lane: 'pooled' | 'serial'; /** * Workers that passed the startup probe and accepted work (0 on the serial lane). * Counts workers that ever came up, not workers still alive at the end — a worker that * died mid-run is still counted here, and its files appear in `workerFallbackFiles`. */ poolSize: number; /** Present only on the serial lane. */ serialReason?: SerialLaneReason; /** * Files whose worker died mid-extraction and which were re-extracted on the main * thread. Non-empty means the pool degraded but the facts are still whole. */ workerFallbackFiles: string[]; /** * Files a worker reported as containing NOTHING where the main thread then found real * facts — a worker-local extraction defect (typically a grammar that loaded on the main * thread but not in that thread). The facts are whole because the main-thread result is * the one that is used, but this is a genuine defect and is disclosed loudly. */ laneDefectFiles: string[]; /** * How many worker answers were re-checked on the main thread because that worker had not * yet proven it can extract that language — an empty result, or a throw. Routine, not a * degradation: the cost of never trusting an unproven worker. See {@link runPooled}. */ unprovenRechecks: number; /** * Files whose extraction took longer than {@link SLOW_FILE_DISCLOSURE_MS}, with their elapsed * time — sorted slowest first and bounded (change: fix-analyze-native-abort-and-file-cost-budget). * * Attribution, not degradation: these files were extracted normally. Before this existed, a run * that sat for minutes gave no way to learn WHICH file was responsible short of attaching a * debugger. Empty on an ordinary run, so nothing is printed when there is nothing to say. */ slowFiles: Array<{ path: string; ms: number; }>; } /** * Keep the slowest {@link SLOW_FILE_DISCLOSURE_CAP} entries, slowest first, path-tiebroken. * * Deduplicated by path, keeping the worst time. One file can be timed TWICE — once in a worker and * again on the main thread when the pool hands it back (a worker fault, an unproven language, a * dead worker) — and listing it twice would spend two of the five slots on one file and silently * evict genuinely distinct slow files, which is the opposite of the attribution this exists for. */ export declare function boundSlowFiles(slow: Array<{ path: string; ms: number; }>): Array<{ path: string; ms: number; }>; /** * The subset of `worker_threads.Worker` the pool uses. Narrow on purpose: tests * inject a fake handle to drive completion order and worker death deterministically, * without spawning threads. */ export interface ExtractionWorkerHandle { postMessage(message: unknown): void; on(event: 'message', listener: (value: unknown) => void): void; on(event: 'error', listener: (err: Error) => void): void; on(event: 'exit', listener: (code: number) => void): void; terminate(): void | Promise; } /** Creates one worker. Returns the handle, or throws if the worker cannot start. */ export type ExtractionWorkerFactory = () => ExtractionWorkerHandle; /** Caller-side control over the Pass-1 lane. Production passes none of these. */ export interface ExtractionLaneOptions { /** Test-only: drive a stub lane whose completion order and failures are deterministic. */ workerFactory?: ExtractionWorkerFactory; /** Test-only: pin the worker count, bypassing the core/file-count/process-budget sizing. */ poolSize?: number; } /** Handed to a worker at construction. */ export interface ExtractionWorkerData { /** * Marks this thread as an extraction worker spawned by THIS pool. * * The worker entry serves requests on `parentPort` as a top-level side effect of being * imported, so anything else that imports it while running inside some other worker * thread would hijack that thread's message channel. `parentPort !== null` is not enough * to tell the two apart (a test runner using a thread pool looks identical), so the * module stays inert unless it finds this sentinel. */ openloreExtractionWorker: true; /** * The language the worker should prove it can parse before accepting work. Chosen from * the build's own files so the probe never gates on a grammar this repo does not use — * every grammar is an optional dependency. Absent, or a language with no probe snippet, * means no startup probe: the per-language unproven-silence guard still covers it. */ probeLanguage?: string; /** * Whether this build has shed the CFG/def-use overlay under memory pressure (change: * make-analyze-scale-to-any-repo). Read from the main thread's build-scoped decision at spawn and * latched in the worker, since the worker isolate cannot see the main thread's async-context * store. When true the worker builds no CFGs — the overlay is shed everywhere consistently. */ skipCfgOverlay?: boolean; } /** Parent → worker. */ export type ExtractionRequest = { type: 'extract'; id: number; file: ExtractionFile; } | { type: 'shutdown'; }; /** Worker → parent. */ export type ExtractionResponse = { type: 'ready'; } | { type: 'unhealthy'; reason: string; } | { type: 'result'; id: number; value: unknown; } | { type: 'failed'; id: number; message: string; } | { type: 'log'; level: 'warning' | 'debug'; message: string; }; /** * Stable prefix on the message of a per-file failure caused by a WORKER FAULT rather than by the * file itself (change: fix-analyze-native-abort-and-file-cost-budget). * * A throw crossing the `worker_threads` boundary is structured-cloned, so only its message * survives — the same constraint that makes the parse-budget signal message-keyed. This prefix is * what lets the builder record `worker-fault` instead of blaming the source file for a defect in * the thread that was reading it. */ export declare const WORKER_FAULT_MESSAGE_PREFIX = "openlore:extraction-worker-fault"; /** Read a failure message as a worker fault. Any other message is not one. */ export declare function isWorkerFaultMessage(message: string | undefined): boolean; /** Resolved worker entry: the module a worker loads, plus any exec args it needs. */ export interface ResolvedWorkerEntry { specifier: URL; execArgv?: string[]; } /** * Locate the worker entry module for the current runtime. * * Compiled (`dist/`) is the production case: the sibling `.js` exists and needs no exec * args. Running from TypeScript source (dev, vitest) needs a loader, so the `.ts` sibling * is used with `--import tsx` — and only when `tsx` actually resolves. When neither is * available the pool is simply not offered; the serial lane covers it. */ export declare function resolveWorkerEntry(): ResolvedWorkerEntry | undefined; /** * How many workers Pass 1 should use for `fileCount` files, or `0` for the serial lane. * One core is left for the main thread (which merges results and runs Pass 2+), and the * process-wide budget is honored so concurrent builds cannot multiply the isolate count. */ export declare function plannedPoolSize(fileCount: number): number; /** Test-only: forget the sticky unavailability and the live-worker count. */ export declare function __resetExtractionPoolStateForTests(): void; /** * Run Pass-1 extraction over `files`, on the worker pool when it is available and worth * it, otherwise serially. `serialExtract` is both the reference implementation and the * fallback executor — extraction logic never forks between lanes. * * The returned `outcomes` array is always the same length as `files` and in input order. */ export declare function extractFilesForPass1(files: ExtractionFile[], serialExtract: (file: ExtractionFile) => Promise, isEmptyResult: (value: T | undefined) => boolean, opts?: ExtractionLaneOptions): Promise<{ outcomes: Array>; disclosure: ExtractionLaneDisclosure; }>; /** * One-line disclosure of the lane Pass 1 ran on, or `undefined` when there is nothing to * disclose (the pool ran clean, or the serial lane was the ordinary choice). Only genuine * degradations speak: a lane that ran as designed says nothing, so a message here always * means analysis was slower than it should have been — never that it was less complete. */ export declare function describeExtractionLane(d: ExtractionLaneDisclosure): string | undefined; //# sourceMappingURL=extraction-pool.d.ts.map