# Large PDF collections

Ask Omnius to process the collection in ordinary language, specifying the result
you need: search, ordering, original-page manipulation, summary, or extraction.
For example: “Review every supplied PDF, extract each recorded duration with its
source page, order the documents by the invoice date, and deliver the complete
results.” A field's meaning, units, missing-value handling, and ordering direction
remain part of the request; a date appearing somewhere in a PDF does not by
itself establish that it is the requested invoice date.

The collection is stored outside the model context. Original PDFs, complete
native page text, page geometry, immutable source identities and hashes remain
available in the private source archive. SQL applies scope, version and request
filters before sorting and paging. Retrieval can reopen exact source ranges;
condensing a model prompt never replaces the original evidence.

Search uses cursor-paged lexical matches in the authorized collection. A search
result, including a result with no matches, does not establish that every source
was extracted or reviewed. Exhaustive requests retain the complete selected
population and track source coverage separately from relevance ranking.

There is no application limit on the total number of PDFs, pages, selected
records, or output files. Per-call work widths keep memory and prompts finite;
the model's actual context window, available storage, compute and each provider's
per-file upload limit still apply. Processing the entire collection takes work
proportional to that collection. Targeted search and exhaustive extraction use
different coverage requirements.

| Task | Runtime tools and retained result |
| --- | --- |
| Ingest and search | `source_corpus` retains originals, indexes native pages, and returns scoped inventories and exact ranges. Telegram's `manifest="current_task"` selects the host-admitted attachment population. |
| Scanned pages | Explicit `source_corpus` OCR work retains page images, full engine text, word positions and engine receipts in the separate `ocr_text` channel. Interrupted page work resumes without replacing native text. |
| Summary and extraction | `corpus_work` preserves the complete request and selection, issues bounded input claims, records ranges that actually reached a successful model request, and commits derived records with source anchors. |
| Field ordering | `corpus_work` stores source-anchored typed fields and orders through SQL. Exact decimal and date handling, ties, missing values and multiple values have explicit policies. Model-selected field meaning remains subject to source review. |
| Original-page manipulation | `pdf_corpus_recipe` and `pdf_corpus_transform` plan and execute original-page ordering, merge and split operations without reconstructing PDFs from summaries. |
| Factual acceptance | `source_answer_review` and `corpus_export_review` independently review complete selected output and source coverage. Numeric aggregates reopen every selected native input and verify exact operands and arithmetic. |
| Large result delivery | `corpus_export_review` partition actions retain upload-sized data and reconstruction files. `partition_finalize` inherits a complete current parent review and verifies every child; actual provider receipts still establish delivery. |

Processing and delivery checkpoints survive restart. Committed work and accepted
uploads are reused under the current task; stale owner or request authority must
be renewed. A Telegram upload quota wait retains the exact reviewed candidate
and resumes at the provider retry time. Completion requires receipts for every
selected output, including all reconstruction files. An extracted index, a model
claim of completion, or an intermediate progress message does not establish that
the final answer was delivered.

Directory acquisition preserves enumeration, original-file acquisition and
manifest-writing checkpoints. Restart verifies the previously observed directory
and file identities before continuing; a changing directory cannot silently
become a different collection. This is an observed, verified manifest, not an
atomic filesystem snapshot. Export writing and fixity checks also resume within
an individual JSONL record, so output size does not impose a total export cap.

Verified durable work can continue beyond the default emergency turn budget.
An explicit finite budget still applies. Repeating a call without advancing its
current source-bound checkpoint earns no continuation credit; progress does not
approve facts or fulfill the final-output and delivery requirements.

After restart, the host reopens the current pending SQL job before applying its
retry guard. An authentic job can receive one admission attempt at its existing
producer clock, including an initialized job that has not advanced yet. Reopening
the job creates no progress event. The consumed attempt survives restart and
cannot be renewed by switching job IDs; actual verified work must advance the
clock before another attempt becomes available.

Native text and geometry can be incomplete or ambiguous. Unresolved extraction,
field association or arithmetic stays explicit; visual review can resolve the
relevant page on the configured capable backend. The research basis and
stage-by-stage evaluation criteria are in the
[PDF distillation synthesis](../../.aiwg/research/pdf-knowledge-distillation/synthesis.md).

Numeric review distinguishes populated cells from recorded events. A nonempty
cell can contain zero or an incomplete time placeholder; it does not necessarily
record an event. Bounded examples expose exact distinct native spellings without
frequencies or expected answers. The reviewer selects the meaning from the full
request, and the host verifies arithmetic against the complete original field.
