# Complete web sources and request-bound extraction

Status: implemented and verified in the isolated installed runtime. Added by
the operator during the OMNIUS research stream/progress repair. Publication and
restart of the active session remain with the operator.

## Requirement

Do not silently truncate web pages. When a complete page cannot fit the main
agent's context, use a dedicated isolated extraction pass that accounts for the
whole source and preserves the requested information's depth and breadth.
An archive containing the full bytes is necessary, but is not proof that the
agent or extractor read the complete requested material.

## Gaps found and repaired

- Native HTTP bytes were already complete, content-addressed and hash-bound.
- `web_fetch` nevertheless returned a default 5,000-character prefix, including
  on cache hits, with a 100,000-character display ceiling.
- The pending HTML evidence implementation selected the first article and capped
  article text; supplementary rendering clips other material.
- The runner could admit that short projection without any extraction call.
- The existing evidence branch remains useful targeted discovery, but samples at
  most six windows/220 lines/24,000 characters and two semantic rounds. It
  clips long lines, excerpts and the request brief. It is not a full-source
  extraction mechanism and must not be presented as one.

## Ordered work

1. Return the full requested textual/raw source view on fresh and cached
   fetches. Keep legacy `max_length` accepted without silently clipping the
   source. Retain explicit size failures, immutable raw bytes and native hash
   receipts. Capacity management belongs to the runner.
2. Traverse the complete static document in order, including all articles,
   tables, code/pre whitespace and captions. Distinguish non-rendered code and
   bounded supplementary metadata from the complete semantic source. Sparse
   or JavaScript-dependent pages must explicitly require GUI/rendered evidence
   rather than claiming rendered-page coverage from static HTML.
3. Capacity-admit the complete semantic view when it fits. Otherwise archive
   that complete semantic document separately from raw HTML and dispatch a
   source-wide extraction coordinator through the existing configured backend.
4. Partition every source character into contiguous lossless hash-bound
   ranges, including long minified lines and Unicode. Every worker receives
   the full accepted original request and requested facets, never a clipped
   replacement. Validate quoted findings against exact chunk/source ranges.
5. Track real completed chunk coverage and exact resumable source/request
   identity. Budget, ownership, Stop, invalid response or unavailable inference
   leaves explicit remaining coverage/cursor. Do not infer absence from an
   unread region or declare the source complete because a few excerpts look
   relevant. Worker judgments do not grant objective fulfillment.
6. Preserve all extraction findings in the owned artifact/state. Give the
   parent a complete request-focused result only after coverage is accounted
   for and it fits; otherwise require an explicit reduction/read/resume step,
   retaining the full findings rather than clipping the main view silently.
7. Verify full-content tails, multiple articles, code/tables/captions, fresh
   and cache paths, request clauses beyond old limits, chunk coverage, Stop,
   resume identity, invalid model evidence and true budget exhaustion. Run
   actual remote extraction on synthetic documents with late and distributed
   requested facts; no invented inference replies count as live acceptance.
8. Rebuild and test the isolated installed package, then make scoped commits
   and push. Preserve unrelated changes and the operator's publication files.
9. Repair invalid returned worker replies within the dedicated pass using
   existing finite calls/attempt/deadline budgets. Record static validation
   reasons and reply hashes, reserve correction-prompt capacity in advance,
   and keep exact quote validation. An invalid chunk earns no coverage credit.

Installed acceptance exposed a separate [request-contract/budget repair](WO-fallback-contract-cooldown-20261013.md)
needed before the low-context public path could finish. Keep these dependencies
explicit; a successful isolated worker alone does not certify the final path.
The final scope audit also opened the [browser/crawler work order](WO-web-browser-crawl-full-coverage-20261013.md)
for independent prefix cuts in those producers.

Dependencies: the existing native web-source custody, source artifacts,
configured branch backend, capacity admission and Stop/owner fences. Reuse
them; do not add a separate completion-approval workflow.

## Verification receipts

The [live coordinator receipt](evidence/web-source-wide-coordinator-live-20261013.json)
records eight real Muse HTTP 200 generations in 40.4 seconds. Five contiguous
workers cover all 99,981 characters of a synthetic HTML semantic document;
three cover all 54,551 characters of single-line JSON. Every requested field
is recovered with exact host-validated source quotes, including later sections,
table values, code, Unicode, captions and an attribute-only API destination.
Full original request/facets survive the actual wire adaptation unchanged.

This receipt covers the compiled coordinator and production remote backend.
Installed public Telegram source routing, parent capacity and goal settlement
are covered by the separate final acceptance below. No local inference, fake
model/catalog replies or actual Telegram sends are involved.

The [installed worker diagnostic](evidence/web-source-worker-validation-before-20261013.json)
captures two distinct returned-reply problems: Step normalized whitespace inside
quotes, while Muse returned literal quotes but marked the compound request
uncertain because only part of it appeared in the chunk. Exact quotes remain
mandatory. Valid quoted findings may coexist with an unresolved facet; neither
chunk support nor processing coverage establishes whole-task fulfillment.

The [parent-capacity diagnostic](evidence/web-source-parent-capacity-before-20261013.json)
then records five valid workers covering all 109,910 RFC characters, followed
by completed-extraction resumption and primary turns with only 662/882 output
tokens. The latter returned an incomplete provider response before the
fixture's 20-request cap. This is a main-turn reservation/reduction problem,
not an invalid source worker or an artifact-query helper. Primary turns now
reserve at least 4,096 output tokens, subject to an explicit lower user ceiling,
and completed findings have a compact complete artifact that reopens without
another worker pass. Runtime timeouts remain bounded and unchanged.

The [final installed public-path receipt](evidence/web-source-installed-public-live-20261013.json)
records nine actual model requests in 65.8 seconds, with real catalog, inference
and RFC source responses. The first schema rejection is repaired on the same
model; the remaining eight HTTP requests succeed. Three workers account for
all 109,910 source characters, preserve the complete original request and
produce exact host-validated quotes. The final reply contains all three
requested details, including the late appendix, and preserves mandatory versus
advisory wording. The native host goal is fulfilled and intake is consumed
after the scratch Telegram ACK. The installed bundle SHA256 is
`546ad34240141d7597b979bb298a7381ae3ab4c8356c28a95a9c7955ddcf51ef`.

This run uses a 65,536-token configured context. The separate
[32k capacity receipt](evidence/web-source-installed-capacity-guard-20261013.json)
records complete source processing followed by a typed before-transport pause:
the intact request, safety margin and required output reserve exceed capacity.
It is a successful capacity-guard check, not a successful final-answer run.
No source prefix or fabricated completion is substituted, and automatic replay
stops. Complete extraction cannot guarantee a final reply when the essential
request itself cannot fit the selected model's context.

Execution suites pass 98 tests; runner/extraction/fallback/budget suites pass
264 with one existing opt-in test skipped; Telegram suites pass 115. All
workspace packages rebuild. The 583-file tarball audit has zero warnings and
the compiler-free fresh npm installation passes. Browser producer acceptance
uses the same installed bundle and is recorded in the dependent work order.
Telegram transport acknowledgments are simulated; actual delivery/typing,
historical group transcripts and web-search availability are outside this run.
