# Release 0.4.3 — Fair span coverage and honest "unanalysed" classes

**Coordinated packages:** `ds4-context-core`, `ds4-context-reference-adapter`, and `ds4-context-engine` 0.4.3.
**Implementation commit:** `47100ad`.

## Summary

Diagnostics patch. The class-only report that `0.4.2` attached to a failed
compaction could not answer the question it exists for: in its first production
run, 13 of 23 rejected spans came back as `not-classified-budget`, and the two
`no-near-miss` entries could not be distinguished from spans the classifier
never analysed. The classifier now covers every distinct span with the cheap
transformed-form lookups before any near-miss analysis starts, gives each
near-miss candidate an equal share of what remains, and reports spans it did not
analyse as unanalysed instead of guessing.

Validation strictness, the eight-bullet and 25% repair bounds, the fail-closed
fallback to Pi default compaction, the provider-facing defaults, the SQLite
schema and every configuration key are unchanged. The report still contains
class names and counters only, so it stays safe to log.

## Changes

- `packages/core/src/compaction/summary-contract.ts`:
  - Transformed-form lookups (`escaped-form-present`, `unescaped-form-present`,
    `whitespace-collapsed-present`, `case-variant-present`,
    `typographic-variant-present`) run for every distinct span before the first
    near-miss lookup, so span order can no longer decide which spans are
    classified.
  - Each remaining near-miss candidate may spend an equal share of the lookups
    left, at least 32 and never more than the budget remaining, so one long span
    cannot consume the budget of the others.
  - New relations distinguish "not analysed" from "absent": `not-classified-length`
    above the 96-character near-miss limit, `not-classified-partial` when the
    span's share ran out, and the existing `not-classified-budget` when the
    shared budget was already exhausted. `no-near-miss` now means every
    applicable lookup ran and found nothing.
  - The report adds `spansClassifiedCheap` (span occurrences attributed by a
    transformed-form relation), `corpusSources`, `probeBudget` and `probesUsed`.
    `classificationComplete` is false when the shared budget or a per-span share
    stopped an analysis; a `not-classified-length` span does not clear it,
    because that limit is deterministic rather than a resource shortfall.
  - `UnsupportedSpanClassOptions.probeBudget` can override the 4000-lookup
    budget; values below one or non-finite values are ignored. No configuration
    key was added.
- `tests/unit/summary-contract.test.ts`: the existing bound test now asserts
  `not-classified-partial` and `probesUsed <= probeBudget`, plus new coverage for
  the phase split, the length limit, budget exhaustion and unusable budget
  values.
- `docs/COMPACTION.md`: the diagnostics section documents the three
  not-classified classes, the phase split and the new counters.

No change was needed in `src/pi-adapter/summary-generator.ts` or
`src/pi-adapter/compaction-coordinator.ts`: the existing error and the existing
`custom_fallback` warning already carry the report, so the new fields appear in
`unsupportedSpanClasses` automatically.

## Why

The first production run of the `0.4.2` report (a manual compaction, 23 rejected
spans across 16 bullets) came back as:

| Class | Count |
| --- | --- |
| `escaped-form-present` | 8 |
| `no-near-miss` | 2 |
| `not-classified-budget` | 13 |

Length buckets were dominated by long spans (10 in `len-33-64`, 7 in
`len-65-120`, 2 in `len-121-plus`) and shapes by prose (20 spans containing
spaces, 10 quotes, 10 colons), so the entries looked like quoted excerpts rather
than identifiers. Only the eight escaped-form spans were actionable: the value
was present, only its rendering differed. Two defects in the instrument blocked
the rest:

- **Coverage.** Every lookup costs one scan of each corpus source
  (`sourceText`, `readFiles`, `modifiedFiles`) and the budget was spent in span
  order, while `single-deletion-present` costs up to one lookup per character
  and `composed-two-present-parts` up to two more per length. A handful of long
  spans, or their complete analysis, could therefore exhaust the budget before
  the remaining spans received a single cheap lookup.
- **Ambiguity.** `no-near-miss` was also returned for spans above
  `MAX_NEAR_MISS_SPAN_LENGTH`, where the near-miss analysis is never attempted.

Neither defect weakened validation: the repair bound and the fail-closed
fallback were respected in every one of these runs. The problem was that counts
alone could not separate a summarizer that invents values from one whose
rendering or composition rules differ from the validation domain, and those two
findings call for opposite fixes. This release makes the next report
interpretable before any change to validation, repair, or the summarizer prompt
is considered.

## Privacy and safety

- The classifier remains observation-only and cannot alter validation, repair
  bounds, or the fail-closed decision.
- The report still contains class names and counters only. The new fields are
  integers (occurrence, source and lookup counts); no span text, no evidence
  text, and no hashes of either are added to logs, warnings, or errors.
- No provider call, network access, or configuration change is involved in this
  release; the classifier runs only after a validation failure has already been
  decided.

## Compatibility and persistence

- No migrations, no schema change (still 16), no new configuration key, and no
  change to any default.
- The report shape is additive: existing consumers keep working, but a reader of
  `relations` must tolerate the new keys `not-classified-length` and
  `not-classified-partial`, and can no longer assume that `no-near-miss` covers
  every unanalysed span.
- `classificationComplete: false` is now also possible with the shared budget
  intact, when a per-span share stops an analysis.

## Measured scope and limitations

Measured locally on the development host with the unit fixtures, no provider
calls:

- A 96-character span with no transformed-form match costs 185 lookups when
  analysed completely (96 single-deletion plus 89 prefix probes), confirming that
  two or three such spans could exhaust the whole budget under the `0.4.2` order.
- With 80 distinct spans of 91-92 characters and the default budget, the cheap
  phase spends 80 lookups and each span then receives a 49-lookup share:
  `probesUsed` equals `probeBudget` exactly and every span is reported as
  `not-classified-partial`.
- The knife edge is observable: in a combined fixture, a budget of 190 reports
  `not-classified-partial` while 191 reports `no-near-miss`, because the share is
  exhausted on the lookup that completes the analysis.

Limitations that remain:

- Distinct spans are deduplicated, so `spansClassifiedCheap` counts occurrences
  while the budget is spent per distinct value.
- Spans above 96 characters are still never near-miss analysed; they now say so
  instead of claiming absence.
- The default budget is still 4000 lookups per failure, unchanged from `0.4.2`.
- A `not-classified-*` class still cannot say whether the span was invented;
  it says only that the classifier did not look.

## Validation

- `npm run build:core`, `npm run build:adapters`, and `npx tsc --noEmit` clean.
- `npm run check`: 99 test files, 627 tests, all passing.
- `npm run quality:compare`, `npm run schema:context-persistence` (passed) and
  `npm run pack:check`: core 0.4.3 (247 files), reference adapter 0.4.3
  (7 files) and engine 0.4.3 (101 files) verified in a clean consumer.
- Classifier behavior was additionally checked against the `0.4.2` semantics
  with a synthetic reconstruction of the production histogram shape (long spans
  first, cheap matches after), without touching session data.
- Not verified: the improved histogram on a real failed compaction. That
  requires a natural failure and is expected to appear on the next
  `compaction.custom_fallback` warning.

## Publication and registry verification

Published manually in dependency order from `928f1a4`: `ds4-context-core@0.4.3`,
then `ds4-context-reference-adapter@0.4.3`, then `ds4-context-engine@0.4.3`, all
with the default `latest` tag, after `npm run pack:check` on the same commit.

A first `npm run registry:check -- 0.4.3` failed with `ETARGET` because the local
npm cache still served the previous packument; `npm view --prefer-online`
already reported 0.4.3 with `latest` resolving to it for all three packages, and
the check passed when re-run against an empty cache directory: "Verified all DS4
registry packages at exact version 0.4.3".

Annotated tag `v0.4.3` and the GitHub Release were created from `928f1a4`. No
provider calls are involved in this release procedure; the classifier runs only
after a validation failure has already been decided, and no session data,
credentials, or provider payloads were read or written while preparing it.