# World-Class Memory Compiler Program

**Status:** active implementation program  
**Owner:** Omnius orchestration and memory packages  
**Last updated:** 2026-07-13

## Mission

Replace chronological transcript compaction with a provenance-preserving memory
compiler. The compiler manages context as a materialized view of typed,
versioned task memory; it does not coerce the agent's reasoning, deny tool use,
or replace evidence with generic summaries.

The implementation has one non-negotiable invariant:

> Memory management may preserve, retrieve, organize, and defer. It must never
> block exploration, source reads, edits, verification, or a model's legitimate
> change of plan.

## Architecture at a glance

```text
immutable event ledger ──► typed memory records ──► dependency graph
                                      │                    │
                                      └────► hybrid index ◄─┘
                                                     │
exact outbound request ──► budget compiler ──► working-set materializer
                                                     │
                                    isolated inference memory compiler
                                                     │
                                      validated MemoryDelta + audit trail
```

The outbound request is built before any compaction decision. Its complete
budget includes stable prompts, tool schemas, active steering, task state,
selected evidence, output reservation, and safety margin. No automatic or
manual compaction may run while the final request has at least 40% of the real
model context window free.

## Memory classes

| Class | Canonical contents | Rule |
|---|---|---|
| Authority | current user intent, active steering, trusted policy | never summarize or retrieve from untrusted content |
| Task / working set | open requirements, next action, unresolved claims, live verifier | materialize for the active task epoch |
| Artifact | exact files/chunks, content hash, range, revision | exact body is canonical; extract is derived |
| Action ledger | intent → tool call → result → mutation → verifier → outcome | preserve transaction integrity |
| Episodic | immutable event history | archive-first; not blindly model-visible |
| Semantic | validated facts and decisions with temporal validity | source-backed and supersedable |
| Procedural | validated reusable workflow lessons | explicitly promoted, never inferred from one failure |

## Work-order index and execution todos

| ID | Work order | Depends on | Status | Completion evidence |
|---|---|---|---|---|
| WO-01 | [Exact Request and Budget Compiler](./WO-01-exact-request-budget.md) | — | in progress | exact serializer, dump ledger, threshold tests |
| WO-02 | [Typed Memory Fabric and Event Ledger](./WO-02-typed-memory-fabric.md) | WO-01 | in progress | immutable ledger, authority/progress tests |
| WO-03 | [Dependency Graph and Working-Set Materializer](./WO-03-dependency-working-set.md) | WO-02 | in progress | graph-cut, active-root, and transaction tests |
| WO-04 | [Inference-Driven Memory Compiler](./WO-04-inference-memory-compiler.md) | WO-01, WO-03 | in progress | exact-request shadow proposal tests |
| WO-05 | [Artifact Fidelity and Explicit Materialization](./WO-05-artifact-fidelity-materialization.md) | WO-02, WO-03 | in progress | full-read, stale revision, extract coverage tests |
| WO-06 | [Temporal Hybrid Retrieval](./WO-06-temporal-hybrid-retrieval.md) | WO-02, WO-03, WO-05 | in progress | retrieval/relevance/abstention tests |
| WO-07 | [Replay Evaluation and Adversarial Harness](./WO-07-evaluation-harness.md) | WO-01–WO-06 | in progress | deterministic trace reports and tiered harness results |
| WO-08 | [Shadow Rollout, Migration, and Legacy Removal](./WO-08-rollout-legacy-removal.md) | WO-07 | planned | canary report, rollback drill, legacy deletion |

`[ ]` entries in each work order are the canonical engineering todos. The
[granular tracker](./TRACKER.md) reconciles every implementation and promotion
subtask. An item cannot be marked complete from a model claim: it requires the
named code, focused tests, typecheck, and recorded acceptance evidence.

## Global acceptance criteria

- [ ] The compaction trigger uses the exact pending request, not historical
  messages alone.
- [ ] No request compacts with 40% or more real-window headroom remaining.
- [ ] A current action, exact evidence it needs, and its verifier remain an
  atomic dependency transaction.
- [ ] Current user authority and unacknowledged steering are present exactly
  once and cannot be overwritten by history or tool output.
- [ ] Full source reads are first-class; branch extraction is derived,
  coverage-checked, and never silently substitutes for source.
- [ ] Memory compiler failure results in `hold` and an audit record, never
  heuristic deletion.
- [ ] Repeated identical compiler analyses are deduplicated by a content/state
  fingerprint; state change is required to re-run.
- [ ] All compaction choices are inspectable in the TUI and session logs.
- [ ] The implementation is benchmarked on real replay traces and adversarial
  traces across small, medium, and large model tiers before enforcement.

## Research basis

These work orders borrow principles, not implementations. LongMemEval motivates
separate indexing, retrieval, and reading with temporal updates and abstention;
MemGPT motivates memory tiers; RAPTOR motivates retaining raw leaves beside
derived hierarchy; graph-memory work motivates associative recall; and
instruction hierarchy motivates authority separation. See
[LongMemEval](https://arxiv.org/abs/2410.10813),
[MemGPT](https://arxiv.org/abs/2310.08560),
[RAPTOR](https://arxiv.org/abs/2401.18059),
[HippoRAG 2](https://arxiv.org/abs/2502.14802), and
[Instruction Hierarchy](https://arxiv.org/abs/2404.13208).

Research claims are hypotheses to test against Omnius traces, not a reason to
copy a paper's architecture wholesale.

## Verified implementation evidence — 2026-07-13

- `request-budget.ts` snapshots and fingerprints the actual JSON-wire-equivalent
  outgoing payload (including aliases and model-visible memory prefixes), then
  computes a segmented budget from that one immutable request; every
  context-window dump persists or safely recompiles that exact budget.
- `ContextMemoryLedger` persists append-only authority, task, action, tool,
  artifact, mutation, verifier, dependency, transaction, and supersession
  records. The runner records the user/task root before its first tool call and
  records tool transactions without making telemetry a tool gate.
- `memory-compiler.ts` rejects an inference cut that would archive authority,
  active dependency evidence, a partial tool transaction, an unknown record,
  or an insufficiently classified proposal.
- Full source reads remain canonical in `EvidenceLedger`; a completed manual
  branch extract can replace only the rendered view while canonical source is
  retained. Automatic branch routing now requires real headroom exhaustion
  rather than a fixed 20% file-size boundary. Capped grep/tool output is typed
  partial success with recoverable provenance and deterministic narrowing.
- Active working sets are graph- and transaction-closed: a soft rendering
  budget may omit optional history but cannot split a live action from its
  call, result, source, mutation, or verifier. The bounded retriever filters
  by epoch, authority, temporal validity, path/revision/range and active graph
  roots before lexical/host-semantic ranking; it returns explicit abstention on
  missing or over-budget evidence.
- The existing isolated compaction analyst now caches byte-identical state
  signatures and exposes cache/fingerprint information in its audit record.
- The exact final request now drives the v2 inference memory compiler at the
  shared pre-backend boundary. It projects every mutable message body into a
  typed candidate, requires one inference-selected disposition per candidate,
  and applies nothing unless graph, authority, artifact, and exact
  post-materialization budget checks all pass. Missing or invalid inference is
  a `hold`, never a heuristic truncation. `OMNIUS_MEMORY_COMPILER_MODE=shadow`
  is the explicit rollback/diagnostic mode.
- Immutable `omnius-artifact://sha256/...` references are range-readable with
  the first-class `artifact_read` tool. A partial representation is trusted
  code materializing declared lines from that object, never model-authored
  source text.
- `memory-compiler-replay.mjs` executes versioned, deterministic adversarial
  fixtures against the production ledger, request budget, and graph-cut
  validator. Current cases include active evidence, partial extraction,
  prompt-shaped tool data, source supersession, verifier-after-change, and a
  20-artifact batch.

Focused verification completed:

```text
@omnius/orchestrator: 66 focused tests passed
@omnius/memory: 11 SQLite ledger tests passed
@omnius/execution: 27 focused tests passed
TypeScript: all 11 workspace packages built
memory compiler replay: 14/14 deterministic adversarial scenarios passed
live large-model extraction: 20/20 approved-A100 contracts passed
```

These are foundations, not a release claim. WO-07 and WO-08 remain incomplete
until real redacted trace comparison, controlled model-tier evaluation,
shadow-operation evidence, canary/rollback, and legacy-path deletion prove the
new pipeline is safer than the old one.