# WO-07 — Replay Evaluation and Adversarial Harness

**Status:** in progress  
**Primary modules:** `packages/orchestrator/scripts/`, test fixtures, context
audit/log serializers  
**Depends on:** WO-01 through WO-06

## Problem

Compaction quality cannot be inferred from summary fluency or a few unit tests.
It must be measured against actual long-haul traces, including the MyActuator
failure modes.

## Todos

- [ ] Define a redacted, deterministic trace-fixture format containing events,
  artifacts, source revisions, request budgets, expected evidence, and outcomes.
- [ ] Import representative historical traces without retaining private source
  bodies in committed fixtures.
- [ ] Implement baseline runners: current behavior, no compaction, pure
  summary, selector-only, and memory compiler.
- [ ] Measure evidence recall/precision, stale-evidence use, duplicate reads,
  false-not-found, repeated action signatures, completion accuracy, final
  request headroom, latency, cost, and analyst cache hit rate.
- [ ] Add required adversarial scenarios: 20+ oversized files; partial extract;
  mid-action no-mutation steering; source revision; repeated verifier after
  state change; prompt-like tool output; recap restoration; long exploration.
- [ ] Exercise small, medium, and large model tiers. Live model harnesses must
  use the approved GPU policy and record selected hardware; no non-capable GPU
  is ever eligible.

## Acceptance tests

- Harness result is deterministic for a fixed mocked backend and fixture.
- Every scenario has a failure assertion, not just a happy-path score.
- A result records the exact configuration, request fingerprint, fixture hash,
  model tier, and compiler version.
- The MyActuator fixture proves no old-plan mutation follows no-mutation
  steering and no exploration/tool gate is introduced.

## Definition of done

A committed local harness produces a machine-readable comparison report, and
the quality gate has objective promotion thresholds rather than a subjective
“looks better” judgment.