# Contract Evaluation and Regression Design

## Purpose

System-message and orchestration work is finished only when a model-facing
behavioral harness proves the intended contract. Static source assertions can
guard a literal wire, but they cannot show whether evidence arrived, whether a
tool was blocked, or whether a model could integrate a steering change.

## Four evaluation layers

### 1. Compiler unit tests

Given typed runtime state, assert the request compiler selects the intended
contract IDs, puts dynamic frames in the required order, excludes retired
frames, and reports normalized context accounting. Use fixtures with unique
sentinels so a missing or duplicate frame is unambiguous.

### 2. Orchestration integration tests

Run the real runner path with a deterministic scripted backend and temporary
workspace. The backend must see the same assembled request that an actual
model would see, issue normal tool calls, and receive normal tool results.
Assert user-visible outcomes, not only method calls.

### 3. Adversarial behavioral tests

Test the failure modes the runtime is designed to prevent:

- inject typed “do not mutate; explain next action” steering during a
  multi-tool old-plan response and a brute-force loop;
- issue a full-file read, compact, and reread after exact source body left the
  request;
- run a partial branch extraction with multiple requirements, including an
  unresolved requirement; and
- make an ordinary edit without controller hash/todo preconditions and assert
  the native tool result, not a synthetic admission denial.

Each test must prove both what happens and what does *not* happen: no old-plan
mutation after steering, no extract labelled full, no missing required branch
evidence silently treated as complete, and no exploration gate blocking an
authorized read/edit.

### 4. Live inference acceptance harness

Use a real model only after the deterministic harness passes. The live harness
should use a controlled repository fixture that requires reading more source
than can be conveniently injected at once, then ask a realistic parent task
whose answer requires evidence from at least 20 oversized files.

Before generating a token, record:

```text
endpoint + exact model
selected GPU UUID/model process placement
available context window and token settings
fixture source identities and requirements
contract-selection manifest
```

The harness must refuse to launch when a capable, explicitly approved device
cannot be verified. It must never select a weaker incidental accelerator simply
because it is available.

## Cross-tier acceptance matrix

| Behavior | Small | Medium | Large |
| --- | --- | --- | --- |
| Current objective/source visibility | Required | Required | Required |
| Full read remains first class | Required | Required | Required |
| Branch extraction returns all requirements or justified unresolved items | Required | Required | Required |
| Natural edit result reaches model | Required | Required | Required |
| Generic exploration/todo/verifier gate | Off by default | Off by default | Off by default |
| Optional concise coaching | Allowed, visible, non-fabricating | Advisory only | Advisory only |
| Model-selected valid tool sequence | Preserve where possible | Preserve | Preserve |
| Typed steering safe-boundary reconciliation | Required | Required | Required |

## Measurements

Record metrics that answer whether the system actually helped:

- request-level source-body visibility and rehydration rate;
- full-read versus shell-fallback rate;
- extraction requirement coverage and unresolved-reason quality;
- synthetic-denial count (target: zero outside real authority/atomic/schema/
  resource boundaries);
- repeated action/result signature rate, reset reasons, and actual recovery;
- tool-result-to-next-action grounding rate;
- verification freshness at completion; and
- user-visible timeouts, packet/network load, GPU placement, and daemon boot
  identity for live tests.

Do not optimize a metric by hiding the behavior it measures. For example,
reducing reread count by denying reads is a regression if source visibility was
lost; reducing failures by relabelling truncation as success is a regression if
the unresolved requirement remains.

## Failure triage for the harness

When an evaluation fails, preserve the request dump, tool trace, compacted
audit, active evidence frame, selected contracts, model output, and fixture
state. Determine whether the defect is in contract selection, frame rendering,
context budgeting, tool dispatch, tool implementation, model behavior, or the
test fixture. Fix the earliest causal layer and retain the test as a regression
case.

Do not weaken a behavioral assertion because a prompt change makes a model
avoid the path. The scenario must still force the requested evidence,
steering, edit, or recovery behavior to surface naturally.