# Layer 5 — Objective Guard Scientific Validation

## Hypotheses

### H1 — Size versus Effective Complexity

Controlling task and model, locality, observability, and contracts predict success better than total lines or files.

### H2 — Non-Monotonic Recovery

Functionally equivalent versions with organic growth will show worse performance than small versions or versions later modularized and made observable.

### H2b — Regime Change

The decline will fit better using segmented models or splines than a single continuous slope, and the change point will correlate with blast radius and connectivity.

### H3 — Anticipatory Surface

Incorrect trajectories will exhibit anomalous surface, distance, and dispersion before close.

### H4 — Relative Evidence

The ratio \(J_t=E_t/S_t\) will discriminate better than isolated diff size.

### H5 — Useful Intervention

A calibrated alert will reduce unnecessary tokens, time, and surface without reducing correct resolution.

### H6 — Rewrite

The probability of rewrite attempt will grow more with estimated understanding cost—transitive closure, low locality, and lack of feedback—than with minimal change size.

## Evidence ladder

### E0 — Documentary Dogfooding

The script `scripts/validate-docs.mjs` demonstrates that a cheap structural contract can detect documentary drift. Does not validate the code agent thesis.

### E1 — Golden Corpus of Detectors

Before/after pairs for syntax, weakened tests, stubs, and dependencies. Estimated duration: days.

### E2 — Seeded Benchmark

Approximately 30 tasks with known drift. Measures detection, latency, and false positives. Estimated duration: one to two weeks after runner is available.

### E3 — Real Passive Observation

Instrumented sessions without statistical alerts. Produces distributions by repo, module, and task type. Minimum duration: several weeks and sufficient volume, not fixed calendar.

`oguard evidence report --output evidence/runs/passive-report.json` reconstructs only `round-end` events and summarizes surface, candidate evidence per KR, criteria, check coverage, and cache hits. It also incorporates `trajectory-features-v1`: a per-round vector, discrete AUS, and relative evidence with versioned formulas. `oguard evidence dataset calibration --output ...` and `oguard evidence dataset evaluation --output ...` generate disjoint sets with deterministic partitioning by objective/round/contract-hash group; the partition prevents the same round from appearing in both sets, but it does not yet constitute a sufficient out-of-sample evaluation.

`mode: passive-observation` and `intervention.enabled: false` are contractual: these artifacts do not alter verdicts, policies, thresholds, or execution. They are descriptive, not a causal validation or an active detector.

The `task-trajectories-v1` corpus is a pipeline integrity test: each task carries a contract, steps, distractors, and a known synthetic label. Its 15 in-contract cases and 15 scope deviations are not independent of the builder and must not be used to claim AUPRC, calibration, sensitivity, or effectiveness. Its function is to prevent future statistical evaluation from starting with events lacking provenance or labels.

Organic evidence is collected as comparable events, not as free-form narratives: `round_id`, plan and contract hash, task type, `work_context`, surface, `scope_validity`, signals, checks, criteria, subsequent action, and a minimal human label. Merge/rebase/migration contexts with `scope_validity: not-evaluable` are treated as censored observations: they serve to measure operational limits, but they fall outside precision or drift-detection-effectiveness metrics.

`oguard outcome --outcome confirmed-correct|confirmed-deviation|inconclusive --note <justification>` appends a human label to the last close and preserves contract hash, objective, round, and sequence. It is manual ground truth for later analysis, not automatic classification or external identity approval.

### E4 — Confirmatory Factorial Experiment

The [2026-09-07 passive data audit](09-passive-data-audit.md) establishes current sample and labeling limitations before proceeding to confirmatory work; its analysis is exploratory.

```text
weak structure × weak tests
strong structure × weak tests
weak structure × strong tests
strong structure × strong tests
```

Includes multiple model families, seeds, hidden tests, and variant builder control. Is a multi-month lab program, conditioned on E2 and E3 justifying the cost.

### E5 — Randomized Intervention

- control without alert;
- placebo with neutral pause;
- statistical alert;
- deterministic block when appropriate.

Placebo separates signal value from the simple benefit of interrupting.

## Metrics

### Primary Product Outcome

Drift or regressions detected before the agent closes the task.

### Quality

- correct resolution with hidden tests;
- regressions;
- `unknown` criteria at close;
- false closes;
- violation severity.

### Efficiency

- time to first useful signal;
- tokens between defect and detection;
- surface avoided after alert;
- checks avoided versus full CI;
- p50/p95 latency;
- cache hit rate.

### Statistical Detector

- AUPRC;
- Brier score;
- calibration curve;
- recall at fixed false alert rate;
- anticipation in actions, tokens, and seconds;
- net utility per alert and omission cost.

### Trajectory

- maximum surface;
- AUS;
- hypothesis changes;
- reverted patches;
- rewrite attempts;
- evidence per unit surface.

## Falsification criteria

The thesis must be reformulated or rejected if:

- raw size predicts as well or better than structural metrics out of sample;
- the non-monotonic curve disappears when controlling for difficulty;
- recovery comes only from tests, not structure;
- surface metrics do not generalize to new repositories;
- alerts only identify failures retrospectively;
- placebo pause achieves the same benefit as signal;
- intervention reduces net success;
- universal detectors produce enough noise to be ignored or disabled.

Before the confirmatory set, hypotheses, exclusions, features, statistical models, thresholds, primary metric, seeds, and failure handling are frozen. Later analyses are declared exploratory.

Detailed historical source of the protocol and its revisions is in [initial research](../initial-research/README.md).