# Phase 4 — Passive data audit, 2026-09-07

## Decision

[Fact] This exploratory audit supports improving measurement and adjudication before adding a statistical alert. It does not establish predictive performance. The [scientific validation gate](05-scientific-validation.md) remains unmet.

The reproducible outputs are [frozen sanitized inputs](../../evidence/runs/phase4-input-v1.json) and [analysis with a labeling queue](../../evidence/runs/phase4-analysis-v1.json). The runner is `scripts/passive-data-analysis.mjs`; its regression cases are in `test/passive-data-analysis.test.mjs`.

## Sources and unit of analysis

[Fact] The audit reads three explicitly selected local logs from two repositories. The historical Objective Guard checkout was located through the registered inventory. Schema Forge is the other registered repository; the current Objective Guard checkout was added explicitly. This is a convenience sample, not a complete cross-machine inventory or a random sample of agent work.

| Source alias | Close events | Explicit outcome labels |
| --- | ---: | ---: |
| objective-guard-history | 157 | 1 |
| objective-guard-current | 4 | 0 |
| schema-forge | 50 | 1 |
| Total | 211 | 2 |

[Fact] The older versioned passive report contains 46 closes. It is a derived historical snapshot and is not concatenated with the source logs. Inventory hook events and synthetic task fixtures are also not counted as trajectory closes. Development logs can contain seeded exercises; organic origin remains unreviewed rather than assumed.

Capture retains a SHA-256 of each complete source log and each close event, timestamps, source-local close sequence, hashed identity and allowlisted measurements. It omits source code, filesystem paths, raw objective/round IDs and free-form notes. Source paths stay in a private manifest. Alias and hash metadata provide pseudonymous provenance, not a guarantee of anonymity.

The audit removes byte-equivalent parsed events within a repository, then groups by repository + objective + round. Contract amendments stay within the same group, preventing a changed hash from moving the same round across candidate partitions. Repeated closes remain available in the input; only the latest timestamp per group contributes to the primary descriptive table. Timestamp ties use source ID and sequence text as a deterministic tie-break. Grouping does not prove that different rounds are independent tasks; task families may span several rounds.

## Results

| Measurement | Result |
| --- | ---: |
| Exact duplicate events | 0 |
| Closes without contract identity | 8 |
| Identified round groups | 44 |
| Additional closes within groups | 159 |
| Groups with multiple contract hashes | 7 |
| Latest observations with normal context and valid scope | 21 |
| Latest observations excluded for unestablished scope | 17 |
| Latest observations excluded for non-normal/missing context after scope check | 6 |
| Explicit labeled observations | 2 correct, 0 deviation |
| Labels attached to eligible latest observations | 0 |
| Candidate group partition | 39 calibration / 5 evaluation |

[Fact] Across all closes, 93 lack recorded context and scope validity. Missing metadata is not interpreted as normal context or valid scope. Six latest observations with valid scope have migration context and are excluded conservatively. Eight unidentified legacy events remain in descriptive source counts but cannot form identified groups.

[Fact] Both existing labels match source-local sequence, contract hash, objective and round, and occur after the labeled close. They are preserved on those exact observations. They are not propagated to subsequent closes or elevated to independent adjudication. Guard `pass`, `warn` and `block` are never outcome labels.

| Feature, eligible latest observations | Observed | Median | p95 | Maximum |
| --- | ---: | ---: | ---: | ---: |
| Files | 21 | 7 | 619 | 2,022 |
| Churn | 21 | 162 | 77,139 | 391,757 |
| Modules | 21 | 4 | 81 | 145 |
| Module entropy | 21 | 0.964425 | 2.042541 | 2.153343 |
| Outside-scope ratio | 21 | 0 | 0.990307 | 0.999505 |
| Recorded affected public boundaries | 21 | 0 | 0 | 0 |
| Proven required criteria / required criteria | 19 | 1 | 1 | 1 |

[Fact] Percentiles use linear interpolation at `(n-1)*p`. Evidence is missing when no required criteria are recorded; absent numeric measurements remain null, not zero. Recorded zero affected public boundaries does not establish complete boundary discovery.

[Fact] In 20 eligible observations with all surface terms and nonzero surface, churn contributes a median 93.61% of `F+L+M+P+D+H` (minimum 59.92%, p95 99.51%). One zero-surface observation has no defined share. This is arithmetic composition, not predictive importance. The current heterogeneous sum is numerically dominated by churn in this sample.

## Implications for extending the framework

1. [Inference] Prioritize stable task identity, origin and observation-completeness fields. Seven amended groups and 93 metadata-incomplete closes make sample independence and exclusions material issues.
2. [Inference] Compare churn-only and files-only baselines against surface and relative evidence before adding complexity. Recorded public-boundary counts have no variation here, so these observations cannot establish their incremental value.
3. [Inference] Keep statistical intervention disabled. Two correct labels, no deviation labels and no labeled eligible terminal observation cannot support recall, AUPRC, calibrated probabilities or a causal benefit claim.
4. [Inference] Add prospective action-level observations before testing early warning. A feature measured at close cannot demonstrate that an alert would have anticipated the failure. Summed repeated-close surface must not be interpreted as time exposure or independent work.

These are proposed next extensions, not implemented detector changes. No thresholds, verdicts or production feature formulas change in this round.

## Human adjudication and subsequent comparison

The report contains 44 queue entries with source-local references, every observation ID in the group, existing labels and empty adjudication fields. Work from a private copy: regenerating the report resets the queue. Do not put private evidence or reviewer identities into the versioned snapshot.

For each group, a reviewer must inspect the original contract, task request, patches and independent acceptance evidence; record `confirmed-correct`, `confirmed-deviation` or `inconclusive`, an evidence reference, organic/seeded/unknown origin and the earliest evidenced deviation observation when applicable. A warning alone cannot justify a deviation label. Missing evidence warrants `inconclusive`. The queue is preparatory: this runner does not ingest edited adjudications or write `oguard outcome` events, and that command targets only the latest local close rather than arbitrary historical groups.

First review all 21 metadata-eligible groups, including successes and failures without selecting by guard verdict. Review the 23 excluded groups for recoverable metadata and task-family linkage; preserve the exclusion record. Establish outcomes independently and resolve disagreements before fitting. Reviewers should avoid seeing candidate feature scores and guard decisions until adjudication where feasible.

Before predictive evaluation, freeze target, exclusions, task-family grouping, temporal split, baseline models, transformations and metric. Use deviation as the positive class; compare files, churn, existing surface and relative evidence on the same eligible examples. Fit transformations and thresholds using calibration data only. The 39/5 hash partition is a reproducibility aid, not an approved holdout: this audit has inspected both partitions and has not established family or repository independence. Collect a new prospective holdout before confirmatory claims. Do not rebalance by duplicating closes or report intervals treating them as independent.

## Reproduce

```bash
node scripts/passive-data-analysis.mjs analyze evidence/runs/phase4-input-v1.json /tmp/phase4-analysis.json
cmp evidence/runs/phase4-analysis-v1.json /tmp/phase4-analysis.json
node --test test/passive-data-analysis.test.mjs
```

To capture a new cohort, use a private manifest with `sources: [{id, repository, path}]`, where `path` points to a JSONL log and aliases are lowercase slugs. Give checkouts of the same repository the same repository alias and distinct source IDs. Run `node scripts/passive-data-analysis.mjs capture <private-manifest.json> <new-snapshot.json>`, then analyze that snapshot. Preserve v1 and its source hashes; do not overwrite it with logs that include subsequent dogfooding of this analysis.