# Synthetic Agent merge-quality verification

The [corpus](../test/agent-quality-corpus.ts) and [tests](../test/agent-quality.test.ts) check structural merge safety with synthetic data and fixed resolver responses. They do not measure real-model semantic accuracy. Agent mode remains experimental and default-off; the [settings reference](settings.md#experimental-agent-mode) and [implementation verification](agent-mode-verification.md) retain their authority.

## Deterministic evidence

Each fixture defines base/local/remote bytes, exact disclosed units, fixed result rows, and an accepted byte image or required refusal. `null` means a missing file; `""` means an existing zero-byte file. Text fixtures also define exact code-owned spans before, between, and after model-owned regions. Expected output is not generated by the production merge implementation.

| Cases | Assertions |
| --- | --- |
| Separate and bridged overlapping hunks | Unit boundaries, independent edits from both sides, whitespace, full accepted bytes and file hash. |
| Same-point insertions and insertions at both deletion boundaries | Empty ancestors/local hunks remain valid evidence; boundary ownership is symmetric. A nonempty replacement can retain an insertion while deleting old lines. |
| Delete/modify hunks and existing empty files | Fixture refusal leaves the whole path unresolved. Empty text replacements are invalid, independently of file eligibility. |
| Repeated lines, LF, CRLF, Unicode and missing final newlines | Exact code-owned bytes and exact fixed replacement bytes survive assembly. The validator does not enforce a model-owned newline policy. |
| Opposite hidden timeout policies | Identical disclosed hunks omit the policy needed to choose a value. Both fixtures require refusal, without adding surrounding context. |
| Coupled provider/model and analytics/tracking settings | Select complete existing units from either side, never a mixed pair. Preserve local BOM, CRLF, member spelling, unknown numeric spelling and independent theme changes. |
| Missing-file delete/modify conflicts | No eligible units; the conflict remains unresolved. Unchanged-versus-deleted files still use deterministic deletion, not model output. |

The resolver tests use the existing mock transport and isolated runner. They assert one request, no tools, exactly the declared units in the user payload, and the expected candidate outcome. Unsupported missing-file conflicts are tested locally, not sent to the runner. Candidate tests reject missing, duplicate or unowned IDs, extra fields, invalid refusals, forbidden empty replacements, and settings synthesis. They repeat validation at the persisted-candidate application boundary. One refusal withholds every unit in that file, including otherwise accepted units.

These tests prove fixture routing, contract validation and byte assembly for the listed inputs. A fixed response does not prove that a model can infer that response. Required semantic refusal is an evaluation oracle, not a runtime guarantee: a dedicated test shows that a schema-valid timeout choice can pass validation despite violating the hidden-context refusal oracle. No additional context, tool, production path, provider API or evaluation command is introduced.

## Separately approved provider evaluation

No provider evaluation is authorized or executed by this work. A future evaluation must follow this protocol:

1. Obtain explicit approval for the provider/model, model version when available, request count, cost ceiling, repetitions, deadline and retention policy. Keep Agent and Continuous settings unchanged. Stop after one clear external-service or entitlement failure unless a retry is separately requested.
2. Freeze the repository revision, corpus revision, resolver contract, model settings and scoring rules before requests. Register allowed exact outputs or whole-settings source choices for each resolvable case. The deterministic fixture response is one reference, not evidence that it is the only valid semantic merge. Do not add allowed results after seeing responses; report unregistered alternatives separately for review.
3. Use only the committed synthetic fixtures. Exclude user files, session history, credentials, real remote storage and discovered resources. Send only the current bounded unit payload and existing resolver instructions. Do not send full files, hidden policies, expected answers, scoring notes or neighboring context. Keep unsupported cases as local eligibility checks; they must not cause a provider request.
4. Record each attempt without overwriting failures: fixture ID, approved provider/model, outcome, schema validity, refused IDs, exact accepted bytes, elapsed time and provider-reported usage. Preserve raw synthetic output for review, but exclude authentication and provider error payloads. Do not retry refusals automatically.
5. Score the outcomes with the measures below. Review every semantic mismatch against the frozen oracle. Report counts and denominators, including failures, rather than only a success percentage. Keep provider/model results separate and state the small synthetic sample's limits.

| Measure | Definition |
| --- | --- |
| Incorrect acceptance | A completed acceptance on a required-refusal case, a byte change outside owned regions, a broken atomic settings choice, or output outside the registered accepted set. Report count divided by completed acceptances, plus affected fixture IDs. |
| Incorrect refusal | Any refused required unit in a resolvable case. Report count divided by resolvable attempts. Report correct refusals divided by required-refusal attempts separately. |
| Contract/eligibility failures | Report malformed/incomplete responses, timeouts, transport failures and unauthorized dispatches separately. A partial refusal leaves the path unresolved; it is not a successful acceptance. Do not remove failed attempts from totals or treat them as correct refusals. |
| Latency | Measure wall-clock duration with a monotonic clock from resolver invocation to completion/failure, including validation. Report per-case values, median and tail values with sample size; include timeout duration and do not infer provider-only latency. |
| Usage | Record request count and provider-reported input, output, cache-read and cache-write tokens, plus cost when available. Mark unavailable values as unknown; do not use mock usage or assume cancellation prevented charges. |

For a multi-unit file, an acceptance is complete only when every required unit is resolved and the entire assembled file passes the registered oracle. A required refusal is correct only when the file remains unresolved. Report any real-model observations as results for that provider, contract and synthetic corpus only. They do not establish production reliability or justify wider disclosure or model authority.