# Verification environment design guidance ## Traceability - Spec ID: verification-environment-design - Status: Implemented ## Intent Add a companion reference for `references/project-harness/agent-verify-loop.md` that teaches an AI coding agent how to discover, construct, and calibrate the smallest trustworthy test environment when the real environment is unavailable, unsafe, expensive, or too slow. The guidance must make substitution a claim-driven decision: keep the system under test and the behavior needed by the verification claim real, replace only explicit boundaries, and preserve honest evidence gaps rather than treating a convenient mock as proof of the real system. The reference will be grounded in read-only `qodercli -p` studies across different repositories under `/Users/phodal/eval/harness` plus primary documentation and research on hermetic environments, service virtualization, contract verification, ephemeral real dependencies, browser isolation, and coding-agent evaluation environments. ## Acceptance Scenarios - **VED-AC-1 (discovery before construction):** The reference tells an agent to inventory repository-owned setup commands, fixtures, fakes, emulators, containers, contract tests, CI services, and existing test seams before inventing a new environment. - **VED-AC-2 (explicit environment contract):** The reference defines a durable environment contract that records the verification claim, system under test, dependency boundary, real-versus-substituted decisions, pinned versions and inputs, start/readiness/reset/stop commands, isolation rules, supported platforms, collected evidence, and known fidelity gaps. - **VED-AC-3 (fidelity by claim):** The selection rules distinguish in-process doubles, protocol/service virtualizers, emulators, ephemeral real dependencies, and sandbox/staging checks. A dependency stays real whenever its semantics are part of the claim; an unavailable required boundary yields `blocked` or `unobserved`, not a mock-backed `pass`. - **VED-AC-4 (progressive bootstrap):** The construction loop starts with one tracer-bullet case and progresses from cheap deterministic checks to higher fidelity only where the claim requires it; every level has a runnable readiness probe, reset path, bounded execution, and machine-readable verdict. - **VED-AC-5 (independent oracle):** The environment is not accepted merely because an AI-authored test passes. It must have an oracle derived from an existing contract, reviewed fixture/baseline, provider verification, or externally observed invariant, plus a negative control that proves the harness fails when the relevant behavior is broken. - **VED-AC-6 (drift and calibration):** Recorded or generated substitutes carry provenance and freshness metadata, and are periodically calibrated against a real provider, contract-verification job, or authorized sandbox observation. Calibration gaps remain visible. - **VED-AC-7 (safety and portability):** The default environment uses no production credentials or unsanitized production data, constrains network and filesystem effects, cleans up deterministically, and avoids platform-specific paths or shell assumptions so Windows, macOS, and Linux remain supported. - **VED-AC-8 (cross-stack evidence):** The reference includes a compact set of evidence-qualified patterns derived from at least three materially different repositories, and labels repository observations separately from general design inference. - **VED-AC-9 (routing and link integrity):** `references/project-harness/agent-verify-loop.md` routes environment-bootstrap questions to the new reference, the project-harness index registers it, all relative Markdown links resolve, and the routing graph is current. ## Non-goals - Build a generic mock server, container runtime, fixture generator, or test orchestration CLI. - Promise that a mock, emulator, container, or local clone proves production behavior, performance, concurrency, security, or vendor-specific semantics. - Prescribe one vendor or framework across all stacks; concrete tools are evidence-backed examples of a substitution class. - Run or modify repositories under `/Users/phodal/eval/harness`; those targets are read-only research inputs for this change. - Change report schemas, scoring models, hooks, templates, or host adapters. ## Plan and Tasks 1. Inspect the existing Agent Verify Loop ownership and routing, and preserve unrelated local changes. 2. Run bounded, read-only `qodercli -p` studies over contrasting repositories under `/Users/phodal/eval/harness`, using one stable output schema. 3. Review primary sources for reproducible coding-agent environments, hermetic isolation, service virtualization, contract verification, ephemeral real dependencies, and UI isolation. 4. Add `references/project-harness/verification-environment.md` with the discovery procedure, environment contract, substitution decision rules, fidelity ladder, oracle/sensitivity checks, calibration policy, safety boundary, cross-stack patterns, and anti-patterns. 5. Route `agent-verify-loop.md` and `references/project-harness/README.md` to the new reference without expanding unrelated owner surfaces. 6. Regenerate the documentation routing graph, run focused link/model tests, run the full suite, and complete a Review Readiness Check over the local diff. ## Test and Review Evidence - Four bounded `qodercli 1.1.12 -p` studies completed against `braintree-web`, `delve`, `sample-spring-microservices-transactions`, and `nimara-ecommerce`. The prompts allowed only read/search tools and prohibited edits, installs, network access, service startup, and test execution. - Primary-source review covered SWE-bench/SWE-agent execution environments, Testcontainers, Pact provider verification, WireMock service virtualization, Playwright isolation, and Docker version/digest pinning. - `node --test test/doc-link-graph.test.mjs test/maturity-models.test.mjs` — 9/9 passed after regenerating `docs/better-harness-doc-links.mmd`. - `npm test` — 1149/1149 passed, with zero failures and zero skips. - `git diff --check` — passed before the final readiness review. - VED-AC-8: retain the four Qoder reports in command output and cite exact repository paths in the reference; treat missing or conflicting evidence as `unobserved`. - VED-AC-1..VED-AC-7: manual contract review against the reference headings, decision tables, example manifest, and anti-pattern list. - VED-AC-9: `node scripts/doc-link-graph/cli.mjs skills/better-harness`, then `node --test test/doc-link-graph.test.mjs`. - Consumer integrity: `node --test test/maturity-models.test.mjs`. - Repository regression gate: `npm test`. - Final hygiene: `git diff --check` and a Review Readiness Check that separates this change from the pre-existing Agent Verify Loop work in the dirty tree. ## Risks - **False confidence:** an AI may choose the easiest double and silently overstate fidelity. Mitigate with claim-first selection, explicit gaps, and `blocked/unobserved` verdicts. - **Self-validating tests:** the same agent can encode its implementation bug in the test oracle. Mitigate with independent sources and a negative control. - **Mock drift:** recorded responses and hand-built fakes age. Mitigate with provenance, expiry/calibration metadata, and provider verification. - **Overfitting:** four repositories cannot prove a universal taxonomy. Separate observed examples from the cross-stack inference and keep the decision model open-ended. - **Portability:** container-first advice can exclude hosts without a container runtime. Define containers as one fidelity option, not the universal base.