# Compare harnesses with real coding outcomes ## Traceability - Spec ID: harness-coding-compare - Status: Implemented ## Intent Add a reproducible Qoder-backed comparison workflow to `@qoder-ai/harness` so maintainers can compare two DSL compositions by letting each one modify the same isolated repository fixture and grading the resulting files. The first benchmark creates a repository-grounded `README.md`; success is based on the file diff and deterministic validation, not on whether an SDK returned text. Keep the `.harness` language responsible for agent-harness assembly. A separate versioned experiment manifest owns tasks, trials, runtime budgets, graders, and artifact locations so evaluation policy does not become authored composition syntax. ## Acceptance Scenarios - AC-1: A full-surface `.harness` fixture exercises every component kind, permission domain/access, binding strength, degradation policy, and configuration value type. Tests assert compiled IR and resolution behavior, while focused invalid fixtures assert structured diagnostics. - AC-2: A `harness-compare.v1` manifest parser rejects unknown fields, unsafe paths, unsupported hosts, invalid trial counts, missing compositions, and runtime settings that do not provide a bounded Qoder coding tool policy. - AC-3: A comparison run copies the frozen task repository into a fresh directory for every variant and trial, resolves each composition, invokes the Qoder SDK, captures before/after file state, runs deterministic graders, and emits parser-safe JSON artifacts without changing the source fixture. - AC-4: Qoder execution can receive explicit visible tools, auto-approved tools, disallowed tools, permission mode, model, turn limit, and a permission callback. Its result records the exact non-secret runtime receipt, SDK event trace, duration, turns, token usage, cost, session id, and terminal reason. - AC-5: The README benchmark requires only `README.md` to change, parses its structure, resolves local links, executes its JavaScript example against the fixture package, checks documented public API and behavior against executable package contracts, rejects stale or invented capabilities, and runs the package's existing tests. - AC-6: A deterministic fake-SDK end-to-end test proves the compare runner creates and grades real files. A native Qoder smoke, when local authentication is available, runs the same public compare command and retains its artifacts; unavailable authentication is reported separately from product correctness. - AC-7: Package documentation explains the assembly/evaluation boundary, safe tool policy, CLI command, artifact layout, limitations, and validation commands. The package build, tests, generated-source check, document-link graph, and pack surface pass before commit. ## Non-goals - Enabling Pi coding tools; the Pi v0.1 executor remains advisory and `noTools: "all"`. - Treating one README task as proof of general coding-agent improvement. - Adding a new Coding Agent host adapter or changing existing support claims. - Allowing arbitrary shell commands, network access, destructive operations, or writes outside an isolated trial directory. - Publishing a package, changing package versions, or editing release metadata. - Using an LLM grader as the primary correctness decision. ## Plan and Tasks 1. Add a full-surface language fixture and behavior-focused compiler/resolver coverage. 2. Extend the Qoder executor with explicit coding options, a bounded permission callback seam, trace capture, usage metrics, and runtime receipt. 3. Add the versioned compare manifest contract, isolated trial runner, filesystem snapshot/diff evidence, command grader, aggregation, and CLI. 4. Add the frozen README fixture and a deterministic Markdown/package grader. 5. Test manifest failure modes, source-fixture immutability, variant isolation, real file creation, grader failures, artifacts, and safe tool decisions. 6. Document and run the focused package gates, native Qoder smoke when available, full repository gates, link graph regeneration, and pack checks. The compare implementation remains under `packages/harness/src/compare/` and uses only public compiler, resolver, and executor surfaces. It uses Node and Git argument arrays for cross-platform execution and never invokes a shell string. ## Test and Review Evidence - AC-1: `examples/full-surface.harness` compiles into schema-valid IR and resolves the satisfied and degraded requirement paths. The package suite asserts all eight component kinds, four strengths, permission alternatives, degradation policies, and configuration value types. - AC-2/AC-3/AC-5: `test/compare.test.ts` exercises manifest policy failures, bounded tool decisions, real isolated file creation, standard Git patches, source-fixture immutability, deterministic README grading, redacted paths, and rejection of generated examples that request host capabilities. - AC-4: `test/exec.test.ts` passes explicit Qoder SDK options through the injectable SDK seam and asserts runtime receipts, duration, turns, dollar cost, credits, usage, session/termination data, and recursive credential redaction. - AC-6: A native SDK run used the frozen `performance` model for one H0 and one H1 trial. Both changed only `README.md`, both executable Quick Starts passed, both package test runs passed 3/3, and both deterministic scores were 100. H0 used 6 turns, 519,923 ms, and 82.219797 credits; H1 used 5 turns, 63,633 ms, and 10.768145 credits. The result correctly remained `need_more_work` because a one-trial tie does not establish improvement. Evidence hashes were manifest `a2fc04c095c71152441f2095d731d0c4662f005d1b2970871a09cc364c1e66b5`, fixture `d70ee5ff320be019be3590e78792d262ed70e76139cf93cf88c1b048472453ba`, and harness `d522506d80f2bad6c6dfd4ffdcd0780193825ce461bf64c707d336940b6432df`. The H1 native patch was then replayed through the final Node permission-model grader and again scored 100 with its example and package tests passing. - AC-7: `npm run check` passed 1,324 root tests, 52 package tests, generated source verification, package build, and pack verification (525 npm entries, 547 runtime-zip entries). The focused documentation link graph passed six assertions. The final tarball contained 65 files (51.9 kB, shasum `82401bc9ca61f5df0ba872b2b5d485f224d10302`), installed into a clean temporary consumer, exposed a working `harness-compare` npm bin, and compiled/resolved the shipped full-surface DSL plus loaded the shipped `harness-compare.v1` manifest. - Risk: Agent nondeterminism can create misleading one-shot wins. Preserve each trial independently and require multiple trials before an improvement claim. - Risk: Tool approval mistakes can escape the fixture. Resolve every path against the trial root, allow only a narrow command vocabulary, deny network tools, and record every permission decision. - Risk: Authentication, model availability, SDK workers, and machine resources can fail independently. Classify these as infrastructure failures and retain stderr/trace evidence rather than scoring them as task failures. - Risk: SDK messages may contain sensitive data. Persist structured protocol events only after recursively redacting token/key/authorization fields; never store injected credentials in manifests, receipts, traces, or fixtures. ## Post-Review Corrections A review of the merged change found eight defects. All are fixed, and each behavioral fix is covered by a test in the package suite. 1. Validation commands spawned `npm` without a shell, which cannot start the Windows `npm.cmd` shim and aborted the whole comparison. `npmInvocation()` now resolves an `npm-cli.js` entry point, and a test runs it on the host. 2. A timed-out command killed only the direct child. Commands now run in their own process group and are stopped together with their children. 3. The shared task prompt stated the required README sections, grounding rules, install command, and validation duty, so the baseline received the candidate composition's content and the native AC-6 run tied at score 100. The prompt now states only the goal and the runtime tool policy; the documentation standard moved into the candidate composition's components. The runtime profile experiment keeps its explicit prompt because both of its arms share one composition. 4. Tests derived filesystem paths from `URL.pathname`, which is not a valid path on Windows. They now use `fileURLToPath()`. 5. Trace redaction shared one cycle set across sibling values, so an object referenced twice was recorded as `[Circular]`. Only the ancestor chain is tracked now. 6. The grader imported the agent-modified package entry point into its own process. Exports are now read by a separate `--permission` Node process, and a test proves a tampered entry point is graded without executing inside the grader. 7. Fixture copy, Git setup, patch capture, and grader breakage propagated out of a trial and discarded the whole run. Such failures are now recorded as that trial's `infrastructure_error` with retained evidence. 8. `runtime.network` was validated but never used, and `metrics.json` duplicated the full check list. Network denial now drives the web-tool requirement, and `metrics.json` keeps the grade summary while `validation.json` keeps checks. The AC-6 native manifest, harness, and fixture hashes above belong to the pre-correction prompt and harness files. They remain the honest record of that run, but they no longer describe the current experiment: a fresh native run is required before any comparative claim, and the baseline is now expected to score lower than the candidate rather than tie. Re-verified after the corrections: `npm run harness:generated`, `npm run harness:build`, `npm run harness:test` (60 package tests), `npx vitest run test/skills-docs/doc-link-graph.test.mjs` (6 assertions), and root `npm test` (1,322 of 1,324; `test/plugins/host-support.test.mjs` and `test/reporting/report-source-review.test.mjs` hit the 120 s limit under full suite load and both pass in isolation). `npm run harness:generated` and `npm pack --dry-run` were not re-established after these corrections: a separate uncommitted change to the grammar, resolver, and `test/sugar.test.ts` is in flight in the same worktree, so the generated-source diff is dirty and one sugar test is red. Re-run both gates once that change is complete.