# Reconciled AC Evidence Backfill Date: 2026-05-17 Task: AC-EVIDENCE-BACKFILL-20260517 Scope: first evidence-generation batch for reconciled current acceptance criteria only. Superseded and duplicate historical criteria from `docs/reports/ac-evolution-reconciliation-20260517.md` are intentionally excluded. ## Evidence Standard Each evidence item must include: - acceptance criterion or domain covered - command or artifact used - observed result - happy path coverage - edge case coverage - known failures or unresolved gaps ## Batch 1 Result | Area | Command | Result | Evidence value | | --- | --- | --- | --- | | Docs adoption | `node --test test/docs-adoption.test.js` | Fail: 2 pass, 1 fail | Confirms current docs/site generated-content drift. | | Tracker sync | `node --test test/github.test.js` | Pass: 18 pass | Covers GitHub, MCP fallback, unsupported tracker, conflicts, close guard. | | Release readiness | `node --test test/release-check.test.js test/release-candidate.test.js` | Pass: 20 pass | Covers ready, blocked, accepted risk, rollback-required, dirty tree, version ahead/behind, private package state. | | QA coverage and CLI evidence | `node --test test/cli-output-evidence.test.js test/qa-coverage.test.js` | Pass: 7 pass | Covers stdout/stderr/exit code mapping, skipped criteria, missing assertions, unresolved QA gaps. | | Workflow governance | `node --test test/autonomous-workflow-cli.test.js test/autonomous-workflow-sizing.test.js test/autonomous-workflow-phase-status.test.js` | Pass: 54 pass | Covers gates, skips, invalid inputs, timeout, block/changes return, provider failure, budget abort, QA e2e generation. | | Runtime, skills, memory | `node --test test/skills-memory-domain.test.js test/delegation-decision.test.js test/runtime-adapters.test.js` | Pass: 47 pass | Covers role-only non-activation, runtime brief without API execution, provider setup docs, delegation guardrails, memory compaction. | | Diagram and MCP visual validation | `node --test test/visual-validation.test.js test/diagram-mcp.test.js` | Pass: 15 pass | Covers Mermaid lint success/failure, missing tool guidance, visual defect detection, TLS/no-secret OAuth behavior. | | Core CLI/API contracts | `node --test test/orchestra.test.js` | Pass: 153 pass | Covers init, target dir, task metadata, QA gates, risk gates, evidence, Playwright plan/evidence, policy, budget, validation, docs audit. | | Browser E2E | `npm run test:e2e` | Fail: 11 pass, 4 fail | Produces screenshots and traces for current web/site failures. | ## Batch 2 Result | Area | Command | Result | Evidence value | | --- | --- | --- | --- | | Generated guidance and refresh | `node --test test/instruction-updates.test.js test/runtime-bootstrap.test.js test/health-checks.test.js` | Pass: 29 pass | Covers unchanged, drifted, blocked, forced, missing, unsupported targets, managed block preservation, dry-run, force, and target filtering. | | Estimation and benchmark | `node --test test/benchmark-*.test.js test/performance-benchmark.test.js` | Pass: 40 pass | Covers automatic/manual estimates, missing history, calibration, burndown, dashboard output, persistent bias lessons, and performance budgets. | | Phase playbooks, project detection, release matrix | `node --test test/phase-playbooks.test.js test/project-detection.test.js test/release-test-matrix.test.js` | Pass: 21 pass | Covers QA/docs/release/UX playbooks, deployment risk challenge, redo/sign-off observation protocol, config overrides, scaffolding, project profile detection, and release matrix commands. | | Web APIs, QA readiness, automation evidence | `node --test test/web-*.test.js test/automation-evidence.test.js test/qa-readiness.test.js` | Pass: 22 pass | Covers observable assertion requirements, deferred external validation, QA plan ready gate, web write authorization, artifact canvas, runtime sessions, provider profiles, recovery repair, and workflow progress summaries. | ## Edge Cases Covered ### Tracker Sync - CLI transport available. - CLI missing with MCP-skill fallback. - Unsupported tracker. - Failed fallback. - Existing issue/task conflict. - Issue close blocked without readiness or accepted risk. - Issue close allowed when release readiness passes. ### Release Readiness - Ready release path. - Accepted-risk path. - Rollback-required path. - Dirty working tree. - Version ahead of latest tag. - Version behind latest tag. - Private package state in npm pack output. - Noisy command output. - Nested web server permission failure guidance. ### QA Evidence - CLI stdout, stderr, and exit code are captured. - Evidence must map to acceptance criteria. - Not-applicable criteria are skipped explicitly. - Missing observable assertions are surfaced. - Release readiness exposes unresolved automation gaps. ### Workflow Governance - `gates=none`, `gates=phase`, and `gates=all`. - Skipped phases. - Invalid skipped phases. - Gate approve idempotency. - Gate block/changes returns to responsible phase. - Timeout marks active phase failed. - Provider failure diagnostics. - Budget exceeded before provider execution. - Placeholder provider output is rejected. - QA phase generates browser/shell E2E artifacts or skipped evidence. ### Runtime, Skills, Memory - Runtime delegation falls back to brief-only when subagents are unsupported. - Runtime packet renders without provider API execution. - Role eligibility alone does not activate skills. - Runtime task signals activate QA evidence and collection standards. - Memory hooks compact long lessons and expose trimming rationale. - Runtime handoff rejects non-workflow-local artifacts. ### Diagram, Visual, MCP - Mermaid lint success and failure. - Missing diagram tool remediation. - Visual validator detects common diagram defects. - Visual validation blocks missing validation artifacts. - MCP OAuth requires TLS and opt-in secure storage. - OAuth refresh does not expose secrets. - Bearer injection is limited to HTTPS MCP calls. ## Current Failures Captured ### Docs Adoption Drift Command: ```sh node --test test/docs-adoption.test.js ``` Failure: - `test/docs-adoption.test.js:51` - Expected `/First visible value/i` in `site/src/App.jsx`. - Actual site copy is now generated from `site/src/generated-site-content.js` and `docs/site-manifest.json`. Classification: - test drift caused by dynamic site content, or missing assertion against generated content. ### Project Site Architecture Viewer Command: ```sh npm run test:e2e ``` Failures: - `e2e/project-site.spec.js:61` - `e2e/project-site.spec.js:187` Observed: - Timeout waiting for `[data-architecture-diagram] svg`. Artifacts: - `test-results/project-site-project-site--5d3bd-nd-interactive-on-localhost-chromium/test-failed-1.png` - `test-results/project-site-project-site--5d3bd-nd-interactive-on-localhost-chromium/trace.zip` - `test-results/project-site-project-site--69905-ontent-fits-mobile-viewport-chromium/test-failed-1.png` - `test-results/project-site-project-site--69905-ontent-fits-mobile-viewport-chromium/trace.zip` Classification: - current browser-visible failure. The architecture diagram container exists, but the SVG never becomes visible. ### Web Ops Task Detail Command: ```sh npm run test:e2e ``` Failure: - `e2e/web-console.spec.js:487` Observed: - Timeout waiting for `Release readiness passed`. Artifacts: - `test-results/web-console-WEB-OPS-accept-d24db-k-detail-and-recovery-state-chromium/test-failed-1.png` - `test-results/web-console-WEB-OPS-accept-d24db-k-detail-and-recovery-state-chromium/trace.zip` Classification: - UI copy/state drift or regression in task detail release readiness rendering. ### Web Lifecycle Gate Controls Command: ```sh npm run test:e2e ``` Failure: - `e2e/web-console.spec.js:636` Observed: - Timeout waiting for button `Block Gate`. Artifacts: - `test-results/web-console-WEB-LIFECYCLE--fdcf8-s-workflow-from-the-browser-chromium/test-failed-1.png` - `test-results/web-console-WEB-LIFECYCLE--fdcf8-s-workflow-from-the-browser-chromium/trace.zip` Classification: - current browser-visible failure in gate decision controls or expected button label/state. ## Evidence Backfill Status This batch generated usable evidence for these reconciled domains: - tracker sync - release readiness - QA coverage and CLI output evidence - workflow governance - runtime/skills/memory - diagram/visual/MCP controls - core CLI/API contracts - generated guidance refresh - estimation and benchmark calibration - phase playbooks and redo/sign-off protocol - project detection and release test matrix - web API security, recovery, provider profiles, and workflow progress This batch produced failing evidence for: - docs adoption generated-site drift - site architecture diagram render - web ops release readiness copy/state - web lifecycle gate block control ## Final Batch Disposition The reconciled evidence candidates are now split into three buckets: | Bucket | Count | Disposition | | --- | ---: | --- | | Demonstrated by passing command evidence | 48 | Covered by the batch 1 and batch 2 commands above. | | Demonstrated by failing evidence | 4 | Evidence exists, but proves current non-compliance. Bugfix stories should be created. | | Requires manual/product acceptance or doc review linkage | 4 | Evidence is mostly present as docs/contract tests, but should be explicitly accepted or linked before release. | The four demonstrated failures are: 1. `GH-296` / docs adoption: generated site content moved visible-value copy out of `site/src/App.jsx`, but `test/docs-adoption.test.js` still asserts literal content in that file. 2. `ARCH-SALESFORCE-INTEGRATION-DIAGRAM` / project site: architecture viewer does not render `[data-architecture-diagram] svg` in desktop or mobile Playwright runs. 3. `GH-298` / web ops: task detail no longer renders `Release readiness passed` where the E2E expects it. 4. `WEB-LIFECYCLE` / workflow UI: `Block Gate` control is unavailable after the request-changes path. The four manual/product acceptance candidates are: 1. `DEV-STANDARDS-GENERALIZE`: confirm stack-agnostic standards examples remain current and intentionally contain no cross-project references. 2. `SETUP-IMPORT-ROLE-ALIASES`: confirm PM/PO alias behavior is still a product-supported import compatibility contract. 3. `RUN-VS-WORKFLOW`: confirm public copy around `run` versus `workflow run` remains accepted after site/docs restructuring. 4. `ARCH-SALESFORCE-INTEGRATION-DIAGRAM`: after the render bug is fixed, BA/Architect should visually accept systems, connectors, boundaries, and directionality. The next remediation batch should create bugfix stories for the four current failures, then rerun `npm test` and `npm run test:e2e`.