# GH-428 Test Coverage And Context Review

Date: 2026-05-22
Task: GH-428-TEST-COVERAGE-CONTEXT-REVIEW
Scope: CLI, web console/site, runtime workflow gates, and context-token performance.

## Summary

Architect and QA review found no release-blocking source test gap for the current implemented behavior. The current suite has broad unit/integration coverage, browser E2E coverage for the web console and site, runtime E2E coverage for init/target guidance and queue behavior, and context-pack coverage for bounded runtime delegation.

The main remaining gaps are backlog improvements: full multi-squad E2E, workflow lifecycle CLI E2E as a standalone installed-user battery, security-boundary E2E, and stronger context budget telemetry over real-world repositories.

## Evidence Reviewed

- `npm run precommit`: passed. Remaining warnings are historical `workflow-runs.jsonl` references to archived/unknown tasks.
- `npm run release:matrix`: passed and lists required release flows.
- `npm run performance:bench`: passed with 250 tasks. `graph-plan` 118 ms, `health` 120 ms, web `/api/graph/plan` 119 ms.
- `node --test test/context-index.test.js test/context-search.test.js test/context-pack.test.js test/runtime-scheduler.test.js test/runtime-adapters.test.js`: 80 passing.
- GH-419 CI passed after push. GH-419 and prior GH-424 Sonar workflows failed at the quality-gate step after importing Orchestra evidence.

## Coverage Matrix

| Area | Current evidence | Gaps | Priority | Recommendation |
| --- | --- | --- | --- | --- |
| Source quality gate | `npm run precommit` covers lint, typecheck, secret scan, security audit, build, unit tests, workflow validation | Historical workflow-run warnings add noise | Medium | Create cleanup task for stale workflow-run references, not a release blocker |
| CLI onboarding | `e2e/init-onboarding.test.js` covers `/tmp` workspaces, first-use flow, target guidance refresh, acceptance mapping | Installed-package journey is listed in matrix but should be mandatory in release checklist | Medium | Promote installed-package init to release-blocking before publish |
| Runtime target guidance | `e2e/runtime-instruction-flow.test.js` and runtime adapter tests cover Codex, Claude, Cursor, VS Code/GitHub Copilot, Windsurf guidance contracts | Real Claude CLI validation remains user-environment dependent | Medium | Keep GH-439 for real Claude validation; add optional local smoke recipe |
| Runtime delegation/queue | `e2e/runtime-manual-queue.test.js`, runtime scheduler tests, parent-action tests | Multi-squad E2E listed in docs but no standalone `e2e/runtime-multi-squad.test.js` file | High | Create E2E for async multi-squad, queue promotion, independent completion, and no parent blocking |
| Workflow lifecycle CLI | Unit/integration workflow tests cover gates, resume, failback, no-go, handoffs | Release matrix references `e2e/workflow-lifecycle-cli.test.js`, but the file does not exist | High | Add standalone E2E for installed-user lifecycle: run, gate, request changes, failback, resume, release readiness |
| Web console | `e2e/web-console.spec.js` covers task creation, dashboard, providers, recovery, evidence, lifecycle, accessibility, artifact canvas | Full visual regression and screenshot review are not release-blocking by default | Medium | Add snapshot/screenshots for high-risk UI states when frontend changes |
| Public site | `e2e/project-site.spec.js` covers docs/site navigation and public docs behavior | Docs search/content freshness should be tied to generated docs manifest in CI | Medium | Add docs content freshness check before site build |
| Acceptance evidence | `test/qa-coverage.test.js`, `test/cli-output-evidence.test.js`, automation evidence tests | E2E file listed as `e2e/acceptance-evidence.test.js` does not exist | Medium | Add cross-surface acceptance evidence E2E for CLI/API/browser/deferred validation |
| Security boundaries | Unit tests cover web action security, secret scan, path handling | Release matrix lists `e2e/security-boundaries.test.js`, but the file does not exist | High | Add E2E for path traversal, unsafe writes, secret redaction, no raw stack traces |
| Context-token performance | Context index/search/pack tests and `performance:bench` pass. Context packs are bounded and fail closed on stale indexes | No large-repo telemetry baseline checked into reports | Medium | Add benchmark fixture/report for large repo context pack size, redaction count, selected files, and token budget |
| Sonar quality loop | Local/import contracts exist and release matrix includes Sonar | GH-419 and GH-424 Sonar workflows failed at the quality-gate step after importing Orchestra evidence | High | Treat Sonar workflow failure as operational quality-gate follow-up before release candidate promotion |

## Architect Review

The test architecture is layered correctly:

- Unit and service tests cover domain contracts, workflow gates, runtime scheduler, context pack services, and evidence rules.
- E2E tests run against isolated `/tmp` workspaces and validate stdout/stderr, generated files, events, and browser-visible state.
- Runtime context work now has a bounded context index/search/pack path, reducing full-transcript and full-file context pressure.

Architecture gaps are mostly orchestration-level:

- The release matrix names some E2E batteries that are not implemented as standalone files yet.
- Runtime/provider validation is intentionally mixed: deterministic source tests prove contracts, while real provider/Claude validation remains environment dependent.
- Sonar is part of the quality loop, but failing CI Sonar should be made visible as a release readiness blocker or accepted operational risk.

## QA Review

QA coverage is sufficient for the current GH-419/GH-424/GH-451 implemented behavior based on precommit and focused suites.

QA gaps to convert into backlog:

- Multi-squad runtime E2E must prove multiple queued/active squads, parent remains conversational, completions reconcile independently, and queued work promotes after capacity frees.
- Workflow lifecycle CLI E2E must prove phase handoffs, blocked transitions, return to owner, resume after correction, and release artifact content in a disposable workspace.
- Security boundary E2E must prove path traversal, unsafe writes, stack trace leakage, and secret-like payload redaction across CLI/API surfaces.
- Context performance evidence should include large-repo budgets, selected-file rationale, redaction counts, stale-index behavior, and token budget deltas.

## Follow-Up Candidates

1. High: Add standalone multi-squad runtime E2E battery: GH-455.
2. High: Add standalone workflow lifecycle CLI E2E battery: GH-453.
3. High: Add security-boundary E2E battery: GH-454.
4. High: Make Sonar workflow failure visible in release readiness.
5. Medium: Promote installed-package CLI init to release-blocking publish evidence.
6. Medium: Add large-repo context-pack benchmark report: GH-452.
7. Medium: Add docs/site content freshness check against generated docs manifest.
8. Medium: Add cross-surface acceptance evidence E2E.

## Release Assessment

No source implementation was performed for GH-428. Current implemented behavior is not blocked by this review. The high-priority follow-ups should be scheduled as hardening work before the next broad release candidate, especially multi-squad E2E, workflow lifecycle CLI E2E, security-boundary E2E, and Sonar release visibility.