# End-to-End Test Batteries Open Orchestra needs E2E coverage that proves installed-user journeys, browser journeys, runtime delegation, and release gates work together. Unit tests can prove helpers; E2E batteries must prove observable behavior from the public entry points a user or CI runner actually executes. ## Policy - Every E2E scenario must run against an isolated temporary workspace unless it is explicitly validating this repository's site or web console. - Tests must validate command output, exit code, JSON contracts, filesystem artifacts, workflow events, and release-readiness effects when applicable. - Browser tests must validate visible UI state, persisted API effects, keyboard or responsive behavior when relevant, and evidence artifacts for user-facing flows. - External provider, GitHub, Sonar, or network-dependent paths must be opt-in and must report skipped or deferred evidence when offline. - Local provider smokes must default to mock or loopback/private endpoints, reject public internet egress unless a trusted-provider policy explicitly allows it, and record redacted provider provenance instead of credentials, raw prompts, hostnames, or ports. - Stubbed provider/MCP batteries are contract evidence only. They may block PRs when CLI/API/workflow contracts regress, but they must not satisfy real-provider acceptance criteria for hosted auth, latency, rate limits, provider-side failures, or production model behavior. - A release can ship only when P0 batteries pass or a release-manager accepted risk records the unavailable environment and compensating evidence. ## Product Journey Coverage Matrix | Journey | Primary surface | Environment model | Expected result | Evidence strategy | Current battery | Child story recommendation | | ------------------------------- | ------------------------------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | ------------------------------------- | | First workspace setup | CLI | Local `/tmp` workspace, source and packaged binary | User can initialize a project, preserve existing files, and see target runtime guidance | Assert exit codes, generated files, managed block boundaries, package version, and human/JSON output | Local and installed CLI onboarding | GH-533-A source/package setup parity | | Runtime-specific guidance regen | CLI + generated docs | Simulated Codex, Claude, Cursor, VS Code, Windsurf, generic workspaces | `init --force` regenerates missing managed guidance without mixing target-specific instructions | Filesystem diffs plus QA coverage JSON proving regenerated artifacts map back to acceptance criteria | Init refresh environments | GH-533-B runtime regen matrix | | Workflow delivery lifecycle | CLI workflow | Isolated local workspace | Task, estimate, phase handoffs, gate pause/resume, QA failback, evidence, and release readiness work end to end | JSON event stream, handoff artifacts, review/evidence records, and release-readiness before/after assertions | Workflow lifecycle CLI | GH-533-C workflow evidence lifecycle | | Runtime delegation | CLI runtime actions | Local runtime queue with detached sessions | Manual and multi-squad delegation queue safely, preserve parent availability, and reconcile lifecycle state | Spawn request artifacts, lifecycle commands, runtime session lists, queue state, and completion events | Runtime manual queue, multi-squad runtime | GH-533-D runtime delegation contracts | | Web console operator flow | Browser app + API | Local web server and persisted test workspace | Operator can inspect tasks, costs, providers, delegation, workflow progress, recovery, and artifacts | Playwright visible assertions, API persistence checks, responsive/keyboard coverage, screenshots/traces on failure | Browser console | GH-533-E web console operator journey | | Chat and provider flow | Browser app + API + provider | Stubbed provider by default, opt-in local provider | Chat sends scoped messages, streams usable status, records provider provenance, and handles failures safely | API response assertions, SSE/event trace, DOM state, provider request fixture, redacted logs | Browser console, provider opt-in | GH-533-F chat provider E2E | | Protected real-provider smoke | GitHub Actions + providers | Protected branch or trusted manual dispatch with hosted provider keys | OpenAI/GPT and Anthropic/Claude smoke calls run only in trusted contexts; missing secrets defer with sanitized evidence | Workflow guard assertions, masked CI logs, `reports/protected-real-provider-smoke-evidence.json` | Protected real-provider smoke | GH-550 GH-528B protected smoke | | Stubbed provider + MCP contract | CLI + web API + workflow | Built-in fake provider and scoped MCP config in an isolated workspace | PR-safe provider/MCP contract proves fake-provider routing, scoped MCP status, redacted evidence, and hosted provider fail-closed policy | CLI/API JSON comparisons, workflow executor provenance, redacted contract evidence report, hosted-provider negative assertion | Stubbed provider MCP contract | GH-551 GH-528A PR-safe contract E2E | | Adversarial provider + MCP | CLI + web API + workflow | Built-in fake provider, scoped MCP config, and hostile local fixtures | Hostile provider/MCP fixture text remains inert data, cannot alter runtime rules, and produces safe redacted CLI/API-visible errors | CLI/API JSON comparisons, workflow events, redacted adversarial evidence report, path traversal negative assertion | Adversarial provider MCP fixtures | GH-547 GH-528D adversarial fixtures | | Evidence redaction policy | CLI + API + MCP + browser log + trace | Stubbed leak fixtures in normal PR CI; protected/manual provider jobs only in trusted contexts | CI separates stubbed PR, protected real-provider, local provider, adversarial, and evidence-redaction suites; forks and Dependabot never receive hosted provider secrets | Unit sanitizer assertions for representative provider chat, MCP, browser log, CLI/API, and trace leaks; deferred skips include owner, reason, environment requirement, and next command | Provider evidence redaction | GH-549 GH-528E redacted evidence | | Public docs and site discovery | Browser site | Local static/site build | Users can find docs, navigate core concepts, inspect architecture, and read mobile-safe pages | Playwright navigation, search, docs catalog, responsive text-fit, and no raw GitHub dead-end assertions | Public site, docs/site content source | GH-533-G docs discovery E2E | | Security-sensitive operations | CLI + API + browser | Local sandbox with hostile fixtures | Unsafe file paths, shell patterns, secrets, stack traces, and telemetry leaks are blocked or redacted | Hostile fixture assertions, redacted command/API evidence, browser-visible safe errors | Security-sensitive operations | GH-533-H security boundary E2E | | Packaged renderer behavior | CLI package path | Built package artifacts | Diagram/report commands use packaged paths and produce user-safe output without stale source-only assumptions | Compiled command output, package file list, XML/SVG/HTML/source invariant checks, and explicit missing messaging | Renderer packaged paths | GH-540 packaged artifact E2E | | Benchmark/dashboard reporting | CLI metrics + dashboard JSON | Isolated workflow telemetry with completed and measurable stories | Velocity, calibration, and dashboard distinguish completed stories from measurable actuals without fake duration | CLI JSON assertions, dashboard metric assertions, actual source checks, and calibration sample-size checks | Benchmark dashboard duration smoke | GH-541 benchmark dashboard E2E | | MCP scoped OAuth diagnostics | CLI + web API | Runtime-scoped MCP config in an isolated workspace | CLI and web API report matching runtime OAuth state, redact secrets, and hide cross-workspace MCP config | CLI/API JSON comparisons, redaction assertions, scoped workspace config checks | MCP scoped OAuth smoke | GH-542 MCP OAuth E2E | | Release candidate readiness | CLI + CI artifacts | Local dry run, CI/self-hosted runner | Candidate package contents, version/tag policy, release matrix, and quality gates are release-ready | `npm pack --dry-run`, release check JSON, CI run links/logs, and accepted-risk records for unavailable environments | Package release dry run, source quality | GH-533-J release candidate E2E | ## Regeneration Plan Regenerate E2E scenarios when product behavior changes one of these contracts: 1. Public CLI command shape, command manifest, JSON output, generated files, or package entry points change. 2. Runtime profiles, phase playbooks, agent skills/rules, spawn request shape, or managed guidance files change. 3. Web console routes, visible task/provider/workflow states, API persistence, SSE events, or recovery flows change. 4. Provider integrations, MCP configuration, local/cloud execution policy, secrets handling, or redaction boundaries change. 5. Release, CI, package, tag, or self-hosted runner policy changes. For each regeneration event, QA must update the matrix row, add or revise the expected result, choose the real evidence surface, and record why lower-level tests are or are not enough. Stubbed tests can stay P0 only when the product contract is deterministic without external services; cloud, GitHub, Sonar, provider, mobile, or desktop-dependent evidence must be P1/P2 opt-in with a clear skip/deferred signal and owner. ## P0 Release-Blocking Batteries | Battery | Scope | Command | Minimum Assertions | Evidence | | --------------------------- | -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | Source quality | Static checks, build, unit tests, workflow validation, secret scan, security audit | `npm run ci:quality` | exit code 0, no leaks, no audit blockers, workflow valid | command log | | Local CLI onboarding | Current source CLI in `/tmp` workspaces | `ORCHESTRA_NODE_SCRIPT=$PWD/bin/orchestra.js npm run test:e2e:init` | `--version`, `init`, `status`, `validate`, first-use task, handoff, evidence, release readiness | stdout/stderr, JSON output, filesystem assertions | | Installed CLI onboarding | Installed or packaged CLI in `/tmp` workspaces | `npm run test:e2e:init` after installing the candidate package | same assertions as local CLI onboarding, proving the packaged binary matches source behavior | stdout/stderr, JSON output, filesystem assertions, package version | | Browser console | Web console task, cost, provider, delegation, recovery, evidence, workflow, accessibility, artifacts | `npm run test:e2e` | visible state, API persistence, evidence attachment, lifecycle transitions, responsive/keyboard behavior | Playwright report, screenshots/traces on failure | | Public site | Documentation/site navigation, docs catalog, architecture viewer, mobile fit | `npm run test:e2e` | navigation order, local docs catalog search, no raw GitHub redirect for docs, mobile content fit | Playwright report | | Runtime manual queue | Manual runtime delegation in a `/tmp` workspace | `npm run test:e2e:runtime` | two active sessions, third manual `spawn-request` materializes `queued`, artifact includes lifecycle commands, `runtime sessions` lists queued session | stdout/stderr, JSON output, artifact content | | Init refresh environments | Simulated generic, Codex, Claude, Cursor, VS Code/GitHub Copilot, and Windsurf workspaces | `npm run test:e2e:init` | missing runtime guidance files regenerate on `init --force`, user content is preserved, managed blocks/frontmatter are updated only inside managed ranges, target-specific metadata excludes wrong-target content, generated-artifact evidence maps back to acceptance criteria | filesystem diff assertions, QA coverage JSON | | Workflow lifecycle CLI | CLI workflow run, gate, resume, QA failback, release readiness | `node --test e2e/workflow-lifecycle-cli.test.js` | task phases create handoffs, blocked QA routes back, routine gate resumes immediately, release readiness maps acceptance to evidence | JSON output, events, handoffs | | Renderer packaged paths | Diagram lint command, artifact previews, and report renderer package contracts | `npm run build && node --test test/renderer-compiled-path.test.js e2e/packaged-renderer-smoke.test.js` | compiled CLI path emits Mermaid install guidance without stack traces, candidate package includes compiled renderer modules, previews preserve draw.io/Mermaid/ERD/TSX invariants, report render command absence is explicit | stdout/stderr, JSON output, package file list, XML/SVG/HTML/source invariant checks | | Stubbed provider MCP | Built-in fake provider, scoped MCP fixture, provider/MCP web API status, and contract evidence | `npm run test:e2e:init` or `npm run build && node --test e2e/stubbed-provider-mcp-contract.test.js` | fake provider routing is visible through CLI/API, scoped MCP CLI/API status matches, hosted provider execution fails before direct provider API use, evidence is redacted and labeled contract evidence, and no hosted provider secret is required | CLI/API JSON output, workflow events, `.agent-workflow/evidence/*-report.md`, `reports/stubbed-provider-mcp-contract-evidence.json` | | Adversarial provider MCP | Hostile provider prompt, MCP diagnostics, MCP args, path traversal, fake OAuth tokens, and hostile artifact fixtures | `npm run build && node --test e2e/adversarial-provider-mcp.test.js` | fixture instructions are treated as data, not promoted to runtime rules, CLI/API-visible diagnostics are redacted, unsafe artifact paths fail safely, and no hosted provider secret or network call is required | CLI/API JSON output, workflow events, `.agent-workflow/evidence/*-report.md`, `reports/adversarial-provider-mcp-evidence.json` | | Provider evidence redaction | Provider chat, MCP diagnostics, browser logs, CLI/API output, traces, and deferred provider skips | `npm run build && node --test test/provider-evidence-redaction.test.js test/automation-evidence.test.js` | representative leak fixtures are redacted before evidence is recorded, forks and Dependabot rely only on stubbed fixtures, protected real-provider jobs are trusted/manual only, and every deferred skip names owner, reason, environment requirement, and next command | sanitizer report, redacted output assertions, deferred metadata assertions | ## P1 High-Risk Regression Batteries | Battery | Scope | Command | Minimum Assertions | Evidence | | ------------------------------ | --------------------------------------------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | | Multi-squad runtime | Parallel squad delegation with queue and threshold policy | `npm run test:e2e:runtime` | independent sessions, non-blocking parent, queued sessions do not fall back to parent, completion order reconciles | JSON output, lifecycle events | | Acceptance evidence | CLI, API, browser, and deferred integration evidence | `node --test e2e/acceptance-evidence.test.js` | evidence maps to named acceptance criteria, deferred external validation requires owner and rationale | evidence artifacts | | Recovery and repair | Interrupted runs, stale locks, failed provider phases | `node --test e2e/recovery-cli.test.js` plus browser recovery coverage | recovery detects issue, repair requires confirmation, repaired state is observable | JSON output, before/after state | | Docs/site content source | Site content generated from docs and manifest | `npm run site:build && npm run test:e2e -- --grep docs` | docs render as human-friendly catalog, no markdown-only dead ends, search works | Playwright report | | Security-sensitive operations | File paths, shell execution, web writes, secrets, telemetry redaction | `npm run test:e2e:security` | path traversal blocked, unsafe writes rejected, secret-like data redacted, no raw stack traces | command/API evidence | | Ollama provider-backed runtime | Local OpenAI-compatible Ollama provider route in a `/tmp` workspace | `npm run test:e2e:runtime:ollama` | `model connect --provider ollama`, provider-backed developer phase, OpenAI-compatible request shape, provider provenance, no runtime subagent credentials in artifacts | stdout/stderr, JSON output, mock provider request, event log | | Benchmark dashboard duration | Benchmark-derived velocity, calibration, and delivery dashboard JSON | `npm run build && node --test e2e/benchmark-dashboard-duration.test.js` | completed stories are counted separately from stories with actual duration, completed-without-duration is visible, calibration sample size excludes null actuals | CLI JSON output, dashboard JSON, calibration JSON | | MCP scoped OAuth smoke | Runtime-scoped MCP integration status through CLI and web API | `npm run build && node --test e2e/mcp-scoped-oauth-smoke.test.js` | CLI and web API agree for `--runtime claude-cli`, diagnostics redact OAuth/API-key material, cross-workspace MCP config is not exposed | CLI/API JSON output, scoped config fixture, redaction checks | ## P2 Extended Confidence Batteries | Battery | Scope | Command | Minimum Assertions | Evidence | | -------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | Tracker and GitHub sync | Issue import/export and close readiness | opt-in CI job with network credentials | labels, comments, close gate, release readiness, no secret exposure | sanitized logs | | Sonar quality loop | Local or remote Sonar import and release gate mapping | configured Sonar workflow or local compose job | insights imported, release readiness reflects quality gate, unavailable token is explicit | artifact import report | | Protected real-provider | Hosted OpenAI/GPT and Anthropic/Claude provider smoke on trusted protected refs only | `npm run test:e2e:providers:protected` from `.github/workflows/protected-real-provider-smoke.yml` | workflow has no PR trigger, rejects forks/Dependabot/unprotected refs, masks configured secrets, records deferred evidence when secrets are missing, and stores only redacted provider smoke evidence | sanitized workflow summary and `reports/protected-real-provider-smoke-evidence.json` | | Provider-backed delegation | OpenAI, Anthropic, Gemini, Ollama, fake/local provider-backed routes plus runtime-native separation | opt-in provider E2E plus focused wrapper/unit coverage | registry routing, explicit direct API policy, forbidden fallback, local-provider public egress rejection, timeout handling, budget and scheduler blocks, redacted evidence, no silent runtime-native fallback | redacted provider provenance | | Package release dry run | npm package contents and release check | `npm pack --dry-run --json && orchestra release check --json` | generated/private state excluded, version/tag policy valid, release readiness complete | package list, release report | ## Required `/tmp` Fixture Patterns Each temporary-workspace E2E must create its own root with a deterministic prefix and must cleanly assert the following when relevant: - workspace classification before and after `init`; - generated `.agent-workflow` state and target runtime guidance files; - command JSON shape and human stdout where users rely on it; - event records for task, estimate, workflow, handoff, evidence, review, and runtime lifecycle; - artifact contents for handoffs, evidence, spawn requests, and generated prompts; - release-readiness output before and after required evidence exists. CLI E2E must run in two modes before release: source mode through `ORCHESTRA_NODE_SCRIPT=$PWD/bin/orchestra.js`, and installed-package mode through the candidate package binary. A pass in source mode and a fail in installed mode means the implementation is correct locally but the installed binary is stale or the packaging/install path is wrong. ## Implementation Order 1. Keep `e2e/runtime-manual-queue.test.js` release-blocking as runtime delegation evolves. 2. Keep `e2e/init-onboarding.test.js` covering Codex, Claude, Cursor, VS Code, GitHub Copilot, Windsurf, and generic project simulations. 3. Add `e2e/workflow-lifecycle-cli.test.js` for workflow run, gate, failback, resume, and release readiness. 4. Add `e2e/runtime-multi-squad.test.js` for async background squad behavior. 5. Add `e2e/stubbed-provider-mcp-contract.test.js` for PR-safe provider/MCP contract evidence that cannot be confused with real-provider evidence. 6. Add focused security and acceptance-evidence E2E only where unit tests cannot prove the user-visible contract. ## Executable Child Story Recommendations | Story | Scope | First executable check | Acceptance criteria seed | | -------- | ---------------------------- | --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | GH-533-A | Source/package setup parity | Extend `e2e/init-onboarding.test.js` to run source and packaged modes from the same fixture table | Source and installed binaries initialize equivalent state, report equivalent JSON contracts, and identify candidate package version | | GH-533-B | Runtime regen matrix | Add fixture rows for each supported runtime target and compare regenerated managed blocks | Regeneration preserves user content, writes only managed ranges, and never emits wrong-target instructions | | GH-533-C | Workflow evidence lifecycle | Expand `e2e/workflow-lifecycle-cli.test.js` around AC-to-evidence and QA failback assertions | Release readiness stays blocked until evidence maps to acceptance criteria and blocked reviews return to the responsible role | | GH-533-D | Runtime delegation contracts | Add queue and lifecycle assertions shared by manual and multi-squad runtime tests | Parent remains available, queued actions are explicit, child lifecycle is observable, and no fallback silently runs in parent | | GH-533-E | Web console operator journey | Create a Page Object backed journey for task, provider, workflow, artifacts, and recovery panels | Browser UI proves visible state, persisted API effects, keyboard/responsive behavior, and failure recovery | | GH-533-F | Chat provider E2E | Add stubbed API/SSE/DOM checks plus opt-in local provider smoke | Chat scope, provider provenance, streamed status, redacted errors, and retry/recovery behavior are asserted | | GH-533-G | Docs discovery E2E | Split docs/site grep coverage into catalog, search, architecture, and mobile scenarios | Users can find docs without raw repository dead ends, and mobile pages have no clipping or overlap | | GH-533-H | Security boundary E2E | Add hostile fixture coverage for paths, shell-like input, secrets, telemetry, and stack traces | Unsafe operations fail closed and all user-facing/API evidence is redacted | | GH-540 | Packaged artifact E2E | Promote compiled renderer/package assertions into a package-candidate smoke | Packaged commands do not depend on source-only paths and produce inspectable artifacts or explicit unavailable messages | | GH-541 | Benchmark dashboard E2E | Add velocity, calibration, and dashboard JSON assertions for completed stories without measured duration | Reports count completed work without inventing duration and calibration uses only measurable actuals | | GH-542 | MCP OAuth scoped E2E | Add CLI/API status parity assertions for runtime-scoped MCP OAuth diagnostics | Scoped OAuth status is consistent across CLI/API and diagnostics stay redacted without cross-workspace leakage | | GH-551 | Stubbed provider/MCP E2E | Add fake-provider workflow plus MCP dry-run CLI assertions in an isolated workspace | PR-safe contract evidence proves provider/MCP integration without secrets or network calls and never replaces real-provider smoke | | GH-533-J | Release candidate E2E | Add a release dry-run evidence bundle around package contents, tag readiness, CI logs, and release matrix | Release candidate proves package contents, version/tag policy, quality gate, rollback evidence, and accepted-risk records | ## Opt-In Provider Runtime Batteries Provider-backed runtime batteries are not part of default CI because they may need local services or paid credentials. They must still be deterministic enough to run on a developer machine. `npm run test:e2e:runtime:ollama` uses a local OpenAI-compatible mock endpoint by default to prove the Ollama adapter contract, workflow provenance, and no-secret behavior without requiring a real Ollama daemon. A separate real-model smoke can be run with `ORCHESTRA_OLLAMA_SMOKE=1` when validating a local model installation. Ollama internal-agent evaluation is broader than this adapter smoke. Future eval batteries must separate: - fake/mock contract evidence that may block PRs; - manifest-governed prompt/context evaluation evidence; - opt-in loopback/private real Ollama smoke evidence; - adversarial fixture evidence that treats model output as hostile data; - future training-readiness evidence that fails closed unless every fixture is training-allowed, reviewed, redacted, and checksum-pinned. Mock or fake provider evidence must be labeled contract evidence and must not be used as acceptance for real local model quality, training readiness, hosted provider behavior, or production agent safety. See [Internal Ollama Agent Evaluation And Tuning Design](internal-ollama-agent-design.md). ## Stubbed Provider And MCP Contract Evidence `e2e/stubbed-provider-mcp-contract.test.js` is the default PR/push-safe provider and MCP contract suite. It is included in `npm run test:e2e:init`, which the normal CI dogfood job runs without hosted provider secrets. The focused command is: ```bash npm run build && node --test e2e/stubbed-provider-mcp-contract.test.js ``` The suite uses the built-in `fake` provider and workspace-scoped MCP config in a temporary workspace. It asserts real CLI and web API outputs, records a redacted report through `orchestra evidence add`, and labels that report `contract evidence`. That evidence proves the local provider/MCP contract only. It must not be used as acceptance evidence for real OpenAI, Anthropic, Gemini, Ollama, or hosted MCP behavior. Real-provider acceptance remains covered only by protected/manual smoke suites with explicit secrets and trusted-run policy. ## Definition Of Done An E2E battery is complete only when it has: - isolated fixtures; - acceptance criteria listed in the test; - assertions for all criteria; - failure output that names the broken user contract; - evidence expectations documented in the test or matrix; - CI placement documented as P0, P1, or P2.