# Story: Chaos testing stack strategy and deterministic fault harness

Backlog Item ID: CHAOS-TESTING-STACK-STRATEGY-20260517

## User Story

As an architect, QA, SRE, security reviewer, or developer, I want a progressive
chaos testing stack so that Open Orchestra can validate failure behavior locally
first and adopt heavier tooling only when the runtime or SaaS architecture needs
it.

## Problem

The `chaos-resilience-testing` skill defines how to reason about deterministic
failure scenarios, but the project also needs a clear tooling strategy. Without
that strategy, teams may either under-test provider/API failure modes or add
heavy infrastructure dependencies too early for the npm package.

## Stack Strategy

### Layer 1: MVP Local Deterministic Fault Harness

Use this layer first. It must work offline and should not add heavyweight
runtime dependencies.

Recommended tools:

- Node test runner for domain, service, CLI, and workflow tests.
- Fake model providers for timeout, unavailable provider, malformed response,
  fallback, and budget exhaustion.
- Fake storage/repository adapters for partial writes, corrupted records,
  stale reads, and audit write failures.
- Controlled timers, `AbortController`, and injected clocks for bounded timeout
  behavior.
- Local fixtures for malformed imports, stale registry data, missing approvals,
  and policy-denied activation.
- Orchestra evidence reports for scenario matrix, expected behavior, actual
  behavior, final state, events, and recovery path.

Use for:

- persona registry and activation;
- provider routing and model fallbacks;
- approval and gate behavior;
- budget envelope enforcement;
- policy fail-closed behavior;
- local/offline mode.

### Layer 2: Web/API Resilience

Use this when user-facing web or API flows need visible degraded states.

Recommended tools:

- Playwright route stubs for API timeout, 500, stale response, empty response,
  slow response, and malformed payload.
- Existing web console E2E patterns for screenshots, traces, and visible state.
- Optional mock handlers such as MSW only if repeated browser/API mocking
  becomes noisy with native Playwright stubs.

Use for:

- web console personas;
- dashboard/runtime status;
- approval and gate UX;
- evidence browsing;
- settings/provider/budget panels.

### Layer 3: Integration And SaaS Simulation

Use this when tests need real network behavior or service boundaries.

Recommended tools:

- Docker Compose profiles for optional integration test stacks.
- Toxiproxy for latency, disconnects, resets, bandwidth constraints, and
  intermittent network behavior.
- WireMock or equivalent HTTP stub server for external APIs when receiver-side
  behavior matters.
- Pact only when consumer/provider contract testing is part of the product
  contract.
- k6 for API load and resilience checks once SaaS endpoints exist.
- OpenTelemetry for correlation IDs, spans, metrics, and failure traceability.

Use for:

- SaaS supervisor or registry APIs;
- external prompt/persona registry;
- GitHub worker execution;
- tenant-scoped APIs;
- long-running runtime scheduler behavior.

### Layer 4: Production-Grade Infrastructure Chaos

Do not include this in the npm package MVP. Use only if Open Orchestra operates
managed services on orchestration platforms.

Recommended tools:

- Chaos Mesh or LitmusChaos for Kubernetes.
- Cloud provider failure testing where permitted.
- Synthetic monitoring and SLO/error-budget checks.

Use for:

- SaaS production control plane;
- multi-tenant worker infrastructure;
- managed model gateway or registry services.

## First Implementation Slices

1. Provider timeout and fallback harness.
2. Registry corruption and import failure harness.
3. Approval race and missing approver policy harness.
4. Policy engine fail-closed harness.
5. Audit/event write failure harness.
6. Budget exhaustion harness.
7. Offline mode with optional sources unavailable.
8. Web console stale/error/timeout state harness.

## Acceptance Criteria

- MVP local chaos validation uses deterministic in-process faults and offline
  tests first.
- Heavy tools such as Toxiproxy, Docker Compose, k6, OpenTelemetry, Chaos Mesh,
  and LitmusChaos are optional progressive layers, not package MVP
  dependencies.
- Every fault scenario maps to acceptance criteria, expected behavior, final
  state, evidence, and recovery path.
- Security, compliance, tenant isolation, regulated authority, payment, secrets,
  policy, and approval failures fail closed by default.
- Optional enrichment and advisory features may degrade only with clear user or
  operator rationale.

## Non-Goals

- Adding Toxiproxy, k6, OpenTelemetry, Chaos Mesh, or LitmusChaos dependencies
  in the first implementation.
- Randomized failure injection in local unit tests.
- Running production chaos experiments from the npm package.

## Suggested Size

S for strategy.

Future implementation slices should be estimated independently.