# Resilience E2E Test Suite

This document defines the automated browser test suite that should validate the realtime resilience behavior of the Sofya Transcription SDK under poor and unstable network conditions.

The goal is to cover the same user-visible recovery flows that are currently validated manually, while also proving the internal guarantees that matter to the PO:

- audio spoken during network instability is not silently lost
- recoverable transport failures do not become terminal failures too early
- terminal failures are surfaced when the retry budget is actually exhausted
- buffered audio is eventually drained after recovery
- buffered audio is cleared after successful completion or after a new user-started session
- stop and resume flows still behave correctly during and after recovery
- the public events and `getResilienceStatus()` snapshot remain consistent with the real runtime state

## Recommended Stack

Use a layered strategy instead of trying to force every guarantee into one test type:

1. `Jest` unit and integration tests for deterministic state-machine coverage.
2. `Playwright` browser tests in Chromium for realistic browser behavior.
3. A controllable local websocket test server for deterministic transport faults.

The current repository already has strong low-level coverage in:

- [__tests__/WhisperTranscriptionAdapter.test.ts](/Users/gabriel/Projects/lib.sofya.transcription/__tests__/WhisperTranscriptionAdapter.test.ts)
- [__tests__/reconnectingWebSocket.integration.test.ts](/Users/gabriel/Projects/lib.sofya.transcription/__tests__/reconnectingWebSocket.integration.test.ts)

What is missing is a browser-level suite that exercises:

- real `AudioContext`
- real `IndexedDB`
- real browser websocket behavior
- real offline and online transitions
- real lifecycle timing between audio production, buffering, reconnect, drain, and finalize

## Why Playwright

Playwright is the best fit for this suite because it gives us:

- browser-level offline and online transitions
- Chromium DevTools Protocol access for latency and bandwidth emulation
- websocket inspection in browser tests
- traces and videos that are useful for PO sign-off
- reliable CI execution

## Test Harness Architecture

The suite should not depend on the production transcription backend. It should run against a deterministic local test harness.

### Browser fixture

Create a browser fixture page that:

- loads the built SDK bundle
- creates a `SofyaTranscriber` instance pointed at a local websocket server
- starts transcription from a deterministic audio source
- records all public events in order
- samples `getResilienceStatus()` on every status event
- records all final and partial transcripts
- exposes a small `window.testApi` object so the Playwright test can drive the session

The page should expose methods such as:

```ts
window.testApi = {
  start(),
  pause(),
  resume(),
  stop(),
  getEvents(),
  getStatuses(),
  getTranscript(),
  getPartials(),
  getFinals(),
  clearEventLog(),
};
```

### Audio fixture

Do not rely on microphone input.

Use a deterministic audio source so every run is reproducible:

- preferred: a short WAV fixture played through an `AudioContext` and routed into a `MediaStreamDestination`
- fallback: synthetic generated audio chunks if the server only needs binary audio flow and not speech recognition quality

If PO sign-off depends on transcript semantics, the local websocket server should not perform real ASR. It should emit deterministic transcript messages for known input phases so assertions are stable.

### Local websocket test server

Create a local websocket server that can behave like a scripted transcription backend.

It must support:

- accepting binary audio frames
- accepting `{"action":"finish"}`
- emitting partial and final transcript messages
- recording all frames received from the browser
- simulating connection close, timeout, silent stall, delayed drain, and recovery

The server should expose scenario controls such as:

- `closeSocket(code, reason)`
- `stallIncomingDrain()`
- `resumeDrain()`
- `delayAllFrames(ms)`
- `ignoreFinishUntilOnline()`
- `returnFinalOnFinish(text)`

### Assertions collected per test

Every E2E resilience test should assert all applicable dimensions below:

- event sequence
- final transcript completeness
- absence of transcript duplication
- resilience snapshot transitions
- buffered audio growth during outage
- buffered audio drain after recovery
- final cleanup state
- websocket server receipts

## Standard Acceptance Contract

Unless a case explicitly expects terminal failure, every recoverable test should prove all of the following:

1. `connected` is emitted at least once.
2. At least one `reconnecting` event is emitted during the fault.
3. `reconnected` is emitted after recovery.
4. `disconnected` is not emitted.
5. The transcript contains all expected spoken segments.
6. No spoken segment appears twice in the final transcript.
7. `totalBufferedAudioBytes` becomes greater than zero during the outage when audio continues.
8. `totalBufferedAudioBytes` eventually returns to zero after recovery and drain.
9. The final resilience state is `stopped` after explicit stop.
10. The backend receives a single logical completion for the session when stop is requested.

## Standard Observability Contract

For each test, record:

- chronological event log
- chronological resilience status log
- backend frame receipt log
- backend control actions log
- Playwright trace

This data should be attached to CI artifacts when the suite fails.

## Audit Baseline Workflow

The `R100` audit sweep writes `test-results/resilience-audit.json` and compares it against the committed baseline at [e2e/fixtures/resilience-audit.baseline.json](/Users/gabriel/Projects/lib.sofya.transcription/e2e/fixtures/resilience-audit.baseline.json).

By default, the comparison is informational only so exploratory runs stay usable. To make the baseline comparison blocking, run the audit with `RESILIENCE_AUDIT_ENFORCE_BASELINE=1`. In that mode, the test fails only when the current run regresses versus the committed baseline.

## Scenario Matrix

The matrix below is the full recommended suite. The first group should be mandatory for merge protection. The second group can start as nightly if runtime becomes too high.

## Execution Profiles

Use two resilience configurations in the suite:

1. `production-like` for merge-gating cases, with reconnect and health-check timings close to real product behavior.
2. `stress` for audit sweeps, with shorter timers that surface failure modes quickly.

The point of the split is to keep the merge gate representative while still preserving a fast audit matrix for critical-point discovery.

### Group A: Mandatory merge-gating cases

#### R001: Happy path baseline

Purpose:
Establish a clean reference run and prove the harness itself is healthy.

Fault injection:
No fault.

Steps:

1. Start transcription.
2. Play a deterministic audio fixture with 3 logical phrases.
3. Wait for all expected partial and final events.
4. Stop transcription.

Must assert:

- no `reconnecting`
- no `disconnected`
- final transcript contains phrase 1, phrase 2, and phrase 3
- buffered audio remains zero throughout
- backend receives live audio frames and a `finish` message

#### R002: Short connection flap while user keeps speaking

Purpose:
Automate the current manual case of "desliga e liga" while speech continues.

Fault injection:
Toggle browser offline for a short recoverable interval, then back online.

Steps:

1. Start transcription.
2. Begin audio fixture with phrase 1, phrase 2, and phrase 3 spoken continuously.
3. During phrase 2, switch browser offline.
4. Keep audio playing while offline.
5. After a short interval, switch browser online.
6. Wait for recovery.
7. Stop transcription.

Must assert:

- `reconnecting` is emitted
- `reconnected` is emitted
- `disconnected` is not emitted
- `connectionState` reaches `reconnecting` and later returns to `connected`
- buffered audio grows during the offline window
- buffered audio returns to zero after recovery
- final transcript contains all phrases, including the one spoken during the outage
- no phrase is duplicated in the final transcript

#### R003: Multiple short flaps in the same session

Purpose:
Prove the logic is stable across repeated recoveries, not just one.

Fault injection:
Three short offline and online cycles during continuous audio.

Steps:

1. Start transcription.
2. Begin a longer audio fixture with 5 logical phrases.
3. Toggle offline and online three times across phrases 2 to 4.
4. Stop transcription after phrase 5.

Must assert:

- at least three reconnect attempts occur
- at least one successful `reconnected` follows each recoverable outage
- no terminal `disconnected`
- final transcript contains phrases 1 through 5
- buffered audio repeatedly grows and drains
- no stale buffered data survives after stop

#### R004: Long outage while user keeps speaking, recover only after speech ends, then finalize

Purpose:
Automate the second current manual case exactly.

Fault injection:
Stay offline for most of the audio and only recover after the user has finished speaking.

Steps:

1. Start transcription.
2. Start audio fixture.
3. Switch browser offline near the beginning of phrase 2.
4. Keep the browser offline until all speech is finished.
5. Switch browser online.
6. Wait for backlog drain.
7. Trigger stop.

Must assert:

- recoverable reconnect behavior occurs during the outage
- audio spoken entirely while offline is still represented in the final transcript
- buffered audio remains above zero during the long outage
- persisted buffered audio is used, not only in-memory pending audio
- buffered audio drains after recovery and before final completion
- final transcript contains all expected phrases with no duplicate phrasing
- stop completes successfully after recovery

#### R005: Retry budget exhaustion becomes terminal disconnect

Purpose:
Prove that the SDK does not pretend to recover forever and properly emits terminal disconnection when the configured retry budget ends.

Fault injection:
Keep browser offline or keep the websocket unavailable until retries are exhausted.

Steps:

1. Start transcription with a reduced `maxReconnectAttempts` for fast execution.
2. Force a connection loss.
3. Keep the transport unavailable long enough to exhaust retries.
4. Continue sending audio in the browser during the retry window.

Must assert:

- at least one `reconnecting` event occurs
- `disconnected` is emitted exactly once when retries are exhausted
- no later `reconnected` occurs
- once `connectionState` is `disconnected`, new audio is no longer buffered
- the final resilience snapshot reports no remaining reconnect attempts

#### R006: Recovery after websocket upstream backpressure stall

Purpose:
Validate the specific resilience behavior that detects an `OPEN` websocket that stopped draining audio.

Fault injection:
Keep the websocket technically open but prevent the backend from draining frames so client `bufferedAmount` grows.

Steps:

1. Start transcription.
2. Put the server into a stalled-drain mode while keeping the socket open.
3. Continue audio production.
4. Let the client detect sustained backpressure.
5. Restore backend drain behavior.
6. Wait for reconnect and drain.
7. Stop transcription.

Must assert:

- reconnect is triggered without a normal close from the backend
- the session enters `reconnecting` because of stalled upstream, not because of offline mode
- offline buffering starts when live send becomes unsafe
- recovery drains the buffered backlog
- final transcript remains complete

#### R007: Stop during recovering state

Purpose:
Prove finalization remains safe if the user clicks finalize during a network problem.

Fault injection:
Trigger a recoverable outage and click stop before recovery finishes.

Steps:

1. Start transcription.
2. Begin audio fixture.
3. Force a recoverable outage.
4. While still in `reconnecting`, call `stop`.

Must assert:

- stop resolves and does not hang forever
- final state becomes `stopped`
- no reconnect loop continues after stop
- buffered data is cleared after stop completes
- no extra websocket reconnections happen after intentional stop

#### R008: Pause and resume after recovery

Purpose:
Validate the special `resumeTranscription()` logic that refreshes the websocket before trusting live audio again.

Fault injection:
Recover from a short outage, then exercise pause and resume.

Steps:

1. Start transcription.
2. Force a short recoverable outage and restore connectivity.
3. After recovery, call `pause`.
4. Call `resume`.
5. Continue audio production.
6. Stop transcription.

Must assert:

- pause sends `finish`
- resume forces websocket refresh
- audio produced immediately after resume is buffered until the refresh completes
- final transcript includes post-resume audio
- no duplicate transcript is produced around the resume boundary

### Group B: Strongly recommended nightly cases

#### R009: Long outage large enough to require persisted offline segments

Purpose:
Prove that long speech during outage crosses the in-memory threshold and is persisted and later drained.

Fault injection:
Stay offline long enough to exceed `OFFLINE_AUDIO_SEGMENT_BYTES`.

Must assert:

- `persistedBufferedAudioBytes` becomes greater than zero
- `persistedBufferedAudioSegments` becomes greater than zero
- persisted data later drains back to zero
- the final transcript still contains the long offline-spoken section

#### R010: New session clears previous buffered leftovers

Purpose:
Prove a fresh user-started session does not inherit stale buffered audio from a prior interrupted session.

Steps:

1. Start session A.
2. Force buffering during an outage.
3. Abort or end session A before successful recovery.
4. Start session B as a new user-started session.

Must assert:

- session B starts with zero buffered audio
- no transcript from session A appears in session B
- server receipts for session B contain only session B audio

#### R011: Stop after long outage without prior online recovery

Purpose:
Validate the fallback finalization path when stop is requested while the backend has not yet returned.

Fault injection:
Remain offline through stop.

Must assert:

- stop resolves by timeout rather than hanging indefinitely
- if a partial transcript existed, it is surfaced as final fallback output
- final state is `stopped`
- no reconnect continues after stop

#### R012: Flaky reconnect with short-lived successful opens

Purpose:
Prove the backoff and recovery logic behaves correctly when connections reopen briefly and fail again.

Fault injection:
Allow the socket to open briefly below the stable uptime window and fail repeatedly.

Must assert:

- reconnect attempts continue using the expected retry progression
- terminal disconnect does not happen too early
- the session eventually either recovers or cleanly exhausts retries, depending on the scenario script

#### R013: High latency and low throughput without full offline mode

Purpose:
Model the more realistic "poor connection" case where the network is degraded rather than fully down.

Fault injection:
Use Chromium network emulation with high RTT and very low throughput.

Must assert:

- the session either remains healthy or enters the intended resilience flow
- if buffering starts, it later drains
- no transcript corruption or duplication occurs

#### R014: Backend closes socket unexpectedly after receiving buffered backlog

Purpose:
Prove recovery can survive a second failure during backlog drain.

Fault injection:
Recover from outage, start draining backlog, then force another close before drain completes.

Must assert:

- buffered audio is not lost between consecutive failures
- a second reconnection can still succeed
- final transcript remains complete once recovery eventually succeeds

#### R015: Recovery status contract

Purpose:
Validate the public `resilience_status` data contract as an API, not just behavior.

Must assert:

- status snapshots include the expected fields throughout the run
- `connectionState`, `websocketState`, retry counters, and buffered byte counters change coherently
- `lastDisconnect` is populated for fault scenarios
- the final snapshot after stop is stable and consistent

## Assertions Per Scenario

To keep coverage complete and reviewable, every scenario should explicitly document five assertion sections in the test file:

1. Event sequence assertions
2. Transcript assertions
3. Buffer lifecycle assertions
4. Backend receipt assertions
5. Final cleanup assertions

Example structure:

```ts
test("R002 short connection flap while user keeps speaking", async ({ page }) => {
  // arrange
  // act
  // assert event sequence
  // assert transcript completeness
  // assert buffer growth and drain
  // assert backend receipts
  // assert final cleanup
});
```

## Transcript Assertion Strategy

Avoid asserting one opaque final paragraph string when possible. Instead, use semantic segments.

Define a phrase list per audio fixture:

```ts
const expectedPhrases = [
  "phrase one",
  "phrase two",
  "phrase three",
];
```

Then assert:

- every expected phrase appears
- each expected phrase appears once in final results
- phrases spoken during outage are still present
- phrases spoken before outage are not duplicated after drain

## Backend Receipt Assertion Strategy

The local backend should log:

- timestamps for connection open and close
- timestamps and sizes of binary frames
- whether frames arrived before outage, during outage, and after recovery
- receipt of `finish`
- final transcript messages sent to the browser

This proves whether recovery solved the intended transport problem instead of only making the UI look healthy.

## Suggested File Layout

```text
e2e/
  resilience/
    resilience.spec.ts
    fixtures/
      resiliencePage.ts
      scriptedTranscriptionServer.ts
      audioFixtures.ts
      assertions.ts
  pages/
    resilience-harness.html
playwright.config.ts
```

## Suggested Rollout Order

Implement in this order:

1. R001 baseline
2. R002 short flap
3. R004 long outage then finalize
4. R005 retry exhaustion
5. R006 stalled upstream backpressure
6. R008 pause and resume after recovery
7. Remaining nightly cases

This order gives fast confidence on the highest-value business flows first.

## Exit Criteria For PO Confidence

The resilience feature should not be considered fully covered until:

1. All Group A scenarios pass reliably in CI.
2. At least R009, R011, and R013 run nightly.
3. Failing tests provide enough artifacts to explain whether the issue was transcript loss, duplicate output, stalled drain, retry exhaustion, or cleanup failure.
4. The PO can see a direct mapping from each manual scenario to at least one automated browser case.

## Mapping Current Manual Tests To Automated Cases

Current manual case:
"Iniciar a transcrição e causar pequenas interrupções de conexão. Durante as interrupções, manter a fala acontecendo."

Automated coverage:

- R002 short connection flap while user keeps speaking
- R003 multiple short flaps in the same session
- R013 high latency and low throughput without full offline mode

Current manual case:
"Causar uma interrupção longa enquanto o usuário fala e retomar somente no final. Após retomar, clicar em finalizar."

Automated coverage:

- R004 long outage while user keeps speaking, recover only after speech ends, then finalize
- R009 long outage large enough to require persisted offline segments
- R011 stop after long outage without prior online recovery

## Non-Goals

This suite is not meant to replace the current Jest coverage. It is meant to prove the recovery behavior under realistic browser conditions.

Keep these responsibilities separate:

- Jest proves deterministic logic and edge conditions cheaply.
- Playwright proves the full browser recovery path and business confidence.