/** * Audio playback pipeline with A/V sync: receives Opus frames from the * server, decodes via WebCodecs AudioDecoder, and plays through an * AudioContext with rate-adjusted resampling to stay in sync with video. * * Decode runs in a dedicated Worker that also owns the worklet's * MessagePort (transferred), so decoded PCM reaches the audio thread * without a main-thread hop — heavy video work (decode callbacks, * full-screen draws, multi-megabyte WebSocket frames) can no longer * starve the jitter buffer. The main thread only relays the ~50 tiny * encoded frames per second and keeps the AudioContext lifecycle, rate * servo, and health checks. Falls back to inline (main-thread) decode * when Workers or in-worker WebCodecs are unavailable. * * Audio and video frames share a common server-side wall-clock timestamp * (milliseconds since compositor creation). The worklet performs linear- * interpolation resampling at a variable rate (±MAX_RATE_OFFSET) so audio * can speed up or slow down to track video. Video is never delayed. * * Playback uses an AudioWorkletNode with an inline processor registered * from a Blob URL — no external file needed. */ /** * Maximum pre-worklet staging depth in decoded frames (~20 ms each). * * Audio decoded while AudioWorklet.addModule() is still loading must be * staged somewhere, but keeping half a second here defeats the worklet's * lower latency bound before its servo even starts. Keep only the newest * 400 ms (20 whole Opus frames), enough to fill the adaptive-buffer * ceiling below. */ export declare const MAX_STAGING_FRAMES = 20; /** * Adaptive jitter buffer: the worklet starts at MIN_BUFFER_SAMPLES, grows * on the leading edge of each underrun event, and shrinks back one frame * at a time after DECAY_STABLE_SAMPLES of underrun-free playback. * Hysteresis is provided by the MIN floor: once bufferTarget hits it, * shrinking stops. Floor is three frames (60 ms) to absorb two * back-to-back late arrivals before the buffer empties; stable * connections steady-state at 60 ms while jittery ones self-size to * whatever headroom they need, up to MAX_BUFFER_TARGET_SAMPLES. */ export declare const MIN_BUFFER_SAMPLES = 2880; /** * Hard ceiling on the adaptive jitter buffer. * * Growth is per-underrun and decay is per-several-seconds-of-calm, so * without a ceiling the target ratchets: an underrun buys 100 ms of * latency back in one event and gives it up over tens of seconds. A * client that underruns faster than it decays (Safari on iPadOS missing * render-quantum deadlines, or a server driving several concurrent video * streams) walks the target into the seconds, and the servo then *holds* * it there — `drift` is measured against the target, so a large target * is defended by slowing playback down to refill it. That is the * "audio falls further and further behind" failure. * * 400 ms leaves 340 ms of adaptive headroom above the 60 ms floor. That * covers browser scheduling stalls and transport batching that can exceed * 250 ms even when protocol RTT and server-side queues look healthy. The * servo must preserve that adaptive target; forcing every client back to the * floor causes repeated gaps. The ceiling still prevents a stressed client * from ratcheting into seconds of latency. */ export declare const MAX_BUFFER_TARGET_SAMPLES = 19200; /** * Ceiling on the *learned* floor, well under MAX_BUFFER_TARGET_SAMPLES. * * The target may still spike to the maximum to ride out something awful; the * floor is what a link is held at afterwards, and a bad minute should not pin * playback 400 ms behind live for the rest of the session. */ export declare const MAX_LEARNED_FLOOR_SAMPLES = 9600; /** * Ceiling on the floor taken from the *output device*. * * The learned floor above is what the link turned out to need. This is the * other half: what the sink needs, which no amount of network health can * reduce. A device with a deep buffer does not consume audio smoothly — it * wakes rarely and asks for everything at once, and a buffer holding less * than one of those bites empties on every single one. Bluetooth is the case * that matters: 150–300 ms is ordinary there against 5–20 ms wired. * * The floor is taken from `AudioContext.outputLatency`, which is an upper * bound on a bite rather than the bite itself — the safe direction to err, * and the reason wired playback is untouched: every wired sink reports well * under the 60 ms MIN, so its floor stays exactly where it was. Only a device * whose own latency already exceeds MIN moves at all, and for that device the * added latency is not really added: it was already paying it downstream. * * Capped so that a bogus or pathological reading cannot walk playback into * seconds of latency the way an uncapped adaptive target once did. */ export declare const MAX_DEVICE_FLOOR_SAMPLES = 19200; /** Minimum number of audio frames received before we start sync adjustment. */ export declare const SYNC_WARMUP_FRAMES = 10; export declare const UNDERRUN_REBUFFER_THRESHOLD: number; /** Samples per 20 ms Opus frame at 48 kHz (per-channel). */ export declare const SAMPLES_PER_20_MS = 960; /** * How many 20 ms frames to grow bufferTarget by on each underrun event. * Transport head-of-line blocking (audio serialized behind video bulk * writes on the same TCP stream) produces arrival gaps proportional to * the video bulk-write time — typically 100–200 ms on keyframes. Growing * by a single frame makes convergence take dozens of audible underruns; * 5 frames (100 ms per event) reaches a buffer depth that absorbs those * bursts within a handful of events. Decay (DECAY_STABLE_SAMPLES of * clean playback per frame shrunk) claws back any overshoot, and * MAX_BUFFER_TARGET_SAMPLES bounds how far it can run. */ export declare const GROW_FRAMES_PER_UNDERRUN = 5; /** * Excess buffered depth over the current adaptive target that triggers a * hard `skip` instead of waiting for the rate servo to drain it. * * The servo can only drain at MAX_RATE_OFFSET, so reclaiming a second of * accumulated latency by rate alone takes about a minute — and latency * arrives in bursts (a backlogged server queue flushing, a catch-up * replay on resubscribe, the tab being unthrottled after a stall) far * faster than that. Above this threshold we drop samples outright: * a single ~1.3 ms fade (see FADE_SAMPLES) beats staying seconds behind. * * Keep this above the 100 ms position-report cadence and ordinary transport * batching. A smaller threshold can turn a stale depth report into a skip * that consumes the current safety margin and induces an underrun. */ export declare const SKIP_EXCESS_MS = 200; /** * Minimum interval between `skip` messages. * * The worklet's buffered depth reaches us via ~100 ms position reports, * so the report following a skip can still describe the pre-skip depth. * Without a cooldown that stale reading triggers a second skip and we * discard twice what we meant to. One second is far longer than the * port round-trip and still bounds a genuinely runaway buffer quickly. */ export declare const SKIP_COOLDOWN_MS = 1000; /** * Inline AudioWorkletProcessor source. * * Runs on the audio render thread. Receives Float32Array PCM frames * (f32-planar: [L...L, R...R]) via the MessagePort and drains them into * the output buffers using linear-interpolation resampling at a variable * rate. Silence is output on underrun. * * Messages IN: * Float32Array — PCM frame to enqueue * "flush" — clear buffer * { type: "rate", value: number } — set playback rate (default 1.0) * * Messages OUT: * { type: "pos", value: number } — cumulative source samples consumed * (reported every ~100 ms) */ export declare const WORKLET_SRC: string; export declare class AudioPlayer { private ctx; private decoder; private worklet; private gain; /** * Dedicated decode worker (see WORKER_SRC). When non-null, the decoder * lives in the worker and the worklet's port is transferred there, so * the main thread only relays ~50 tiny Opus frames per second. Null * means inline mode: decode + worklet feed on the main thread (either * Worker is unavailable, or the worker declared itself broken). */ private worker; /** Set when the worker path failed — never try it again this player. */ private workerBroken; private _muted; private _subscribed; private _destroyed; /** Pending decoded PCM frames waiting to be posted to the worklet. */ private buffer; private listeners; /** * True while an `initAudioContext()` call is in flight. Guards against * concurrent re-init attempts (e.g. two rapid `handleAudioFrame` calls * both detecting a dead context). */ private initializingContext; /** Number of audio frames received (for warmup). */ private framesReceived; /** Current playback rate sent to the worklet. */ private currentRate; /** Smoothed rate — exponentially filtered to avoid wow/flutter. */ private smoothedRate; /** Worklet's current adaptive bufferTarget (samples), mirrored from reports. */ private currentBufferTarget; /** Floor imposed by the output device (samples), from its reported latency. */ private deviceFloorSamples; /** Last observed buffered depth (samples, from pos reports) — feeds the drift servo. */ private lastBufferedSamples; /** Timestamp (ms) of the last `skip` posted to the worklet; gates SKIP_COOLDOWN_MS. */ private lastSkipAt; /** * What the jitter buffer has been through this session. * * The worklet already decides everything about buffering and says so in its * events, but nothing kept the tally, so "the audio glitches occasionally" * had no number attached and no way to tell an over-tight buffer from a * genuinely bad link. `peakTargetSamples` is the interesting one: it is the * headroom this connection actually turned out to need, which the decay * gives back within half a minute of calm. * * Underruns are only half the story, and reporting them alone is actively * misleading: a buffer that runs *dry* underruns, but a buffer that runs * *deep* is cut back by `skip`, and a buffer whose pipeline is rebuilt loses * everything in it. Both of those are audible gaps with no underrun * attached, so a link that glitches constantly can read zero underruns * forever. `skippedSamples` is the honest measure there — how much audio was * thrown away, not how many times we decided to throw some. */ private stats; /** Timestamp (ms) of the last audio frame received via handleAudioFrame. */ private lastFrameAt; /** Timestamp (ms) of the last worklet position report. */ private lastWorkletReportAt; /** Periodic health-check timer for stall detection. */ private healthTimer; /** Timestamp (ms) of the last automatic pipeline reset. */ private lastAutoResetAt; /** Number of frames sent to decoder.decode(). */ private decodesRequested; /** Number of decoded frames received from the decoder output callback. */ private framesDecoded; /** Snapshot of decodesRequested at the last health check. */ private lastHealthDecodesRequested; /** Snapshot of framesDecoded at the last health check. */ private lastHealthFramesDecoded; /** * Whether a health check saw the decoder receive frames but produce no * output. A single silent check (2 s) triggers a reset. */ private decoderSilentLastCheck; /** Timestamp (ms) of the last decoded audio frame output. */ private lastDecodedAt; /** Timestamp (ms) when the AudioContext entered "suspended" state. */ private suspendedSince; /** Registered visibilitychange handler, for cleanup. */ private visibilityHandler; /** * Jitter-buffer health, in milliseconds and counts. * * `peakMs` against `targetMs` is the diagnosis: a peak well above the * current target means the link needed that headroom and the decay has * since given it back, which is why rare glitches keep recurring instead * of the buffer settling somewhere that survives them. */ get bufferStats(): { targetMs: number; peakMs: number; received: number; decoded: number; underruns: number; rebuffers: number; shrinks: number; skips: number; skippedMs: number; resets: number; outputLatencyMs: number; baseLatencyMs: number; sampleRate: number; deviceFloorMs: number; }; get muted(): boolean; get subscribed(): boolean; /** Whether the browser supports WebCodecs AudioDecoder for Opus. */ static get supported(): boolean; /** Whether this browser can route playback to a chosen output device. */ static get outputSelectionSupported(): boolean; /** * Route playback to a specific output device — `""` is the system default. * * Remembered rather than applied once: the context is torn down and rebuilt * whenever the browser closes it (device removal, resource pressure), and * the choice has to survive that. */ setOutputDevice(deviceId: string): void; /** The viewer's choice, whether or not it could be honoured. */ private _outputDeviceId; /** * The id this context's sink has been set to, or null after an attempt that * failed. * * Claimed before `setSinkId` is awaited rather than after, because the choice * is re-applied far more often than it changes — several times a second while * anything on the far side is moving — and a burst of those must collapse to * one call, not one per re-apply. Dropped again if the call rejects, which is * what makes the failure retryable: `setSinkId` can refuse a device that has * just gone away (a headset walking off) without disturbing playback, leaving * audio on the old sink, and until this was tracked apart from the choice the * guard above pinned it there for good. * * Starts and returns to `""` because that is where a freshly constructed * AudioContext plays: the system default. */ private _sinkDeviceId; private applyOutputDevice; onChange(fn: () => void): () => void; private emit; /** Toggle mute. When unmuting, creates the AudioContext (requires user gesture). */ setMuted(muted: boolean): void; /** * Resume a suspended AudioContext. Browsers block AudioContext.resume() * unless it happens inside a user-gesture event handler. When called * outside a gesture (e.g. on page load from persisted config) we * install a one-shot listener for the first click/keydown/touchstart so * that audio starts as soon as the user interacts with the page. */ private resumeOnGesture; /** Mark as subscribed (called by connection after sending C2S_AUDIO_SUBSCRIBE). */ setSubscribed(subscribed: boolean): void; /** * A port the transport worker can push encoded frames into. * * Returns null when there is no decode worker to reach — no `Worker`, or it * failed to start — in which case the caller must keep feeding frames the * ordinary way rather than sending them somewhere that cannot decode them. * * The decoder is what moves off the main thread here; the AudioContext * cannot, so `handleAudioFrame` still runs for every frame, just without a * payload to carry. */ /** * Called when the decode worker dies, for whoever is bypassing the main * thread to reach it. Set by the owner of the transport, since only that * side knows how to revoke the shortcut it asked for. */ onDecodeWorkerLost?: () => void; transportAudioPort(): MessagePort | null; /** Handle an incoming S2C_AUDIO_FRAME. */ handleAudioFrame(timestamp: number, _flags: number, data: Uint8Array): void; /** Called on connection reset / disconnect. */ reset(): void; /** * Full pipeline reset: tears down the AudioContext, decoder, and all * state. Everything rebuilds automatically on the next incoming audio * frame. Use this to recover from stalled or broken audio without * reconnecting. Unlike {@link reset}, this keeps the server subscription * intact — no re-subscribe round-trip is needed. */ resetPipeline(): void; /** Permanently destroy the player. */ destroy(): void; /** * Tell the worklet how much headroom this output device needs. * * `outputLatency` is not available until the context has actually rendered * — it reads 0 immediately after construction on every engine that reports * it at all — so this is called again from the health check rather than * only at setup, and re-posts whenever the reading moves by more than a * frame. That also covers the device being swapped underneath us. * * Safari reports neither `outputLatency` nor a useful `baseLatency`, so it * keeps the old fixed floor: no worse than before, but not fixed either. */ private applyDeviceFloor; private resetSync; /** * Close and null the decoder, resetting all stall-detection counters. * The AudioContext and worklet are left intact — only the decode chain * is rebuilt. This avoids the expensive teardown+async-reinit of the * full pipeline when only the decoder is broken. */ private resetDecoder; /** Reset decoder-related counters without touching the decoder itself. */ private resetDecoderState; /** * Called when the worklet reports its consumed-sample position. * Runs the buffer-depth servo: compares actual buffered depth against * the adaptive target and nudges the worklet's playback rate within * ±MAX_RATE_OFFSET to return there. Excess too large for the rate servo * to absorb is dropped outright. */ private onWorkletPosition; private startHealthCheck; private stopHealthCheck; /** * Periodic health check (every 2 s): detects stalled or silently broken * audio and recovers by rebuilding the pipeline. * * Checks for four failure modes: * 1. **Worklet stall** — frames arrive from the server but the worklet * hasn't reported a consumed-sample position in over 5 seconds. The * decode → worklet chain has silently broken. * 2. **Decoder stall** — frames are being sent to the decoder but no * decoded output arrives for two consecutive checks (4 s). The * WebCodecs AudioDecoder has silently stopped producing output * without transitioning to the "closed" state. (Most decoder * stalls are caught earlier by the inline check in handleAudioFrame.) * 3. **AudioContext death** — context is "closed" (resource pressure, * device removal, GPU process crash). The statechange listener * handles this immediately, but this is a safety net. * 4. **Persistent suspension** — context is "suspended" and resume() * fails for > 5 s. Tear down and rebuild from scratch. * * Also resumes a suspended AudioContext (can happen after device * changes or resource pressure without transitioning to "closed"). */ private checkHealth; /** * Tear down the AudioContext and worklet without touching the decoder or * sync state. Used when the context has died (state === "closed") and * needs to be rebuilt. */ private teardownAudioContext; private initAudioContext; /** * Send a control message to the worklet processor, routing through the * decode worker when it owns the worklet's (transferred) port. */ private postToWorklet; /** Handle a worklet-originated message (direct or relayed by the worker). */ private handleWorkletMessage; /** * Spawn the decode worker (idempotent). Called from both * handleAudioFrame() and initAudioContext() so that whichever runs * first decides the mode before the worklet is wired up — the worklet * port must be transferred at creation, not retrofitted. */ private initWorker; private initDecoder; private onDecodedFrame; } //# sourceMappingURL=AudioPlayer.d.ts.map