# odyssey-spatial-comms

> WebRTC spatial audio & video SDK — built on mediasoup, Web Audio API, and TensorFlow noise suppression.

[![npm version](https://img.shields.io/npm/v/odyssey-spatial-comms)](https://www.npmjs.com/package/odyssey-spatial-comms)
[![license](https://img.shields.io/npm/l/odyssey-spatial-comms)](LICENSE)

---

## Installation

**npm**
```bash
npm install odyssey-spatial-comms
```

**yarn**
```bash
yarn add odyssey-spatial-comms
```

**pnpm**
```bash
pnpm add odyssey-spatial-comms
```

---

## Quick Start

### React / Next.js
```tsx
import { SpatialCommsSDK } from 'odyssey-spatial-comms';

// Create once when the user enters a space — type is inferred automatically
const client = SpatialCommsSDK.create('https://your-mediasoup-server.com');

// Resume audio context on first user gesture (required by all browsers)
await client.resumeAudio();

// Listen for events
client.on('room-joined', (data) => console.log('joined', data));
client.on('participant-joined', (p) => console.log('user joined', p.userId));

// Join a room
await client.joinRoom({
  roomId: 'my-room',
  userId: 'user-123',
  deviceId: 'device-123',
  position: { x: 0, y: 0, z: 0 },
  direction: { x: 0, y: 0, z: 1 },
  userName: 'Alice',          // optional — shown to other participants
  userEmail: 'alice@co.com', // optional
});

// Publish microphone
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
await client.produceTrack(stream.getAudioTracks()[0]);

// Update 3D position every time the user moves (~10Hz)
client.updatePosition(
  { x: 1.0, y: 0.0, z: 5.0 },    // world position in meters
  { x: 0, y: 0, z: 1 },           // forward direction (unit vector)
  { rot: { x: 0, y: 45, z: 0 } }  // rotation: pitch, yaw, roll (degrees)
);

// Leave
client.leaveRoom();
```

### Vue 3 (Composition API)
```ts
import { SpatialCommsSDK } from 'odyssey-spatial-comms';
import type { OdysseySpatialCommsHandle } from 'odyssey-spatial-comms';
import { onMounted, onUnmounted, ref } from 'vue';

const client = ref<OdysseySpatialCommsHandle | null>(null);

onMounted(() => {
  client.value = SpatialCommsSDK.create('https://your-mediasoup-server.com');

  client.value.on('room-joined', () => console.log('connected!'));
  client.value.on('participant-joined', (p) => console.log('user joined', p.userId));

  client.value.joinRoom({
    roomId: 'my-room',
    userId: 'user-123',
    deviceId: 'device-123',
    position: { x: 0, y: 0, z: 0 },
    direction: { x: 0, y: 0, z: 1 },
  });
});

onUnmounted(() => client.value?.leaveRoom());
```

### Plain TypeScript / Vanilla JS
```ts
import { SpatialCommsSDK } from 'odyssey-spatial-comms';

const client = SpatialCommsSDK.create('https://your-mediasoup-server.com');

client.on('room-joined', () => console.log('connected!'));

await client.joinRoom({
  roomId: 'lobby',
  userId: 'user-1',
  deviceId: 'device-1',
  position: { x: 0, y: 0, z: 0 },
  direction: { x: 0, y: 0, z: 1 },
});
```

### Next.js — dynamic import (browser-only SDK)
```ts
// pages/space.tsx or app/space/page.tsx
import dynamic from 'next/dynamic';

// Must be dynamic — SDK uses Web Audio API (browser only)
const SpaceRoom = dynamic(() => import('../components/SpaceRoom'), { ssr: false });
```

```ts
// components/SpaceRoom.tsx
import { useEffect, useRef } from 'react';
import { SpatialCommsSDK } from 'odyssey-spatial-comms';
import type { OdysseySpatialCommsHandle } from 'odyssey-spatial-comms';

export default function SpaceRoom() {
  const clientRef = useRef<OdysseySpatialCommsHandle | null>(null);

  useEffect(() => {
    const client = SpatialCommsSDK.create('https://your-server.com');
    clientRef.current = client;

    client.on('room-joined', async () => {
      await client.resumeAudio();
      const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
      await client.produceTrack(stream.getAudioTracks()[0]);
    });

    client.joinRoom({
      roomId: 'space-1',
      userId: 'user-1',
      deviceId: 'dev-1',
      position: { x: 0, y: 0, z: 0 },
      direction: { x: 0, y: 0, z: 1 },
    });

    return () => { client.leaveRoom(); };
  }, []);

  return <div>Space loaded</div>;
}
```

### TypeScript — if you need the type explicitly
```ts
import { SpatialCommsSDK } from 'odyssey-spatial-comms';
import type { OdysseySpatialCommsHandle } from 'odyssey-spatial-comms';

// Option A: let TypeScript infer it (recommended — no import needed)
const client = SpatialCommsSDK.create('https://your-server.com');

// Option B: explicit type annotation (only needed for refs/stores)
const clientRef: OdysseySpatialCommsHandle | null = null;

// Option C: derive the type from the factory (no import needed)
type OdysseyClient = ReturnType<typeof SpatialCommsSDK.create>;
```

---

## Peer Dependencies

These are **not bundled** — install them in your own project:

```bash
npm install socket.io-client mediasoup-client @tensorflow/tfjs webrtc-adapter
```

---

## Key API

| Method | Description |
|---|---|
| `SpatialCommsSDK.create(serverUrl)` | Create a new SDK client |
| `client.joinRoom({ roomId, userId, deviceId, position, direction })` | Join a room |
| `client.leaveRoom()` | Leave the current room |
| `client.updatePosition(position, direction)` | Update 3D position in space |
| `client.produceTrack(track, appData?)` | Publish a local audio/video track |
| `client.setMasterMuted(muted)` | Mute/unmute all incoming audio |
| `client.initializeMLNoiseSuppression(modelPath)` | Enable TF.js noise suppression |
| `client.setListenerPosition(pos, orientation)` | Set spatial audio listener |
| `client.on(event, handler)` | Subscribe to SDK events |
| `client.off(event, handler)` | Unsubscribe from SDK events |

---

## Events

```ts
client.on('room-joined', (data: RoomJoinedData) => { ... });
client.on('participant-joined', (p: Participant) => { ... });
client.on('participant-left', (p: Participant) => { ... });
client.on('participant-updated', (p: Participant) => { ... });
client.on('track-added', ({ participantId, track, kind }) => { ... });
client.on('huddle-invite', ({ from }) => { ... });
client.on('space-live-started', (data) => { ... });
client.on('space-live-stopped', () => { ... });
```

---

## Complete Flow: Frontend → Server → SDK

```
[1] UNREAL ENGINE
    Sends: pos=(4130, 220, 700) cm  (X=forward, Y=right, Z=up)

         │
         ▼

[2] FRONTEND (Vue / React)
    Transform Unreal coords → Standard + cm → meters:

    position = {
      x: unrealPos.y / 100,  // UE Y (right)   → X (right)   = 2.2m
      y: unrealPos.z / 100,  // UE Z (up)       → Y (up)      = 7.0m
      z: unrealPos.x / 100   // UE X (forward)  → Z (forward) = 41.3m
    }

    Calls:
      sdk.updatePosition(position, direction, { rot })
      sdk.setListenerFromLSD(position, cameraPos, lookAtPos, rot)

         │
         ▼

[3] SDK → SERVER  (socket.emit "update-position")
    Sends: { participantId, position, direction, rot }

         │
         ▼

[4] SERVER (pass-through mode)
    1. Receive position from client
    2. Auto-detect units: if maxAxis > 50 → divide by 100 (cm→m safety net)
    3. No smoothing: pass real-time position straight through
    4. Broadcast normalized position (meters) to all other clients

         │
         ▼

[5] SDK — receiving remote participant positions
    socket.on("participant-position-updated") triggers:
    1. normalizePositionUnits()  — backup unit check (maxAxis > 50 → /100)
    2. snapPosition()            — ignore movements < 15cm (anti-jitter)
    3. computeHeadPosition()     — add +1.6m Y for head height
    4. calculateLogarithmicGain()— cubic falloff 100%→0% over 0.5m→15m
    5. calculatePanning()        — sin(atan2) projection onto listener right-ear axis
    6. Web Audio nodes updated   — GainNode + StereoPannerNode ramped smoothly
```

---

## What Happens on a Sudden 5m Position Jump

```
SCENARIO: Person teleports from 2m away → 7m away instantly

Step 1 — SERVER detects jump > 5m
         Lerps 30% toward new position each frame:

         Frame 1:  2.0m → 3.5m   (30% of 5m gap)
         Frame 2:  3.5m → 4.55m
         Frame 3:  4.55m → 5.29m
         Frame 4:  5.29m → 5.80m
         Frame N:  ... converges to 7m

Step 2 — SDK receives the smoothed intermediate positions
         Recalculates cubic gain for each step:
           3.5m  → ~51% gain
           4.55m → ~38% gain
           5.29m → ~29% gain

Step 3 — Web Audio smooths each gain change
         gainNode.gain.setTargetAtTime(newGain, now, 0.05)
         ≈ 150ms smooth ramp per step  →  zero clicks or pops
```

## Coordinate System (World Space)

All spatial calculations are performed relative to a **world origin (datum)** at `(0, 0, 0)`:

```
                    +Z (Forward/North)
                      ↑
                   10 |
                      |        B (15, 8) ← Speaker
                    8 |       /
                      |      / 5.83m distance
                    6 |     /
                      |    /
                    5 | A (10, 5) ← YOU (Listener, facing 0°)
                      | ↑
                    3 | | Your right ear →
                      | |
                    1 |
                      |
                    0 +--+--+--+--+--+--+--+--+--→ +X (Right/East)
                      0  2  4  6  8 10 12 14 16
                     
                    ↙ (into page)
                  +Y (Up/Height)
```

**Key Points:**
- **Datum (0,0,0)**: World origin - all positions measured from here
- **X-axis**: Right/Left (positive = right, negative = left)
- **Y-axis**: Up/Down (height above ground)
- **Z-axis**: Forward/Back (positive = forward/north, negative = back/south)
- **Distance**: 3D Euclidean distance = `√(Δx² + Δy² + Δz²)`
- **Panning**: Calculated from X-Z plane position relative to listener rotation

**Coordinate Transform (Unreal → Standard):**
```javascript
// Unreal: X=forward, Y=right, Z=up
// Standard SDK: X=right, Y=up, Z=forward
position = {
  x: unrealPos.y / 100,  // UE Y (right) → X (right)
  y: unrealPos.z / 100,  // UE Z (up) → Y (up)
  z: unrealPos.x / 100   // UE X (forward) → Z (forward)
}
```

---

## Feature Highlights

- 🔌 **One factory to rule it all** – `SpatialCommsSDK.create(serverUrl)` wires transports, producers, consumers, and room state.
- 🧭 **Accurate pose propagation** – `updatePosition()` streams listener pose to the SFU while `participant-position-updated` keeps the local store in sync.
- 🎧 **Studio-grade spatial audio** – each remote participant gets a dedicated Web Audio graph: `ML denoiser (ScriptProcessorNode) → limiter → high-pass → low-pass → stereo panner → adaptive gain → master compressor`. ML denoiser is a trained 3-layer GRU model (872K params, val_loss=0.1636) running fully client-side via TensorFlow.js.
- 🎚️ **Crystal-Clear Audio Processing** – A finely-tuned audio pipeline featuring a gentle compressor, multi-stage filtering, and a smart denoiser prevents audio dropouts and echo. The result is a more natural, continuous voice without distracting artifacts.
- 🧭 **Position-based spatial panning** – `updatePosition` forwards positions to Web Audio which calculates panning based on WHERE the speaker is relative to the listener (not which way they face). Uses listener's right-vector projection with 5m pan radius for natural left/right placement.
- 🤖 **ML Noise Suppression (Active)** – TensorFlow.js GRU model (`odyssey_adaptive_denoiser`) runs as a `ScriptProcessorNode` wired as the **first node** in every participant's audio chain. Loads non-blocking in the background; operates in pass-through mode until the model is ready, then switches to ML denoising automatically. No fallback — if it fails, the error is logged to console.
- 🔄 **ICE Connection Stability** – Automatic ICE restart on transport disconnect for robust connections. SDK requests ICE restart from server when transport enters `disconnected` state, enabling faster recovery from network issues without full reconnection.

---

## Audio Flow (Server ↔ Browser)

```
┌──────────────┐   update-position   ┌──────────────┐   pose + tracks   ┌──────────────────┐
│ Browser LSD  │ ──────────────────▶ │ MediaSoup SFU│ ────────────────▶ │ SDK Event Bus     │
│ (Unreal data)│                     │ + Socket.IO  │                   │ (EventManager)    │
└──────┬───────┘                     └──────┬───────┘                   └──────────┬────────┘
       │                                    │                                  track + pose
       │                                    │                                       ▼
       │                           ┌────────▼────────┐                      ┌──────────────────┐
       │ audio RTP                 │  consumer-created│                      │ SpatialAudioMgr   │
       └──────────────────────────▶│  setup per-user │◀──────────────────────│  (Web Audio API)  │
                                   └────────┬────────┘                      │  - Denoiser       │
                                            │                               │  - HP / LP        │
                                            │                               │  - StereoPanner   │
                                            ▼                               │  - Gain + Comp    │
                                       Web Audio Graph                      └──────────┬───────┘
                                            │                                          │
                                            ▼                                          ▼
                                    Listener ears (Left/Right)                  System Output
```

---

## Video Flow (Capture ↔ Rendering)

```
┌──────────────┐   produceTrack   ┌──────────────┐   RTP   ┌──────────────┐
│ getUserMedia │ ───────────────▶ │ MediaSoup SDK│ ──────▶ │ MediaSoup SFU│
└──────┬───────┘                  │ (Odyssey)    │         └──────┬───────┘
       │                          └──────┬───────┘                │
       │                   consumer-created │ track                │
       ▼                                  ▼                       │
┌──────────────┐                   ┌──────────────┐                │
│ Vue/React UI │ ◀─────────────── │ SDK Event Bus │ ◀──────────────┘
│ (muted video │                   │ exposes media │
│  elements)   │                   │ tracks        │
└──────────────┘                   └──────────────┘
```

**Video Track Flow:**
1. **Capture**: `getUserMedia()` captures video from camera or screen
2. **Produce**: `sdk.produceTrack(track, { isScreenshare: true })` sends to SFU
3. **Route**: MediaSoup SFU routes video RTP to other participants
4. **Consume**: SDK receives `consumer-created` event with video track
5. **Render**: UI attaches track to muted `<video>` element (audio handled separately)

---

## Web Audio Algorithms

**Coordinate normalization** – Unreal sends centimeters; SpatialAudioManager auto-detects large values and converts to meters once.

**360° angle-based stereo panning** – `setListenerFromLSD()` calculates the listener's right-ear vector from their yaw (rot.y). When `updateSpatialAudio()` runs, it uses `atan2` to calculate the angle from listener to speaker, then applies `sin(angle)` for natural panning. This gives **full left/right separation** at ±90° angles. Speaker's rotation is ignored – only their position relative to listener matters.

**Dynamic distance gain** – `updateSpatialAudio()` measures distance from listener → source and applies a **CUBIC EXPONENTIAL** falloff (0.5m-15m range). Voices gradually fade from 100% (0.5m) to **complete silence at 15m+** (hard cutoff). The cubic `(1-normalized)³` formula creates clearly noticeable volume changes as you move. Distance calculated from listener's HEAD position to participant's HEAD position (body + 1.6m height). Master compressor is **DISABLED** to ensure gain changes are audible.

**Noise handling** – a TensorFlow.js GRU model (`odyssey_adaptive_denoiser`, 872K params, val_loss=0.1636) runs in a ScriptProcessorNode as the FIRST node in every participant's chain, applying a learned spectral mask before the high/low-pass filters. Audio passes through unchanged until the model finishes loading, then ML denoising becomes active automatically with no user action required.

---

## Spatial Audio System (CLOCKWISE Rotation)

### Core Algorithm (Full 360° Support)

The panning calculation uses **position-based projection** onto the listener's right-ear axis:

```typescript
// Step 1: Calculate listener's right vector from yaw (CLOCKWISE rotation)
const yawRad = (rot.y * Math.PI) / 180;
listenerRight = {
  x: Math.cos(yawRad),
  z: -Math.sin(yawRad)  // NEGATIVE sine for CLOCKWISE rotation
};

// Step 2: Vector from listener to speaker
vecToSource = {
  x: speakerPos.x - listenerPos.x,
  z: speakerPos.z - listenerPos.z
};

// Step 3: Calculate forward vector (90° CW from right)
listenerForward = { x: -listenerRight.z, z: listenerRight.x };

// Step 4: Project onto both axes
dxLocal = vecToSource.x * listenerRight.x + vecToSource.z * listenerRight.z;  // Right/Left
dzLocal = vecToSource.x * listenerForward.x + vecToSource.z * listenerForward.z;  // Front/Back

// Step 5: Calculate angle using atan2 (gives -π to +π radians)
angleToSource = Math.atan2(dxLocal, dzLocal);

// Step 6: Convert to pan value using sine (-1 to +1)
// 90° (right) = +1.0, 270° (left) = -1.0, 0°/180° (front/back) = 0.0
rawPan = Math.sin(angleToSource);

// Step 7: Apply smoothing to prevent jitter
smoothedPan = smoothPanValue(participantId, rawPan);
```

### Key Principles

| Principle                  | Description                                                 |
|----------------------------|-------------------------------------------------------------|
| **Position-based**         | Panning based on WHERE speaker is, NOT where they're looking |
| **Listener yaw matters**   | Your `rot.y` determines which direction is "right"          |
| **Speaker rotation ignored** | Their facing direction does NOT affect panning            |
| **Full 360° support**       | cos/sin trigonometry handles any angle automatically        |

### Listener Right Vector by Yaw (CLOCKWISE Rotation)

| Yaw   | Facing    | listenerRight (x, z) | Right Ear Faces | Left Ear Faces |
|-------|-----------|----------------------|-----------------|----------------|
| 0°    | +Z (fwd)  | (1.0, 0.0)           | +X              | -X             |
| 90°   | +X (right)| (0.0, -1.0)          | -Z              | +Z             |
| 180°  | -Z (back) | (-1.0, 0.0)          | -X              | +X             |
| 270°  | -X (left) | (0.0, 1.0)           | +Z              | -Z             |

### Pan Value to Left/Right Gain

| panValue | Left Ear | Right Ear | Angle        | Description    |
|----------|----------|-----------|--------------|----------------|
| -1.0     | 100%     | 0%        | 270° (left)  | Full LEFT      |
| -0.71    | 85%      | 15%       | 315°/225°    | Diagonal LEFT  |
| 0.0      | 50%      | 50%       | 0°/180°      | CENTER         |
| +0.71    | 15%      | 85%       | 45°/135°     | Diagonal RIGHT |
| +1.0     | 0%       | 100%      | 90° (right)  | Full RIGHT     |

---

## Anti-Jitter Smoothing (3 Layers)

### Layer 1: Gain Change Threshold Filter (2.5%)
```typescript
const GAIN_CHANGE_THRESHOLD = 0.025; // 2.5%
if (Math.abs(newGain - currentGain) / 100 < GAIN_CHANGE_THRESHOLD) {
  return currentGain; // Ignore micro-jitter (movements ≤40cm)
}
```

### Layer 2: Adaptive EMA for Pan
```typescript
// Normal: 70% smoothing for stability
smoothedPan = previousPan * 0.7 + newPan * 0.3;

// Near center: 50% smoothing for moderate response
if (bothNearCenter) {
  smoothedPan = previousPan * 0.5 + newPan * 0.5;
}

// Full flip (likely jitter): 85% HEAVY smoothing
if (signFlipped && panChange > 1.0) {
  smoothedPan = previousPan * 0.85 + newPan * 0.15;
}
```

### Layer 3: Audio API Ramp Time
```typescript
stereoPanner.pan.setTargetAtTime(panValue, currentTime, 0.08);  // 80ms pan
gainNode.gain.setTargetAtTime(gainValue, currentTime, 0.05);    // 50ms gain
```

---

## Distance-Based Gain: CUBIC EXPONENTIAL Falloff (HARD CUTOFF at 15m)

| Distance       | Gain   | Description                      |
|:---------------|:------:|:---------------------------------|
| `0.0 - 0.5m`   | 100%   | Full volume (intimate)           |
| `1.0m`         | ~90%   | Very close - still loud          |
| `2.0m`         | ~72%   | Normal talking - NOTICEABLE      |
| `3.0m`         | ~57%   | Across table - CLEARLY QUIETER   |
| `5.0m`         | ~33%   | Across room - MUCH QUIETER       |
| `7.0m`         | ~17%   | Far end of room - very faint     |
| `10.0m`        | ~4%    | Barely audible                   |
| `≥15.0m`       | **0%** | **Silent (HARD CUTOFF)**         |

**CUBIC EXPONENTIAL falloff formula:**
```typescript
private calculateLogarithmicGain(distance: number): number {
  const minDistance = 0.5;   // Full volume at 0.5m or closer
  const maxDistance = 15.0;  // Silent at 15m or farther - HARD CUTOFF

  if (distance <= minDistance) return 100;  // Full volume
  if (distance >= maxDistance) return 0;    // Silent - HARD CUTOFF

  // CUBIC: (1 - normalized)³ for NOTICEABLE volume changes
  const range = maxDistance - minDistance;  // 14.5m
  const normalizedDistance = (distance - minDistance) / range;
  const remainingRatio = 1 - normalizedDistance;
  
  return 100 * remainingRatio * remainingRatio * remainingRatio;
}
```

**Why Cubic (not Linear or Quadratic)?**
- **Linear**: Too gradual - hard to notice volume changes
- **Quadratic**: Not steep enough for 15m range  
- **Cubic**: Perfect balance - clearly noticeable with proper 15m silence

**Smoothing:** Web Audio's `setTargetAtTime()` handles all smoothing:
```typescript
// Time constant 0.05 = ~150ms smooth transition (no clicks)
gainNode.gain.setTargetAtTime(gainValue, currentTime, 0.05);
```

**Note:** Master compressor is **DISABLED** to ensure gain changes are clearly audible.

---

## Audio Stability System

### Layer 1: Gain Change Threshold Filter (2.5%)
```typescript
const GAIN_CHANGE_THRESHOLD = 0.025; // 2.5%
if (Math.abs(newGain - currentGain) / 100 < GAIN_CHANGE_THRESHOLD) {
  return currentGain; // Ignore micro-jitter (movements ≤40cm)
}
```

### Layer 2: SDK Position Snapping  
```
positionSnapThreshold = 40cm
If movement < 40cm → use cached position (ignores pixel streaming jitter)
```

### Layer 3: Web Audio Smoothing
```typescript
// Gain changes are smoothed by Web Audio API directly
// 50ms time constant for smooth transitions
gainNode.gain.setTargetAtTime(gainValue, currentTime, 0.05);  // 150ms smooth
stereoPanner.pan.setTargetAtTime(panValue, currentTime, 0.08);  // 240ms smooth
```

**Why simplified?** Previous rate-limiting was causing gain to get stuck at low values. Web Audio's built-in smoothing is sufficient and more reliable.

---

## Enterprise-Grade Gain Smoothing

**Problem Solved:** With 40+ people in a room, rapid position updates (60+ Hz) caused **instantaneous gain changes** that created audible clicks, pops, and "pit pit" crackling noise. This was caused by `setValueAtTime()` creating waveform discontinuities.
```
Throttling = "Wait a bit before doing the same thing again" to prevent overload! 🎯
Person walking 10 meters:

Without Throttling:
|||||||||||||||||||||||||||||||||||||  600 updates
↑ Every single tiny movement = update
Result: Clicks, pops, CPU overload 😖

With Throttling (16ms):
|  |  |  |  |  |  |  |  |  |  60 updates
↑  ↑  ↑  ↑  ↑  ↑  ↑  ↑  ↑  ↑
16ms gaps between updates
Result: Smooth, efficient, perfect 😎
```
### The Solution: Intelligent Throttling + Adaptive Ramping

```typescript
// OLD (Causes Clicks):
nodes.gain.gain.setValueAtTime(gainValue, currentTime);  // ❌ Instant jump

// NEW (Butter Smooth):
nodes.gain.gain.cancelScheduledValues(currentTime);
nodes.gain.gain.setValueAtTime(lastGain, currentTime);
nodes.gain.gain.linearRampToValueAtTime(gainValue, currentTime + rampTime);  // ✅ Smooth transition
```

### Performance Characteristics

| Participant Count | Position Updates/sec | Throttled Updates/sec | CPU Impact | Audio Quality |
|:-----------------:|:--------------------:|:---------------------:|:----------:|:-------------:|
| **2-5**           | ~300                 | ~60                   | Low        | Perfect ✅    |
| **10**            | ~600                 | ~120                  | Low        | Perfect ✅    |
| **20**            | ~1,200               | ~240                  | Medium     | Perfect ✅    |
| **40**            | ~2,400               | ~480                  | Medium     | Perfect ✅    |
| **100**           | ~6,000               | ~600                  | Medium     | Perfect ✅    |

### Intelligent Throttling Logic

```typescript
// Throttle: Skip update if too recent AND gain change is small
const isSignificantChange = gainDelta > 0.1;  // >10% change
if (timeSinceLastUpdate < 16ms && !isSignificantChange) {
  return;  // Skip this update, wait for next frame
}
```

**Key Features:**
- ✅ **Time-based throttling**: Maximum 60Hz per participant (16ms interval)
- ✅ **Significance bypass**: Large changes (>10%) bypass throttle immediately
- ✅ **Per-participant tracking**: Each person has independent throttle state
- ✅ **Standing participants**: Minimal updates when not moving (saves CPU)

### Adaptive Ramp Time

The system automatically adjusts ramp time based on gain change magnitude:

| Gain Change | Ramp Time | User Experience |
|:-----------:|:---------:|:----------------|
| **< 5%**    | 15ms      | Instant feel, imperceptible smoothing |
| **5-20%**   | 15-35ms   | Smooth transition, natural |
| **20-30%**  | 35-45ms   | Very smooth, no artifacts |
| **> 30%**   | 50ms      | Ultra smooth, prevents any clicking |

**Formula:**
```typescript
rampTime = Math.min(
  0.015 + (gainDelta * 0.1),  // Base 15ms + scaled by change
  0.050                        // Max 50ms cap
);
```

### Real-World Scenarios

| Scenario | Behavior | Result |
|----------|----------|--------|
| **Person walking nearby** | Small gain changes → 15-25ms ramps | Feels instant, zero clicks |
| **Person runs past you** | Large gain changes → 40-50ms ramps | Smooth volume sweep |
| **40 people, 20 moving** | ~1200 updates → throttled to ~240 | Perfect audio, low CPU |
| **Person stands still** | Updates skipped entirely | Zero CPU usage |
| **Person teleports close** | >10% change bypasses throttle | Immediate volume update |

### Error Handling & Fallback

```typescript
try {
  // Smooth ramping
  nodes.gain.gain.linearRampToValueAtTime(gainValue, currentTime + rampTime);
} catch (err) {
  // Fallback: Direct value setting (rare edge case)
  console.warn(`Gain scheduling failed, using instant set:`, err);
  nodes.gain.gain.value = gainValue;
}
```

### Why This Works

**Root Cause:** Instantaneous gain changes create **waveform discontinuities**:
```
Old Method:                    New Method:
Volume                         Volume
  ↑                              ↑
  │     ╱╲    ╱╲                │     ╱╲    ╱╲
  │    ╱  ╲  ╱  ╲               │    ╱  ╲  ╱  ╲
  │   ╱    ╲╱    ╲              │   ╱    ╲╱    ╲
  │  ╱            ╲             │  ╱            ╲
  │ ╱              ╲            │ ╱              ╲
──┼────────────────────→ Time  ──┼────────────────────→ Time
  0  ← JUMP! Click here!        0  ← Smooth ramp here!
```

**Technical Details:**
- Rapid gain jumps = discontinuous waveform = audible click
- With 60Hz position updates × 40 people = 2400 potential clicks/sec
- Linear ramping = continuous waveform = zero artifacts
- Throttling reduces update frequency by ~60% (saves CPU + audio thread)

### Network Resilience

**Server-Side:**
```typescript
// Opus codec with Forward Error Correction
useinbandfec: 1  // Automatically recovers lost packets
ptime: 20        // 20ms frames for low latency
```

**Why Non-Spatial Audio Worked Fine:**
- Non-spatial audio: Single static gain value, rarely changes
- Spatial audio: Per-frame position updates = rapid gain changes
- **The issue wasn't network** - it was rapid gain value changes in Web Audio API

---

## 🎛️ Audio Processing Settings

> **Design Goal:** Crystal clear voice with no echo, pumping, or bathroom effect.

### 🔊 Master Compressor

| Setting       | Value     | Purpose                              |
|:--------------|:---------:|:-------------------------------------|
| Threshold     | `-18 dB`  | Only compress loud peaks             |
| Knee          | `40 dB`   | Soft knee for natural sound          |
| Ratio         | `3:1`     | Gentle compression, no pumping       |
| Attack        | `10 ms`   | Fast enough to catch peaks           |
| Release       | `150 ms`  | Fast release prevents echo tail      |
| Master Gain   | `1.0`     | Unity gain for clean signal          |

### 🎚️ Filter Chain

| Filter          | Frequency   | Q Value | Purpose                         |
|:----------------|:-----------:|:-------:|:--------------------------------|
| Highpass        | `100 Hz`    | `0.5`   | Remove room boom/rumble         |
| Lowpass         | `10 kHz`    | `0.5`   | Open sound, no ringing          |
| Voice Boost     | `180 Hz`    | `0.5`   | ❌ **Disabled** (prevents echo) |
| Dynamic Lowpass | `12 kHz`    | `0.5`   | Natural treble preservation     |

### 🛡️ Per-Participant Limiter

| Setting    | Value     | Purpose                               |
|:-----------|:---------:|:--------------------------------------|
| Threshold  | `-6 dB`   | Only activate near clipping           |
| Knee       | `3 dB`    | Hard knee = true limiter              |
| Ratio      | `20:1`    | High ratio catches peaks cleanly      |
| Attack     | `1 ms`    | Ultra-fast peak catching              |
| Release    | `50 ms`   | Fast release = no pumping             |

### 🎤 Denoiser (ML — GRU ScriptProcessorNode)

| Parameter         | Value                     | Purpose                                           |
|:------------------|:-------------------------:|:--------------------------------------------------|
| Model             | `odyssey_adaptive_denoiser` | 3-layer GRU, UINT8 quantized TF.js              |
| Params            | 872,448                   | Trained 100 epochs, val_loss=0.1636               |
| Buffer size       | 4096 samples (~85ms)      | ScriptProcessorNode synchronous processing        |
| Backend           | WebGL (GPU)               | Reported at load: `[MLNoiseSuppressor] TF.js backend ready: webgl` |
| Pass-through      | Yes (while loading)       | Audio unaffected until model is ready             |
| Normalization     | mean=0.3953, std=0.1442   | Stats loaded from `normalization_stats.json`      |

---

## 🔗 Audio Chain

```
┌──────────────────────────────────────────────────────────────────────────────────────────┐
│                              AUDIO PROCESSING CHAIN                                      │
├──────────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                          │
│  MediaStream  ML Denoiser    Per-Participant                        Spatial    Master    │
│    Source   → (GRU model) →    Limiter     → Filters →             Panner  → Compressor │
│      │            │               │             │                    │           │       │
│      ▼            ▼               ▼             ▼                    ▼           ▼       │
│  [WebRTC]  [ScriptProcessor  [Peak Catch]  [HP 100Hz]          [Stereo      [3:1 Ratio] │
│   Track     872K GRU model]   [-6dB]        LP 10kHz]           L/R Pan]     Output     │
│             (pass-through                                                                │
│              while loading)                                                              │
└──────────────────────────────────────────────────────────────────────────────────────────┘
```

**Detailed Chain:**
```
Source → MLScriptProcessor (GRU denoiser) → Limiter → HighPass(100Hz) → VoiceBand → LowPass(10kHz) → 
DynamicLP(12kHz) → MonoDownmix → StereoUpmix → StereoPanner → Gain → MasterCompressor → Output
```

---

## Spatial Audio Flowchart

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                           SPATIAL AUDIO PIPELINE                            │
└─────────────────────────────────────────────────────────────────────────────┘

    ┌──────────────────┐         ┌──────────────────┐
    │   LISTENER DATA  │         │   SPEAKER DATA   │
    │  pos, rot (yaw)  │         │   pos (x, y, z)  │
    └────────┬─────────┘         └────────┬─────────┘
             │                            │
             ▼                            │
    ┌──────────────────┐                  │
    │ Calculate Right  │                  │
    │     Vector       │                  │
    │ cos(yaw), -sin() │                  │
    └────────┬─────────┘                  │
             │                            │
             ▼                            ▼
    ┌─────────────────────────────────────────────┐
    │           VECTOR TO SPEAKER                 │
    │  vecToSource = speakerPos - listenerPos     │
    └──────────────────────┬──────────────────────┘
                           │
                           ▼
    ┌─────────────────────────────────────────────┐
    │            DOT PRODUCT (Projection)         │
    │  dxLocal = vecToSource · listenerRight      │
    │  (positive = RIGHT, negative = LEFT)        │
    └──────────────────────┬──────────────────────┘
                           │
              ┌────────────┴────────────┐
              ▼                         ▼
    ┌──────────────────┐      ┌──────────────────┐
    │   NORMALIZE PAN  │      │ CALCULATE DIST   │
    │ sin(atan2(dx,dz))│      │ dist = |vecTo|   │
    │  range: -1 to +1 │      │ 0.5m → 15m range │
    └────────┬─────────┘      └────────┬─────────┘
             │                         │
             ▼                         ▼
    ┌──────────────────┐      ┌──────────────────┐
    │ ANTI-JITTER      │      │ EXPONENTIAL GAIN │
    │ - Threshold 2.5% │      │ gain = (1-norm)² │
    │ - EMA 70%        │      │ × 100%           │
    │ - Ramp 80ms      │      │ 0% at 15m        │
    └────────┬─────────┘      └────────┬─────────┘
             │                         │
             ▼                         ▼
    ┌──────────────────┐      ┌──────────────────┐
    │   StereoPanner   │      │    GainNode      │
    │  L/R balance     │      │   volume         │
    └────────┬─────────┘      └────────┬─────────┘
             │                         │
             └──────────┬──────────────┘
                        ▼
              ┌──────────────────┐
              │  AUDIO OUTPUT    │
              │  (headphones)    │
              └──────────────────┘
```

---

## 360° Spatial Audio Diagram (Top View)

```
                        0° (Front)
                     L100%  R100%
                         ↑
                         │
          315°           │           45°
        L100% R58%       │       L58% R100%
              ↖         │         ↗
                ↖       │       ↗
                  ↖     │     ↗
                    ↖   │   ↗
  270° ←─────────────── 🎧 ───────────────→ 90°
 (Left)            Listener            (Right)
L100% R40%          yaw=0°           L40% R100%
                    ↙   │   ↘
                  ↙     │     ↘
                ↙       │       ↘
              ↙         │         ↘
        L100% R58%       │       L58% R100%
          225°           │           135°
                         │
                         ↓
                     L100%  R100%
                     180° (Behind)

    Legend:
    - 🎧 = Listener at origin, facing 0° (forward)
    - Angles = Speaker position around listener
    - L/R % = Left/Right ear volume for speaker at that position
```

---

## Configuration

| Parameter              | Value  | Description                 |
|------------------------|--------|-----------------------------|
| `positionPanRadius`    | 5.0m   | Distance for full L/R pan   |
| `nearDistance`         | 0.5m   | Full gain threshold         |
| `farDistance`          | 10.0m  | Silence threshold           |
| `panSmoothingFactor`   | 0.5    | Normal smoothing            |
| `panChangeThreshold`   | 0.02   | Jitter ignore threshold     |
| `panRampTime`          | 0.15s  | Audio transition time       |
| `headHeight`           | 1.6m   | Added to body Y             |

---

## Console Logs Reference

```javascript
// Mediasoup server URL being used
[Odyssey] Connecting to MediaSoup server: https://...

// ML model loading
[MLNoiseSuppressor] Initializing TF.js backend: webgl
[MLNoiseSuppressor] Model loaded — 872,448 params | backend: webgl
[Odyssey] ML Noise Suppression loaded and active

// ML model failure (audio still works — pass-through mode)
[Odyssey] ML Noise Suppression failed to load: <error>

// ML active per participant
[SpatialAudioChannel] ML noise suppression ACTIVE — model loaded from <url>

// Listener position update
📍 [SDK Listener] pos=(x, y, z) rot=(pitch, yaw, roll)

// Speaker position received
🎧 [SDK Rx] <id> bodyPos=(x, y, z) rot=(pitch, yaw, roll)

// Spatial audio calculation
🎧 SPATIAL AUDIO [<id>] dist=Xm dxLocal=Xm rawPan=X smoothPan=X pan(L=X%,R=X%) gain=X% listenerRight=(x,z) vecToSrc=(x,z)
```

---

## Server Contract (Socket.IO Events)

| Event                            | Direction        | Payload                                                                      |
|----------------------------------|------------------|-----------------------------------------------------------------------------|
| `join-room`                      | client → server  | `{roomId, userId, deviceId, position, direction}`                           |
| `room-joined`                    | server → client  | `RoomJoinedData` (router caps, participants snapshot)                       |
| `update-position`                | client → server  | `{participantId, conferenceId, position, direction, rot, cameraDistance}`   |
| `participant-position-updated`   | server → client  | `{participantId, position, direction, rot, mediaState, pan}`                |
| `consumer-created`               | server → client  | `{participantId, track(kind), position, direction, appData}`                |
| `participant-media-state-updated`| server → client  | `{participantId, mediaState}`                                               |
| `all-participants-update`        | server → client  | `{roomId, participants[]}`                                                  |
| `new-participant`                | server → client  | `{participantId, userId, position, direction}`                              |
| `participant-left`               | server → client  | `{participantId}`                                                           |

### Position Data Types (Critical for Spatial Audio)

The SDK sends **three separate data types** to the server for accurate spatial audio:

| Data Type    | Structure                        | Description                                                   |
|--------------|----------------------------------|---------------------------------------------------------------|
| `position`   | `{x, y, z}` in meters            | World coordinates - WHERE the player is located               |
| `direction`  | `{x, y, z}` normalized vector    | Forward direction - which way the player is LOOKING (unit vector) |
| `rot`        | `{x, y, z}` in degrees           | Euler rotation angles - pitch(x), **yaw(y)**, roll(z)         |

**IMPORTANT**: `rot.y` (yaw) is **critical** for spatial audio left/right ear calculation:
- The listener's yaw determines their ear orientation
- `listenerRight = { x: cos(yaw), z: -sin(yaw) }`
- Speakers are panned based on their position projected onto listener's right axis

```typescript
// Frontend sends all 3 data types:
sdk.updatePosition(position, direction, {
  rot,           // Rotation angles (pitch, yaw, roll) in degrees
  cameraDistance,
  screenPos,
});

// Server broadcasts to other clients:
socket.emit("participant-position-updated", {
  position,   // World coordinates
  direction,  // Forward vector
  rot,        // Rotation angles - yaw used for L/R audio
  ...
});
```

---

## Noise-Cancellation Stack (What's Included)

| Layer                         | Purpose                                                                                                                         |
|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| **Adaptive denoiser worklet** | Learns each participant's noise floor in real time, applies multi-band downward expander plus dynamic low/high-pass shaping    |
| **speechBoost**               | Lifts the low/mid band only when speech confidence is high, keeping consonants bright without reintroducing floor noise        |
| **highBandGate**              | Clamps constant fan hiss in the 4–12 kHz band whenever speechPresence is low                                                   |
| **Silence gate**              | If energy stays below `silenceFloor` for configurable hold window, track ramps to true silence, wakes instantly on voice return|
| **Classic filters**           | Fixed high-pass (80Hz) / low-pass (8kHz) shave off rumble and hiss before signals reach the panner                             |

**Configuration example:**
```typescript
const sdk = SpatialCommsSDK.create(serverUrl, {
  denoiser: {
    threshold: 0.008,
    maxReduction: 0.88,
    hissCut: 0.52,
    holdMs: 260,
    voiceBoost: 0.65,
    voiceSensitivity: 0.33,
    voiceEnhancement: true,
    silenceFloor: 0.00075,
    silenceHoldMs: 520,
    silenceReleaseMs: 160,
    speechBoost: 0.35,
    highBandGate: 0.7,
    highBandAttack: 0.25,
    highBandRelease: 0.12,
  },
});
```

---

## How Spatial Audio Is Built

| Step                          | Description                                                                                                                                                           |
|-------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **1. Telemetry ingestion**    | Each LSD packet is passed through `setListenerFromLSD(listenerPos, cameraPos, lookAtPos, rot)` so the Web Audio listener matches the player's real head/camera pose  |
| **2. Per-participant graph**  | When `consumer-created` yields a remote audio track, `setupSpatialAudioForParticipant()` spins up: `Source → Compressor → Denoiser → HP → LP → StereoPanner → Gain`   |
| **3. Position updates**       | Every `participant-position-updated` event calls `updateSpatialAudio(participantId, position, rot)`. Position feeds panning, rot provides listener's yaw             |
| **4. Distance-aware gain**    | The manager computes Euclidean distance to each remote participant and applies inverse distance law with exponential falloff (0.5m–15m range) for more perceptible volume changes |
| **5. Anti-jitter smoothing**  | 3-layer system: threshold filter (0.02), EMA smoothing (0.5), SNAP behavior (0.2 for direction changes)                                                              |
| **6. Left/right rendering**   | StereoPannerNode outputs processed signal with accurate L/R separation based on position projection                                                                  |

---

## Integration Checklist

- [ ] **Instantiate once per page/tab** and keep it in a store (Vuex, Redux, Zustand, etc.)
- [ ] **Pipe LSD/Lap data** from your rendering engine into `updatePosition()` + `setListenerFromLSD()` at ~10 Hz
- [ ] **Render videos muted** – never attach remote audio tracks straight to DOM; let `SpatialAudioManager` own playback
- [ ] **Resume audio context** – call `sdk.resumeAudio()` on first user interaction (required by browsers)
- [ ] **Handle consumer-created** – attach video tracks to UI, audio is handled automatically by spatial audio
- [ ] **Monitor logs** – browser console shows `🎧 SDK`, `📍 SDK`, and `🎚️ [Spatial Audio]` statements for every critical hop
- [ ] **Push avatar telemetry** back to Unreal so `remoteSpatialData` can render minimaps/circles

---

## Core Modules

| File                        | Purpose                                                                              |
|-----------------------------|--------------------------------------------------------------------------------------|
| `src/index.ts`              | `SpatialCommsSDK.create()` – socket lifecycle, producers/consumers, event surface    |
| `src/core/MediasoupManager.ts`        | Transport helpers for produce/consume/resume                                                                    |
| `src/channels/spatial/SpatialAudioChannel.ts` | Web Audio orchestration (listener transforms, per-participant chains, ML denoiser node) |
| `src/audio/MLNoiseSuppressor.ts`      | TensorFlow.js GRU denoiser — `odyssey_adaptive_denoiser` model, 872K params, val_loss=0.1636               |
| `src/core/EventManager.ts`            | Lightweight EventEmitter used by the entire SDK                                                                 |
| `src/types/index.ts`                  | TypeScript interfaces for Position, Direction, Participant, MediaState, etc.                                    |

---

## Development Tips

- Run `npm install && npm run build` inside `odyssey-mediasoup-sdk` to publish a fresh build
- Use `npm run dev` while iterating so TypeScript outputs live under `dist/`
- The SDK targets evergreen browsers; Safari <16.4 needs WebGL support for TF.js (all modern Safari versions have this)
- Have questions or want to extend the SDK? Start with `SpatialAudioManager` – that's where most of the "real-world" behavior (distance feel, stereo cues, denoiser) lives
- **ML Noise Suppression**: Initialized automatically at SDK startup using the `odyssey_adaptive_denoiser` model from `public/odyssey_adaptive_denoiser/model.json`. No manual call needed. Watch browser console for `[Odyssey] ML Noise Suppression loaded and active`

---

## Development

```bash
# Install dependencies
npm install

# Build
npm run build

# Watch mode
npm run dev

# Type check only
npm run typecheck
```

---

## Related Documentation

- [HEAD_POSITION_DATA_FLOW.md](HEAD_POSITION_DATA_FLOW.md) – Detailed panning algorithm with 360° tables
- [SPATIAL_AUDIO_IMPLEMENTATION.md](SPATIAL_AUDIO_IMPLEMENTATION.md) – Implementation summary with examples
- [audio-position-wisecalulation.md](audio-position-wisecalulation.md) – Position-wise calculation reference