# MultiSet VPS WebXR

TypeScript SDK for integrating MultiSet's Visual Positioning System (VPS) into WebXR applications. Provides 6-DOF localization by matching camera frames against cloud-hosted maps, and object tracking by matching camera frames against registered 3D objects.

## Contents

- [Architecture](#architecture)
- [Installation](#installation)
- [Requirements](#requirements)
- [CORS Configuration](#cors-configuration)
- [Quick Start](#quick-start)
  - [VPS — Three.js](#vps-localization--threejs)
  - [VPS — Vanilla / WebGL2](#vps-localization--without-threejs-webgl2--vanilla)
  - [Object Tracking — Three.js](#object-tracking--threejs)
  - [Needle Engine](#needle-engine)
    - [Unity Inspector Workflow](#unity-inspector-workflow)
    - [MapAnchor — zero-code object placement](#mapanchor--zero-code-object-placement-unity)
- [Canvas Visibility During AR](#canvas-visibility-during-ar)
- [API Reference](#api-reference)
  - [MultisetClient](#multisetclient)
  - [XRSessionManager](#xrsessionmanager)
  - [ThreeAdapter](#threeadapter)
  - [NeedleAdapter](#needleadapter)
  - [MapAnchor](#mapanchor)
  - [MultisetVPS](#multisetvps)
- [Placing Content at Map Coordinates](#placing-content-at-map-coordinates)
- [Styling the AR Button](#styling-the-ar-button)
- [Custom Start / Stop Button](#custom-start--stop-button)
- [Type Definitions](#type-definitions)
- [Troubleshooting](#troubleshooting)
  - [AR button doesn't appear](#ar-button-doesnt-appear)
  - [MapAnchor not appearing after localization](#mapanchor-not-appearing-after-localization-needle)
  - [Component start() never called](#component-start-never-called-needle)

---

## Architecture

The SDK is split into independent entry points so you only install what you need:

| Entry point | Contents | Peer deps |
|---|---|---|
| `@multisetai/vps/core` | `MultisetClient` + `XRSessionManager` | None |
| `@multisetai/vps/three` | `ThreeAdapter` | `three >=0.169.0` |
| `@multisetai/vps/needle` | `NeedleAdapter` | `@needle-tools/engine` (brings its own `three`) |

`XRSessionManager` owns the full vanilla WebXR session lifecycle — frame loop, camera capture, localization, tracking-loss recovery — with zero Three.js dependency. `ThreeAdapter` and `NeedleAdapter` wire it to a renderer and scene.

## Installation

```bash
# Core only (no Three.js)
npm install @multisetai/vps

# With Three.js adapter
npm install @multisetai/vps three

# With Needle Engine — no separate `three` install needed.
# Needle Engine ships its own three.js fork as a dependency.
npm install @multisetai/vps
```

## Requirements

- **HTTPS** — WebXR requires a secure context (`https://` or `http://localhost`).
- **Android + ARCore** — Chrome or Edge on an ARCore-capable Android device (Android 8+, Chrome 81+).
- **Three.js ≥ 0.169.0** — only required when using `@multisetai/vps/three`. Compatible through the latest release (r184+).
- **Needle Engine ≥ 5.0.0** — only required when using `@multisetai/vps/needle`. Needle ships its own `three` fork — do **not** install `three` separately.

> **iOS is not supported.** This SDK requires the `camera-access` WebXR feature. Safari on iOS does not implement it.

## CORS Configuration

The SDK makes direct browser-to-API requests, so your domain must be whitelisted in the MultiSet dashboard.

1. Open the [MultiSet Dashboard](https://docs.multiset.ai/basics/credentials/configuring-allowed-domains-cors)
2. Go to **Credentials → Settings → Domains**
3. Click **Add +** and enter your origin (e.g. `https://localhost:5173` for dev, `https://your-app.com` for prod)

Without this, the browser will block every API request with a CORS error.

---

## Quick Start

### VPS Localization — Three.js

```typescript
import * as THREE from 'three';
import { MultisetClient, XRSessionManager } from '@multisetai/vps/core';
import { ThreeAdapter } from '@multisetai/vps/three';

// Check support before showing any AR UI
const supported = await ThreeAdapter.isSupported();
if (!supported) {
  console.warn('WebXR immersive-ar is not supported on this device.');
}

const client = new MultisetClient({
  clientId: 'CLIENT_ID',
  clientSecret: 'CLIENT_SECRET',
  code: 'MAP_OR_MAPSET_CODE',
  mapType: 'map',
});

await client.authorize();

const renderer = new THREE.WebGLRenderer({ antialias: true });
document.body.appendChild(renderer.domElement);

const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.01, 1000);

const session = new XRSessionManager(renderer.getContext() as WebGL2RenderingContext, {
  client,
  overlayRoot: document.body,
  autoLocalize: true,
  confidenceCheck: true,
  confidenceThreshold: 0.5,
  onSessionStart: () => {
    // Hide the canvas during AR — the XR compositor owns the display.
    renderer.domElement.style.display = 'none';
  },
  onSessionEnd: () => {
    renderer.domElement.style.display = 'block';
  },
  onLocalizationResult: (result) => console.log('Localized:', result.localizeData.position),
  onLocalizationFailure: (reason) => console.warn('Failed:', reason),
  onError: (err) => console.error(err),
});

const adapter = new ThreeAdapter({ session, renderer, scene, camera, showMesh: true });
adapter.initialize(); // mounts the built-in START AR / STOP AR button

// Add your 3D content
scene.add(new THREE.Mesh(
  new THREE.BoxGeometry(0.1, 0.1, 0.1),
  new THREE.MeshBasicMaterial({ color: 0xff0077 })
));
```

### VPS Localization — Without Three.js (WebGL2 / Vanilla)

In this mode the SDK manages the session lifecycle, camera capture, and localization. You are responsible for all rendering: each XR frame, draw your scene into `event.baseLayer.framebuffer` using the provided `gl` context. This approach works with any WebGL2-based renderer — Babylon.js, raw WebGL, or your own engine.

```typescript
import { MultisetClient, XRSessionManager } from '@multisetai/vps/core';

const supported = await XRSessionManager.isSupported();
if (!supported) {
  console.warn('WebXR immersive-ar is not supported on this device.');
}

const client = new MultisetClient({ clientId: '...', clientSecret: '...', code: '...', mapType: 'map' });
await client.authorize();

// A WebGL2 context is required — WebXR renders into a GL framebuffer, not a 2D canvas.
const gl = document.querySelector('canvas')!.getContext('webgl2')!;

const session = new XRSessionManager(gl, {
  client,
  overlayRoot: document.body,
  autoLocalize: true,
  onLocalizationResult: (result) => console.log('Localized:', result.localizeData.position),
  onError: (err) => console.error(err),
});

// Wire your render loop — called every XR frame with the current pose and framebuffer.
session.setXRFrameHandler((event) => {
  // Bind event.baseLayer.framebuffer and render your scene using event.view for camera matrices.
});

document.body.appendChild(session.createButton());
```

### Object Tracking — Three.js

Object tracking detects and poses registered 3D objects by matching a captured camera frame against the MultiSet cloud.

```typescript
import * as THREE from 'three';
import { MultisetClient, XRSessionManager } from '@multisetai/vps/core';
import { ThreeAdapter } from '@multisetai/vps/three';

const client = new MultisetClient({
  clientId: 'CLIENT_ID',
  clientSecret: 'CLIENT_SECRET',
  mapType: 'object-tracking',
  code: ['YOUR_OBJECT_CODE'],
});

await client.authorize();

const renderer = new THREE.WebGLRenderer({ antialias: true });
document.body.appendChild(renderer.domElement);
const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.01, 1000);

const session = new XRSessionManager(renderer.getContext() as WebGL2RenderingContext, {
  client,
  overlayRoot: document.body,
  autoTracking: true,          // detect once on session start
  confidenceCheck: true,
  confidenceThreshold: 0.5,
  onSessionStart: () => { renderer.domElement.style.display = 'none'; },
  onSessionEnd:   () => { renderer.domElement.style.display = 'block'; },
  onObjectTrackingSuccess: (result) => {
    console.log('Detected', result.objectCodes, 'at', result.position);
  },
  onObjectTrackingFailure: (reason) => console.warn('Tracking failed:', reason),
  onError: (err) => console.error(err),
});

const adapter = new ThreeAdapter({
  session,
  renderer,
  scene,
  camera,
  showObjectMeshes: true,       // load and display the 3D outline mesh
  onObjectMeshLoaded: (code) => console.log('Mesh loaded for', code),
});
adapter.initialize();

// Trigger tracking manually from a button
trackButton.addEventListener('click', () => {
  void adapter.trackObjects();
});
```

### Needle Engine

`NeedleAdapter` is a Needle Engine `Behaviour` component. Add it to a GameObject via `addNewComponent()` from your own component's `start()`. The built-in START AR / STOP AR button mounts automatically — no `initialize()` call needed.

> **Important:** Do **not** add a Needle `WebXR` component to the same scene. `NeedleAdapter` must own the WebXR session directly because it bypasses Three.js's built-in XR manager (`renderer.xr`). This is required to work around a Chrome/WebXR regression where enabling `camera-access` corrupts Three.js's internal texture state. Adding a Needle `WebXR` component would start a second session and conflict with this setup. Delete the `WebXR` component from the scene hierarchy before adding `MultisetVPS`.

#### Unity Inspector workflow

`MultisetVPS` and `MapAnchor` are ready-made components. Copy a few template files into your project, fill in credentials in the Inspector — no code required.

> **Why template files?** Needle Engine's Unity codegen scans only your local `src/scripts/` folder to generate C# stubs — it skips `node_modules`. Copying the templates into your project gives Unity the component definitions it needs, while all SDK utilities (`NeedleAdapter`, `MultisetClient`, etc.) are still imported from the package.

1. **Install** in your Needle web project:
   ```bash
   npm install @multisetai/vps
   ```

2. **Copy the TypeScript templates** into your Needle project's scripts folder (the folder Needle scans for components — `web/src/scripts/` by default, but use whatever path your project uses):

   ```bash
   cp node_modules/@multisetai/vps/templates/MultisetVPS.ts <your-web-folder>/src/scripts/
   cp node_modules/@multisetai/vps/templates/MapAnchor.ts   <your-web-folder>/src/scripts/
   cp node_modules/@multisetai/vps/templates/MapType.ts     <your-web-folder>/src/scripts/
   ```

   > `MapType.ts` must be present alongside `MultisetVPS.ts` — Needle's component compiler resolves the enum from this file to generate the **Map Type** dropdown. Without it the component will not appear in the Inspector.

   Needle Engine automatically picks up everything in your scripts folder and generates the matching C# stubs in Unity.

3. **Copy the C# enum** into your Unity project's `Assets/` folder (one-time setup — this file is never overwritten by Needle). Run this from inside your web folder:

   ```bash
   cp node_modules/@multisetai/vps/templates/MapType.cs ../Assets/
   ```

   This gives you a **Map Type** dropdown in the Inspector (`SingleMap / MapSet / ObjectTracking`). Without it, Needle generates the field but Unity can't compile the enum reference.

4. **In Unity**, add a `MultisetVPS` component to any GameObject and fill in `clientId`, `clientSecret`, and `mapCode` in the Inspector.

5. **Add `MapAnchor`** to every GameObject you want placed in AR. Configure `offset`, `matchOrientation`, `rotationOffset`, `hideUntilLocalized`, and `isRightHanded` in the Inspector. `MultisetVPS` discovers all `MapAnchor` instances automatically at startup — no manual wiring needed.

6. **Export to web** — the START AR / STOP AR and CAPTURE FRAME buttons appear automatically.

For runtime-spawned objects (loaded from an API, instantiated mid-session), call `adapter.registerAnchor(anchor)` — see [MapAnchor — zero-code object placement](#mapanchor--zero-code-object-placement-unity) below.

#### Code-based setup (without Unity Inspector)

```typescript
import { Behaviour, serializable, addNewComponent } from '@needle-tools/engine';
import { MultisetClient } from '@multisetai/vps/core';
import { NeedleAdapter } from '@multisetai/vps/needle';
import * as THREE from 'three';

export class MyARComponent extends Behaviour {
  async start() {
    const client = new MultisetClient({
      clientId: 'CLIENT_ID',
      clientSecret: 'CLIENT_SECRET',
      code: 'MAP_CODE',
      mapType: 'map',
    });

    const adapter = new NeedleAdapter({
      client,
      showMesh: true,
      sessionOptions: {
        autoLocalize: true,
        onSessionStart: () => console.log('AR started'),
        onSessionEnd:   () => console.log('AR ended'),
        onLocalizationFailure: (reason) => console.warn('Failed:', reason),
        onError: (err) => console.error(err),
      },
      onLocalizationSuccess: (result, worldFromMap) => {
        // Place content anchored to the scanned map
        const marker = new THREE.Mesh(
          new THREE.SphereGeometry(0.05),
          new THREE.MeshBasicMaterial({ color: 0x00ff88 })
        );
        marker.position.applyMatrix4(worldFromMap);
        this.context.scene.add(marker);
      },
    });

    // addNewComponent triggers awake() — button mounts automatically
    addNewComponent(this.gameObject, adapter);
    await client.authorize();
  }
}
```

#### MapAnchor — zero-code object placement (Unity)

`MapAnchor` is a Needle Engine component that anchors any GameObject to the VPS map origin after a successful localization. Add it to the object you want to place in AR from the Unity Inspector — no code required.

| Inspector field | Type | Description |
|---|---|---|
| **Offset** | `Vector3` | Position offset from the map origin in metres. Enter Unity Inspector values directly — X is negated automatically unless **Is Right Handed** is enabled. |
| **Match Orientation** | `bool` | Align the object's rotation to the map's orientation. |
| **Rotation Offset** | `Vector3` (degrees) | Additional rotation applied on top of the map orientation. Only used when **Match Orientation** is enabled. Enter Unity Inspector values — Y and Z are negated automatically unless **Is Right Handed** is enabled. |
| **Hide Until Localized** | `bool` | Hide the object until the first successful localization. Re-hides it when the AR session ends. |
| **Is Right Handed** | `bool` | Off by default. When off, **Offset** and **Rotation Offset** are treated as Unity (left-handed) values and converted automatically. Enable only if you are entering Three.js (right-handed) values directly. |

`MultisetVPS` (or your own component calling `addNewComponent`) automatically discovers all `MapAnchor` instances in the scene at startup and wires them to the adapter. You do not call any method manually.

> **Important:** When `hideUntilLocalized` is `true`, Needle sets `visible = false` on the GameObject in `awake()`. An invisible GameObject is inactive in Needle — every component on it, not just `MapAnchor`, will have its `start()` skipped. Keep `MapAnchor` on the object you want placed in AR. Put any logic components (session listeners, custom UI) on a **separate, always-visible GameObject**.

For objects spawned **at runtime** (e.g. from API data or a prefab instantiated mid-session), register them explicitly so they receive localization events and are placed immediately if localization has already succeeded:

```typescript
import { MapAnchor } from '@multisetai/vps/needle';

// Spawn a new object at runtime and anchor it to the map
const obj = instantiate(myPrefab);
const anchor = addNewComponent(obj, new MapAnchor());
anchor.hideUntilLocalized = true;
anchor.offset.set(0.5, 0, -1.0);

// Connect it — places immediately if localization already succeeded
this.adapter.registerAnchor(anchor);
```

**Using `useDefaultButton: false`** (custom button):

```typescript
const adapter = new NeedleAdapter({
  client,
  useDefaultButton: false,
  sessionOptions: { autoLocalize: true },
});

addNewComponent(this.gameObject, adapter);
await client.authorize();

myButton.addEventListener('click', () => {
  if (adapter.isActive()) {
    adapter.stopSession();
  } else {
    void adapter.startSession();
  }
});
```

---

## Canvas Visibility During AR

> **Important** — the SDK renders the Three.js scene into the **XR framebuffer**, not the canvas element. During an active AR session the canvas element is not updated; it retains whatever was last drawn by the preview loop. If the canvas is visible during AR (e.g. as part of the WebXR DOM overlay) it will appear as a frozen image on top of the AR scene.

Always hide the canvas when the session starts and restore it when it ends:

```typescript
onSessionStart: () => { renderer.domElement.style.display = 'none'; },
onSessionEnd:   () => { renderer.domElement.style.display = 'block'; },
```

---

## API Reference

### `MultisetClient`

Pure HTTP client for auth, localization, and object tracking. No WebXR or rendering concerns.

```typescript
new MultisetClient(config: IMultisetClientConfig)
```

#### `IMultisetClientConfig`

**VPS mode (`mapType: 'map'` or `mapType: 'map-set'`)**

| Parameter | Type | Description |
|---|---|---|
| `clientId` | `string` | Your MultiSet client ID |
| `clientSecret` | `string` | Your MultiSet client secret |
| `mapType` | `'map' \| 'map-set'` | Whether `code` is a single map or a map set |
| `code` | `string` | Map or map-set code |
| `endpoints?` | `Partial<IMultisetSdkEndpoints>` | Override default API endpoints |
| `isRightHanded?` | `boolean` | Handedness sent to the API. Default `true` |
| `convertToGeoCoordinates?` | `boolean` | Request geographic coordinates in the response |
| `hintPosition?` | `string` | Local-space position hint `"x,y,z"` |
| `hintRadius?` | `number \| string` | Search radius in metres (1–100). Requires `hintPosition` or `passGeoPose` |
| `hintMapCodes?` | `string[]` | Narrow candidates by map code. Only valid when `mapType: 'map-set'` |
| `passGeoPose?` | `boolean` | Send a `geoHint` from the Geolocation API with each request |
| `use2DFiltering?` | `boolean` | Skip altitude in geo filtering. Only valid when `passGeoPose: true` |

**Object tracking mode (`mapType: 'object-tracking'`)**

| Parameter | Type | Description |
|---|---|---|
| `clientId` | `string` | Your MultiSet client ID |
| `clientSecret` | `string` | Your MultiSet client secret |
| `mapType` | `'object-tracking'` | Enables object tracking mode. No map code required. |
| `code` | `string[]` | Object codes to detect and track |
| `isRightHanded?` | `boolean` | Handedness sent to the API. Default `true` |
| `endpoints?` | `Partial<IMultisetSdkEndpoints>` | Override default API endpoints |

#### Methods

| Method | Returns | Description |
|---|---|---|
| `authorize()` | `Promise<string>` | Authenticate and cache an access token. Call before any other method. |
| `localizeWithFrame(frame, intrinsics)` | `Promise<ILocalizeAndMapDetails \| null>` | Submit a captured frame for VPS localization. |
| `trackObject(frame, intrinsics)` | `Promise<IObjectTrackingResponse \| null>` | Submit a captured frame for object detection. Uses `code` from the client config. Returns `null` when no object is detected. |
| `downloadObjectMesh(objectCode)` | `Promise<string \| null>` | Fetch a signed download URL for the 3D mesh of an object. |
| `fetchMapDetails(mapCode)` | `Promise<IGetMapsDetailsResponse \| null>` | Fetch map metadata by code (result is cached). |

---

### `XRSessionManager`

Owns the WebXR session lifecycle — frame loop, camera capture, localization, object tracking, and tracking-loss recovery. Zero Three.js dependency.

```typescript
import { XRSessionManager } from '@multisetai/vps/core';

new XRSessionManager(gl: WebGL2RenderingContext, options: IXRSessionOptions)
```

#### `IXRSessionOptions`

| Parameter | Type | Description |
|---|---|---|
| `client` | `MultisetClient` | Required. |
| `overlayRoot?` | `HTMLElement` | Root element for the WebXR DOM overlay. |
| `autoLocalize?` | `boolean` | Run one localization automatically when the session starts. |
| `relocalization?` | `boolean` | Re-localize whenever tracking is lost and then recovered. |
| `confidenceCheck?` | `boolean` | Only accept results with `confidence >= confidenceThreshold`. Applies to both VPS and object tracking. |
| `confidenceThreshold?` | `number` | Minimum confidence (0.2–0.8). Default `0.5`. |
| `poseTimeoutMs?` | `number` | Max ms to wait for a valid viewer pose before failing. Default `10000`. |
| `localizationTrackingTimeoutMs?` | `number` | **Deprecated** — use `poseTimeoutMs`. Will be removed in v3. |
| `backgroundLocalization?` | `boolean` | Periodically send localization/tracking requests in the background while the session is active. |
| `bgLocalizationInterval?` | `number` | Interval in seconds between background attempts. Clamped to 10–180 s. Default `30` for VPS modes, `10` for object tracking. |
| `autoTracking?` | `boolean` | Call `trackObjects()` once automatically when the session starts. Requires `mapType: 'object-tracking'` on the client. |
| `restartTracking?` | `boolean` | Re-attempt tracking whenever XR tracking is lost and then recovered. |
| `trackingCaptureDelayMs?` | `number` | Milliseconds to wait before capturing a frame when `trackObjects()` is called. Useful for camera stabilisation. Default `0`. |
| `referenceSpaceType?` | `XRReferenceSpaceType` | XR reference space type. Default `'local'`. Use `'local-floor'` for floor-relative tracking if the device supports it. |
| `framebufferScaleFactor?` | `number` | XR framebuffer scale relative to native resolution. Values `< 1` reduce GPU load; values `> 1` supersample. |
| `onSessionStart?` | `() => void` | Called when the AR session starts. |
| `onSessionEnd?` | `() => void` | Called when the AR session ends. |
| `onLocalizationInit?` | `() => void` | Called at the start of a VPS localization run. |
| `onLocalizationResult?` | `(result: ILocalizeAndMapDetails) => void` | Called when VPS localization succeeds (and passes the confidence check, if enabled). |
| `onLocalizationSuccess?` | `(result: ILocalizeAndMapDetails) => void` | **Deprecated** — use `onLocalizationResult`. If using `ThreeAdapter`, use its `onLocalizationSuccess` which also provides `worldFromMap`. Will be removed in v3. |
| `onLocalizationFailure?` | `(reason?: string) => void` | Called when VPS localization fails or falls below the confidence threshold. |
| `onFrameCaptured?` | `(frame: IFrameCaptureEvent) => void` | Called when a camera frame is captured for localization. |
| `onCameraIntrinsics?` | `(intrinsics: ICameraIntrinsicsEvent) => void` | Called with camera intrinsic parameters for the captured frame. |
| `onPoseResult?` | `(pose: IPoseResultEvent) => void` | Called with the raw pose result from the VPS backend. |
| `onObjectTrackingInit?` | `() => void` | Called at the start of an object tracking run. |
| `onObjectTrackingRequested?` | `(frame: IFrameCaptureEvent, intrinsics: ICameraIntrinsicsEvent) => void` | Called just before the tracking request is sent, with the captured frame and intrinsics. |
| `onObjectTrackingSuccess?` | `(result: IObjectTrackingResponse) => void` | Called when object tracking succeeds and passes the confidence check (if enabled). |
| `onObjectTrackingFailure?` | `(reason?: string) => void` | Called when object tracking fails or falls below the confidence threshold. |
| `onError?` | `(error: unknown) => void` | Called when any error occurs. |
| `onContextLost?` | `() => void` | Called when the WebGL context is lost. The active session is ended automatically. |
| `onContextRestored?` | `() => void` | Called when the WebGL context is restored. The user may restart the session. |

#### Static methods

| Method | Returns | Description |
|---|---|---|
| `XRSessionManager.isSupported()` | `Promise<boolean>` | Returns `true` if the browser supports `immersive-ar` WebXR sessions. Use this to conditionally show AR UI before creating any objects. |

#### Methods

| Method | Returns | Description |
|---|---|---|
| `createButton()` | `HTMLButtonElement` | Create the built-in styled AR button. Shows **START AR** / **STOP AR** and toggles the session on click. |
| `startSession()` | `Promise<void>` | Start an AR session programmatically. **Must be called from within a user gesture handler** (click/tap). |
| `stopSession()` | `void` | Stop the active AR session. No-op if no session is running. |
| `localizeFrame()` | `Promise<ILocalizeAndMapDetails \| null>` | Capture and localize one frame against the configured map. Requires an active session. |
| `trackObjects()` | `Promise<IObjectTrackingResponse \| null>` | Capture one frame and run object detection. Requires an active session and `mapType: 'object-tracking'` on the client. |
| `isActive()` | `boolean` | Whether an XR session is currently running. |
| `isLocalizing` | `boolean` | Whether a localization or tracking run is currently in progress. |
| `getClient()` | `MultisetClient` | Access the underlying `MultisetClient`. |
| `dispose()` | `void` | End the session, clear background timers, remove context loss listeners, and release all resources. |

#### Adapter hooks

Used internally by `ThreeAdapter`. Only call these when building a custom renderer adapter.

| Method | Description |
|---|---|
| `setXRFrameHandler(fn)` | Called every XR frame with pose, view, viewport, and framebuffer info. |
| `setAdapterResultHandler(fn)` | Called after a successful VPS localization with the result and tracker-space matrix. |
| `setAdapterObjectTrackingHandler(fn)` | Called after a successful object tracking result with the result and tracker-space matrix. |
| `setAdapterSessionHandlers(onStart, onEnd)` | Called on session start/end, before user callbacks. |

---

### `ThreeAdapter`

Wires `XRSessionManager` to a Three.js renderer. Handles XR framebuffer binding, camera matrix sync, preview loop, resize, and optional map mesh / gizmo / object mesh display.

```typescript
import { ThreeAdapter } from '@multisetai/vps/three';

new ThreeAdapter(options: IThreeAdapterOptions)
```

#### `IThreeAdapterOptions`

| Parameter | Type | Description |
|---|---|---|
| `session` | `XRSessionManager` | Required. |
| `renderer` | `THREE.WebGLRenderer` | Required. |
| `scene` | `THREE.Scene` | Required. |
| `camera` | `THREE.PerspectiveCamera` | Required. |
| `showMesh?` | `boolean` | Show the VPS map mesh after localization. Default `false`. |
| `showGizmo?` | `boolean` | Show a transform gizmo after localization. Default `true`. |
| `showObjectMeshes?` | `boolean` | Load and display a 3D outline mesh for each detected object. Default `false`. |
| `useDefaultButton?` | `boolean` | Mount the built-in START AR / STOP AR button. Default `true`. Set to `false` to drive the session via `startSession()` / `stopSession()`. |
| `buttonContainer?` | `HTMLElement` | Where to append the built-in button. Defaults to `overlayRoot` or `document.body`. |
| `onButtonCreated?` | `(button: HTMLButtonElement) => void` | Called after the built-in button is created. |
| `onXRFrame?` | `(event: IXRFrameEvent) => void` | Called every XR frame after camera matrices are synced, before the scene is rendered. Use this to update scene objects each frame. |
| `onLocalizationSuccess?` | `(result: ILocalizeAndMapDetails, worldFromMap: THREE.Matrix4) => void` | Called immediately after a successful VPS localization. `worldFromMap` transforms map-space coordinates to Three.js world space — use it to place content at known map coordinates. |
| `onObjectMeshLoaded?` | `(objectCode: string) => void` | Called when a detected object's 3D mesh has been loaded and placed in the scene. Only fires when `showObjectMeshes: true`. |

#### Static methods

| Method | Returns | Description |
|---|---|---|
| `ThreeAdapter.isSupported()` | `Promise<boolean>` | Returns `true` if the browser supports `immersive-ar` WebXR sessions. |

#### Methods

| Method | Returns | Description |
|---|---|---|
| `initialize(buttonContainer?)` | `HTMLButtonElement \| null` | Start the preview render loop, attach resize handler, and mount the built-in button. Returns `null` when `useDefaultButton: false`. |
| `isActive()` | `boolean` | Whether an XR session is currently running. |
| `isLocalizing` | `boolean` | Whether a localization or tracking run is currently in progress. |
| `startSession()` | `Promise<void>` | Start an AR session. **Must be called from within a user gesture handler.** |
| `stopSession()` | `void` | Stop the active AR session. No-op if no session is running. |
| `localizeFrame()` | `Promise<ILocalizeAndMapDetails \| null>` | Capture and localize one frame. |
| `trackObjects()` | `Promise<IObjectTrackingResponse \| null>` | Capture one frame and run object detection. Requires `mapType: 'object-tracking'` on the client. |
| `clearObjectMeshes()` | `void` | Remove all object meshes from the scene that were placed by `showObjectMeshes`. |
| `dispose()` | `void` | Stop loops, remove listeners, dispose Three.js resources, and end the session. |

---

### `NeedleAdapter`

Needle Engine `Behaviour` that wires `XRSessionManager` to Needle's renderer and scene. Handles XR framebuffer binding, camera matrix sync, AR passthrough (transparent background), and optional map mesh / gizmo / object mesh display.

```typescript
import { NeedleAdapter } from '@multisetai/vps/needle';

new NeedleAdapter(options: INeedleAdapterOptions)
```

Add to a scene via `addNewComponent(gameObject, adapter)` — this triggers `awake()`, which creates the session and mounts the button.

#### `INeedleAdapterOptions`

| Parameter | Type | Description |
|---|---|---|
| `client` | `MultisetClient` | Required. |
| `sessionOptions?` | `Omit<IXRSessionOptions, 'client'>` | All session options and callbacks — forwarded directly to `XRSessionManager`. See [IXRSessionOptions](#ixrsessionoptions) for the full list. |
| `showMesh?` | `boolean` | Show the VPS map mesh after localization. Default `false`. |
| `showGizmo?` | `boolean` | Show a transform gizmo after localization. Default `false`. |
| `showObjectMeshes?` | `boolean` | Load and display a 3D outline mesh for each detected object. Default `false`. |
| `useDefaultButton?` | `boolean` | Mount the built-in START AR / STOP AR button automatically in `awake()`. Default `true`. Set to `false` to drive the session via `startSession()` / `stopSession()`. |
| `buttonContainer?` | `HTMLElement` | Where to append the built-in button. Defaults to `overlayRoot` or `document.body`. |
| `onButtonCreated?` | `(button: HTMLButtonElement) => void` | Called after the built-in button is created. |
| `onXRFrame?` | `(event: IXRFrameEvent) => void` | Called every XR frame after camera matrices are synced, before the scene is rendered. |
| `onLocalizationSuccess?` | `(result: ILocalizeAndMapDetails, worldFromMap: THREE.Matrix4) => void` | Called after a successful VPS localization. `worldFromMap` transforms map-space coordinates to Three.js world space. |
| `onObjectMeshLoaded?` | `(objectCode: string) => void` | Called when a detected object's 3D mesh has been loaded and placed in the scene. Only fires when `showObjectMeshes: true`. |

#### Static methods

| Method | Returns | Description |
|---|---|---|
| `NeedleAdapter.isSupported()` | `Promise<boolean>` | Returns `true` if the browser supports `immersive-ar` WebXR sessions. |

#### Methods

| Method | Returns | Description |
|---|---|---|
| `isActive()` | `boolean` | Whether an XR session is currently running. |
| `isLocalizing` | `boolean` | Whether a localization or tracking run is currently in progress. |
| `startSession()` | `Promise<void>` | Start an AR session. **Must be called from within a user gesture handler.** |
| `stopSession()` | `void` | Stop the active AR session. No-op if no session is running. |
| `localizeFrame()` | `Promise<ILocalizeAndMapDetails \| null>` | Capture and localize one frame. |
| `trackObjects()` | `Promise<IObjectTrackingResponse \| null>` | Capture one frame and run object detection. Requires `mapType: 'object-tracking'` on the client. |
| `clearObjectMeshes()` | `void` | Remove all object meshes placed by `showObjectMeshes`. |
| `registerAnchor(anchor)` | `void` | Connect a dynamically created `MapAnchor` to the adapter. Registers localization listeners and immediately applies the last localization result if the session is already active. |
| `addLocalizationListener(fn)` | `void` | Subscribe to localization events from any Needle component. `fn` receives `(result, worldFromMap)`. Prefer this over the constructor `onLocalizationSuccess` when multiple components need to react. |
| `removeLocalizationListener(fn)` | `void` | Unsubscribe a previously added localization listener. Call from `onDestroy`. |
| `addSessionStartListener(fn)` | `void` | Subscribe to session-start events. Useful for showing in-session UI (buttons, overlays) when AR begins. |
| `removeSessionStartListener(fn)` | `void` | Unsubscribe a previously added session-start listener. Call from `onDestroy`. |
| `addSessionEndListener(fn)` | `void` | Subscribe to session-end events. Useful for hiding anchored content between sessions. |
| `removeSessionEndListener(fn)` | `void` | Unsubscribe a previously added session-end listener. Call from `onDestroy`. |
| `getSession()` | `XRSessionManager` | Access the underlying session manager directly. Use only when you need low-level session control. |

---

### `MapAnchor`

Needle Engine `Behaviour` that anchors a GameObject to the VPS map origin after localization. Exported from `@multisetai/vps/needle`.

In Unity, add the component via the Inspector — `MultisetVPS` discovers and wires all `MapAnchor` instances at startup automatically.

For runtime-spawned objects use `NeedleAdapter.registerAnchor()` (see [MapAnchor — zero-code object placement](#mapanchor--zero-code-object-placement-unity) above).

#### Fields

| Field | Type | Default | Description |
|---|---|---|---|
| `offset` | `THREE.Vector3` | `(0, 0, 0)` | Position offset from the map origin in metres. When `isRightHanded` is `false` (default), enter Unity Inspector values directly — X is negated automatically. When `true`, supply Three.js values as-is. |
| `matchOrientation` | `boolean` | `true` | Align the object's rotation to the map's orientation. |
| `rotationOffset` | `THREE.Euler` | `(0°, 0°, 0°)` | Additional rotation applied on top of the map orientation. Only applied when `matchOrientation` is `true`. When `isRightHanded` is `false` (default), enter Unity Inspector values — Y and Z are negated automatically. |
| `hideUntilLocalized` | `boolean` | `true` | Hide the object until the first successful localization. Re-hides on session end. |
| `isRightHanded` | `boolean` | `false` | When `false` (default), `offset` and `rotationOffset` are treated as Unity (left-handed) values and converted automatically (negate X in position; negate Y and Z in rotation). Set to `true` if you are supplying Three.js (right-handed) values directly. |

#### Methods

| Method | Description |
|---|---|
| `connectAdapter(adapter)` | Called automatically by `MultisetVPS`. Registers localization and session-end listeners on the adapter. |
| `applyLocalization(result, worldFromMap)` | Apply a localization result directly — positions and shows the object. Called internally by `registerAnchor`; also useful for testing or custom placement logic. |

---

### `MultisetVPS`

Inspector-driven Needle Engine component that bootstraps the full VPS stack — credentials, session, UI buttons, and `MapAnchor` discovery — from Unity Inspector fields alone. Exported from `@multisetai/vps/needle`.

```typescript
import { MultisetVPS, MapType } from '@multisetai/vps/needle';
```

> See [Unity Inspector workflow](#unity-inspector-workflow) for setup. The template files handle TypeStore registration automatically — no manual `register()` call needed.

#### Inspector fields

| Field | Type | Default | Description |
|---|---|---|---|
| `clientId` | `string` | `""` | Your Multiset client ID. |
| `clientSecret` | `string` | `""` | Your Multiset client secret. |
| `mapCode` | `string` | `""` | Map/map-set code, or comma-separated object codes for object-tracking mode. |
| `mapType` | `MapType` | `SingleMap` | `SingleMap`, `MapSet`, or `ObjectTracking`. |
| `showMesh` | `boolean` | `true` | Show the VPS map mesh after localization (VPS modes only). |
| `showGizmo` | `boolean` | `true` | Show a transform gizmo after localization (VPS modes only). |
| `showObjectMeshes` | `boolean` | `false` | Load and display a 3D outline mesh for each detected object (OT mode only). |
| `autoLocalize` | `boolean` | `true` | Run one localization automatically when the session starts (VPS modes). |
| `relocalization` | `boolean` | `false` | Re-localize whenever tracking is lost and then recovered. |
| `backgroundLocalization` | `boolean` | `false` | Periodically localize in the background while the session is active. |
| `bgLocalizationInterval` | `number` | `0` | Seconds between background localization attempts (10–180). 0 = SDK default. |
| `confidenceCheck` | `boolean` | `false` | Only accept results above `confidenceThreshold`. |
| `confidenceThreshold` | `number` | `0.5` | Minimum confidence (0.2–0.8). |
| `poseTimeoutMs` | `number` | `0` | Max ms to wait for a valid viewer pose. 0 = SDK default (10 000 ms). |
| `convertToGeoCoordinates` | `boolean` | `false` | Request geographic coordinates in the localization response. |
| `passGeoPose` | `boolean` | `false` | Send a geolocation hint with each request. |
| `use2DFiltering` | `boolean` | `false` | Skip altitude in geo filtering. Only valid when `passGeoPose` is enabled. |
| `hintPosition` | `string` | `""` | Local-space position hint `"x,y,z"`. |
| `hintRadius` | `number` | `0` | Search radius in metres around `hintPosition`. |
| `hintMapCodes` | `string` | `""` | Comma-separated map codes to restrict search (map-set mode only). |
| `autoTracking` | `boolean` | `false` | Detect objects automatically when the session starts (OT mode). |
| `restartTracking` | `boolean` | `false` | Re-attempt tracking when XR tracking is lost and recovered (OT mode). |
| `trackingCaptureDelayMs` | `number` | `0` | Ms to wait before capturing a frame for tracking. |

#### Properties

| Property | Type | Description |
|---|---|---|
| `adapter` | `NeedleAdapter \| null` | The underlying `NeedleAdapter` after `start()` completes. Use this to call `registerAnchor()`, add listeners, or access the session directly. `null` until start resolves. |

---

## Placing Content at Map Coordinates

After localization, the `onLocalizationSuccess` callback on `ThreeAdapter` provides a `worldFromMap` matrix that converts any point from VPS map space into Three.js world space. Use this to anchor scene objects to specific physical locations in the scanned map — independently of where the user started the AR session.

```typescript
const adapter = new ThreeAdapter({
  session,
  renderer, scene, camera,
  onLocalizationSuccess: (result, worldFromMap) => {
    // mapPoint is a position you measured from the scanned map (in metres)
    const mapPoint = new THREE.Vector3(1.5, 0, -2.0);

    const marker = new THREE.Mesh(
      new THREE.SphereGeometry(0.05),
      new THREE.MeshBasicMaterial({ color: 0x00ff88 })
    );
    marker.position.copy(mapPoint.applyMatrix4(worldFromMap));
    scene.add(marker);
  },
});
```

> **Note** — `worldFromMap` is recomputed on every successful localization. If you re-localize, update or re-add your objects so they stay in sync with the latest result.

---

## Styling the AR Button

The built-in button ships with minimal inline styles. Use these CSS classes to override appearance from your own stylesheet:

| Class | When present |
|---|---|
| `.multiset-ar-button` | Always — use for base styles |
| `.multiset-ar-inactive` | No active session (START AR state) |
| `.multiset-ar-active` | During an active AR session (STOP AR state) |

```css
.multiset-ar-button {
  font-family: 'Your Font', sans-serif;
  border-radius: 8px;
}

.multiset-ar-inactive {
  background: rgba(0, 0, 0, 0.5);
}

.multiset-ar-active {
  background: rgba(200, 0, 0, 0.6);
  border-color: #ff4444;
}
```

---

## Custom Start / Stop Button

Set `useDefaultButton: false` to manage the session yourself:

```typescript
const adapter = new ThreeAdapter({
  session,
  renderer, scene, camera,
  useDefaultButton: false,
});

adapter.initialize();

myButton.addEventListener('click', () => {
  if (adapter.isActive()) {
    adapter.stopSession();
  } else {
    // startSession() must be the first call inside the gesture handler —
    // any await before it consumes the browser's user activation token.
    void adapter.startSession();
  }
});
```

---

## Type Definitions

```typescript
import type {
  IMultisetClientConfig,
  IMultisetSdkEndpoints,
  IXRSessionOptions,
  IXRFrameEvent,
  IFrameCaptureEvent,
  ICameraIntrinsicsEvent,
  IPoseResultEvent,
  ILocalizeAndMapDetails,
  IObjectTrackingResponse,
  IGetMapsDetailsResponse,
  MapType,
} from '@multisetai/vps/core';

import type { IThreeAdapterOptions } from '@multisetai/vps/three';

import type { INeedleAdapterOptions } from '@multisetai/vps/needle';
import { NeedleAdapter, MapAnchor, MultisetVPS, MapType } from '@multisetai/vps/needle';
```

### `IObjectTrackingResponse`

Returned by `trackObjects()` and `MultisetClient.trackObject()` when an object is detected.

| Field | Type | Description |
|---|---|---|
| `poseFound` | `boolean` | Whether the API found a valid pose for the object. |
| `position` | `IPosition` | Position of the detected object in tracker space `{ x, y, z }`. |
| `rotation` | `IRotation` | Orientation of the detected object as a quaternion `{ x, y, z, w }`. |
| `confidence` | `number` | Detection confidence score (0–1). |
| `objectCodes` | `string[]` | The object code(s) that were matched. |

---

## Troubleshooting

### AR button doesn't appear

- **`useDefaultButton: false` set?** — you must call `startSession()` from your own button. For `ThreeAdapter`, ensure you still call `adapter.initialize()`. For `NeedleAdapter`, the button mounts automatically in `awake()` unless `useDefaultButton: false`.
- **`authorize()` not called?** — call `client.authorize()` before any session or localization attempt.
- **`WebXR` component conflict (Needle)?** — remove any Needle `WebXR` component from the same scene. `NeedleAdapter` must own the WebXR session directly (it bypasses `renderer.xr` to avoid a Chrome/WebXR `camera-access` regression). A Needle `WebXR` component starts a second session that conflicts with this setup. Delete the `WebXR` component from the scene hierarchy before adding `MultisetVPS`.

### Camera passthrough not showing (only 3D scene visible)

The XR compositor shows the camera feed wherever the rendered output has alpha = 0. If you see only the 3D scene with an opaque background, `ThreeAdapter` or `NeedleAdapter` handle this automatically. If you're using a custom renderer, set `scene.background = null` and `renderer.setClearAlpha(0)` at the start of each XR frame.

### CORS error on every API request

Your domain is not whitelisted. Go to the [MultiSet Dashboard](https://docs.multiset.ai/basics/credentials/configuring-allowed-domains-cors) → **Credentials → Settings → Domains** and add your origin (e.g. `https://localhost:5173`).

### `authorize()` throws or returns 401

- Verify `clientId` and `clientSecret` are correct.
- Tokens expire — call `authorize()` again if you receive a 401 after a long session.

### Localization always fails / low confidence

- Ensure the physical space matches the scanned map (lighting, object placement).
- Enable `confidenceCheck: true` and lower `confidenceThreshold` (default 0.5) if you want to accept lower-quality results.
- Try `relocalization: true` to automatically re-localize when tracking is lost.

### WebGL context lost

The browser reclaims the GPU context on mobile when the app is backgrounded. `onContextLost` fires and the session ends automatically. Re-prompt the user to start a new session when `onContextRestored` fires.

### MapAnchor not appearing after localization (Needle)

- **Runtime-spawned anchors** — `MapAnchor` instances created after `MultisetVPS.start()` (or your equivalent) are not auto-discovered by `findObjectsOfType`. Call `adapter.registerAnchor(anchor)` explicitly after adding the component. If localization has already succeeded the anchor is placed immediately via last-localization replay.
- **`connectAdapter` never called** — if you are not using `MultisetVPS`, ensure your setup code calls `adapter.registerAnchor(anchor)` (or `anchor.connectAdapter(adapter)`) for every `MapAnchor` in the scene. Without this the anchor never receives localization events.
- **Wrong GameObject** — confirm the `MapAnchor` component is on the exact object (or a parent of the object) you intend to place. The anchor repositions `this.gameObject` — not the scene root.

### Component `start()` never called (Needle)

A component whose `start()` never fires is almost always on a GameObject that was marked invisible before `start()` ran. In Needle Engine, `visible = false` makes the whole object inactive — every component on it skips `start()`.

The most common cause: the component is on the same GameObject as a `MapAnchor` that has `hideUntilLocalized: true`. `MapAnchor.awake()` sets `visible = false` immediately, which silently disables all sibling components including your own.

**Fix:** keep `MapAnchor` on the object you want placed. Move any other logic components (custom UI, session listeners, test utilities) to a **different, always-visible GameObject** in the scene.

### `startSession()` throws "must be called from a user gesture"

Browsers only allow WebXR session creation in direct response to a user gesture (click/tap). Do not `await` anything before calling `startSession()` inside a click handler — any `await` consumes the activation token.

```typescript
// Correct
button.addEventListener('click', () => {
  void adapter.startSession(); // no await before this
});

// Wrong — the await before startSession loses the gesture token
button.addEventListener('click', async () => {
  await someOtherThing();
  void adapter.startSession(); // will throw
});
```

---

## License

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)