# AGENTS.md

## Project Overview

XR Blocks (`xrblocks`) is a cross-platform JavaScript/TypeScript SDK for rapidly
prototyping **AI + XR** applications on the open web. It is built on [three.js](https://threejs.org),
targets Chrome 136+ with WebXR on Android XR, and ships a **desktop simulator** so the same
code runs in an ordinary browser. The published npm package exposes one ESM bundle
(`build/xrblocks.js`) plus per-addon bundles under `build/addons/*`.

> [!IMPORTANT]
> **Only call APIs that exist.** The entire public surface is re-exported from
> [`src/xrblocks.ts`](src/xrblocks.ts) — if a symbol is not there, it is internal. The
> framework's own evaluation found that hallucinated/inconsistent APIs are the #1 cause of
> broken generated apps. When unsure, grep `src/xrblocks.ts` and the relevant subfolder, or
> copy a working pattern from `samples/`, `demos/`, or `templates/`.

> [!NOTE]
> **three.js is a peer dependency** (`^0.184.0`). Apps load it via an importmap; never bundle
> a second copy, and keep the version aligned with `package.json`.

## Build & Test

```bash
npm ci                 # install from the lockfile
npm run dev            # rollup watch + http-server on http://127.0.0.1:8080
npm run build          # production build (rollup -c --failAfterWarnings)
npm run serve          # serve the repo without rebuilding
npm test               # vitest run
npm run lint           # eslint src
npm run format         # prettier --write
```

> [!IMPORTANT]
> Run `npm run lint`, `npm run format`, and `npm test` before opening a PR. The build fails
> on warnings (`--failAfterWarnings`) — fix them, don't suppress. Tests are colocated
> `*.test.ts` files next to the code under test (Vitest, `jsdom` where DOM is needed).

`build/` is generated by Rollup — never hand-edit it; regenerate with `npm run build`.

## Architecture

XR Blocks is a **singleton engine driven by a script lifecycle**:

- **`Core`** ([`src/core/Core.ts`](src/core/Core.ts)) is a singleton, exposed as `xb.core`.
  `xb.init(options)` builds the renderer, camera, WebXR session, and every subsystem, then
  runs the frame loop. It owns the dependency-injection `Registry`.
- **`Script`** ([`src/core/Script.ts`](src/core/Script.ts)) is the developer extension point —
  a `THREE.Object3D` with Unity-`MonoBehaviour`-style hooks (`init`, `update`, `onSelectEnd`,
  `initPhysics`...). Apps are added via `xb.add(new MyScript())` before `xb.init()`.
- **`Options`** ([`src/core/Options.ts`](src/core/Options.ts)) aggregates per-subsystem config
  and exposes chainable `enable*()` methods.
- **Barrel export**: every public symbol flows through [`src/xrblocks.ts`](src/xrblocks.ts).
  New public symbols MUST be added there.
- **Build**: Rollup ([`rollup.config.js`](rollup.config.js)) emits `build/xrblocks.js` (ESM),
  `.min.js`, and `.d.ts`, plus each addon under `build/addons/*`. Heavy deps (three,
  `@google/genai`, `@mediapipe/*`, `@pmndrs/uikit`, Rapier, Spark, lit) stay external.

### Repository layout

| Path          | Purpose                                                                                                             |
| ------------- | ------------------------------------------------------------------------------------------------------------------- |
| `src/`        | SDK source — one folder per subsystem (overview: [`src/SKILL.md`](src/SKILL.md))                                    |
| `src/addons/` | Opt-in modules built separately: `uiblocks`, `netblocks`, `glasses`, `simulator`, `volumes`, `virtualkeyboard`, ... |
| `samples/`    | Focused feature examples (served by the docs site)                                                                  |
| `demos/`      | Richer showcase apps (Ballpit, XR-Emoji, XR-Object, Gemini Icebreakers, ...)                                        |
| `templates/`  | Minimal starting points (`0_basic`, `1_ui`, `2_hands`, `6_ai`, ...)                                                 |
| `docs/`       | Docusaurus manual + samples/templates pages                                                                         |
| `skills/`     | Agent skill registry (`xb-*`) — focused, task-oriented guides                                                       |
| `build/`      | **Generated** bundle output — do not edit                                                                           |

### `src/` subsystems

| Folder                                      | Responsibility                                                                               |
| ------------------------------------------- | -------------------------------------------------------------------------------------------- |
| `core/`                                     | `Core` singleton, `Script`, `Options`, `User`, DI `Registry`, `XRButton`, WebXR session mgmt |
| `input/`                                    | controllers, hands, gaze, mouse, gamepad; `gestures/`; `strokes/`                            |
| `world/`                                    | `World` + `planes/`, `mesh/`, `objects/` (Gemini & MediaPipe backends), `sounds/`            |
| `depth/`                                    | depth sensing, depth mesh, `occlusion/` shaders & passes                                     |
| `ai/`                                       | `AI` facade over `Gemini` + `OpenAI` (query / live / image gen)                              |
| `agent/`                                    | agent framework: tools, memory, context (WIP)                                                |
| `ui/`                                       | core spatial UI: `SpatialPanel`, `Grid`/`Row`/`Col`, views, `ModelViewer`, `Reticle`         |
| `ux/`                                       | `DragManager`, reusable interaction behaviors                                                |
| `simulator/`                                | desktop XR simulator (virtual user/hands/depth/planes, control modes)                        |
| `sound/`                                    | spatial audio, speech recognizer/synthesizer                                                 |
| `physics/`                                  | Rapier3D integration                                                                         |
| `lighting/`, `camera/`, `video/`, `stereo/` | light estimation, device camera, video streams, stereo utils                                 |
| `utils/`                                    | `ModelLoader`, dependency injection, helpers                                                 |

## Conventions

- **TypeScript**, strict. Public symbols re-exported via `src/xrblocks.ts`.
- **Tests**: colocated `*.test.ts`, run with Vitest.
- **Lint/format**: ESLint flat config ([`eslint.config.mjs`](eslint.config.mjs), with
  `eslint-plugin-tsdoc`) + Prettier ([`.prettierrc.json`](.prettierrc.json)).
- **Imports**: some intra-`src` imports carry `.js` extensions for module resolution — follow
  the surrounding file rather than "fixing" them ad hoc.
- **Dependency injection over globals**: prefer `static dependencies = {...}` on a `Script`
  (resolved via `core.registry`) to reaching into singletons, so subsystems stay testable.
- Keep PR diffs focused and add/adjust colocated tests for new branches.

## Adding a Feature / Subsystem

1. Add config to the relevant `*Options` class (or a new one) and register it in `Core.init()`.
2. Expose a chainable `enable*()` on `Options` if it is user-facing.
3. Wire the subsystem into `Core` and the frame loop / DI `Registry`.
4. Re-export public types from `src/xrblocks.ts`.
5. Add colocated `*.test.ts`, a `samples/`/`templates/` example, and — for a distinct
   capability — a `skills/xb-*` entry (see [`skills/README.md`](skills/README.md)).
6. Update `CONTEXT.md`, `README.md`, `AGENTS.md` and `src/SKILL.md` with the changes.

## AI & API-Key Security

> [!IMPORTANT]
> Never commit API keys, and never ship a key in client-side code for production. The `?key=`
> URL parameter and `keys.json` fallback in [`src/ai/AI.ts`](src/ai/AI.ts) are for **local
> prototyping only** — production apps must proxy AI calls through a server you control. AI
> features (Gemini Live/Flash) send data to Google servers; WebXR and on-device (LiteRT)
> features process data locally.

## Agent Docs & Skills

- [`CONTEXT.md`](CONTEXT.md) — rules of engagement for building apps with the SDK.
- [`src/SKILL.md`](src/SKILL.md) — in-tree SDK overview for app developers.
- [`skills/`](skills/) — focused `xb-*` skills (UI, hands, gestures, depth, world, AI,
  physics, simulator, sound, multiplayer). Index + naming convention in
  [`skills/README.md`](skills/README.md).