# AGENTS.md

## Project Overview

XR Blocks (`xrblocks`) is a cross-platform JavaScript/TypeScript SDK for rapidly
prototyping **AI + XR** applications on the open web. It is built on [three.js](https://threejs.org),
targets Chrome 136+ with WebXR on Android XR, and ships a **desktop simulator** so the same
code runs in an ordinary browser. The published npm package exposes one ESM bundle
(`build/xrblocks.js`) plus per-addon bundles under `build/addons/*`.

> [!IMPORTANT]
> **Only call APIs that exist.** The entire public surface is re-exported from
> [`src/xrblocks.ts`](src/xrblocks.ts) — if a symbol is not there, it is internal. The
> framework's own evaluation found that hallucinated/inconsistent APIs are the #1 cause of
> broken generated apps. When unsure, grep `src/xrblocks.ts` and the relevant subfolder, or
> copy a working pattern from `samples/`, `demos/`, or `templates/`.

> [!NOTE]
> **three.js is a peer dependency** (`^0.184.0`). Apps load it via an importmap; never bundle
> a second copy, and keep the version aligned with `package.json`.

## Build & Test

```bash
npm ci                 # install from the lockfile
npm run dev            # rollup watch + http-server on http://127.0.0.1:8080
npm run build:sdk      # preferred product build for testing without building demos/samples
npm run serve          # serve the repo without rebuilding
npm test               # vitest run
npm run lint           # eslint src
npm run format         # prettier --write
```

> [!IMPORTANT]
> Run `npm run lint`, `npm run format`, and `npm test` before opening a PR. The build fails
> on warnings (`--failAfterWarnings`) — fix them, don't suppress. Tests are colocated
> `*.test.ts` files next to the code under test (Vitest, `jsdom` where DOM is needed).

`build/` is generated by Rollup — never hand-edit it; regenerate with `npm run build:sdk`.

## Architecture

XR Blocks is a **singleton engine driven by a script lifecycle**:

- **`Core`** ([`src/core/Core.ts`](src/core/Core.ts)) is a singleton, exposed as `xb.core`.
  `xb.init(options)` builds the renderer, camera, WebXR session, and every subsystem, then
  runs the frame loop. It owns the dependency-injection `Registry`.
- **`Script`** ([`src/core/Script.ts`](src/core/Script.ts)) is the developer extension point —
  a `THREE.Object3D` with Unity-`MonoBehaviour`-style hooks (`init`, `update`, `onSelectEnd`,
  `initPhysics`...). Apps are added via `xb.add(new MyScript())` before `xb.init()`.
- **`Options`** ([`src/core/Options.ts`](src/core/Options.ts)) aggregates per-subsystem config
  and exposes chainable `enable*()` methods. `enableAutomationMode()` is the shared
  desktop simulator and context preset for automation and external runs, also available
  through `?xrAutomation=1`. Separately, `?debug=1` exposes `window.xb` and
  `window.xbReady` for in-page browser drivers.
- **Barrel export**: every public symbol flows through [`src/xrblocks.ts`](src/xrblocks.ts).
  New public symbols MUST be added there.
- **Build**: Rollup ([`rollup.config.js`](rollup.config.js)) emits `build/xrblocks.js` (ESM),
  `.min.js`, and `.d.ts`, plus each addon under `build/addons/*`. Heavy deps (three,
  `@google/genai`, `@mediapipe/*`, `@pmndrs/uikit`, Rapier, Spark, lit) stay external.

### Repository layout

| Path          | Purpose                                                                                                                                      |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `src/`        | SDK source — one folder per subsystem                                                                                                        |
| `src/addons/` | Opt-in and experimental modules built separately: `testing`, `netblocks`, `lipsync`, `agenthands`, `embodied-control`, `remote-control`, ... |
| `samples/`    | Focused examples and richer showcases under `samples/advanced/` (served by the docs site)                                                    |
| `demos/`      | Experimental applications and development showcases                                                                                          |
| `templates/`  | Starting points for new projects (`00_basic`, `01_spatial_ui`, `06_ai_query`, `13_typescript_vite`, ...)                                     |
| `docs/`       | Docusaurus manual + samples/templates pages                                                                                                  |
| `skills/`     | Portable consumer task skills for agents building XR Blocks applications                                                                     |
| `build/`      | **Generated** bundle output — do not edit                                                                                                    |

### `src/` subsystems

| Folder                                      | Responsibility                                                                                                 |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `core/`                                     | `Core` singleton, `Script`, `Options`, `User`, DI `Registry`, `XRButton`, WebXR session mgmt                   |
| `input/`                                    | controllers, hands, gaze, mouse, gamepad; `gestures/`; `headGestures/`; `strokes/`                             |
| `world/`                                    | `World` + `planes/`, `mesh/`, `objects/` (Gemini & MediaPipe backends), `sounds/`, `anchors/`                  |
| `context/`                                  | Agent-facing scene context: semantic tree, visible objects, Set-of-Mark screenshots                            |
| `depth/`                                    | depth sensing, depth mesh, `occlusion/` shaders & passes                                                       |
| `ai/`                                       | `AI` facade over `Gemini` + `OpenAI` (query / live / image gen)                                                |
| `agent/`                                    | agent framework: tools, memory, context (WIP)                                                                  |
| `ui/`                                       | core spatial UI: cards, overlays, panels, text, images, buttons, sliders, themes, `ModelViewer`                |
| `interaction/`                              | unified ray, gaze, mouse, direct-touch, selection, capture, and manipulation pipeline                          |
| `simulator/`                                | desktop XR simulator (manifest environments, physical objects, virtual user/hands/depth/planes, control modes) |
| `sound/`                                    | spatial audio, speech recognizer/synthesizer                                                                   |
| `physics/`                                  | Rapier3D integration                                                                                           |
| `lighting/`, `camera/`, `video/`, `stereo/` | light estimation, device camera, video streams, stereo utils                                                   |
| `utils/`                                    | `ModelLoader`, dependency injection, helpers                                                                   |

## Conventions

- **TypeScript**, strict. Public symbols re-exported via `src/xrblocks.ts`.
- **Tests**: colocated `*.test.ts`, run with Vitest.
- **Lint/format**: ESLint flat config ([`eslint.config.mjs`](eslint.config.mjs), with
  `eslint-plugin-tsdoc`) + Prettier ([`.prettierrc.json`](.prettierrc.json)).
- **Imports**: some intra-`src` imports carry `.js` extensions for module resolution — follow
  the surrounding file rather than "fixing" them ad hoc.
- **Dependency injection over globals**: prefer `static dependencies = {...}` on a `Script`
  (resolved via `core.registry`) to reaching into singletons, so subsystems stay testable.
- Keep PR diffs focused and add/adjust colocated tests for new branches.

## Adding a Feature / Subsystem

1. Add config to the relevant `*Options` class (or a new one) and register it in `Core.init()`.
2. Expose a chainable `enable*()` on `Options` if it is user-facing.
3. Wire the subsystem into `Core` and the frame loop / DI `Registry`.
4. Re-export public types from `src/xrblocks.ts`.
5. Add or adjust the smallest colocated `*.test.ts` proof and one executable
   sample or template when the public pattern needs teaching.
6. Update the one manual page or agent workflow that owns the changed concept.
   A new API capability does not require a new skill or blanket edits to every
   overview.

## AI & API-Key Security

> [!IMPORTANT]
> Never commit API keys, and never ship a key in client-side code for production. The `?key=`
> URL parameter and `keys.json` fallback in [`src/ai/AI.ts`](src/ai/AI.ts) are for **local
> prototyping only** — production apps must proxy AI calls through a server you control. AI
> features (Gemini Live/Flash) send data to Google servers; WebXR and on-device (LiteRT)
> features process data locally.

## Agent Docs & Skills

- [`CONTEXT.md`](CONTEXT.md) — compact contract for agents building apps with the SDK.
- [`docs/docs/manual/`](docs/docs/manual/) — canonical concepts, setup, behavior,
  limits, and examples.
- [`skills/`](skills/) — portable consumer task workflows using the Agent
  Skills directory format.
- [`skills/xb-contribute-sdk/`](skills/xb-contribute-sdk/) — repository
  complete-seam contribution workflow.

A skill is justified by an independently useful task process and distinct
invocation, not by a source subsystem. Exact public API truth remains in
[`src/xrblocks.ts`](src/xrblocks.ts), addon public entries, and source TSDoc.