# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project overview

`mtr-facemesh` is a TypeScript library that overlays 3D make-up (lipsticks, eye shadows, blush, foundation) on a face in real time, driven by either the webcam or a static image. It is consumed as a bundled ES module: `dist/lib/mtr_facemesh.js` exports the `FaceMesh` class plus a re-export of `THREE`.

## Build & development

```bash
npm run build       # production bundle: esbuild → dist/lib/mtr_facemesh.js (minified, ESM, es2020)
npm run watch       # same bundle, non-minified, rebuild on change
npm run make-dev    # copy dist/ → dev/ (for local serving from a separate path)
```

There is **no test suite, linter script, or formatter script wired in package.json**. `gts` and `prettier` are listed as devDependencies but are not run by any npm script — don't claim tests pass; there are none to run.

### Type checking

`esbuild` strips types without checking them, so nothing in the normal build catches a type
error. To check, you cannot use the pinned compiler: `npx tsc` **crashes** —
`TypeError: Cannot read properties of undefined (reading 'kind')` in `checkJSDocLinkLikeTag`,
because `typescript@4.8` chokes on the 217 files of `@types/three` that use `{@link}`. Use a
modern one instead, which needs no change to `package.json`:

```bash
npx -y -p typescript@5.9 tsc --noEmit --skipLibCheck
```

`--skipLibCheck` is required: `@types/node@18` against a modern `lib.dom` produces a wall of
`Buffer`/`AbortSignal` errors, and both `@types/stats` and `@types/stats.js` are installed so
`Stats` is a duplicate identifier. Expect **10 pre-existing errors under `src/`** even on a
clean tree, all of them implicit-`any`s in `utils.ts` — the `save` / `saveString` /
`saveArrayBuffer` / `exportGLTF` helpers, which are dead code: no callers in `src/` or
`demos/`, and not on the bundle's public surface (which is `FaceMesh`, `THREE`,
`DEFAULT_FRAME_LIGHT_CONFIG`, `DEFAULT_FRAME_LIGHT_PROBE_CONFIG`). Annotating them would
silence the checker without improving anything; deleting them is the open question. Judge a
change by whether it adds to that list.

The bundle is what every consumer (including the demos) imports. After editing anything under `src/`, you must rebuild for the demos to see the change. The committed `dist/lib/mtr_facemesh.js` is part of the published artifact, so build output changes belong in commits.

### Running the demos

Demos in `demos/*/` are static HTML pages that `import` from `../../dist/lib/mtr_facemesh.js`. They need an HTTP server (file:// won't load ES modules or the MediaPipe WASM). Use any static server from the repo root — e.g. `npx http-server .` — then open `http://localhost:8080/demos/photo/`.

`demos/glitter-studio/` is the glitter authoring bench: pick a palette, shape the
flakes, watch them on a tracked face, download the pair of PNGs, and copy the
command that reproduces them. Serve the **repo root**. Two traps it exists to get
right, both already handled: the canvases go into
`material.uniforms.glitterNormalMap.value` (never a property — same rule as the
eyeliner studio and `alphaMap`), and they need `RepeatWrapping` set explicitly,
because the glitter maps are the **only** ones in the library that tile. Also note
`--count` is an absolute flake count, not a density, so preview size must equal
export size or the preview is a different map; the panel warns in amber.

`demos/eyeliner-studio/` is the eyeliner authoring bench: paint a strip-space mask,
watch it on the tracked ribbon, tune the layer, download the PNG. It needs the **repo
root** served, not `demos/` — it `fetch`es `../../src/geometry.ts` and regex-parses
FACES/UVS/VERTICES out of it, the same way `tools/bake_eye_strip.py` does, because the
bundle does not export them. Its only two layers are the two ribbons, so the indices are
0 = upper, 1 = lower (`window.STUDIO_LAYERS` is the mapping and the demo addresses layers
through it). No `src/` change was needed to build it and none should be: the drawn texture
goes into `material.uniforms.alphaMap.value`, and the layer declares an `alphaMapDir` at
construction purely so `USE_ALPHAMAP` gets compiled.

Note that some demos point `binaryDir` at a jsDelivr CDN URL rather than the local `dist/lib`. If you've changed anything under `dist/lib` that the MediaPipe/TFJS runtime loads at runtime (binarypb, wasm), edit the demo's `config.js` to point at the local `./lib` path or you'll be testing against a stale CDN copy.

### Tooling (`tools/`)

Standalone helper scripts, not part of the bundle and not on any npm script.

**The step-by-step recipe lives in README.md, "Authoring an eyeliner"** — flags,
what `--emit mask` is for, and how to check the result. Read that rather than
re-deriving it; the two traps it documents (`alphaMapDir` reads GREEN so a colour
map multiplies `color` away, and `--stroke` must equal the layer's
`eyeStrip.strokeWidth`) are easy to hit twice.

The bake and the parametric generator now exist **twice**: as the two Python scripts
here and as `demos/eyeliner-studio/strip_bake.js`, a dependency-free ES module the
studio runs in the browser (and node runs headless). That is a deliberate duplication —
an artist should not need a shell — but it is only tolerable while the two agree, so
`python3 tools/check_strip_bake.py` diffs them over 18 flag combinations plus the ported
curve. Change one side, change the other, re-run it. Measured parity: `makeLinerStrip`
is **bit-exact** (it even reproduces the shipped `eye_liner_lower_mask.png` byte for
byte), `bakeAtlasToStrip` differs by at most **1/255 on ≤0.012% of texels** — float
accumulation order, because numpy means over the whole supersampled buffer at the end
while the JS accumulates per column — and the ported `CentripetalCatmullRom` matches
three's `CatmullRomCurve3('centripetal')` to **1.8e-15**. Do not chase the last bit;
do notice if that table gets worse.

- `tools/bake_eye_strip.py` — reprojects an atlas-space eyeliner map into the
  strip space used by `region: 'eyeLidUpper'` / `'eyeLidLower'`. Existing eyeliner art is painted in
  the canonical face-UV layout; the ribbon wants a flat strip, and this transfers
  one to the other so nothing has to be redrawn. Needs only numpy + Pillow.
  `python3 tools/bake_eye_strip.py <atlas.png> --smooth-u 6`
  `--emit mask` writes the shape into RGB for `alphaMapDir` instead of RGBA for
  `colorMapDir`; that is what lets a layer's `color` tint the stroke, since every
  eyeliner asset here has RGB = pure black inside the stroke and `colorMapDir` is
  multiplied as a vec4. `--region lower` bakes onto the lower lash line.
- `tools/make_liner_strip.py` — generates a parametric liner strip mask when there
  is no art to bake: thickness ramping from the inner canthus outward, soft far
  edge, tapering through the wing. Also mask form, for `alphaMapDir`.
- `tools/check_strip_bake.py` — the parity check described above. Shells out to both
  Python tools for the reference and drives `strip_bake.js` through node for the
  candidate; exits non-zero on drift. Needs numpy + Pillow + node, and
  `node_modules/three` for the curve case (skipped if absent).
- `tools/make_glitter_map.py` — generates the two paired maps a `glitter` layer
  needs (normal+coverage, hue+film thickness). **Uses no numpy and no `np.random`,
  on purpose**: it is duplicated as `demos/glitter-studio/glitter_gen.js` so an
  artist can try a palette in the browser, and PCG64 has no JS equivalent, so both
  sides run the same explicit mulberry32 and the Python inner loop is plain Python
  so its max-wins ordering can be mirrored. Do not vectorise it or restore
  `np.random` — either would silently divorce the studio from the tool.
- `tools/check_glitter_gen.py` — the parity check for that pair. **Tolerance is
  zero** (unlike `check_strip_bake.py`, which has a real accumulation-order
  difference): measured, all 13 cases are byte-identical and the JS reproduces the
  shipped 1024² PNGs exactly. It also checks the PRNG stream, `rint` and
  `hsvToRgb` head-on, because pixel diffing provably does not cover them — the only
  `rint` tie a real map produces is 127.5, where every rounding rule agrees, so a
  broken tie-break passed all 12 image cases until those probes were added.
- `tools/bisect_perf.sh` — attributes a frame-rate problem by measurement. Steps through
  a control (main's `src/` + demo config), four one-line probes (transmission,
  `autoLightIntensity`, the added layer, `willReadFrequently`) and the `src/` commits that
  affect a demo without `region` layers, rebuilding each and prompting for fps plus
  `renderer.info` counters. Refuses to start on a dirty tree; restores everything on exit,
  including Ctrl-C. `--list` to see the steps.
- `tools/smoke_eye_strip.mjs` — exercises the real `EyeStripGeometry` in node and
  prints the numbers the eyeliner design rests on (arc-length evenness, faceting,
  which side of the aperture the ribbon leaves on, and the texel-density invariant
  across a blink). Takes the bundle path and a region:
  `node tools/smoke_eye_strip.mjs /tmp/esg.mjs eyeLidLower`. Bundle the module
  with esbuild first; the header has the one-liner. Not a test suite — there still
  isn't one.

### Performance: what the demos already pay, before you add anything

Measured on this tree, and largely **pre-existing** — worth knowing before blaming a change:

- **Never call `getImageData` on `canvasVideo`.** Chrome permanently demotes an accelerated
  2D canvas to a software one after ~3 readbacks, on any cadence, and after that the
  `drawImage(video, ...)` that `animate()` does into it every 30 frames stops being a
  GPU blit and becomes a full CPU download of the frame: measured 0.21 ms -> 17.7 ms at
  1920x1080. Coinciding with `luminance()` every 60 frames it produced a **~40 ms hitch once
  a second**, which is the "laggy every ~1 s after ~6 s" report (the 6 s is just how long
  until a face is tracked and the readbacks start). `frameReadback()` exists to prevent
  this — it owns a downscaled `willReadFrequently` canvas and is the only readback path;
  `luminance()`, `exposure()` and `getFaceClosestColor()` all go through it.
- `luminance()` runs every 60 frames when `autoLightIntensity` is on, which it is by default
  whenever a demo has a `modelConfig` or any layer without `legacy: false` (see
  `needsSceneLights`) — ON in webcam/photo/vto/facetracker, OFF in controls. It and
  `exposure()` both go through `frameReadback()`, which now hands back the GPU-read copy
  (`grabFrameFromGpu`) rather than downloading the frame: 15.2 ms -> 0.15 ms per call.
  The catch is that the GPU readback is asynchronous, so **the copy is as fresh as how often
  you ask for it** — `exposure()` polled at 100 ms sees a ~100 ms old frame, the
  once-per-60-frames auto-exposure sees the frame from its previous tick. Pass
  `fullResolution: true` for pixels that are current and calibrated, which is what
  `getFaceClosestColor()` does.
  The CPU download it replaced is irreducible through the canvas API (a 1080p frame costs
  10.9 ms into a 96x96 software canvas and 18.0 ms into a 1920x1080 one — the destination
  size barely matters, the download dominates), so don't try to make it cheap; route around it.
- `FaceMesh.exposure()` samples all 478 landmarks. `demos/common/index.js` polls it on a
  `setInterval(1500)` in **every** demo; production polls it every **100 ms**, but only inside
  the "test your light" screen, never in the mirror. At 15.5 ms a call that was ~15% of wall
  clock on that screen, which is where routing it through the GPU readback pays most. It used
  to be 478 separate `getImageData(x, y, 1, 1)` calls; those were cheap (0.006 ms each) but
  they were the fastest way to trigger the demotion above.
- The frame light probe **used to be** the biggest single cost in `demos/webcam` — 20% of wall
  clock, 12.9 ms per call every `updateInterval` frames (default 3) — because
  `drawImage(this.video, ...)` into its `willReadFrequently` canvas paid that same 1080p
  download. It now reads the frame off the GPU instead (`FaceMesh.grabProbeFrame`: render the
  video texture into a `sampleSize²` target, `readRenderTargetPixelsAsync` it through a
  PIXEL_PACK_BUFFER + fence), which is **0.13 ms** on the main thread, so the probe is 1.9% of
  wall. Two things to know if you touch it:
  - Raising `updateInterval` was **never** the lever (measured at 12 on the old path: probe
    time 20% -> 5.7%, fps unchanged — the cost just moved to whoever next touched a software
    canvas). Don't reach for it.
  - `readPixels`' origin is bottom-left and `getImageData`'s is top-left, so `grabProbeFrame`
    flips the rows. Verify that empirically after any change — correlate the per-row luma
    profile against the canvas path both ways round (1.0 vs -0.63 when it is right).
  The probe is still only constructed when a layer sets `legacy: false`
  (`ensureFrameLightProbe` requires `frameLightMaterials.length !== 0`), so a legacy-material
  demo never pays even the 1.9%. `faceMesh.frameLightProbeEnabled = false` turns it off at
  runtime; the per-layer `frameLightEnable` uniform does NOT — that only silences the shader
  term.
- `transmission > 0` on any material makes three run `renderTransmissionPass` in each
  `renderer.render` call, but it has **no measurable effect** on frame rate here. Two
  independent tests: `demos/webcam`'s gloss layer zeroed from 0.9 gave 39.3 fps against a
  36.6-38.9 baseline, and `demos/controls` with three of its eight DynamicMaterial layers
  transmissive gave 56.2 fps against 55.8 with none (medians of three interleaved runs;
  `renderer.info.triangles` confirms the pass really runs, 2888 -> 4896). `demos/controls`
  had these zeroed as a perf fix in `16584a4` and they are restored — zeroing them also made
  that demo's gold shadow and liner all but invisible (12455 changed pixels, all in the eye
  region), so transmission there is load-bearing for the look, not decoration. Set it by what
  the look needs. Do not spend time on it as a perf lever before re-measuring.
- `fpsContainer.innerHTML = screenLogs.log()` runs **every frame regardless of
  `showScreenLogs`**, rebuilding 12 DOM nodes. `screenLogs` also does ~22 linear string
  scans and ~22 `new Date()` allocations per frame.
- Nothing sets `alphaTest`, so a make-up layer shades its whole face silhouette even where
  the mask is transparent. `demos/webcam`'s eyeliner layer blends 3.56% coverage that way.
- Textures are not shared: each `textureLoader.load` makes its own `Texture` and
  `THREE.Cache` is off, so `sombra_normal_map_glitter_v9.png` is resident **twice** in
  `demos/webcam` (16.8 MiB duplicated). `color_map_blushv7.png` is 4096^2 = 67 MiB decoded
  for a layer at `opacity: 0.12`.

One more, and it is the one that survived a full bisect: `demos/webcam/index.html`
**hard-requires `min: 1920 x 1080`** from `getUserMedia`. MediaPipe inference, the video
texture upload, the blur render targets, `drawImage` and `luminance` all scale with it. It
lives in `index.html`, so anything that only edits `src/` or `config.js` leaves it untouched.

`tools/bisect_perf.sh` walks all of these one at a time, rebuilding as needed and restoring
the tree on exit; `tools/frame_phases.js` is a console snippet that samples the library's own
per-phase timers for 12 s and reports mean / p95 / max, which is what actually localises a
hitch. Use them instead of reasoning from the diff — two confident diagnoses from reading the
code alone were wrong, and a 10-step bisect then showed the branch was not responsible at
all: `main`'s own webcam demo, built locally, stutters identically.

### Make-up alignment: the displayed frame must be the analysed frame

`animate()` samples the camera twice — MediaPipe reads the `<video>` element inside
`updateFaceMesh()`, and three's `VideoTexture` uploads it later, inside `renderVideoPlane()`'s
`renderer.render`. `VideoTexture` bumps its version from `requestVideoFrameCallback`, i.e.
whenever the browser decodes a frame, asynchronously. So without intervention the video plane
shows the newest frame available at RENDER time while the make-up is drawn from the frame that
was newest when INFERENCE STARTED, and the gap between them is however long inference took.
`animate()` now uploads the frame up front (`renderer.initTexture`) and restores
`videoTexture.version` after inference so the render reuses that upload.

Measured on the rendered output, against a face panned at constant speed with the direction
alternating (a temporal lag reverses sign with direction, a spatial error does not, so the
triangle wave separates them without differentiating anything):

| inference cost | before | after |
|---|---|---|
| ~15 ms (desktop) | 6.0 ± 0.3 ms | 3.1 ± 0.2 ms |
| ~55 ms | 33.4 ± 0.5 ms | 0.8 ± 0.3 ms |
| ~95 ms | 47.8 ± 1.3 ms | 4.4 ± 0.4 ms |

It tracks the inference cost 1:1, which is why it is invisible on a desktop and obvious on a
phone — the reported "the mesh is always late, it only grabs my face when I stop moving". The
**trade** is that the video is now as old as the frame that was analysed, so the mirror as a
whole sits one inference behind reality instead of the make-up sitting one inference behind the
video. Mutual alignment is what a viewer judges, but if total latency ever matters more, the
lever is inference cost itself — i.e. the camera resolution in `demos/webcam/index.html`.

**Every consumer of the frame has to be pinned, not just `this.videoTexture`.** Three places
made their own `VideoTexture` of the same `<video>`: `mtr_facemesh.ts` (the video plane),
`face_retouch_material.ts`, and `dynamic_material.ts` (one *per layer*). three keeps each one
up to date from its own `requestVideoFrameCallback`, so any left out of the pin shows whatever
frame was newest when its pass ran. That put the retouch mask 16 ms ahead of the video on this
desktop and 98 ms ahead with inference at 75 ms — while the video and the make-up stayed in
step, which is exactly how it was reported. `animate()` now pins everything `frameTextures()`
finds.

`DynamicMaterial` takes the shared texture (`config.videoTexture`), which also collapses
`demos/controls` from eight uploads of every camera frame to one: **31.8 -> 33.6 fps**, medians
of three interleaved runs with non-overlapping ranges. The retouch keeps its **own** texture on
purpose — it tags it `SRGBColorSpace` where the rest of the pipeline uses
`LinearSRGBColorSpace`, and that tag is not cosmetic: handing it the shared texture moved 4.6%
of the canvas across the whole face region. Pinning it in place changes nothing visible (0.22%,
against a 0.31% floor measured by rendering the *same build* in two sessions — always take that
control before attributing a pixel diff to a change; MediaPipe's landmark jitter and the
probe's EMA state alone move ~0.3% of the canvas).

Ruled out by measurement, so don't re-derive them: the `OneEuroFilter` is inert on the
`useNewMediapipe: true` path (`mtr_facemesh.ts:2728` — the filter is in the `else` arm), there
is no other temporal smoothing anywhere in `src/`, MediaPipe itself adds none (the landmark
amplitude matches the frame's to within 3% at 0.5, 2 **and** 4 Hz, where any low-pass would
damp progressively), and the `<video>` element is never added to the DOM so the browser is not
compositing it separately. A residual **spatial** error remains: the make-up sits ~8 px off on
average with ~14 px of spread across a 1000 px sweep at 1920 wide, direction-independent and
unaffected by the above. That one is unexplained.

Two traps when measuring this yourself:

- The library's own phase timers use `new Date().getTime()`, so they are integer-millisecond
  and they cannot see GPU work or a canvas demotion. Wrapping the suspects with
  `performance.now()` is what localised both costs above. The `probe` timer no longer covers
  the probe's readback either — that lands in a promise continuation outside the rAF callback.
- To separate a temporal lag from a spatial error, pan the subject at **constant speed with
  alternating direction** and difference the two directions at matched positions. Differencing
  a sinusoid needs a velocity estimate, which aliases badly once the frame rate drops — an
  emulated slow device reported a peak velocity of 297 px/s where the truth was 880. And when
  emulating a slow device, put the added time **after** the real `detectForVideo`: a slower
  machine reads the frame and then computes, and busy-waiting first inverts the very skew you
  are trying to measure.
- Measure the **synchronous** part of an async call, not the part after the `await`. A bench
  that timed only the post-await work made `createImageBitmap(video, {resizeWidth: 96})` look
  like 0.12 ms; in the app the call itself blocks for 14.5 ms, i.e. no better than the
  `drawImage` it was meant to replace. That is why the probe went to `readPixels` instead.
- `faceMesh.landmarks` is **pre-allocated with 478 zero entries** at construction
  (`mtr_facemesh.ts:1322`), so `landmarks.length > 0` is true before any face is tracked.
  Gate on an actual non-zero coordinate, or you will measure a pipeline that is drawing no
  make-up at all. This machine also runs OBS; check `uptime` before trusting an fps delta
  under ~10%.

### WASM module (`wasm/`)

Optional native OpenCV pipeline compiled to wasm via Emscripten — produces `mtr_xr_face.wasm` consumed by `src/mtr_xr_face.js`. Build steps (from README):

```bash
source /path/to/emsdk/emsdk_env.sh
cd wasm/build && emcmake cmake . && emmake make
```

`wasm/build/CMakeLists.txt` hardcodes OpenCV paths under `/home/merligus/...` — adjust before building on another machine.

## Architecture

### One class does everything

`src/mtr_facemesh.ts` (~4000 lines) defines a single `FaceMesh` class that owns the entire pipeline: AI model loading, camera/video plumbing, three.js scene, the animation loop, photo/share UX, and the foundation-color matching algorithms. New features almost always land as new methods on this class plus a new field in `FaceMeshConfig`. Don't split it — downstream consumers depend on the single public surface and the bundled artifact path.

The pipeline per frame, in `animate()` → `updateFaceMesh()`:

1. Pull frame from `<video>` or `<img>`.
2. Run face landmarker (MediaPipe or TFJS — see "two backends" below) to get 478 keypoints.
3. Smooth landmarks with `OneEuroFilter` (one filter per coordinate) — **only on
   the old TFJS path**. The `useNewMediapipe: true` branch does no smoothing at all
   (`mtr_facemesh.ts:2522` sits inside the TFJS `else` arm), so `oneEuroFilterConfig`
   is inert on the path every demo now uses. The `oEuroData[478][3]` filter bank is
   built and never touched there.
4. Feed landmarks into `FaceMeshFaceGeometry` (`src/face_mesh_geometry.ts`) which mutates a `BufferGeometry` in place — vertices are the landmarks, UVs come from the canonical face-mesh UV table in `src/geometry.ts`. Then update any generated geometry in `this.regionGeometries` (see "Two kinds of geometry" below), which reads the positions this step just wrote.
5. Render with three.js using one material per make-up layer — `DynamicMaterial` when `legacy: false`, otherwise `MeshPhysicalMaterial` — plus separate retouch meshes using `FaceRetouchMaterial` (blur shader for skin smoothing / foundation tint).

### Two kinds of geometry

Most make-up layers share **one** `FaceMeshFaceGeometry`, instantiated exactly once
(`mtr_facemesh.ts:825`) and handed to every `new Mesh(...)`. Retouch meshes reuse it too.

A layer that sets `region: 'eyeLidUpper'` or `'eyeLidLower'` instead gets its own
`EyeStripGeometry` (`src/eye_strip_geometry.ts`): a per-eye ribbon rebuilt each frame from
the corresponding tracked lash line, carrying a rectangular parametrisation — `u` is arc
length along the lash line, `v` is offset away from the aperture over the authored stroke
width. The rings live in one `RINGS` table; one region per layer, so an upper and a lower
liner are two `meshConfigs` entries. Both eyes live in one geometry
because layers are addressed **by index** all over the library (`getMeshes()[i]`,
`setFrameLight(index)`, the demos' hardcoded GUI folders, and the retouch-tail split in
`animate()`), so one `meshConfigs` entry must stay one mesh. Region meshes are built inside
the `meshConfigs` loop so they land in the head of `this.meshes`, ahead of that split.

It exists because an eyeliner cannot work on the shared atlas: the lid band gets ~11 px of
atlas height for ~2 mm of skin, and the two defects below squash and smear the artwork
regardless of how it is painted. `region` requires `legacy: false` —
`MeshPhysicalMaterial` ignores the vertex `color` attribute unless `vertexColors` is set,
which would silently drop the ribbon's seam feather.

`tools/bake_eye_strip.py` converts atlas-space art to strip space so nothing has to be
redrawn, and `demos/eyeliner-studio/` does the same reprojection in the browser as part
of an authoring loop (draw, bake into the live material, download).

### Do not remove the `arrayExpand` eye block

`src/face_mesh_geometry.ts` has an `{index, under, sign, constant}` displacement table applied
to the landmarks each frame. Its eye entries (`sign: -0.5`, added in `83e9f6e`) move each lid
crease vertex to the **midpoint** toward the lash vertex under it, which halves the lid band
while the UV table stays put. Consequences worth knowing:

- It compresses every eye texture **exactly 2×** across the band, and stretches the band above
  it ~1.26×, putting a scale discontinuity through the top edge of any liner.
- It is nonetheless **load-bearing**: it is what keeps the shipped eye-shadow art off the
  sclera, and every calibrated config depends on it. Removing or retuning it moves that art.

So route around it rather than fixing it. `EyeStripGeometry` does: it consumes the crease ring
only as a single **averaged direction**, and since the hack scales every offset by the same
0.5 along its own direction, that mean direction is exactly invariant to it.

### Two face-detector backends, gated by `useNewMediapipe`

- `false` (default): old `@tensorflow-models/face-landmarks-detection` (MediaPipe-attention-mesh under TFJS). Uses local binaries from `binaryDir` and the TFJS backend (`webgl` / `wasm` / `cpu`).
- `true`: new `@mediapipe/tasks-vision` `FaceLandmarker` (WebGL-only inside MediaPipe Tasks). Fetches wasm from jsDelivr.

Both code paths exist in `updateFaceMesh()` and the model-loading block in `build()`. When changing landmark handling, update both branches or guard explicitly. The `meshConfig.legacy` flag selects between `MeshPhysicalMaterial` and `DynamicMaterial` per layer, independently of the detector backend — note it is **opt-out**: the test is `legacy === false`, so an absent flag means the legacy material.

Per-layer flags on `ThreeMeshConfig` are strongly preferred over global `FaceMeshConfig` flags; the flag is read exactly once, at construction, and thereafter per-frame code dispatches on `material instanceof DynamicMaterial` rather than re-reading config.

### Two cameras, ortho composited over perspective

The renderer holds an `OrthographicCamera` (for the background video plane and 2D HUD) and a `PerspectiveCamera` (for the face mesh and any 3D model). `setWindow` / `setWindowFactor` implement the "curtain" by scissor-clipping the perspective render. The `flipCamera` config mirrors the video plane and adjusts the geometry accordingly — flipping is not just a CSS transform; landmark X coordinates are reflected too.

### Colour management is OFF — check this before reasoning about colour

`src/mtr_facemesh.ts:126` runs `ColorManagement.enabled = false` at module scope.
That makes **every colour-space argument in the codebase a no-op**: with it off,
`setStyle(hex, LinearSRGBColorSpace)`, `setStyle(hex, SRGBColorSpace)`,
`new Color(0xhex)` and `Color.set(hex)` all produce the *same* linear values, so a
hex string and a hex number parse identically.

Verify that line before concluding anything about a colour bug. Measured with
three's own default (management **on**) the paths diverge a lot — `'#F2D4B5'`
gives 0.9490 linear against 0xF2D4B5's 0.9473, and `Color.set` gives 0.4851 where
`parseColor`'s string branch gives 0.7255 — which makes it very easy to "find" an
asymmetry that this codebase does not actually have. `parseColor` in `src/utils.ts`
carries a longer note on what re-enabling management would break.

### Materials & shaders

- `src/face_retouch_material.ts` — custom `ShaderMaterial` that blurs the video texture under an alpha mask to simulate foundation/skin retouch.
- `src/dynamic_material.ts` — the `legacy: false` material. A `ShaderMaterial` that masquerades as physical (`isMeshStandardMaterial`/`isMeshPhysicalMaterial = true`), so three's `refreshUniformsStandard/Physical` overwrites most of its uniforms from the plain material properties **every frame**. Consequence: drive anything three knows about via the property (`material.roughness`), and only write `uniforms.x.value` directly for genuinely custom uniforms (`reflectivity`, `normalScale`, `frameLight*`, `hasEnvMap`, `cubeUV*`). Its convention is **compile always, gate with a float uniform** — `USE_TRANSMISSION`, `USE_CLEARCOAT` and the env-map defines are unconditional so sliders never trigger a recompile. Do not re-gate them.
- `src/frame_light_probe.ts` — CPU luminance measurement of the camera frame (96×96 canvas readback): mean and p85 of a landmark region plus p95 of the whole frame. Global, one per `FaceMesh`, throttled. It calibrates the `frameLight` illumination inside `DynamicMaterial`, whose premise is that a photograph of a face is already a record of real light transport, so the light should be **read out of the frame** rather than reconstructed from it. The measured shading feeds `irradiance` and `radiance` — the BRDF's *inputs* — so every PBR parameter shapes it.
- `src/shaders/` — `DynamicMaterialShader` (the big one) and `HorizontalBlurShader` / `VerticalBlurShader` (two-pass gaussian/sobel feeding `blurredVideoTexture`, which is where every video-derived map comes from).

- **Glitter is a separate lobe, not a normal-map perturbation** — `glitter_lobe()` in
`DynamicMaterialShader.ts`, gated by the `glitterIntensity` float, added to `outgoingLight`
before the clearcoat attenuation. It samples two authored, *tiling* maps
(`tools/make_glitter_map.py` writes them as a pair: normal+coverage, hue+film thickness) and
feeds them `getIBLRadiance` + `frame_light_radiance` + a share of `irradiance`. There is
deliberately no direct-light term — with an env map bound the directional light is already
zeroed. It could not be done through the existing maps for three measured reasons: the
`normalMap` slot is `mix(mapN1, mapN2, 0.5)`'d with the video normal, there is one
`material.roughness` per layer (a flake needs ~0.06 inside a 0.8 film), and nothing in the
library supports a UV transform while the lips are 1.02% of the UV square. Three things worth
knowing before touching it: flakes are **metallised film**, so `glitterMetalness` defaults to
1 and F0 is the flake's own colour — built as a dielectric it came out ~11x too dim and
vanished; **mipmapping does not kill the sparkle**, which was the expected failure —
across a 16x flake-footprint sweep the total glitter energy stayed within 77-142%; and the
lobe is **additive plus optional occlusion**, because purely additive glitter cannot tint a
light base. On a white lipstick the flake contributes ~13 levels over a base at ~83, so the
flake's own hue (saturation 0.75) is outvoted 86/14 — `glitterOcclusion` lets it hide the
film instead, and that is the only thing that fixes it (measured colour lift +0.007 additive
-> +0.027 occluded). Note `glitterCoverage` scales the map's ALPHA, so a low value makes
every flake transparent rather than leaving fewer solid ones, and it caps occlusion by the
same factor — density belongs in the generator's `--count`. Read
README "Authoring a glitter" rather than re-deriving any of this.

**Two removed webcam-lighting attempts.** `envMap()` (whole frame → 360° equirect → PMREM) and then `DynamicEnvMap` (threshold for lights → frontal-cap equirect → PMREM, plus a screen-space `videoLight` term) both tried to *reconstruct* a light field from the camera. Both failed on the same premise: a webcam does not see the lights, it sees the subject — so the reconstructed "light" was largely the user's own lit face, re-lighting that same face. Don't reintroduce this shape. Static env maps (`envMapDir`, `loadEnvMap`) remain the way to get real IBL.
- The retouch path runs through an `EffectComposer` (`postprocessing/EffectComposer`) — the blur composer is a separate render target the retouch material samples from.

### Foundation-color matching (`getFaceClosestColor`)

Lives in `mtr_facemesh.ts` and uses helpers from `src/color_utils.ts` (RGB↔LAB, euclidean distance) and `skmeans`. There are multiple algorithms selected by `foundationMatchConfig.option`; the Python prototypes live in `skin_color/` (not part of the JS bundle). The optimization machinery (`optimizeParameters`, `simulatedAnnealing`, `adamOptimizer`, `renderMeshWithParameters`, `renderRetouchWithParameters`, `getDifferenceImage`) is used to fit material parameters to a target image — it renders the mesh into an offscreen canvas, diffs against the target, and iterates.

### What's vendored vs imported

- `src/html2canvas.js`, `src/profiler.js`, `src/mtr_xr_face.js` are vendored JS — not generated, not on npm. Edit in place.
- `src/mtr_xr_face.js` is the Emscripten-generated loader for the OpenCV wasm module; regenerated by the `wasm/` build.
- The big externals — `three`, `@tensorflow/tfjs-*`, `@mediapipe/*`, `skmeans`, `onnxruntime-web`, `html-to-image`, `stats.js`, `jsdom` — are all bundled into `dist/lib/mtr_facemesh.js` by esbuild. There is no separate runtime dependency to install for consumers.

## Configuration surface

`FaceMeshConfig` (defined at top of `src/mtr_facemesh.ts`) is the single entry point — the README documents the user-facing fields. When adding a config option, define it in the interface, default it in the constructor (around line 496+), and propagate to the relevant subsystem. Public getters (`getRenderer`, `getScene`, `getOrthographicCamera`, `getPerspectiveCamera`, `getGroup`, `getMeshes`) are the supported way for callers to reach into the three.js scene — keep them stable.