# mtr-facemesh
## Contents
- [Description](#description)
- [Usage](#usage)
- [Features](#features)
- [Authoring an eyeliner](#authoring-an-eyeliner)
- [Authoring a glitter](#authoring-a-glitter)
- [Demo](#demo)
- [WASM Compilation](#wasm)
## Description
The current repository is the mtr-facemesh project. The module projects 3D masks (in this case make ups) on a face from a video element detected by an AI model.
## Usage
In order to use the module, add the div with **id="facemesh-holder"** and set the width and height of the canvas to show.
```html
```
After that, you can create the FaceMesh object with the following.
```js
import {FaceMesh, THREE} from './lib/mtr_facemesh.js';
...
const config = {
divName: "facemesh-holder",
modelConfigs: undefined,
meshConfigs: [
{
alphaMapDir: './assets/imgs/face/mask_lips.jpg',
ambientOcclusionMapDir: './assets/imgs/face/ambient_occlusion.png',
normalMapDir: './assets/imgs/face/normal_texture_lips_3.png',
color: 0xff0000,
roughness: 0.25,
metalness: 0.1,
opacity: 0.3,
clearcoat: 0.5,
clearcoatRoughness: 1.0,
}
],
binaryDir: './lib',
refineLandmarks: true,
backend: 'webgl',
flipCamera: true,
verbose: true,
showFPSPanel: true,
showScreenLogs: true,
loadedCallback,
errorCallback,
cameraStatusCallback,
oneEuroFilterConfig: {
mincutoff: 0.005,
beta: 0.12,
dcutoff: 1,
},
faceRetouchConfig: {
retouchAlphaMask: './assets/imgs/face/facemesh_alpha_base_v4.jpg',
opacity: 1.0,
blurRadius: 10,
},
};
faceMesh = new FaceMesh(config);
```
Where config is an object with properties:
- `divName` HTML div ID of the AR holder. Default is facemesh-holder.
- `imageID` HTML image element ID that will replace the camera.
- `modelConfig` object of configuration for the 3D model:
- `dir`: 3D model file path. File must be .glb or .gltf. No default.
- `animationSpeed`: 3D model animation speed. (0, 1) for slower animations and animationSpeed > 1 for faster animations. Defaults to 1.
- `scale`: Sets how big the 3D model will be. Defaults to 1.
- `position`: position of the model considering camera is at {0, 0, 0}. Defaults to global {0, 0,-distance}.
- `rotation`: rotation of the model. Defaults to {0, 0, 0}.
- `loop`: if the model should loop. Default is true.
- `showMannequin`: show the head occluder to better position the object. Default is false.
- `meshConfigs` Array of configuration for each make up mesh:
- `legacy`: if set to false the layer renders through the custom DynamicMaterial
shader (video-derived relief normal and ambient occlusion, PMREM env map,
frame light probe) instead of three's MeshPhysicalMaterial. Note this is
opt-out: anything other than an explicit false uses the legacy material.
Default is true.
- `region`: render this layer on its own geometry, generated each frame from the
landmarks, instead of on the shared face mesh. `'eyeLidUpper'` and
`'eyeLidLower'` build a ribbon along the corresponding tracked lash line of
both eyes, with a rectangular parametrisation of its own: `u` is normalised
arc length along the lash line (0 at the inner canthus, `eyeStrip.cornerU` at
the outer one, 1 at the wing tip) and `v` is the offset away from the
aperture, 0 at the lash margin and 1 at the far edge of the stroke. Artwork is
therefore authored as a flat strip — see
`demos/assets/imgs/face/eye_strip_uv_template.png` — rather than as a sliver
of the canonical face atlas.
One region per layer. An upper and a lower liner are two `meshConfigs`
entries, which keeps one entry = one mesh and lets each carry its own artwork
and stroke width; a lower liner is normally thinner and gets no wing by
default.
Note the shape must reach the shader through `alphaMapDir`, not
`colorMapDir`, if you want `color` to tint it. `colorMapDir` is multiplied as
a vec4, so a map whose RGB is black inside the stroke — which is true of every
eyeliner asset in this repo — multiplies `color` away. `alphaMapDir` is read
on the green channel; `tools/bake_eye_strip.py --emit mask` and
`tools/make_liner_strip.py` both write that form.
This is what an eyeliner needs, because on the shared face mesh a liner
inherits two defects that no amount of UV work removes. The `arrayExpand`
table in `src/face_mesh_geometry.ts` moves each lid crease vertex to the
midpoint toward the lash vertex under it while the UV table stays put, which
compresses the artwork exactly 2x across the band; and the band's 3D area
swings about 2.6x during a blink against a fixed UV area, which smears the
artwork down the lid. Both live in a free scale factor that defining `v`
against a pose-invariant length removes.
Requires `legacy: false`: `MeshPhysicalMaterial` ignores the vertex `color`
attribute unless `vertexColors` is set, which would silently drop the seam
feather. The layer is skipped with a `verbose` note otherwise. No default.
- `eyeStrip`: tuning for `region: 'eyeLidUpper'`. All fields optional.
- `columns`: spine samples per eye, wing included. Sets the residual
faceting: 48 gives 36 samples over the eye and about 3.5 deg of turn per
segment, against 14.8 for the 8-segment landmark polyline. Default is 48.
- `rows`: samples across the band. Default is 4.
- `cornerU`: `u` of the outer canthus; everything above it is the wing. It is
snapped to the nearest real column and the snapped value is readable back
off the geometry. Default is 0.75.
- `strokeWidth`: band width as a fraction of eye width (inner canthus to
outer canthus). Eye width is pose-invariant, so this is a constant physical
stroke: 0.11 is about 2.1 mm on an adult. Default is 0.11.
- `wingLength`: wing length, also a fraction of eye width. Default is 0.2.
- `wingLift`: wing direction, 0 straight out along the lash tangent, 1 fully
up the lid. Default is 0.5.
- `lift`: offset toward the camera along the surface normal, as a fraction of
eye width. Parallax only — depth correctness comes from `depthWrite: false`
plus render order. Default is 0.02.
- `foldOpenFrac`: fraction of the authored stroke visible with the eye wide
open. Default 1.0, i.e. off. Lid skin rolls under the supratarsal crease as
the eye opens, so lowering this makes a blink *reveal* more of the stroke
instead of stretching a fixed range over more skin. The direction is
established; the magnitude is not, so it ships off — `v` = 1 is the band
exposed at the canonical rest pose, which is the open-eye reference
`tools/bake_eye_strip.py` targets, and a value below 1.0 hides art the bake
placed. Tune it live in `demos/controls` against a real blinking face.
Width and `v` range always scale together, so skin offset =
`v * strokeWidth * eyeWidth` holds at every lid state. Leave it at 1.0 on
`'eyeLidLower'`: the fold is an upper-lid phenomenon, the lower lid barely
moves during a blink, and the aperture signal cannot tell the two apart.
- `foldAttackMs` / `foldDecayMs`: time constants of the running open-eye
reference that normalises the aperture across people — fast to learn a
wider eye, slow to forget one, so a blink never drags it down. Defaults 150
and 8000.
- `seamFeather`: alpha ramp at the ribbon's free boundaries, as a fraction of
the band (for `v` = 1) and of the strip length (for the two ends). The lash
margin at `v` = 0 is never feathered. Set to 0 if the artwork already fades
at its edges. Default is 0.12.
- `colorMapDir`: texture file path. File must be an image. No default.
- `alphaMapDir`: alpha mask file path. File must be an image. No default.
- `ambientOcclusionMapDir`: illumination mask file path. File must be an image. No default.
- `normalMapDir`: normal map file path. File must be an image. No default.
- `color`: make up color in hexadecimal. Default is 0xffffff.
- `metalness`: how much the material is like a metal. Wood or stone use 0.0, metallic use 1.0. Default is 0.0.
- `opacity`: float in the range of 0.0 - 1.0 indicating how transparent the material is. Default is 1.0.
- `clearcoat`: use clear coat related properties to enable multilayer materials
that have a thin translucent layer over the base layer. Default is 0.0.
- `roughness`: represents the intensity of the clear coat layer, from 0.0 to 1.0. Default is 0.0.
- `clearcoatRoughness`: roughness of the clear coat layer, from 0.0 to 1.0. Default is 0.0.
- `ior`: index of refraction, which sets how reflective the layer is —
`F0 = pow2((ior - 1) / (ior + 1))`, so the default 1.5 gives the usual 4%
dielectric reflectance and raising it reflects more. Default is 1.5.
- `reflectivity`: **inert** — the uniform exists but no shader line reads it. It
is still accepted so existing configs keep working unchanged. Use `ior` for how
reflective the layer is, and `specularIntensity` to scale it.
- `envMapDir`: static HDR environment map file path for this layer, giving it
image-based lighting. No default.
- `normalScale`: strength of the combined video-derived and asset normal
perturbation. Lower it toward 0 to flatten the relief and calm reflection
noise at low roughness. `legacy: false` only. Default is 1.0.
- `frameLight`: if set, the camera frame itself becomes the light source for
this layer: the real per-pixel shading in the frame is fed to the material as
irradiance and radiance, so roughness, metalness, IOR and clearcoat shape the
measured light exactly as they shape environment-map light. `legacy: false`
only. Merely being present switches the illumination on, so `frameLight: {}`
is enough — every field below then takes its value from
`DEFAULT_FRAME_LIGHT_CONFIG` (exported by the library), which is the single
source of truth for these defaults. Omit `frameLight` entirely to leave the
illumination off.
Because the probe replaces the scene lights rather than supplementing them,
the defaults are calibrated with the directional and ambient intensities at 0
(see `autoLightIntensity` below).
Each field has exactly one job, listed below with what it acts on. Changing one
should never alter another's territory.
- 'irradiance' (diffuse): ambient level taken from the frame. The frame's own
per-pixel shading, normalised by the measured region mean so it is independent
of skin tone and exposure. This is the term that lights the layer; it adds to
the ambient light, so zero the scene lights to see the probe alone.
Default is 2.64.
- 'gain' (specular): overall highlight brightness. A plain monotone multiplier
over the whole highlight — it does not move or resize it. Default is 1.5.
- 'hdr' (specular): highlight contrast, i.e. how much brighter the peak is than
the rest of the lit region. 0.0 lights the region evenly. It shapes the
brightness profile and never the extent. Default is 4.0.
- 'region' (specular): how much of the make-up is highlighted, in units of the
measured mean-to-p85 range. 1.0 means only pixels above the p85, which is
already a very small core; useful values sit below that, and 0.0 highlights
everything above the mean. Default is 0.6.
- 'softness' (specular): edge softness of that region, in the same units. The
region's centre does not move as this changes. Default is 0.35.
- 'detail' (specular): amount of fine real specular sparkle — the high-frequency
texture the broad region averages away. Added on top of the region rather than
traded against it, so the two are independently visible. Default is 0.35.
- 'radius': radius of the local average that separates the fine detail from the
broad region, in x UV units (scaled by aspect in y). Too small and 'detail' is
only sensor noise; this cannot reuse the shared blurred video texture, which is
about 1.6 px because it feeds the relief normal. Requires
`gaussianSobelSmoothing` to be negative (a blur). Default is 0.0135.
- 'sheer' (albedo): how much of the real lip or skin chroma shows through the
make-up colour. Hue only — the brightness already arrives as irradiance, so
this does not double-count the shading. 0.0 is opaque. Default is 0.5.
Only 'irradiance' and 'sheer' are calibrated values; the specular defaults are
starting points. `demos/controls` exposes all of them live, plus a **DEBUG view**
that replaces the layer's output with one intermediate term (the shading field,
the region mask, the detail, the irradiance or the full radiance) so a single
parameter can be checked in isolation.
- `binaryDir` Path to the face mesh binary files. No default.
- `refineLandmarks` Refine landmarks option. Default is true.
- `backend` Tensorflow backend: webgl, wasm or cpu. Default is webgl.
- `flipCamera` Flips the feedback camera. Default is false.
- `verbose` If set to true shows logs on the console. Default is false.
- `loadedCallback` Function to call after the AI model is loaded. No default.
- `cameraStatusCallback` Function to call after the video permission (no default):
- 'requesting': when user is requested for the camera
- 'hasStream': when user is granted access for the camera
- 'failed': when something went wrong with the camera
- `errorCallback` Function to call after any error. No default.
- `showFPSPanel` If set to true shows the FPS on the main loop. Default is false.
- `showScreenLogs` If set to true shows the FPS for every step on the main loop. Default is false.
- `oneEuroFilterConfig` One Euro Filter parameters.
- 'mincutoff': minimum cutoff frequency. Decreasing the minimum cutoff frequency decreases slow speed jitter. Default is 0.001.
- 'beta': speed coefficient. Increasing the speed coefficient decreases speed lag. Default is 0.007.
- 'dcutoff': cutoff frequency. Default is 1.
- `faceRetouchConfig` face retouch parameter (one object).
- 'retouchAlphaMask': path to the alpha mask file. Only apply the retouch on the mask. No default.
- 'blurRadius': Face retouch intensity. Default is 10.
- 'color': Foundation color. Default is 0xffffff (white).
- 'opacity': Foundation opacity. Default is 0.0 (transparent).
- `faceRetouchConfigs` face retouch parameters array (multiple face retouch objects). Each object of the array must contain:
- 'retouchAlphaMask': path to the alpha mask file. Only apply the retouch on the mask. No default.
- 'blurRadius': Face retouch intensity. Default is 10.
- 'color': Foundation color. Default is 0xffffff (white).
- 'opacity': Foundation opacity. Default is 0.0 (transparent).
- `foundationMatchConfig` If set, executes the foundation match algorithm after togglePhoto method.
- 'option': tone algorithm option.
- 'colors': array of foundation colors in RGB and its names.
- `gaussianSobelSmoothing` Smoothing applied to the video before it is used to
derive the relief normal, the ambient occlusion and the frame light probe's
specular residual. Negative is a blur, positive is a sharpen. The residual needs
a blur, so keep this below 0 when using `frameLight`. Default is -2.0.
- `frameLightProbeConfig` Overrides for the frame light probe, the CPU luminance
measurement that calibrates `frameLight`. Omit to take
`DEFAULT_FRAME_LIGHT_PROBE_CONFIG` (exported by the library), which is the single
source of truth for the defaults.
- 'sampleSize': side of the square offscreen canvas read back. Default is 96.
- 'updateInterval': frames between readbacks. Default is 3.
- 'regionRungs': outer/inner landmark pairs defining the measured region.
Defaults to the 22 lip rungs.
The scene's directional and ambient lights are switched on or off automatically at
construction. They exist for the legacy lighting path, so they are **on** when any
mesh is legacy (`legacy !== false`) or a `modelConfig` is present, and **off** —
both intensities parked at 0 — when every layer is `legacy: false` and therefore
lit by the frame light probe and/or an env map. They are shared by the whole scene,
so a single legacy layer anywhere keeps them on for everything. Set
`autoLightIntensity` to override.
While the flag is true, **luminance** re-derives both intensities from frame
brightness every 60 frames. To hold your own values, clear the flag first inside
**loadedCallback**, otherwise they are overwritten within a couple of seconds:
```js
faceMesh.autoLightIntensity = false;
faceMesh.directionalLight.intensity = 0.0;
faceMesh.ambientLight.intensity = 0.0;
```
The method **requestPermissions** must be called after the model is loaded using the **loadedCallback** config property, since the face detector only starts predicting after the video is set. The method receives the camera configuration as a parameter, the same as seen at [getUserMedia](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia). Inside **loadedCallback** you must also add lights to the scene since the textures are very sensible to it. The following code is an example of one **loadedCallback**.
```js
const loadedCallback = function () {
scene = faceMesh.getScene();
const hemiLight = new THREE.HemisphereLight(0xffffff, 0x080820, 0.5);
scene.add(hemiLight);
const ambientLight = new THREE.AmbientLight(0x404040, 0.1);
scene.add(ambientLight);
faceMesh.requestPermissions({
facingMode: {ideal: 'user'},
width: window.innerHeight,
height: window.innerWidth,
});
};
```
With these lines of code it's already possible to run the face mesh. If you want to personalize the threejs scene, you can do it by using the get methods `faceMesh.getRenderer()`, `faceMesh.getScene()`, `faceMesh.getOrthographicCamera()`, `faceMesh.getPerspectiveCamera()`, `faceMesh.getGroup()` and `faceMesh.getMeshes()`. For example, you can change material params with `faceMesh.getMeshes()[0].material.color = new THREE.Color(0xff0000);`.
## Features
The AR object also include other functionalities.
### Partial Render
The partial render works like a curtain for the threejs renderer and only renders what is from 0 to `windowSlider.value * canvas.width`. For this, you need to create a checkbox and a slider from 0 to 1 and then use them as follows.
```js
const windowCheckbox = document.getElementById('window-checkbox');
const windowSlider = document.getElementById('window-slider');
windowCheckbox.oninput = function () {
faceMesh.setWindow(windowCheckbox.checked);
windowSlider.style.display = windowCheckbox.checked ? 'block' : 'none';
};
windowSlider.oninput = function () {
faceMesh.setWindowFactor(windowSlider.value);
};
```
### Photograph
The photograph functionality is simple and works like it is said. When the method **togglePhoto** is called, the animation loop freezes/unfreezes depending on the actual state. You only need to create on html and use the following code. The method also returns a promise of a canvas of the photograph.
```js
const screenshotButton = document.getElementById('photo-button');
screenshotButton.onclick = async function () {
const frozenVideoCanvas = await faceMesh.togglePhoto();
};
```
### Share
The next feature complements the photograph feature. When the **share** method is called, the object takes a screenshot and then share it through the selected social media. For that, you only need to create a button to trigger the method.
```js
const shareButton = document.getElementById('share-button');
shareButton.onclick = function () {
faceMesh.share('screenshot');
};
```
### Stop and Continue
The application can be stopped or continued after calling **.stop()** and **.continue()** methods. They are useful when you want to stop the application with a close button or start the application instantly after the user clicks a start button. The demo shows an example of usage.
```js
const closeButton = document.getElementById('close-button');
var toggleCloseContinue = true;
closeButton.onclick = function () {
if (toggleCloseContinue) {
faceMesh.stop();
closeButton.textContent = 'Continue';
} else {
faceMesh.continue();
closeButton.textContent = 'Close';
}
toggleCloseContinue = !toggleCloseContinue;
};
```
### Foundation Color
If `faceRetouchConfig` and/or `faceRetouchConfigs` are set, it is possible to set a color on the users face as if there is a foundation using **.setFoundation** method. The function receives two parameters, color and opacity. The `color` is a number array with 3 positions in RGB space, and the `opacity` is a floating point number between 0.0 and 1.0 that is the strength of the color. If there are multiple face retouch objects, you can set the `color` and `opacity` with the third parameter `index`. The following example will make up the face with red foundation.
```js
faceMesh.setFoundation([255, 0, 0], 0.5, 0);
```
### Light Exposure
Light exposure is a feature related to face color match since it detects the illumination on the users face. If the user face is under or over exposed, the `faceMesh.exposure()` will return `-2` and `2` respectively.
```html
```
```js
function anim() {
document.getElementById(
'exposure-log'
).textContent = `${faceMesh.exposure()}`;
}
setInterval(anim, 500);
```
### Face Color Match
To use it, call the method `.getFaceClosestColor()` to get the object of closest color. This feature matches the users face color to the closest color in `foundationMatchConfig.colors` and return an object with attributes `foundationID`, `skinColor` and `distances`.
- `foundationID` is the index of the best foundation ID in `foundationMatchConfig.colors` array;
- `skinColor` is the extracted color of the user skin in RGB space;
- `distances` This attribute is also an object with attributes `distance` and `index`. The `index` is the index to the `foundationMatchConfig.colors` array and the `distance` is the euclidean distance from `foundationMatchConfig.colors[foundationID]` to `foundationMatchConfig.colors[distances[..].index]`. This array is sorted in ascending order by the `distance` attribute.
```js
FaceMesh(
...,
foundationMatchConfig: {
option: 1,
colors: [
{color: [203, 176, 145], name: '00'},
{color: [169, 129, 83], name: '50'},
{color: [77, 52, 48], name: '100'},
]
}
)
```
```js
window.afterScreenshot = function afterScreenshot() {
// matches the face color with one of the 3 colors set in foundationMatchConfig
const distances = faceMesh.getFaceClosestColor().distances;
// second best color distance, and index
console.log(distances[1].distance, distances[1].index);
const foundationId = faceMesh.getFaceClosestColor().foundationID;
console.log(window.foundations[foundationId], foundationId);
// uses the setFoundation feature to set the matched color on faceRetouch as a foundation
faceMesh.setFoundation(window.foundations[foundationId].color, 0.3, 0);
```
### Using an image instead of camera
If faceMesh is created with `imageID` set, the video camera will be replaced with the image loaded on html with `id=` and **.requestPermissions()** method will not be necessary to call. For example, an image tag is loaded with id "photo-id", and faceMesh is created with it replacing the camera.
```html
```
```js
faceMesh = new FaceMesh({
imageID: "photo-id",
...
});
```
## Authoring an eyeliner
A `region: 'eyeLidUpper'` / `'eyeLidLower'` layer takes its artwork as a flat
**strip**: `x` is arc length along the lash line, `y` is offset across the lid, and
`v = 0` — the lash margin — is the **bottom** row of the image. You do not have to
draw in that space; if the art already exists in the canonical face-UV atlas, one
command reprojects it.
There are two routes to a strip: the studio, in the browser, or the two Python
tools at a prompt. They do the same arithmetic — `tools/check_strip_bake.py`
diffs them — so pick whichever suits the person doing the work.
### The studio (`demos/eyeliner-studio/`)
```sh
npx http-server . # from the repo root, so /src is reachable
# then open http://localhost:8080/demos/eyeliner-studio/
```
Three panels: paint the strip on the left, watch it on a tracked face in the
middle, tune the layer on the right. **Bake → face** pushes the drawing into the
running material with no file round trip (and *auto after each stroke* does it
for you), so a taper can be judged on a blinking eye rather than guessed at.
- **Import** takes either space. A wide image is loaded as a strip; a squarish one
is treated as canonical-atlas art and reprojected on the way in — the same bake
`tools/bake_eye_strip.py` does, using whatever `strokeWidth`, `cornerU` and
`wingLength` the ribbon is currently set to. Both mask conventions are
detected: an image with no transparent pixel is read as a mask (shape in RGB),
anything else as art (shape in alpha).
- **Generate a base stroke** is `tools/make_liner_strip.py` in the page, for when
there is no art to start from.
- The **guides** are read live off the geometry, and they are the point of doing
this in a browser at all: `v` labelled in millimetres of skin, where the outer
canthus falls, which bands the ribbon feathers away, and — the one that catches
people — the `foldOpenFrac` line, above which nothing shows while the eye is
open.
- **Download strip PNG** writes the same file the Python tools write, and **copy
config snippet** gives you the `meshConfigs` entry with the `strokeWidth` the
art was actually drawn against, which is the trap below made survivable.
The paint canvas is 1024×128 whatever it is displayed at, so the pointer maps
straight to strip texels; *zoom v* only stretches the view, because a strip shown
at its true 8:1 is unpaintable.
### Bake existing atlas art into a strip
```bash
python3 tools/bake_eye_strip.py demos/assets/imgs/face/color_map_eye_liner.png \
--emit mask --smooth-u 6 \
--out demos/assets/imgs/face/eye_liner_upper_mask.png
```
Needs only `numpy` and `Pillow`. Then point a layer at it:
```js
{
legacy: false, // required with region
region: 'eyeLidUpper',
alphaMapDir: '../assets/imgs/face/eye_liner_upper_mask.png',
color: 0x14100f, // this is what tints it
envMapDir: '../assets/imgs/face/studio2.hdr',
eyeStrip: {strokeWidth: 0.11}, // MUST match the bake's --stroke
}
```
The flags that matter:
- **`--emit mask`** — almost always what you want. It writes the shape into RGB and
makes the image opaque, which is the form `alphaMapDir` needs, because the shader
reads `alphaMap` on the **green** channel. Without this you would reach for
`colorMapDir`, which is multiplied as a **vec4** — and every eyeliner asset in
this repo is pure black inside the stroke, so it would multiply the layer's
`color` away and the stroke would be black whatever you set. Use the default
`--emit color` only for art whose RGB you actually want.
- **`--region lower`** — bakes onto the lower lash line instead. Must match the
layer's `region`.
- **`--stroke F`** — where `v = 1` lands, as a fraction of eye width. Default 0.11
(about 2.1 mm). **It must equal the layer's `eyeStrip.strokeWidth`**, or `v` means
two different things in the art and in the geometry.
- **`--smooth-u 6`** — optional. The atlas assets here carry a 1–13 texel stroke, so
their edge is quantised to whole texels and that transfers as a ~70 µm wobble
along the top edge. It is not artistic intent — a synthetic smooth source bakes
with a measured 0.0 µm wobble. Leave it off (`0`) for a strictly faithful bake.
- **`--eye average|left|right`** — `average` (default) cancels left/right asymmetry
in the source. The art in this repo is genuinely asymmetric: 15.6% of pixels
differ by more than 32 alpha between the two eyes.
It prints where the ink landed, which is the number to sanity-check:
```
ink occupies v 0.000..0.656 (0.00..1.40 mm from the lash margin on a 19.4 mm eye)
ink occupies u 0.055..0.976 (outer canthus at u=0.745, wing beyond it)
```
If the stroke renders thinner than expected, that report is the first place to
look — a thin band of ink inside a wide `strokeWidth` band renders thin, and
raising `strokeWidth` only buys more empty band.
### Or generate one parametrically
With no art to bake:
```bash
python3 tools/make_liner_strip.py --region lower \
--out demos/assets/imgs/face/eye_liner_lower_mask.png
```
Also mask form. `--start-u` / `--end-u` bound the stroke along the lash line,
`--thick-inner` / `--thick-outer` ramp its thickness, `--edge` softens the far edge.
Note `--end-u` defaults to the outer canthus on the lower lid: with `wingLength: 0`,
the lower-lid default, every column past the canthus collapses onto it, so ink
painted out there is never rendered.
### Checking the result
Load `demos/controls/`, set `USE_STRIP_TEMPLATE = true` in its `config.js`, and both
ribbons render `eye_strip_uv_template.png` instead. The 32 ticks should be evenly
spaced along the eye, the diagonal should read as one smooth curve, the band should
hold its width through a blink, and the green bar should sit on the outer canthus.
Check that before judging artwork — a geometry problem otherwise reads as an art
problem.
## Authoring a glitter
Glitter is a **separate specular lobe**, not a bumpier make-up layer. Any
`legacy: false` layer can carry it — lipstick, blush, eye shadow, an eyeliner
ribbon — by adding a `glitter` block to its `meshConfigs` entry. It is off unless
asked for: `intensity` defaults to 0 and the shader skips the whole block.
### Why it is not just a map
The three reasons a flake field cannot be expressed through the existing maps,
all measurable in this tree:
- `normalMapDir` is blended **50/50** with the video-derived normal
(`DynamicMaterialShader.ts`, `mapN = mix(mapN1, mapN2, 0.5)`), so authored
flake normals arrive at half strength, and `normalScale` scales both together.
- There is **one `material.roughness` per layer**. A flake is a near-mirror
(~0.06) suspended in a matte film (~0.8). One scalar cannot be both.
- Every sampler reads bare `texture2D(map, tUV)` — **no UV transform anywhere** —
and the lips are only **1.02% of the UV square** (78 of 880 triangles, 483x177
px at 2048²). Flake density would be capped by atlas resolution. `glitter.scale`
is the one place in the library that tiles a map.
### The studio (`demos/glitter-studio/`)
Pick a palette, shape the flakes, watch it on a tracked face, download the two
PNGs. Nothing touches disk until you press Download.
```sh
npx http-server . # from the repo ROOT
# then open http://localhost:8080/demos/glitter-studio/
```
The generator runs in the browser as `glitter_gen.js`, a dependency-free port of
`tools/make_glitter_map.py`. The panel prints the command line that reproduces
exactly what you downloaded, plus the `glitter` block to paste into a layer —
the studio is for **finding** the numbers, the tool is for reproducing them in a
build.
That duplication is only tolerable while the two agree, so
`python3 tools/check_glitter_gen.py` diffs them. Change one side, change the
other, re-run it. Unlike `check_strip_bake.py` the **tolerance is zero**: there
is no supersample collapse here, just a fixed sequence of IEEE double operations
both languages perform identically. Measured: all 13 cases byte-identical, and
`glitter_gen.js` reproduces the shipped 1024² `glitter_multi` PNGs exactly.
Agreeing at all cost numpy. `default_rng` is PCG64 with no JS equivalent, so
**neither side uses its language's RNG** — both run the same explicit mulberry32
and draw the same seven values per flake in the same order. Don't "simplify" the
Python back to `np.random`; it would silently divorce the studio from the tool.
The checker also compares the PRNG stream, the rounding and the HSV conversion
head-on, because pixel diffing does not cover them: the only `rint` tie a real
map produces is 127.5, where every rounding rule agrees, so a broken tie-break
passes every image case.
**The frame light is on a preset dropdown in both glitter demos**, because it is
the thing most likely to bury the flakes. The studio opens on a `calm` preset —
its job is judging flakes, not judging a look — while `demos/glitter` opens on
the validated `tuned` values; each has the other one click away, plus
`calibrated (demos/controls)` and `off`.
If the illumination runs away, look at the read-only **`measured by the probe`**
block before touching a slider. Every specular term in `frame_light_radiance` is
`(luma - mean) / max(hi - mean, 0.02)`, so the probe's two measurements are the
input the sliders merely scale, and `hdr` and `detail` both multiply a value that
is **deliberately not clamped at the top**. A `hi` sitting almost on top of `mean`
means the camera is seeing a flat lip region and the 0.02 floor, not your slider,
is deciding the result. `probe running` freezes the measurement, which is the only
way to tell a jumpy reading apart from a bad parameter. Debug views 4 and 5 show
the irradiance and radiance terms on their own.
One thing to know before judging density: **`--count` is an absolute number of
flakes, not a density.** Flake positions scale with the tile while the radius
stays in pixels, so the same count at 2048² spreads over four times the area at a
quarter of the coverage. Preview size therefore defaults to the export size, and
the panel turns amber when you make them differ.
### Generate the two maps
They are written together and paired **texel by texel**, so they must always be
used and swapped as a pair:
| file | RGB | A |
|---|---|---|
| `_normal.png` | tangent-space flake normal, `(128,128,255)` between flakes | flake coverage |
| `_color.png` | that flake's base hue | that flake's film thickness, 0..1 |
```sh
python3 tools/make_glitter_map.py --out-prefix demos/assets/imgs/face/glitter_festa --size 1024 --count 6000 --radius-px 3.0 --slope 0.75 --palette '#ff3d8b,#ffd23d,#3dffd2,#8b3dff,#ff5f3d,#3dff7a'
```
`--palette random` spreads saturated hues around the wheel instead. The map
tiles: flakes crossing an edge are redrawn on the opposite side, so a 1024 tile
repeated a few times beats a 4096 one-shot at 1/16 the memory. Needs numpy +
Pillow, nothing else.
### Wire it to a layer
This is the look `demos/glitter` ships, tuned on a live camera rather than
assembled from defaults — start here rather than from the defaults:
```js
{
legacy: false,
alphaMapDir: '../assets/imgs/face/mask_lips.jpg',
envMapDir: '../assets/imgs/face/studio2.hdr',
// ior 1.0 puts the base's F0 at exactly 0, so it has NO dielectric specular
// and all its shine comes through the metallic path. That is what stops the
// film competing with the flakes for the same highlight — the pair of
// `ior: 1.0` + `metalness: 0.72` go together.
color: '#f66151', opacity: 0.37, roughness: 0.6,
metalness: 0.72, ior: 1.0, clearcoat: 0, envMapIntensity: 0.68,
frameLight: {
irradiance: 2.64, gain: 1.16, hdr: 2.86, region: 0.66,
softness: 0.44, detail: 0.93, radius: 0.021, sheer: 0.5,
},
glitter: {
normalMapDir: '../assets/imgs/face/glitter_multi_normal.png',
colorMapDir: '../assets/imgs/face/glitter_multi_color.png',
intensity: 4.0,
coverage: 0.9,
scale: 2.9, // how many times the tile repeats over the layer's UV
normalScale: 1.43, // flake tilt, widened — the real "more sparkle" lever
roughness: 0.47, // the flakes' own, independent of the layer's
metalness: 0.48, // part-way from metallised film toward a pearl bead
iridescence: 0.68, // 0 = fixed hues only, 1 = + full thin-film travel
thicknessRange: [180, 640],
ambient: 0.8,
},
}
```
The env-map orientation is **not** a `meshConfigs` field — it lives on the
`FaceMesh`. That look also applies
`faceMesh.setEnvMapRotation(0.12, 0.12, -2.32)` at startup; the roll does most of
the work, swinging the studio HDR's key light so it rakes across the lips rather
than hitting them head on, which is what gets flakes at different tilts to light
up at different moments instead of all together.
Every field is live-tunable without a recompile — `faceMesh.setGlitter(index, {...})`
and `faceMesh.setGlitterMaps(index, normalDir, colorDir)`. Unlike `alphaMapDir` /
`colorMapDir`, the glitter samplers are compiled unconditionally, so a layer built
**without** a `glitter` block can still be given one at runtime.
### Two colour mechanisms, and how to tell them apart
- **Base hue** — the flake's own pigment, from the colour map's RGB. Fixed: a
magenta flake stays magenta from every angle, its neighbour stays gold. This is
what separates one flake from the next in a still frame.
- **Angle shift** — thin-film interference through the same Belcour & Barla
`evalIridescence` the layer's own `iridescence` uses, with a **per-flake**
thickness from the colour map's alpha. This is what makes a flake *travel*
(magenta → gold) as the head turns, and because the thickness is drawn per
flake, two flakes of the same base hue travel to different colours.
Sweep `iridescence` from 1 to 0 to see them separately: at 0 only the fixed hues
remain.
### Traps
- **`metalness` defaults to 1, and it should.** A real cosmetic glitter particle
is metallised polyester, so its F0 *is* its colour, around 0.9. Built as a
dielectric at `ior` 1.8 instead, a flake reflects 8% — measured about **11x too
dim**, and it vanished under the lipstick entirely. Drop it toward 0 only for a
soft pearl.
- **The layer's `opacity` fades the flakes**, because the lobe is added to
`outgoingLight` and blending multiplies that by the layer's alpha. That is also
what keeps the flakes inside the lip mask, so it is not a bug — raise
`glitter.intensity` on a sheer layer. On the `demos/controls` lipstick
(`opacity: 0.18`) the flakes need about **6** where an opaque base needs 1.
- **Off-palette flakes are expected, and they are the loudest ones.** A red-only
palette still renders some yellow/cyan/blue, because a flake *reflects the
environment*: its colour on screen is its own tint **times the colour of
whatever it happens to be reflecting**. A red flake's F0 is about
`(0.39, 0.06, 0.08)` — the green and blue channels are not zero — so a flake
catching a bright cool highlight in `studio2.hdr` can come out blue. That is
correct PBR, the same reason a red mirror pointed at a blue lamp looks blue.
What makes it conspicuous is that those flakes are also the **brightest**
(measured 1.8x the rest overall, 2.8x for the env-map term alone), because they
are the ones catching the brightest light — so 3-4% of flakes dominate what the
eye picks out. Measured on a four-red palette, share of flakes far from red
(hue 45-260): **3.4% with thin film off, 4.2% at `iridescence: 0.62`**.
Attribution with thin film off — env map 2.7%, ambient 4.3%, frame light 1.2%.
Levers, in the order worth trying: `iridescence` toward 0 (it is what makes the
gold and cyan specifically), `saturation` 1.5-2 (tightens mean hue error from
16.5 to 12.8 degrees by zeroing the off-hue channels of F0 so light cannot leak
through them), a warmer env map or lower `envMapIntensity`, then
`frameLightDetail`, which injects the camera's own residual chroma.
- **On a light base, `occlusion` is the control — not `intensity`, not
`saturation`.** The lobe is *added* to the make-up, and adding light can
brighten something already near white but cannot tint it. Measured on a white
lipstick: the flake contributes ~13 levels over a base at ~83, so the composite
is 86% base and the flake's own hue (saturation 0.75, matching the palette) is
dragged down to the base's. `occlusion` lets the flake **hide** the film under
it, which is what an opaque metallised particle really does. Measured colour
lift over the whole lip: **+0.007 additive → +0.027 at occlusion 1.0 → +0.037
with `saturation: 2.0`** on top. Expect flakes darker than the base in places —
an opaque red flake on a white film *is* darker, except where it catches a
highlight. Defaults to 0, so no look tuned before it existed moves.
- **`coverage` thins flakes toward transparency; it does not remove them.** It
scales the map's alpha channel, so `coverage: 0.08` makes every flake 92%
transparent rather than leaving 8% of the flakes solid — and since occlusion is
weighted by the same value, it also caps occlusion at 0.08. Compensating with a
big `intensity` then fights the transparency instead of fixing it. For fewer,
solid flakes lower the generator's `--count` and leave `coverage` near 1.
Measured: `coverage 1.0` + `occlusion 1.0` at `intensity 1.0` beat
`coverage 0.08` at `intensity 7.18` (+0.027 against +0.014) at a seventh of the
intensity.
- **`intensity` saturates, so it is not the lever for "more sparkle".** Measured on
that same sheer base, total glitter energy goes 0.25M → 0.35M → 0.55M → 0.53M →
0.56M for intensity 1 → 3 → 6 → 10 → 16: it knees at ~6 and is flat after. The
peak per-pixel delta is pinned at its maximum from intensity 1 onward, which is
the explanation — lit flakes already clip to white, and raising intensity clips
them harder rather than lighting the flakes that are facing nothing bright. For
more sparkle use `coverage` or `scale` (more flakes) or `normalScale` (a wider
spread of facing directions).
- **A hot base hides them.** With `demos/controls`' calibrated frame-light values
(`gain: 1.5`, `detail: 0.35`) the lip goes near-white and the flakes disappear
into it. `demos/glitter` deliberately runs a much quieter base.
- **Both maps must change together.** A normal map from one palette with a colour
map from another gives every flake the wrong hue.
- **Mipmapping does not kill the sparkle** — this was expected to be the hard
problem and it measurably is not. Sweeping `scale` 1 → 16 (which shrinks the
flake footprint exactly as walking away from the camera does), total glitter
energy stayed within **77-142%** of the scale-1 value and the peak stayed flat.
What changes is character: blobs at 1, distinct coloured specks at 4, fine even
dust at 16, where sub-pixel flakes average their colours toward each other.
Around 4 keeps the hues telling each other apart. **Not measured:** temporal
flicker as the head moves — the test feed was a still image, so it says nothing
about flakes popping between frames.
### Checking the result
The glitter shares the `frameLightDebug` switch, and the `demos/glitter` GUI
exposes all of it. Slots 6, 7 and 8 are the glitter's:
- **6 coverage** — where the flakes are. It must sit still on the lip as the head
moves; if it crawls, it is being sampled in video space instead of atlas space.
- **7 flake normals** — a normal-map view. A flat lilac field means the tilt is
not arriving.
- **8 glitter only** — the lobe with the make-up removed. This is where you can
see that only *some* flakes are lit at any pose, which is the twinkle.
To prove the flakes are lit by indirect light and not by a lamp: with an env map
bound the shader already zeroes the directional light
(`directLight.color *= (1.0 - hasEnvMap)`). Zero the ambient too and rotate the
env map — the sparkle must move. Then `clearEnvMap()` and enable the frame light
probe — the sparkle must survive, now driven by the real light in the room.
## Demo
The [demos](./demos/) folder have demos with all the features. The [photo](./demos/photo/) shows an example of using an image loaded on html and the [webcam](./demos/webcam/) uses the user camera. The application loads a canvas and shows blush, lipsticks and eye shadows make ups on the face with the face retouch. It is also possible to render partially with the **setWindow** method, freeze the video like a photograph and share the photo.
The [eyeliner-studio](./demos/eyeliner-studio/) demo is the authoring bench for the
eyelid ribbons — draw a strip, see it on a tracked face, tune the layer, download
the PNG. See [Authoring an eyeliner](#authoring-an-eyeliner). Serve the **repo
root**, not `demos/`: it reads `src/geometry.ts` for the canonical model.
The [glitter-studio](./demos/glitter-studio/) demo is the authoring bench for the
flakes — pick a palette, shape them, see them on a tracked face, download the
PNGs, and copy the command that reproduces them. See
[Authoring a glitter](#authoring-a-glitter). Serve the **repo root**, not
`demos/`.
The [glitter](./demos/glitter/) demo is a **temporary** bench for the glitter
lobe: one lipstick layer, a lil-gui panel for every glitter knob, a palette
dropdown, and the env-map / frame-light switches needed to prove the flakes are
lit by indirect light. See [Authoring a glitter](#authoring-a-glitter).
The [controls](./demos/controls/) demo is the parameter sandbox: it runs the webcam through the custom DynamicMaterial shader and exposes every knob live in a lil-gui panel — full PBR per layer, the video-derived normal scale, the Gaussian/Sobel smoothing, environment-map upload, and the frame light probe (with a read-only readout of what the illumination is currently measuring). Upload an image from the same page to A/B against a still instead of the camera.
## WASM
1. Activate o EMSDK
```sh
source /path/to/emsdk/emsdk_env.sh
```
2. Generate the Makefile with cmake command
```sh
cd ./wasm/build/
emcmake cmake .
```
3. Compile
```sh
emmake make
```
4. The **wasm** and **js** files will be inside **output** directory