# @bperel/face-mesh-detection
Unofficial Capacitor plugin for [ML Kit Face Mesh Detection](https://developers.google.com/ml-kit/vision/face-mesh-detection).[^1]
## Installation
```bash
npm install @bperel/face-mesh-detection
npx cap sync
```
### Android
#### Variables
This plugin will use the following project variables (defined in your app’s `variables.gradle` file):
- `$mlkitFaceMeshDetectionVersion` version of `com.google.mlkit:face-mesh-detection` (default: `16.0.0-beta1`)
## Configuration
No configuration required for this plugin.
## Demo
A working example can be found here: [robingenz/capacitor-mlkit-plugin-demo](https://github.com/robingenz/capacitor-mlkit-plugin-demo)
## Usage
```typescript
import { FaceMeshDetection, UseCase } from '@bperel/face-mesh-detection';
const processImage = async () => {
const { faceMeshs } = await FaceMeshDetection.processImage({
path: 'path/to/image.jpg',
useCase: UseCase.FaceMesh,
});
return faceMeshs;
};
```
## API
* [`processImage(...)`](#processimage)
* [Interfaces](#interfaces)
* [Enums](#enums)
### processImage(...)
```typescript
processImage(options: ProcessImageOptions) => Promise
```
Detects face mesh from the supplied image.
Only available on Android.
| Param | Type |
| ------------- | ------------------------------------------------------------------- |
| **`options`** | ProcessImageOptions |
**Returns:** Promise<ProcessImageResult>
**Since:** 5.3.0
--------------------
### Interfaces
#### ProcessImageResult
| Prop | Type | Description | Since |
| --------------- | ----------------------- | ------------------------ | ----- |
| **`faceMeshs`** | FaceMesh[] | The detected face meshs. | 5.3.0 |
#### FaceMesh
Represents a face mesh detected by `FaceMeshDetector`.
When `BoundingBoxOnly` is selected, `FaceMesh` only contains valid bounding box.
When `FaceMesh` is selected, `FaceMesh` also contains a group of 468 3D face mesh points and related triangle information.
Each point is represented by `FaceMeshPoint` describing a specific position in detected face.
The triangle information is a group of 3 `FaceMeshPoint`s representing a valid surface on Face (e.g. a valid small surface on nose tip).
| Prop | Type | Description | Since |
| -------------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| **`bounds`** | Rect | Returns the axis-aligned bounding rectangle of the detected face mesh. | 5.3.0 |
| **`contours`** | Contours | Returns contours with a list of `FaceMeshPoint` representing the detected face. | 5.3.0 |
| **`faceMeshPoints`** | FaceMeshPoint[] | Returns a list of `FaceMeshPoint` representing the whole detected face. | 5.3.0 |
| **`triangles`** | Triangle[] | Returns a list of `Triangle` representing logical triangle surfaces of detected face. Each `Triangle` contains 3 `FaceMeshPoint`, representing 3 points of the triangle surface. The sequence of the 3 points are constant and always counter clockwise in face mesh. | 5.3.0 |
#### Rect
Rect holds four integer coordinates for a rectangle.
| Prop | Type | Description | Since |
| ------------ | ------------------- | ---------------------------------------------------- | ----- |
| **`left`** | number | The X coordinate of the left side of the rectangle. | 5.3.0 |
| **`top`** | number | The Y coordinate of the top of the rectangle. | 5.3.0 |
| **`right`** | number | The X coordinate of the right side of the rectangle. | 5.3.0 |
| **`bottom`** | number | The Y coordinate of the bottom of the rectangle. | 5.3.0 |
#### Contours
Represents contours with their face mesh points.
| Prop | Type | Description | Since |
| ------------------------ | ---------------------------- | -------------------------------------------------------- | ----- |
| **`faceOval`** | FaceMeshPoint[] | Returns all points for the `FaceOval` contour. | 5.3.0 |
| **`leftEyebrowTop`** | FaceMeshPoint[] | Returns all points for the `LeftEyebrowTop` contour. | 5.3.0 |
| **`leftEyebrowBottom`** | FaceMeshPoint[] | Returns all points for the `LeftEyebrowBottom` contour. | 5.3.0 |
| **`rightEyebrowTop`** | FaceMeshPoint[] | Returns all points for the `RightEyebrowTop` contour. | 5.3.0 |
| **`rightEyebrowBottom`** | FaceMeshPoint[] | Returns all points for the `RightEyebrowBottom` contour. | 5.3.0 |
| **`leftEye`** | FaceMeshPoint[] | Returns all points for the `LeftEye` contour. | 5.3.0 |
| **`rightEye`** | FaceMeshPoint[] | Returns all points for the `RightEye` contour. | 5.3.0 |
| **`upperLipTop`** | FaceMeshPoint[] | Returns all points for the `UpperLipTop` contour. | 5.3.0 |
| **`upperLipBottom`** | FaceMeshPoint[] | Returns all points for the `UpperLipBottom` contour. | 5.3.0 |
| **`lowerLipTop`** | FaceMeshPoint[] | Returns all points for the `LowerLipTop` contour. | 5.3.0 |
| **`lowerLipBottom`** | FaceMeshPoint[] | Returns all points for the `LowerLipBottom` contour. | 5.3.0 |
| **`noseBridge`** | FaceMeshPoint[] | Returns all points for the `NoseBridge` contour. | 5.3.0 |
#### FaceMeshPoint
Represents a 3D point in face mesh.
The index is an unique ID meaning a fixed position on face, ranging from 0 to 467.
In `Point3D`, `x` and `y` are pixel location of detected face in `InputImage`.
`z` is also scaled to image size, while the origin will be somewhere in the center of all 468 face mesh points.
| Prop | Type | Description | Since |
| ----------- | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| **`index`** | number | Gets the index of the face mesh point, ranging from 0 to 467. For each specific point, the index is a constant value. | 5.3.0 |
| **`point`** | Point3D | Gets a 3D point in face mesh. Inside `Point3D`, `X` and `Y` means a 2D position in original image. More information on the `Z` value: - The unit of measure for the `Z` value is the same as `X` and `Y`. - The smaller the `Z` value, the closer that landmark is to the camera. - The `Z` origin is approximately at the center of all 468 face mesh points. `Z` value will be negative if the point is close to camera and will be positive if the point is away from the camera. | 5.3.0 |
#### Point3D
Represents a 3D point.
| Prop | Type | Description | Since |
| ------- | ------------------- | --------------------------------- | ----- |
| **`x`** | number | Returns the X value of the point. | 5.3.0 |
| **`y`** | number | Returns the Y value of the point. | 5.3.0 |
| **`z`** | number | Returns the Z value of the point. | 5.3.0 |
#### Triangle
Represents a triangle with 3 generic points.
| Prop | Type | Description | Since |
| ------------ | ---------------------------- | ----------------------------------------------------------------- | ----- |
| **`points`** | FaceMeshPoint[] | Returns all points inside the `Triangle`. | 5.3.0 |
#### ProcessImageOptions
| Prop | Type | Description | Default | Since |
| ------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------- | ----- |
| **`path`** | string | The local path to the image file. | | 5.3.0 |
| **`useCase`** | UseCase | Sets the use case. When `BoundingBoxOnly` is selected, the returned `FaceMesh` only contains bounding box. When `FaceMesh` is selected, the returned `FaceMesh` contains bounding box as well as 468 `FaceMeshPoint` and triangle information. It detects at most 2 faces in this case and it is slower than `BoundingBoxOnly`. | UseCase.FaceMesh | 5.3.0 |
### Enums
#### UseCase
| Members | Value | Description | Since |
| --------------------- | -------------- | ------------------------------------------------------------------------------------- | ----- |
| **`BoundingBoxOnly`** | 0 | Return bounding box for detected face. | 5.3.0 |
| **`FaceMesh`** | 1 | Return face mesh info for detected face. It detects at most 2 faces in this use case. | 5.3.0 |
## Terms & Privacy
This plugin uses the [Google ML Kit](https://developers.google.com/ml-kit):
- [Terms & Privacy](https://developers.google.com/ml-kit/terms)
- [Android Data Disclosure](https://developers.google.com/ml-kit/android-data-disclosure)
- [iOS Data Disclosure](https://developers.google.com/ml-kit/ios-data-disclosure)
## Changelog
See [CHANGELOG.md](https://github.com/capawesome-team/capacitor-mlkit/blob/main/packages/face-mesh-detection/CHANGELOG.md).
## License
See [LICENSE](https://github.com/capawesome-team/capacitor-mlkit/blob/main/packages/face-mesh-detection/LICENSE).
[^1]: This project is not affiliated with, endorsed by, sponsored by, or approved by Google LLC or any of their affiliates or subsidiaries.