# mujoco-react
Composable [React Three Fiber](https://docs.pmnd.rs/react-three-fiber) wrapper around the official [@mujoco/mujoco](https://www.npmjs.com/package/@mujoco/mujoco) WASM bindings. Load any MuJoCo model, step physics, render bodies, and write controllers as React components.
[](https://www.npmjs.com/package/mujoco-react)
**[Demo](https://mujoco-react-example.pages.dev)** | **[Docs](https://dadd.mintlify.app)** | **[npm](https://www.npmjs.com/package/mujoco-react)** | **[Example Source](https://github.com/noah-wardlow/mujoco-react-example)** | **[llms.txt](https://dadd.mintlify.app/llms.txt)**
## Install
```bash
npm install mujoco-react three @react-three/fiber @react-three/drei
```
## Vite Plugin and Type-Safe Names
Use the Vite plugin to generate declaration merging for actuator, sensor, body, joint, site, geom, and keyframe names:
```ts
// vite.config.ts
import { defineConfig } from "vite";
import { mujocoReact } from "mujoco-react/vite";
export default defineConfig({
plugins: [
mujocoReact({
models: {
franka: "models/panda/scene.xml",
},
}),
],
});
```
The plugin writes `src/mujoco-register.gen.ts` during dev and build. Commit that generated file. Vite auto-loads it, so app code imports generated values from `mujoco-react`, not from the generated file:
```ts
// src/mujoco-register.gen.ts
// Auto-generated by mujoco-react. Do not edit.
import { registerModelResources } from "mujoco-react";
const generatedModelResources = {
franka: {
actuators: { joint1: "joint1", joint2: "joint2", joint3: "joint3", gripper: "gripper" },
sensors: { force_sensor: "force_sensor", torque_sensor: "torque_sensor" },
bodies: { link0: "link0", link1: "link1", hand: "hand" },
joints: { joint1: "joint1", joint2: "joint2", joint3: "joint3" },
sites: { tcp: "tcp" },
geoms: { floor: "floor" },
keyframes: { home: "home" },
},
};
registerModelResources(generatedModelResources);
declare module "mujoco-react" {
interface Register {
models: {
franka: {
actuators: "joint1" | "joint2" | "joint3" | "gripper";
sensors: "force_sensor" | "torque_sensor";
bodies: "link0" | "link1" | "hand";
joints: "joint1" | "joint2" | "joint3";
sites: "tcp";
geoms: "floor";
keyframes: "home";
};
};
actuators: "joint1" | "joint2" | "joint3" | "gripper";
sensors: "force_sensor" | "torque_sensor";
bodies: "link0" | "link1" | "hand";
}
}
```
Once generated, hooks like `useCtrl`, `useSensor`, `useBodyState`, and API methods like `setCtrl`, `applyForce`, `getSensorData` accept the global union. For model-scoped code, use package exports such as `ModelActuators.franka.gripper`, `ModelSites.franka.tcp`, and `ModelBodies.franka.hand`. Generic type helpers like `ModelActuators<"franka">` are still available for reusable library code. When no `Register` augmentation is present, names fall back to `string`.
Non-Vite projects can generate the same file with:
```bash
npx mujoco-react codegen franka=models/panda/scene.xml
```
## Load a Model
```tsx
import { MujocoProvider, MujocoCanvas } from "mujoco-react";
import type { SceneConfig } from "mujoco-react";
const panda: SceneConfig = {
src: "https://raw.githubusercontent.com/google-deepmind/mujoco_menagerie/main/franka_emika_panda/",
sceneFile: "scene.xml",
homeJoints: [1.707, -1.754, 0.003, -2.702, 0.003, 0.951, 2.490],
};
export function App() {
return (
);
}
```
## Add Scene Tools
```tsx
import { OrbitControls } from "@react-three/drei";
import { IkGizmo, ModelSites, useIkController } from "mujoco-react";
function PandaTools() {
const ik = useIkController({ siteName: ModelSites.franka.tcp });
return (
<>
{ik && }
);
}
```
Use it as a child of ``:
```tsx
```
## Gaussian Splat Environments
Gaussian splats are visual context; MuJoCo XML remains the source of physics, contacts, and task fixtures. Use visual-only splats when you only need rendered environment context, and add collision proxy metadata when a workflow needs simplified contact geometry.
Use `VisualScenarioEffects` when the same MuJoCo task should render under
different camera exposure, fog/background, and deterministic material variants:
```tsx
import { ScenarioLighting, VisualScenarioEffects } from "mujoco-react";
object.name.startsWith("prop_")}
/>
;
```
Use the renderer-agnostic boundary from the main package. If your app stores
visual scenarios as data, pass the scenario directly; the component resolves the
splat asset and any paired MJCF collision proxy metadata for you. Visual-only
splats are valid, and readiness tells you whether a collision proxy is required
for your training/physics handoff.
```tsx
import { MujocoCanvas, SplatEnvironment, useSplatSceneConfig } from "mujoco-react";
const splat = useSplatSceneConfig({ sceneConfig, scenario });
{splat.environment ? (
) : null}
;
```
When a splat scenario includes paired MJCF collision proxy metadata, render a
generic wireframe preview from that XML with `SplatCollisionProxyPreview`. The
component parses MJCF primitives such as planes, boxes, spheres, capsules, and
mesh placeholders from any fetchable proxy XML; the tabletop example is just one
possible environment.
```tsx
import {
SplatCollisionProxyPreview,
SplatEnvironment,
useSplatCollisionProxyGeoms,
useSplatSceneConfig,
} from "mujoco-react";
const splat = useSplatSceneConfig({ sceneConfig, scenario });
const proxy = splat.environment?.collisionProxy;
: undefined
}
/>;
```
Use `useSplatCollisionProxyGeoms()` when your app wants to inspect or style the
proxy primitives itself:
```tsx
const proxyPreview = useSplatCollisionProxyGeoms({
collisionProxy: splat.environment?.collisionProxy,
});
proxyPreview.geoms.map((geom) => geom.type);
```
Use `splat.readiness` or `getSplatEnvironmentReadiness(scenario)` to gate
authoring and import flows. The status distinguishes disabled scenarios,
missing visual assets, missing collision proxies, unsupported renderer formats,
and ready environments.
Use `createSplatSceneConfig()` when the same scene composition needs to run
outside React, such as codegen, import validation, backend handoff metadata, or
tests:
```tsx
import { createSplatSceneConfig } from "mujoco-react";
const splat = createSplatSceneConfig({
sceneConfig,
scenario,
renderer: "spark",
});
```
Use `createVisualScenarioExecutionContext()` or
`useVisualScenarioExecutionContext()` when recording rollouts or exporting
LeRobot/HF Jobs handoff artifacts. It resolves the scenario seed, camera
exposure/noise/blur/jitter, material randomization, splat source, collision
proxy, and readiness into one serializable object.
```tsx
import { createVisualScenarioExecutionContext } from "mujoco-react";
const visualContext = createVisualScenarioExecutionContext({
scenario,
renderer: "spark",
variantId,
});
writeEpisodeManifest({
task,
visualExecutionContext: visualContext,
});
```
For MuJoCo + 3DGS composition, derive the optional collision environment from
the same splat metadata and pass the resulting config to ``:
```tsx
const sceneConfig = withSplatEnvironment(
{
src: "/models/xlerobot/",
sceneFile: "xlerobot.xml",
},
kitchenScenario
);
```
For first-class Spark rendering, install Spark and import the optional adapter:
```bash
npm install @sparkjsdev/spark
```
```tsx
import {
SparkSplatEnvironment,
useSparkSplatEnvironment,
} from "mujoco-react/spark";
function Scene() {
const splat = useSparkSplatEnvironment({ sceneConfig, scenario });
return (
{splat.environment ? (
) : null}
);
}
```
`SparkSplatEnvironment` currently renders `.spz` assets. Use the renderer-agnostic
`SplatEnvironment` for `.ply`/`.splat` metadata or when wiring a different renderer.
Tune live rendering and snapshots separately with `renderTuning` and
`captureTuning`:
```tsx
```
## Write Controllers
```tsx
import { ModelActuators, useBeforePhysicsStep, useCtrl } from "mujoco-react";
function MyController() {
const shoulder = useCtrl(ModelActuators.franka.actuator1);
useBeforePhysicsStep(({ data }) => {
shoulder.write(Math.sin(data.time));
});
return null;
}
```
Controllers are just React children that read sensors, write named controls, apply forces, or call the `MujocoSimAPI` at physics-step time.
With generated resource values, reusable controllers can be scoped to one robot without hand-typing names:
```tsx
import { ModelActuators, ModelJoints, ModelSites, useCtrl, useIkController } from "mujoco-react";
function FrankaTypedControls() {
const gripper = useCtrl(ModelActuators.franka.gripper);
const ik = useIkController({
siteName: ModelSites.franka.tcp,
joints: [
ModelJoints.franka.joint1,
ModelJoints.franka.joint2,
ModelJoints.franka.joint3,
ModelJoints.franka.joint4,
ModelJoints.franka.joint5,
ModelJoints.franka.joint6,
ModelJoints.franka.joint7,
],
actuators: [
ModelActuators.franka.actuator1,
ModelActuators.franka.actuator2,
ModelActuators.franka.actuator3,
ModelActuators.franka.actuator4,
ModelActuators.franka.actuator5,
ModelActuators.franka.actuator6,
ModelActuators.franka.actuator7,
],
});
return null;
}
```
## Use the Sim API
```tsx
import { useMujoco } from "mujoco-react";
function ResetButton() {
const sim = useMujoco();
if (!sim.isReady) return null;
return ;
}
```
## Map Controls to Joints
Use named controls when a controller should write actuator commands without
hard-coding `data.ctrl` offsets:
```tsx
import {
ModelActuators,
defineControls,
useBeforePhysicsStep,
useControls,
} from "mujoco-react";
const arm = defineControls({
shoulder: ModelActuators.so101.shoulder_pan,
lift: ModelActuators.so101.shoulder_lift,
elbow: ModelActuators.so101.elbow_flex,
});
function PolicyDriver({ action }: { action: number[] }) {
const controls = useControls(arm);
useBeforePhysicsStep(() => {
controls.write(action);
controls.patch({ elbow: 1.2 });
});
return null;
}
```
`defineControls()` preserves the alias keys for `set()`, `patch()`, `get()`,
and `read()`, while `write()` accepts an ordered vector in the same order as the
object. The Vite plugin injects the generated register module automatically, so
application code imports generated names from `mujoco-react`, not from
`src/mujoco-register.gen.ts`.
Use lower-level control groups when a robot's actuator order does not match a
simple `qpos[0..n]` layout or you want actuator-name keys directly:
```tsx
import { controlGroup, ModelActuators, useControlGroup } from "mujoco-react";
const gripper = controlGroup([ModelActuators.so101.gripper]);
function HoldGripper() {
const controls = useControlGroup(gripper);
controls.set(ModelActuators.so101.gripper, -0.17453);
return null;
}
```
For selector-based introspection, `resolveControlGroup()` accepts `{ siteName }`,
`{ bodyName }`, `{ joints }`, or `{ actuators }`. Selectors can be a name,
ordered name array, regex, or predicate.
## Build Observations
Build policy-ready observation vectors from common MuJoCo state without hard-coding offsets:
```tsx
import { buildObservation, useBeforePhysicsStep } from "mujoco-react";
function PolicyDriver() {
useBeforePhysicsStep(({ model, data }) => {
const obs = buildObservation(model, data, {
qpos: true,
qvel: true,
ctrl: true,
sensors: ["imu_gyro", "imu_accel"],
sites: ["tcp"],
projectedGravity: "torso",
});
runPolicy(obs.values, obs.layout);
});
return null;
}
```
Use `output: "float64"` when a downstream model expects double precision. Named resources are skipped when absent, so `obs.layout` is the source of truth for the current model.
## WebSocket Control
Stream actuator commands over a WebSocket and send simulation state back. Use a schema validator such as Zod at this boundary because socket messages are untrusted app input (`npm install zod` for this example):
```tsx
import { useEffect, useRef } from "react";
import { z } from "zod";
import { useMujoco, useBeforePhysicsStep, useAfterPhysicsStep } from "mujoco-react";
const CtrlCommand = z.preprocess((data) => {
try {
return typeof data === "string" ? JSON.parse(data) : data;
} catch {
return undefined;
}
}, z.object({
type: z.literal("ctrl_command"),
ctrl: z.array(z.number()),
}));
function useWebSocketControls(url: string) {
const wsRef = useRef(null);
const latestCtrlRef = useRef(null);
useEffect(() => {
const ws = new WebSocket(url);
wsRef.current = ws;
ws.onmessage = (evt) => {
const command = CtrlCommand.safeParse(evt.data);
if (command.success) latestCtrlRef.current = command.data.ctrl;
};
return () => ws.close();
}, [url]);
// Apply incoming actuator controls each physics step.
useBeforePhysicsStep(({ model, data }) => {
const ctrl = latestCtrlRef.current;
if (!ctrl) return;
for (let i = 0; i < Math.min(ctrl.length, model.nu); i++) {
data.ctrl[i] = ctrl[i];
}
});
// Send simulation feedback back after physics.
useAfterPhysicsStep(({ data }) => {
const ws = wsRef.current;
if (!ws || ws.readyState !== WebSocket.OPEN) return;
ws.send(JSON.stringify({
type: "feedback",
time: data.time,
qpos: Array.from(data.qpos),
qvel: Array.from(data.qvel),
sensordata: Array.from(data.sensordata),
}));
});
}
```
For reusable controllers with typed config, default merging, and children, use the `createController` factory:
```tsx
import {
ModelActuators,
createController,
useBeforePhysicsStep,
useCtrl,
} from "mujoco-react";
export const MyController = createController<{ gain: number }>(
{ name: "MyController", defaultConfig: { gain: 1.0 } },
({ config, children }) => {
const shoulder = useCtrl(ModelActuators.franka.actuator1);
useBeforePhysicsStep(({ data }) => {
shoulder.write(config.gain * Math.sin(data.time));
});
return <>{children};
},
);
//
//
//
```
A `createControllerHook` factory is also available for the hook equivalent — see the [Building Controllers](https://dadd.mintlify.app/guides/building-controllers) guide.
## Architecture
`` wraps R3F `
```
### Custom IK Solvers
The built-in `useIkController()` uses Damped Least-Squares. Pass `ikSolveFn` to swap in your own solver (analytical, learned, etc.):
```tsx
import { ModelSites } from "mujoco-react";
import type { IKSolveFn } from "mujoco-react";
const myIK: IKSolveFn = ({ position, currentQ }) => {
return myAnalyticalSolver(position, currentQ); // return joint angles or null
};
const ik = useIkController({ siteName: ModelSites.franka.tcp, ikSolveFn: myIK });
```
### `useIkController(config | null)`
Hook for interactive end-effector control. Pass `null` to disable IK (safe to call unconditionally):
```tsx
import { IkGizmo, ModelSites, useIkController } from "mujoco-react";
const ik = useIkController({ siteName: ModelSites.franka.tcp });
return ik ? : null;
```
| Config | Type | Default | Description |
|--------|------|---------|-------------|
| `siteName` | `string` | **required** | MuJoCo site to track |
| `joints` | `string \| string[] \| RegExp \| (joint) => boolean` | inferred | Explicit hinge/slide joints for IK |
| `actuators` | `string \| string[] \| RegExp \| (actuator) => boolean` | inferred | Explicit actuators for IK output |
| `numJoints` | `number` | legacy only | Contiguous qpos/ctrl count from older examples |
| `ikSolveFn` | `IKSolveFn` | built-in DLS | Custom solver function |
| `damping` | `number` | `0.01` | DLS damping |
| `maxIterations` | `number` | `50` | Max solver iterations |
Returns `IkContextValue | null` with methods like `setIkEnabled`, `moveTarget`, `syncTargetToSite`, `solveIK`, and `getGizmoStats`.
Pass the returned value to `` or to your own controller as a prop.
By default the controller infers scalar hinge/slide joints by walking from the site body toward the model root. For robots where the MJCF control layout is not a simple chain, pass explicit names:
```tsx
import { ModelActuators, ModelJoints, ModelSites } from "mujoco-react";
const leftArmIk = useIkController({
siteName: ModelSites.franka.tcp,
joints: [
ModelJoints.franka.joint1,
ModelJoints.franka.joint2,
ModelJoints.franka.joint3,
ModelJoints.franka.joint4,
ModelJoints.franka.joint5,
ModelJoints.franka.joint6,
ModelJoints.franka.joint7,
],
});
const gripperIk = useIkController({
siteName: ModelSites.franka.tcp,
actuators: [
ModelActuators.franka.actuator1,
ModelActuators.franka.actuator2,
ModelActuators.franka.actuator3,
ModelActuators.franka.actuator4,
ModelActuators.franka.actuator5,
ModelActuators.franka.actuator6,
ModelActuators.franka.actuator7,
],
});
```
## Loading Models
The loader fetches `src + sceneFile`, parses the XML for dependencies (meshes, textures, includes), recursively fetches those too, and writes everything to MuJoCo's in-memory WASM filesystem.
```tsx
// MuJoCo Menagerie
const franka: SceneConfig = {
src: "https://raw.githubusercontent.com/google-deepmind/mujoco_menagerie/main/franka_emika_panda/",
sceneFile: "scene.xml",
};
// Any URL
const custom: SceneConfig = {
src: "http://localhost:3000/models/my_model/",
sceneFile: "model.xml",
};
```
## SceneConfig
```ts
interface SceneConfig {
src: string; // Base URL for model files
sceneFile: string; // Entry XML/URDF file, e.g. "scene.xml"
files?: File[]; // Local files for browser upload workflows
environmentFiles?: string[]; // Static MJCF environment XMLs merged before compile
sceneObjects?: SceneObject[]; // Objects injected into scene XML at load time
homeJoints?: number[]; // Initial joint positions
xmlPatches?: XmlPatch[]; // Patches applied to XML files during loading
onReset?: ({ model, data }) => void; // Called during reset after mj_resetData
}
```
Use `environmentFiles` to compose reusable physics/collision layers with a robot model. For Gaussian splat scenes, keep the `.spz` as a parallel visual layer and point `environmentFiles` at a paired MJCF proxy scene only when contact geometry is needed:
```tsx
const kitchenRobot: SceneConfig = {
src: "/models/xlerobot/",
sceneFile: "xlerobot.xml",
environmentFiles: ["splats/tabletop/scene.xml"],
};
```
### Local Files and URDF
Load browser-selected MJCF or URDF files directly. Folder uploads preserve `webkitRelativePath`, and flat uploads fall back to matching referenced mesh/texture assets by basename:
```tsx
import { useEffect, useRef } from "react";
import { useMujoco } from "mujoco-react";
function ModelUpload() {
const sim = useMujoco();
const inputRef = useRef(null);
useEffect(() => {
inputRef.current?.setAttribute("webkitdirectory", "");
}, []);
return (
{
if (sim.isReady && event.currentTarget.files) {
sim.api.loadFromFiles(event.currentTarget.files);
}
}}
/>
);
}
```
### Adding Objects to Any Scene
```tsx
const config: SceneConfig = {
src: "https://raw.githubusercontent.com/google-deepmind/mujoco_menagerie/main/franka_emika_panda/",
sceneFile: "scene.xml",
sceneObjects: [
{ name: "ball", type: "sphere", size: [0.03, 0.03, 0.03],
position: [0.5, 0, 0.1], rgba: [1, 0, 0, 1], mass: 0.1, freejoint: true },
{ name: "platform", type: "box", size: [0.2, 0.2, 0.01],
position: [0.4, 0.3, 0], rgba: [0.5, 0.5, 0.5, 1] },
],
};
```
### XML Patching
```tsx
xmlPatches: [{
target: "panda.xml",
replace: ["name=\"actuator8\"", "name=\"gripper\""],
inject: "",
injectAfter: "`
Loads the MuJoCo WASM module. Wrap your entire app in this.
| Prop | Type | Description |
|------|------|-------------|
| `wasmUrl` | `string?` | Custom WASM URL override |
| `mtWasmUrl` | `string?` | Custom multi-threaded WASM URL override |
| `threadedLoader` | `(options?) => Promise` | Optional loader imported from `@mujoco/mujoco/mt` |
| `wasmVariant` | `"single" \| "threaded" \| "auto"` | MuJoCo WASM build. Defaults to `"single"` |
| `timeout` | `number` | WASM load timeout in ms |
| `onError` | `(error: Error) => void` | Called if WASM fails to load |
The official `@mujoco/mujoco` package also ships a multi-threaded WASM build. Import it only in apps that opt into it:
```tsx
import loadMujocoMt from "@mujoco/mujoco/mt";
import mtWasmUrl from "@mujoco/mujoco/mt/mujoco.wasm?url";
```
`"auto"` uses the threaded build only when `threadedLoader` and `mtWasmUrl` are provided and `globalThis.crossOriginIsolated` is true. Forced `"threaded"` mode requires `threadedLoader`, `mtWasmUrl`, `Cross-Origin-Opener-Policy: same-origin` and `Cross-Origin-Embedder-Policy: require-corp`.
### ``
Thin wrapper around R3F `