# Extending ZelPi — writing a `RobotDriver` `server/robot-agent.mjs` is the program that runs on (or beside) your robot. It speaks the HAL protocol (see [`PROTOCOL.md`](PROTOCOL.md)) over WebSocket and delegates everything hardware-specific to one small object: the `RobotDriver`. Swapping the driver is the *only* change needed to go from the bundled simulator to your real robot — the WebSocket handling, watchdog, e-stop latching, and telemetry loop in `robot-agent.mjs` never change. ## The interface ```ts interface RobotDriver { // Advertise drive kind, dof, skills, units, frame, limits, rate. capabilities(): Capabilities; // Register / reset a robot's initial pose. seedPose(id: string, x: number, y: number, theta: number): void; // Apply a unicycle velocity command. Called every control step (20 Hz). applyCmd(id: string, v: number, w: number): void; // Execute a high-level skill; resolve with a terminal status. runSkill(id: string, skill: string, args: object): Promise<{ status: "done" | "failed", error?: string }>; // Read current pose/state, or null if the id is unknown. readTelemetry(id: string): { x: number; y: number; theta: number; vx?: number; w?: number; battery?: number; joints?: number[]; } | null; estop(): void; // immediately zero all motion and latch e-stop release(): void; // clear the e-stop latch, resume accepting commands } ``` `createSimDriver()` in `robot-agent.mjs` is the reference implementation — read it before writing your own; every method above has a working example there. It also implements an optional `_step(dt)` used only by the simulator to advance its own kinematics — a real driver omits it and reads pose from actual hardware odometry instead. ## Writing your driver 1. Add a factory function next to `createSimDriver`, e.g. `createUnitreeDriver()` — a commented-out skeleton for a differential base already exists in `robot-agent.mjs` as a starting point. 2. Register it in `selectDriver()`'s switch statement. 3. Run with `node server/robot-agent.mjs --driver ` (or `PIOS_ROBOT_DRIVER=`). 4. Point ZelPi at it: `npx zelpi hal connect ws://:9091` (or `PIOS_ROBOT_URL=ws://:9091 zelpi up`). ### Example: a differential-drive SDK ```js function createMyRobotDriver() { const sdk = require("my-robot-sdk"); sdk.connect(); return { capabilities: () => ({ protocol: 1, robotKind: "my-robot", drive: "diff", dof: 0, skills: ["move", "stop", "dock"], units: { len: "m", ang: "rad" }, frame: "odom", limits: { vMax: 1.2, wMax: 2.0 }, rateHz: 20, }), seedPose(id, x, y, theta) { sdk.setOdomOrigin(x, y, theta); }, applyCmd(id, v, w) { sdk.setVelocity(v, w); }, async runSkill(id, skill, args) { if (skill === "dock") { await sdk.dock(); return { status: "done" }; } return { status: "failed", error: `unsupported skill: ${skill}` }; }, readTelemetry(id) { const p = sdk.getPose(); return p ? { x: p.x, y: p.y, theta: p.theta, battery: sdk.getBattery() } : null; }, estop() { sdk.stopImmediate(); }, release() { sdk.resume(); }, }; } ``` ### Arms and joint-space control The native `cmd` op is unicycle-only. For continuous joint targets (a VLA or learned policy), implement the `setJoints` skill in `runSkill`: ```js async runSkill(id, skill, args) { if (skill === "setJoints") { // args.joints: number[] (length must match your advertised `dof`) // args.gripper?: number sdk.setJointTargets(args.joints, args.gripper); return { status: "done" }; } // ... } ``` There is deliberately **no magnitude/velocity/torque clamp** at the protocol layer for `setJoints` — your driver, or the SDK beneath it, must be the real safety boundary, exactly as with any other skill. `sim/scripts/hal_client.py` is a minimal Python client if your inference loop lives in Python rather than the JS agent process — see `sim/scripts/vla_bridge.py` for a full example driving a real VLA checkpoint through it. ## Genie Sim — AgiBot's Isaac Sim/Omniverse humanoid platform [AgibotTech/genie_sim](https://github.com/AgibotTech/genie_sim) is AgiBot's open-source Isaac Sim/Omniverse-based humanoid simulation platform (LLM-driven scene generation, a 200+ task benchmark suite, teleoperation, and a large-scale synthetic data collection pipeline). `zelpi geniesim` follows the same honest, capability-gated pattern as `zelpi hyworld`/`zelpi lingbot` — but unlike those, genie_sim's requirements are hard, not just recommended: - **Linux** (Ubuntu 22.04+) — no native Windows support - **Docker**, plus the **NVIDIA Container Toolkit** on Linux for GPU passthrough - An **NVIDIA RTX-class GPU** - **ROS 2 Jazzy** and **Isaac Sim 5.1/6.0** (both ship inside genie_sim's own Docker image) genie_sim is also not pip-installable — it must be cloned and driven through its own `geniesim` CLI (`geniesim bootstrap`/`docker build`/`docker up`). `sim/scripts/geniesim_install.py` only clones the repo and installs that CLI dispatcher into a lightweight venv; it never vendors genie_sim source into this repo. `source/geniesim*` and `source/data_collection` are MPL-2.0; `source/scene_reconstruction` has mixed licenses and nothing in zelpi references or invokes it. **Verified end-to-end on real Linux (WSL2 Ubuntu 22.04), not just designed against docs.** `zelpi geniesim install` was run for real against genie_sim's live `main` branch on a genuine Ubuntu 22.04 host (WSL2, with real Docker + NVIDIA Container Toolkit + GPU passthrough all confirmed working via `docker run --gpus all nvidia/cuda... nvidia-smi`), and completed successfully end-to-end: clone (5,723 files), venv creation, and a real `geniesim_cli` editable install, with the capabilities manifest correctly reporting `docker_present/running: true`, `nvidia_container_toolkit: true`, `nvidia_gpu: true`, `geniesim_cli_installed: true`. Three real, previously undiscovered issues surfaced by that run, all now fixed: 1. **Windows git checkout breaks `pip install -e source/geniesim_cli/`.** `source/geniesim_cli/VERSION` is a symlink to the repo-root `VERSION` file; Windows git checkouts without symlink support turn it into a 13-byte text placeholder (`../../VERSION`), which setuptools' `/`-to-`-` normalization mangles into `Invalid version: '..-..-VERSION'`. `geniesim_install.py`'s `_repair_windows_symlink_placeholder()` now detects and fixes this automatically, install-time-only, never touching genie_sim's tracked history. 2. **Stock Debian/Ubuntu has no bare `python` binary, only `python3`** (the `python-is-python3` package that provides it isn't installed by default) — `cli/geniesim.mjs` hardcoded `"python"` (the same pattern `hyworld`/ `lingbot`/`gpu setup` use elsewhere in this CLI, sharing the same latent bug) and would fail via ENOENT on genie_sim's own required OS. Fixed here by preferring `python3` on non-Windows platforms. 3. **`geniesim doctor`'s own upstream prompt is unsafe to inherit stdio into.** It ends with an interactive "Apply the fixes above? [Y/n]:" that can cascade into `geniesim bootstrap` — installing genie_sim's entire sibling-package stack — and behaved unpredictably on a closed/non-TTY stdin during testing (in one run it began executing the bootstrap install plan before the pipe closed; in another it hung indefinitely). `zelpi geniesim doctor` now explicitly pipes a decline (`"n"`) into that prompt so it always stays a read-only diagnostic — never a silently-triggered stack install — regardless of TTY state. To actually apply genie_sim's suggested fixes, run its real `geniesim bootstrap` binary yourself, directly, where you can see and consent to what it does. Also required, but not something zelpi installs on your behalf (both are standard, low-risk OS packages rather than anything zelpi's own dependency prompts cover): a fresh Ubuntu may need `apt install python3.-venv` (e.g. `python3.10-venv` on 22.04) for `python -m venv` to bootstrap pip at all — `zelpi geniesim install` will fail with a clear `ensurepip is not available` error and the exact apt command if this is missing. ### Walkthrough ```bash zelpi geniesim install # clone + lightweight CLI venv, then walks through # Docker/WSL2/NVIDIA Container Toolkit prompts zelpi geniesim deps # re-run just the dependency prompts later zelpi geniesim up # docker build/up + ros2 launch genie_sim_bringup # (only proceeds once Docker + an NVIDIA GPU are confirmed) zelpi geniesim bridge # wires genie_sim's ROS topics into zelpi HAL ``` ### Or just ask for it — the conversational shell drives the same commands You don't have to type the commands above yourself. `npx zelpi`'s conversational REPL (`cli/oschat.mjs`) is a real tool-calling agent — the LLM sees the OS's actual command surface, decides which commands to run, and reads their genuine output before replying (see `docs/HARDWARE.md`/README's "conversational shell" description; it's the same loop that already handles `hub pull`, `gpu setup`, etc.). Genie Sim is wired into that catalog, so you can just say things like: ``` > set up genie sim > install and get genie sim running > is genie sim installed on this machine? ``` and the agent will actually run `geniesim status` → `geniesim install` → (if you asked for the full stack, not just install) `geniesim deps` → `geniesim up` → `geniesim bridge`, narrating real output at each step — not just describe the steps in prose. **This does not weaken any of the safety gates above — it can't.** The agent's tool loop only decides *which* `zelpi geniesim ...` command to run next; it invokes the exact same code path as if you'd typed the command yourself, including every `confirm()` prompt in `cli/geniesimDeps.mjs`. Those prompts read from your terminal's real stdin and write to your terminal's real stdout — the LLM has no channel to answer them and never sees them as anything but plain text in the command's output. So when `geniesim install` hits "Install Docker now? [y/N]", or `geniesim up` hits "Run \`geniesim docker build\` now?", that prompt appears live in your terminal exactly as it would from a direct CLI run, and the agent will (and must) wait for you to personally type the answer — it always confirms this with you in its own reply, in case you're not watching the raw output for those prompts. There is no configuration, flag, or phrasing that makes the agent (or anything running under it) answer a system-modifying confirmation on your behalf; that boundary is enforced by `ensureGenieSimDeps()` itself, not by the conversational layer, so it holds regardless of how the command was invoked. Before running `geniesim up` specifically (the actual Docker image build — tens of GB), the agent gives you a plain heads-up and asks first, in its own reply, the same way it already does before a multi-GB `hub pull`/`lingbot pull` — a second, conversational check on top of `up`'s own `[y/N]` prompt. **Dependency prompts always require a live confirmation.** `gpu setup`'s CUDA-torch swap only touches an isolated venv, so it's safe to auto-approve via a flag. Installing Docker Desktop, WSL2, or the NVIDIA Container Toolkit modifies host OS state (services, kernel features, often a reboot) — there is deliberately **no** `--yes`/env-var path that skips those confirmations, in `cli/geniesimDeps.mjs`. A missing NVIDIA GPU driver itself is never offered as an auto-install at all (too hardware/OS-specific, real risk of a broken display) — `geniesim install`/`up` just point at the right guidance and stop. **rosbridge placement.** genie_sim's `genie_sim_bringup app.launch.py` doesn't start `rosbridge_server` itself. Rather than forking genie_sim's launch files — which this repo cannot build or test against — `zelpi geniesim up` installs and starts `rosbridge_server` **inside the running container**, as its own explicit, confirmed step, once the container is actually up. **`/joint_command`'s real shape — verified against genie_sim's actual source, not guessed.** An earlier version of this doc speculated `trajectory_msgs/ JointTrajectory` based on MoveIt 2 being in the stack. Cloning genie_sim for real and reading `genie_sim_bringup/scripts/gripper_cmds.py`/`wbc_cmds.py` (`self.pub = self.create_publisher(JointState, "/joint_command", qos)`) and `genie_sim_engine`'s `newton/control.py` shows the actual message type is **`sensor_msgs/JointState`**, and — more importantly — that its handler keys targets **by joint name**, not array position (`apply_commands()`'s docstring: *"ALL joint names land here regardless of joint kind"*). `server/createRosDriver.mjs`'s `setJoints` skill now supports this: set `topics.jointCommandType: "sensor_msgs/JointState"` and `topics.jointNames` (this robot's exact per-joint names, same order as the `joints` array) in your ros-config — `setJoints` rejects with a clear error if the type is `JointState` but `jointNames` is missing or the wrong length, rather than silently publishing a nameless message a name-keyed subscriber would ignore. The default `topics.jointCommandType` remains `std_msgs/Float64MultiArray` for backward compatibility with every other robot's config. The bundled `sim/configs/geniesim_ros.json` ships real joint names for **`scene_pnp_g2_op`'s left arm** — verified against `source/geniesim_benchmark/.../G2_crsB_omnipicker.urdf` (`idx21_arm_l_joint1` .. `idx27_arm_l_joint7`, 7 DOF). This only matches that specific scene/robot: pass a different `--scene` to `zelpi geniesim up` and you must update `jointNames` (and likely `robotId`/topic names) to match whatever robot config that scene actually loads. This integration is **capability-gated, and its `install`/`deps` path is now verified end-to-end on real Linux** (see above) — including real Docker, NVIDIA Container Toolkit, and GPU passthrough all confirmed working via WSL2. Still genuinely unverified: `geniesim up`'s Docker image build (large, untested — Isaac Sim's base image was never actually pulled/built in this project's testing), `ros2 launch genie_sim_bringup`, the in-container rosbridge install, and an actual `setJoints` round-trip against a live, running genie_sim instance. All of that requires either finishing a real `docker build` (tens of GB, possibly gated behind an NVIDIA NGC login for the Isaac Sim base image — untested) or a more powerful GPU than the 6 GB RTX 3060 this was verified against, which is below Isaac Sim's typical recommendation. ## Prefer ROS? Use babyros instead If your robot already publishes odometry/scan/IMU over ROS 1 or ROS 2, you likely don't need a custom `RobotDriver` at all: - `zelpi hal connect` + `--driver ros --ros-config ` bridges HAL to `rosbridge_suite` directly (see `server/createRosDriver.mjs`). - `zelpi slam attach --url ws://:9090 --scan /scan --odom /odom --imu /imu` runs the bundled SLAM/EKF fusion nodes against your robot's real rosbridge with no new code. See [`PROTOCOL.md`](PROTOCOL.md) for both wire formats in full. ## Training-pipeline adapters (`trainpipe`) `zelpi trainpipe` is the unified self-correcting training pipeline: dataset → per-model preprocessing → train → seeded evals → agentic diagnosis → iterate until the eval target is met or budgets run out. v1 ships adapters for the LeRobot family (`smolvla`, `act`, `diffusion-policy` — one adapter, LeRobot v3 datasets) and `qwen2.5-vl` (QLoRA on grounding jsonl). World models are eval/inference-only until someone writes their adapter — which is the point of this section. An adapter is a plain object registered in `cli/trainpipe/adapters.mjs`: ```js { key: "my-world-model", modelKeys: ["cosmos"], // cli/hub.mjs MODELS keys this trains datasetKind: "video-episodes", // what validate/collect produce & expect trainable: true, diskFloorGb: 10, // abort training below this free space collectCommand(run) {}, // StageSpec | null — auto-collect data validateCommand(run) {}, // StageSpec — dataset preflight trainCommand(run, iter) {}, // StageSpec — iter = {n, resume, warmStartCkpt} parseTrainLine(line) {}, // → {step, loss} | null (drives progress) checkpointPath(run) {}, // where eval finds the trained model evalCommand(run, ckpt, seed) {},// StageSpec — ONE seeded rollout parseEvalOutput(stdout) {}, // → {success, distCm} applyHparams(run, hparams) {}, // whitelisted diagnosis adjustments only } ``` A `StageSpec` is `{exe, argv, env, logFile, cwd?}`. Two hard rules the runner enforces for you — do not fight them: 1. **Stage children get file-descriptor stdio, never pipes.** Long trainers holding a pipe to a mortal parent die of SIGPIPE when that session ends. Your stage just writes to stdout/stderr; it lands in `logFile`. 2. **Progress is parsed from the log**, so emit machine-readable lines: either lerobot's `step:NK ... loss:X` shape or `PIPELINE_PROGRESS {"step": N, "loss": X}` — and finish Python stages with a final `PIPELINE_RESULT {"ok": ..., ...}` line (see `sim/scripts/pipeline/validate_dataset.py` for the pattern). State machine, budgets, disk pruning, and the diagnosis loop are shared — an adapter only describes commands and parsers. Run store lives at `/pipelines//run.json`; tests exercise adapters with `PIOS_TRAINPIPE_MOCK=1` (see `test/trainpipe.test.mjs`).