# ZelPi

**A robotics toolkit that runs with one command — `npx zelpi` — plus a fleet-OS
software twin for exploring the full Physical Intelligence architecture.**

ZelPi gives a robotics developer, with zero install beyond Node:

- **babyros** — a built-in pub/sub middleware speaking the **rosbridge v2 wire
  protocol**. roslibjs, `zelros`, and existing rosbridge clients connect to it
  unchanged. No ROS installation required.
- **SLAM + sensor fusion** — occupancy-grid SLAM (correlative scan matching)
  and an EKF fusing odometry + IMU with online gyro-bias estimation, running
  as babyros nodes — or attached to **your real robot's rosbridge**.
- **HAL** — a versioned WebSocket protocol (e-stop, watchdog, skills,
  telemetry) linking the OS to any robot that runs the bundled
  `zelpi-robot-agent`, with a ROS 1/2 driver included.
- **Real models** — `zelpi hub install` downloads real VLA checkpoints
  (SmolVLA, ACT, Diffusion Policy) from HuggingFace and runs real inference;
  `zelpi gpu setup` switches the venv to CUDA.
- **Honest MuJoCo simulations** — a humanoid driven by a learned 40-DoF
  action model (`zelpi sim`), and a Shadow Dexterous Hand doing
  language-commanded pick-and-place (`zelpi demo2`).

## Five minutes to a live SLAM map

```bash
npx zelpi slam launch        # simulated robot + lidar → EKF + SLAM → live ASCII map
```

That boots a babyros broker, a 2D differential-drive world with a 180-beam
lidar (3% odometry slip, gyro bias — deliberately imperfect), an EKF fusion
node, and an occupancy-grid SLAM node, then renders the map live in your
terminal. On a reference run the fused estimate tracked the true pose at
**0.05 m error vs 0.14 m from raw odometry**, with the gyro bias estimated to
within 3%.

Have a real robot publishing `sensor_msgs/LaserScan` + odometry on a
rosbridge? Attach the same nodes to it:

```bash
npx zelpi slam attach --url ws://<robot>:9090 --scan /scan --odom /odom --imu /imu
```

More first commands:

```bash
npx zelpi babyros up                        # standalone middleware (port 9091)
npx zelpi babyros topics                    # zelros + roslibjs connect unchanged
npx zelpi hal connect ws://<robot>:9091     # native robot link (e-stop, skills, telemetry)
npx zelpi hub install smolvla               # real HuggingFace weights + local inference
npx zelpi gpu setup                         # CUDA torch for the models venv
npx zelpi docs hardware                     # the full real-hardware integration guide
```

## Demo mode — the fleet-OS software twin

ZelPi also ships a **complete simulated fleet OS**: a cross-timescale neural
stack (System 2 planning / System 1 skills / System 0 reflexes), an agent
swarm, a Safety Broker with a Human-in-the-Loop gate, and a live fleet
dashboard. It exists to make the architecture explorable — it is a
**deliberate simulation**, not a claim that these commands drive production
robots.

Every command backed by the software twin prints a `▓ SIMULATION` banner so
it can never be mistaken for real capability. Run **`zelpi demo`** for the
exact split, or see the table below. (`PIOS_NO_SIM_BANNER=1` suppresses the
banner.)

`npx zelpi intent "Inventory Aisle 4"` plans with a **real LLM call** on
**Claude**. Bring your own Claude Pro/Max subscription: install the Claude Code
CLI and log in once (`npm i -g @anthropic-ai/claude-code`, then `claude` →
/login); ZelPi drives it headlessly on **Claude Haiku** with no API key, and it's
the zero-config default. OpenAI (`OPENAI_API_KEY`) and a local Ollama are optional
alternatives. Every plan prints `plan source → LLM` or `plan source →
deterministic parser`, so you can tell exactly which System-2 path produced it.
See `.env.example` for provider config.

## What's real vs. simulated — read this before you rely on it

| Real & verified | Simulated / not yet verified end-to-end |
|---|---|
| **babyros** (`lib/babyros/`) — rosbridge v2 wire protocol, verified with zelros and roslibjs clients; SLAM/EKF verified live (0.05 m fused vs 0.14 m raw) | The fleet dashboard, System 2/1/0 tick loop, Safety Broker and agent swarm — the software twin behind the `▓ SIMULATION` banner |
| **HAL protocol** (`cli/hal.mjs`, `server/robot-agent.mjs`) — real WebSocket JSON wire format, e-stop/watchdog, 300+ unit tests | The "fine-tuning on fleet experience" step of `hub pull` — no real fleet-telemetry/RL training pipeline exists behind it |
| **ROS 1/2 bridge** (`server/createRosDriver.mjs`) — verified against a real independent `rclpy` node, both directions | `perceive` / `firewall` (`cli/spe.mjs`) — deterministic, rule-based, no model call of any kind, by design |
| **`zelpi intent`** — really calls an LLM (Claude Haiku on your own Pro/Max plan via the Claude Code CLI by default, or an optional OpenAI key / local Ollama) and labels every plan with its true source; falls back to a keyword parser, visibly, when the LLM is unreachable | Real hardware in the loop — every verification so far is sim-driver-to-sim-driver or sim-to-local-ROS-node; no physical robot has been tested yet |
| **`hub install` / `hub pull`** — download real HuggingFace weights; `vla_bridge.py` runs genuine forward passes (SmolVLA, ACT, Diffusion Policy verified) | `vla_bridge.py` feeds *synthetic* camera observations — it proves the pipeline, not that a pretrained checkpoint is useful on your robot without fine-tuning |
| **GPU inference** — verified on a real RTX 3060 (SmolVLA 56 s → 3.9 s/step) | One GPU model tested; NVIDIA/CUDA only — AMD/Apple accelerators unsupported |
| **Genie Sim `install`/`deps`** (`sim/scripts/geniesim_install.py`, `cli/geniesim.mjs`) — verified end-to-end on real Linux (WSL2 Ubuntu 22.04) with real Docker + NVIDIA Container Toolkit + GPU passthrough all confirmed working; found and fixed 3 real bugs in the process (Windows symlink checkout, missing `python3` fallback, an unsafe upstream interactive prompt) | Genie Sim's actual Isaac Sim runtime (`geniesim up`'s Docker image build/`docker up`/`ros2 launch`, `geniesim bridge`'s live ROS round-trip) — real code, reuses `createRosDriver.mjs` unmodified, but the Docker image itself was never built: it's a large, untested pull possibly gated behind an NVIDIA NGC login, and the one GPU tested against (RTX 3060, 6 GB) is below Isaac Sim's typical recommendation |
| **`zelpi transporterdemo`** (Zelantrix floor-transfer robot) — the real, unmodified `createRosDriver.mjs` genuinely drives, docks, and e-stops a simulation of that robot's documented motor spec over a real `babyros` broker; `computeLimits()` derives velocity limits from motor spec + wheel/track geometry instead of a flat placeholder | The real robot itself — no physical unit was ever connected; its actual `/cmd_vel`/`/odom` topic names, real safe speed limits, and any real hardware e-stop remain unconfirmed, see `docs/HARDWARE.md` |
| **Memory layer** (`cli/memory.mjs`, `lib/memory/polygres.mjs`) — real semantic recall via Polygres (managed Postgres + pgvector) and Google `gemini-embedding-001`; verified live recalling a paraphrase sharing zero literal words with the original episode (81% cosine relevance) | Only exercised against one Polygres project so far; falls back to a local JSON file with keyword-overlap recall when `DATABASE_URL` isn't set — that path has no real semantic matching |

Full detail: [`docs/HARDWARE.md`](docs/HARDWARE.md) (including a Known Gaps
section) and `npx zelpi demo`.

## The CLI

```bash
npx zelpi                      # interactive OS shell — boots a backend automatically
npx zelpi help                 # full command surface
```

Highlights beyond the quickstart: `zelros` (ROS 1/2 CLI), `zelpi hal scan`
(USB/serial hardware detection + model recommendation), `zelpi hyworld` /
`zelpi lingbot` (world-model integrations), `zelpi visualise` (text → 
simulatable world), `zelpi mcp` (expose ZelPi to AI agents over MCP), and the
conversational shell — type plain English at the REPL and it runs real
commands to answer.

Point it at a full backend (live LLM / ROS / MLflow) with `PIOS_HOST` /
`PIOS_WS_URL`. Full reference: [`docs/CLI.md`](docs/CLI.md) and
`npx zelpi help`. Protocol + driver-SDK docs for building on ZelPi:
[`docs/PROTOCOL.md`](docs/PROTOCOL.md) and [`docs/EXTENDING.md`](docs/EXTENDING.md).

## The web Observation Deck

A browser dashboard for the fleet software twin (the demo-mode half):

```bash
npm install
npm run dev      # → http://localhost:3000
```

Type an intent (`Inventory Aisle 4`, `Retrieve the green box from Aisle 3`)
and watch the swarm decompose it, a robot drive the floor, and the
Human-in-the-Loop gate pause a manipulation for your authorization.

- **One engine, one rAF loop.** `PiOsEngine` is framework-agnostic; React
  subscribes via `useSyncExternalStore` (`lib/useEngine.ts`).
- **Deterministic world.** Generated from a fixed seed (`lib/world.ts`) so
  reloads are reproducible.
- Stack: Next.js 15 · React 19 · TypeScript · hand-rolled engine · plain CSS.

## Contributing & releasing

See [`CONTRIBUTING.md`](CONTRIBUTING.md). Releases are tag-triggered
(`git tag v*` → GitHub Actions tests → `npm publish`); the changelog is
[`CHANGELOG.md`](CHANGELOG.md). License: [MIT](LICENSE).

**Maintainers** — build & publish the lean package locally:

```bash
npm run build:pkg            # → cli-dist/ (bundled engine + CLI, no web-app deps)
cd cli-dist && npm publish --access public
```