# @edward40/pi-computer-use

An installable [Pi](https://github.com/earendil-works/pi-mono) extension that exposes one OpenAI-compatible `computer` tool backed by the same-process [`@trycua/cua-driver`](https://github.com/trycua/cua/tree/main/libs/cua-driver/typescript) TypeScript SDK. It controls the visible primary desktop and returns a fresh PNG screenshot after every supported action.

No Cua executable, daemon, or MCP server is required. The package pins Cua Driver `0.18.0` and installs its native library for the current host as an optional npm package.

## Requirements

- Node.js 22.19 or newer
- Pi with extension package support
- One of the native hosts shipped by Cua Driver 0.18.0:

| OS | Architectures | Notes |
| --- | --- | --- |
| macOS | arm64, x64 | macOS 13 or newer; Accessibility and Screen Recording grants required |
| Windows | arm64, x64 | MSVC native package |
| Linux | arm64, x64 | glibc (`gnu`) hosts only |

Other architectures, musl-based Linux distributions, and hosts for which npm skipped the matching optional native package fail at first tool use with a platform-specific loading error. Package loading itself remains lightweight because the native SDK is imported lazily.

## Install

From npm:

```bash
pi install npm:@edward40/pi-computer-use
```

From GitHub:

```bash
pi install git:github.com/SASUKE40/pi-computer-use
```

From a local checkout:

```bash
npm install
pi install ./path/to/pi-computer-use
```

Use `-l` with `pi install` for a project-local installation. Restart Pi after installing or use `/reload` when the package is already available to the current Pi process.

## macOS permissions

The process hosting Pi needs both macOS grants before Cua starts:

1. In Pi, run `/computer-permissions`.
2. Accept the Accessibility request.
3. If Screen Recording is missing, System Settings opens to the appropriate Privacy & Security pane. Enable the terminal or application that hosts Pi.
4. Restart that host application and Pi so macOS applies the grants to the new process.

The extension checks both grants before creating a native session. It never starts Cua first and asks later. If Pi runs inside a terminal, IDE, or desktop wrapper, grant access to that actual host application—not to this npm package.

## Use

The extension registers exactly one tool named `computer`. Pi supplies its schema to the model, so normal prompts are enough:

```text
Take a screenshot and tell me what is open.
```

```text
Open the browser's address bar, type https://example.com, press Enter, and wait for it to load.
```

```text
Scroll down in the document until you find the API section. Do not click anything.
```

The parameter shape is `{ action: { type, ...fields } }`:

| Action | Fields | Behavior |
| --- | --- | --- |
| `screenshot` | none | Capture current primary desktop |
| `click` | `x`, `y`, `button` | `left`, `right`, or `wheel`; wheel becomes a middle click |
| `double_click` | `x`, `y` | Left double-click |
| `scroll` | `x`, `y`, `scroll_x`, `scroll_y` | Positive deltas move right/down |
| `type` | `text` | Type text verbatim |
| `wait` | none | Wait two seconds |
| `move` | `x`, `y` | Move the pointer |
| `keypress` | `keys` | Press one key or a key chord |
| `drag` | `path` | Drag from the first point to the last point |

`click` accepts the provider-compatible `back` and `forward` button names in its flat schema, but returns an explicit validation error because Cua Driver has no side-button action. `META` maps to Command on macOS, Windows on Windows, and Super on Linux. `CTRL`, `ALT`/`OPTION`, arrows, Return, Escape, and Space are normalized too.

## Coordinate and action semantics

Coordinates are physical points in the latest returned primary-desktop screenshot. Send them back unchanged: do not perform Retina, DPI, browser zoom, or logical-pixel scaling. A display arrangement with multiple monitors is not exposed as separate targets; this extension intentionally uses Cua's strict desktop scope for the primary desktop.

Scroll deltas follow OpenAI's sign convention. Each nonzero axis is converted to `ceil(abs(delta) / 120)` Cua line steps, clamped from 1 through 50. Horizontal scrolling runs before vertical scrolling. A drag uses its first and last path points, 500 ms duration, and the number of supplied points as interpolation steps, capped at 200.

Tool calls execute sequentially. Native operations have a 30-second timeout and receive Pi's cancellation signal. The runtime does not retry actions: if Cua reports an error, throws, times out, or is cancelled after dispatch, the result warns that the outcome is unknown and includes a new screenshot. The model should inspect that image before deciding whether a retry is safe.

Cua runs in its standard permission mode, which is promptless for normal automation. The extension adds no per-action confirmation dialog. OS-level permission prompts still apply.

## Compatibility boundaries

The tool mirrors the OpenAI computer action surface listed above, not every Cua MCP tool. It does not expose clipboard, window management, raw session escalation, or Cua's full tool inventory. It also does not provide Codex's app-scoped Sky accessibility operations; it operates the foreground desktop through mouse, keyboard, and screenshots.

One lazy Cua session is created for each Pi extension/session instance, with a unique public session ID. The extension requests Cua's visible agent cursor. If the host cannot provide a cursor overlay—most notably a terminal-hosted same-process runtime on macOS—it reports a warning and keeps desktop capture/input available. Guaranteeing the macOS overlay would require Cua's separately supplied GUI private-worker executable, which is intentionally outside this package's no-executable contract. On Pi `session_shutdown`, cleanup attempts `endSession`, `shutdown`, and `uniffiDestroy` in order, even if an earlier cleanup step fails.

## Troubleshooting

**The native SDK cannot be loaded**

Confirm `node -p 'process.platform + "/" + process.arch'` matches a row above. On Linux, confirm the system uses glibc. Reinstall without `--no-optional`; Cua's platform library is an optional dependency selected by npm.

**macOS says permissions are missing after I enabled them**

Fully quit and restart the terminal, IDE, or app hosting Pi. Run `/computer-permissions` again. If Pi is launched through another host process, macOS may attribute the grant to that host.

**Clicks land in the wrong place**

Request a new screenshot and use its coordinates directly. Do not scale coordinates or reuse locations after a window moved, display settings changed, or the UI reflowed.

**An action reports an unknown outcome**

Inspect the screenshot returned in the same tool result. The action may have completed before the timeout or cancellation; retry only when the screenshot proves it did not land.

**Desktop capture fails**

On macOS, verify Screen Recording. On all platforms, ensure the visible desktop is unlocked and available to the Node process. The extension requires a PNG observation and rejects missing or non-PNG captures.

**The agent cursor overlay is unavailable**

The tool still works and includes the Cua message as a warning. Same-process macOS runtimes hosted by a terminal do not have Cua's certified AppKit main-thread adapter, so they cannot display the overlay. A GUI host with the appropriate adapter is required for that visual affordance.

## Development

```bash
npm install
npm run typecheck
npm test
npm run pack:check
```

The native smoke test is deliberately opt-in because it opens a real desktop session. It only captures one screenshot and never sends mouse or keyboard input; teardown always runs:

```bash
npm run test:smoke
```

To additionally load the extension through Pi's real resource loader, verify its tool and command registrations, and invoke `computer` for a screenshot:

```bash
npm run test:pi-smoke
```

## License

MIT