# pi-agent-browser

English | [日本語](README.ja-JP.md)

Typed browser automation tools for Pi, powered by [`agent-browser`](https://github.com/vercel-labs/agent-browser).

Use this package when Pi needs to operate real web pages, extract rendered context, keep login sessions, or capture visual evidence without falling back to ad-hoc shell commands.

## What you get

- **Rendered-page context extraction** — read text from public pages or the active authenticated browser tab.
- **Reliable interaction handles** — use accessibility snapshots with stable `@ref` targets for clicks and form fills.
- **Evidence capture** — save screenshots and extracted text to predictable artifact paths.
- **Session persistence** — reuse login state with named `agent-browser` sessions and restore support.
- **Safer browsing controls** — constrain work with `allowedDomains`, `maxOutput`, and explicit state management.
- **Typed Pi tools** — each browser action has its own schema instead of one free-form command string.

## Requirements

- Pi / Feynman runtime with package support
- [`agent-browser`](https://github.com/vercel-labs/agent-browser) installed and available on `PATH`
- Chrome for Testing installed through `agent-browser install`

Check your local setup:

```bash
agent-browser doctor
```

If `agent-browser` is installed outside `PATH`, set:

```bash
export AGENT_BROWSER_BIN=/absolute/path/to/agent-browser
```

## Installation

Install from npm:

```bash
pi install npm:@53able/pi-agent-browser
```

Install from GitHub:

```bash
pi install https://github.com/53able/pi-agent-browser
```

Install from a local checkout:

```bash
git clone https://github.com/53able/pi-agent-browser.git
pi install ./pi-agent-browser
```

## Registered tools

- `agent_browser_open` — open a URL or launch `about:blank`
- `agent_browser_read` — extract text or JSON from a URL or active rendered tab
- `agent_browser_snapshot` — get accessibility tree refs for interaction
- `agent_browser_click` — click by `@ref` or CSS selector
- `agent_browser_fill` — fill an input by `@ref` or CSS selector
- `agent_browser_screenshot` — capture screenshots to disk
- `agent_browser_eval` — evaluate JavaScript in the active tab
- `agent_browser_state` — save, load, list, show, rename, clear, or clean saved state
- `agent_browser_close` — close a browser session
- `agent_browser_handoff` — pause and hand control to a human for 2FA, CAPTCHA, or identity verification
- `agent_browser_login_handoff` — spawn an unattended browser for interactive login flows (Google, OAuth)
- `agent_browser_commit` — confirmation gate for irreversible actions (payment, delete, publish, send)
- `agent_browser_doctor` — run `agent-browser doctor`

## Common workflows

### Read a page and save the extracted source

Ask Pi:

```text
Read https://example.com through the browser and save the extracted text to outputs/browser/example.md.
```

Equivalent `agent_browser_read` arguments:

```json
{
  "url": "https://example.com",
  "outputPath": "outputs/browser/example.md",
  "allowedDomains": ["example.com"]
}
```

### Interact with a rendered page

1. Open the page.

```json
{
  "url": "https://example.com",
  "session": "research-example",
  "allowedDomains": ["example.com"]
}
```

2. Call `agent_browser_snapshot` to get fresh `@ref` values.
3. Use `agent_browser_click` or `agent_browser_fill` with those refs.
4. Call `agent_browser_read` for text or `agent_browser_screenshot` for visual evidence.
5. Close the session with `agent_browser_close` when finished.

### Keep login state across runs

Use a stable session name and `restore: true`:

```json
{
  "url": "https://example.com",
  "session": "work-example",
  "restore": true,
  "headed": true
}
```

After logging in once, future opens with the same `session` and `restore` can reuse saved browser state.

### Inspect or clear saved state

List saved states:

```json
{
  "operation": "list"
}
```

Clear one saved session state:

```json
{
  "operation": "clear",
  "sessionName": "work-example"
}
```

Clear all saved states only when intentionally resetting everything:

```json
{
  "operation": "clear",
  "all": true
}
```

## Safety guidance

- Pass `allowedDomains` whenever the browsing scope is known.
- Use `outputPath` when extracted content supports a report, audit, or later verification.
- Take a fresh snapshot after navigation, form submission, modal dismissal, or a failed click.
- Prefer `agent_browser_read` for textual evidence and `agent_browser_screenshot` for visual evidence.
- Use `agent_browser_eval` only when `read` or `snapshot` cannot expose the required state.
- Review saved browser state before clearing broadly.

## Human handoff

Not every step should run unattended. Three tools hand control back to a human at the right moments:

- **Safe boundaries** — `agent_browser_handoff` pauses the agent for steps it cannot or should not perform itself: 2FA/OTP entry, CAPTCHA, identity verification. It captures a screenshot for context, then blocks on a real confirmation dialog until a human resumes or aborts. `agent_browser_read` and `agent_browser_snapshot` also run a lightweight keyword scan and flag likely checkpoints in their output as a nudge to call this tool.
- **Interactive login flows** — `agent_browser_login_handoff` solves the problem that Google, OAuth, and similar providers block sign-in when agent-browser is used, because automation detection happens during the login step itself. This tool spawns an unattended Chrome process (with no CDP attached) so the human can log in normally. After login succeeds, the still-running browser is live-attached to the named session and **kept open as the permanent backing browser**—it remains visible on screen with an open remote-debugging port until closed with `agent_browser_close`. Use this instead of `agent_browser_open` whenever a task is expected to hit an interactive Google or OAuth sign-in flow. Agents should call `agent_browser_close` as soon as the session is no longer needed, rather than leaving the debugging port open indefinitely.
- **Stop before commit** — `agent_browser_commit` is a dedicated gate for irreversible actions: payment, deletion, publishing, sending. It captures evidence, requires explicit human confirmation before clicking, and refuses outright (without clicking) if no interactive UI is available to ask. Use it instead of `agent_browser_click` for the final confirming step of any such action.

All three tools require an interactive session (TUI or RPC with UI support); in headless/unattended runs they refuse rather than proceed silently.

`agent_browser_open` remembers whether each named session was launched with `headed: true`. If `agent_browser_handoff` is called for a session that was not opened headed, it automatically warns the human and includes the exact `agent_browser_open` arguments (session name, `headed: true`, `restore: true`, and the last known URL) needed to reopen a visible window — no more guessing which command to rerun. Prefer `headed: true` up front for any task that might hit a login, 2FA, CAPTCHA, or identity-verification step, since some challenges are single-use and reopening mid-flow may require restarting the login. `agent_browser_login_handoff` records the resulting session as `headed: true` (since the login browser is visible and running); you don't need to reopen it to see it.

## Troubleshooting

Run:

```bash
agent-browser doctor
```

Or ask Pi to run `agent_browser_doctor`. This verifies installation, Chrome, daemon, security, providers, network, and launch tests. It also reports any live login-handoff browsers left open and cleans up stale tracking entries.

If you have sessions created with `agent_browser_login_handoff`, `agent_browser_doctor` will report any that still have live login browsers running, showing the session name, process ID, debugging port, and last known URL. Call `agent_browser_close` for each session when done — these hold an open remote-debugging port until closed. Stale entries (tracking data for processes no longer running) are cleaned up automatically with no notification. `agent_browser_login_handoff` itself also names any *other* sessions still running a live login browser in its success message, so you notice without a separate `agent_browser_doctor` call.

Common fixes:

```bash
npm install -g agent-browser
agent-browser install
```

## Project history

If you maintain or contribute to this package, read the project history in [English](docs/project-history.en.md) or [Japanese](docs/project-history.ja-JP.md) for the package-name decision, npm Trusted Publishing setup, and semantic-release workflow.

## Release automation

Releases are automated with semantic-release. Maintainers merge Conventional Commits into `main`; GitHub Actions runs `npx semantic-release@25`, determines the next version, publishes to npm through Trusted Publishing, creates the Git tag, and writes the GitHub Release.

Commit message examples:

- `fix: handle empty snapshot output` → patch release
- `feat: add browser state export tool` → minor release
- `feat!: rename tool arguments` or `BREAKING CHANGE:` → major release

The npm Trusted Publisher should continue to point at `.github/workflows/publish.yml`. No long-lived `NPM_TOKEN` is required.

## License

MIT