# pi-read-page

Let [pi](https://github.com/earendil-works/pi-coding-agent) read webpages through your local browser and return Markdown.

## What it provides

- One read-only Agent tool: `read-page`.
- Local Chrome/Chromium rendering.
- Manual handoff for login/captcha/blocked states.
- Markdown output with pagination and cache.
- Defensive defaults for untrusted webpages and private-network access.

## Requirements

- pi.
- A local Chrome/Chromium browser.
- Bun only if you are developing or running tests locally.

`pi-read-page` uses `playwright-core`; it does not download a browser. By default it launches the `chrome` channel. On macOS, Chrome is opened through Launch Services in the background and Playwright connects over CDP, so ordinary reads do not take foreground focus. Set `READ_PAGE_CHROME_PATH` or `READ_PAGE_BROWSER_CHANNEL` if needed.

## Installation

Install from npm:

```bash
pi install npm:pi-read-page
```

Try it for one pi run without installing:

```bash
pi -e npm:pi-read-page
```

Install from GitHub if you want the latest repository version:

```bash
pi install https://github.com/Sukitly/pi-read-page
```

Use a local checkout:

```bash
git clone https://github.com/Sukitly/pi-read-page.git
cd pi-read-page
bun install
pi -e .
```

## Usage

Ask pi to read a URL:

```text
Read https://example.com
```

The extension registers one Agent-facing tool:

```text
read-page(url, offset?, limit?, refresh?, preserveQuery?)
```

Parameters:

| Parameter | Default | Description |
| --- | --- | --- |
| `url` | required | HTTP or HTTPS URL to read. |
| `offset` | `1` | 1-based line offset for pagination. |
| `limit` | `300` | Number of lines to return. Maximum `1000`. |
| `refresh` | `false` | Force browser re-extraction and overwrite cache. |
| `preserveQuery` | `false` | Preserve URL query parameters. By default query params are stripped for canonical cache keys. |

Use the returned `Next offset` to continue reading long pages.

## How extraction works

```text
URL normalization and private-network policy
  -> headed browser (background launch on macOS)
  -> DOMContentLoaded + network idle wait
  -> final URL private-network policy
  -> read-only lazy-load scroll
  -> open shadow-root flattening
  -> URL absolutization
  -> Defuddle HTML/Markdown extraction
  -> confidence and handoff detection
  -> local cache write
  -> paginated Markdown output
```

Concurrent reads share one background Chrome process and use independent tabs, with a default limit of four active pages. Each call closes only its own tab. After the final active page closes, the browser waits briefly for related work and then exits.

If a page appears to require a real user action, handoffs are serialized: pi brings only that page to the foreground and shows a confirmation prompt. Complete the login/captcha/manual navigation, then confirm in pi. The same page is settled and extracted again.

## Cache

Successful browser extractions are cached under:

```text
~/.pi/agent/caches/read-page
```

Cache behavior:

- Normal TTL: 30 days.
- User-action TTL: 1 day.
- Cache files: `content.md` and `meta.json`.
- Writes are atomic.
- Cached Markdown is sha256-verified on load.
- If refresh/extraction fails and a cache entry exists, the tool returns cached content with an explicit `refresh-failed-fresh` or `stale-fallback` status.

## Security model

`read-page` treats webpages as untrusted external content.

- The output includes a security notice and document boundary.
- The Agent is instructed not to follow instructions from the page unless the user explicitly asks.
- Private/local hosts and IPs are blocked by default.
- Browser automation is read-only: it may navigate, wait, scroll, extract DOM, and cache content.
- The extension does not expose browser mutation/control tools to the Agent.
- On macOS, background launch opens an unauthenticated DevTools endpoint on a random `127.0.0.1` TCP port while Chrome is running. Any other local process running as the same user can connect to that endpoint, control the browser and access its persistent login state outside this extension's network policy. Set `READ_PAGE_MACOS_BACKGROUND=0` to avoid this local CDP endpoint.
- User handoff is only used for actionable captcha, blocked/interstitial, or explicit login-wall states.

To intentionally allow private/local network URLs:

```bash
READ_PAGE_ALLOW_PRIVATE_NETWORK=1 pi
```

## Configuration

Optional environment variables:

| Variable | Default | Description |
| --- | --- | --- |
| `READ_PAGE_CHROME_PATH` | unset | Explicit Chrome/Chromium executable path. |
| `READ_PAGE_BROWSER_CHANNEL` | `chrome` | Playwright browser channel. |
| `READ_PAGE_MACOS_BACKGROUND` | enabled on macOS | Set to `0` to avoid the unauthenticated loopback CDP port and use Playwright's direct launcher, which may take foreground focus. |
| `READ_PAGE_MAX_CONCURRENCY` | `4` | Maximum number of browser pages extracted concurrently. Clamped to `1`-`16`. |
| `READ_PAGE_IDLE_CLOSE_MS` | `500` | Delay before closing Chrome after the final page lease is released. Clamped to `0`-`10000`. |
| `READ_PAGE_PROFILE_DIR` | `~/.pi/agent/read-page/browser-profile` | Persistent browser profile directory. |
| `READ_PAGE_DISABLE_TEMP_PROFILE_FALLBACK` | unset | Set to `1` to fail instead of using a temporary profile when the persistent profile is locked. |
| `READ_PAGE_ALLOW_PRIVATE_NETWORK` | unset | Set to `1` to allow private/local network access. |
| `READ_PAGE_PARSE_TIMEOUT_MS` | `8000` | Defuddle parse timeout before sync fallback. |
| `READ_PAGE_DEFUDDLE_ASYNC` | unset | Set to `1` to allow Defuddle third-party async extraction. |
| `READ_PAGE_DEFUDDLE_DEBUG` | unset | Set to `1` to include Defuddle debug information. |

## Development

Install dependencies:

```bash
bun install
```

Run deterministic checks:

```bash
bun run lint
bun test
```

Run the browser integration test:

```bash
bun run integration -- https://example.com
bun run integration -- https://example.com https://example.org
```

The integration test opens a real browser in the background on macOS, reads all supplied URLs concurrently, prints extraction metadata, and closes the browser context.

## Publishing

Pi package catalog entries are discovered from public npm packages with the `pi-package` keyword.

Authenticate with npm first:

```bash
npm login
npm whoami
```

Run a release by choosing the semantic version increment:

```bash
bun run release -- patch
bun run release -- minor
bun run release -- major
```

The release script requires a clean, up-to-date `main` or `master` branch. It verifies npm authentication and the currently published version, runs lint, tests, and `npm pack --dry-run`, asks for confirmation, creates the `Release vX.Y.Z` commit and annotated `vX.Y.Z` tag, pushes them, publishes the public npm package, and verifies the npm `latest` tag.

Run only the preflight checks without changing or publishing anything:

```bash
bun run release -- minor --dry-run
```

Use `--yes` or `-y` to skip confirmation in a non-interactive environment. If npm publication fails after the git tag is pushed, fix the npm error and retry `npm publish --access public`; do not create another version.

After publishing, install with:

```bash
pi install npm:pi-read-page
```

## Project layout

```text
extensions/pi-read-page.ts      extension entrypoint
src/tools/read-page.ts          tool orchestration, output formatting, TUI rendering
src/browser/                    browser lifecycle, extraction, handoff, confidence
src/cache/cache.ts              cache, pagination, checksums
src/security/url-policy.ts      URL normalization and private-network policy
test/                           deterministic unit tests
scripts/integration-read-page.ts browser integration runner
```

## Troubleshooting

### Chrome is not found

Install Google Chrome/Chromium, or set:

```bash
READ_PAGE_CHROME_PATH=/path/to/chrome pi
```

### Login state is missing

By default the extension uses a persistent profile at:

```text
~/.pi/agent/read-page/browser-profile
```

If that profile is already locked by another browser process, `read-page` falls back to a temporary profile. The tool output will include a warning when this happens.

### Query parameters were removed

Set `preserveQuery: true` when query parameters are required for the page content, such as search results, filters, or app/detail pages.

### Localhost or private IP is blocked

This is intentional. Use `READ_PAGE_ALLOW_PRIVATE_NETWORK=1` only when you explicitly want to read local/private services.

## License

MIT. See [LICENSE](LICENSE).