# @fadouse/pi-web

A Pi extension that provides three selectable web-search pipelines, pageable artifacts, and Impit-only multi-format resource fetching.

```text
Main Agent
  ├─ web_search
  │    ├─ OpenAI Direct        → /alpha/search → native report
  │    ├─ OpenAI + Summary     → /alpha/search → isolated per-result summaries
  │    └─ Exa Direct           → Exa /search → compact excerpts
  ├─ web_fetch(url, format, offset, limit)
  └─ web_fetch_read(fetch_id, format, offset, limit)
```

## Design goals

- The main Agent calls search directly; no Search Agent or Agent Session is created or dispatched.
- OpenAI Standalone Search sends exactly one `/alpha/search` POST per tool call, with retries explicitly disabled.
- Summary mode places every result in a fresh, single-message completion context. Up to four summaries run concurrently without sharing context.
- Exa mode does not call a summary model. It deterministically returns titles, compact excerpts, exact result IDs, and direct URLs.
- Each search mode has its own tool prompt. The main Agent sees only the currently available capabilities—not plugin configuration, backend composition, or model settings.
- Complete reports and raw responses are stored as separate artifacts. Search reports expose a local `report_path` for Pi's built-in `read`; fetched content remains pageable through `web_fetch_read`.
- `web_fetch` uses Impit exclusively. It does not execute JavaScript, launch a browser, or fall back to another fetch backend.
- Before a model request, successful Web tool bodies from older user turns are replaced with compact receipts. Session entries and on-disk artifacts remain unchanged.

## Installation

```bash
pi install /path/to/pi-web
```

For development:

```bash
npm install
pi -e ./src/index.ts
```

Node.js 22.19 or later is required.

## Search configuration

Run:

```text
/web-config
```

Choose one of three pipelines:

1. **OpenAI direct (no summary)** — returns `/alpha/search` output directly and requires no summary model.
2. **OpenAI + summary model** — splits `/alpha/search` output and summarizes each result in an independent completion.
3. **Exa API (direct results)** — calls the Exa Search API directly and uses no summary model.

Global configuration is stored at `~/.pi/agent/pi-web.json`. Trusted projects may instead use `.pi/pi-web.json`.

### OpenAI Direct

```json
{
  "provider": "openai-codex",
  "model": "<standalone-search-model>",
  "mode": "cached",
  "search_context_size": "medium",
  "allowed_domains": ["example.com"],
  "max_output_tokens": 10000,
  "timeout_ms": 300000
}
```

### OpenAI + Summary

```json
{
  "provider": "openai-codex",
  "model": "<standalone-search-model>",
  "summary_provider": "anthropic",
  "summary_model": "<summary-model>",
  "mode": "live",
  "timeout_ms": 300000
}
```

`summary_provider` and `summary_model` must either both be present or both be omitted.

### Exa Direct

```json
{
  "provider": "exa",
  "exa_api_key": "<your-exa-api-key>",
  "mode": "live",
  "allowed_domains": ["example.com"],
  "timeout_ms": 300000
}
```

When selected, the Exa key is stored in plain text inside `pi-web.json`. The extension forces file mode `0600`, and `/web-config show` displays only `[configured]` instead of the key. Never commit a project-level `.pi/pi-web.json` file.

Configuration precedence:

```text
environment > trusted project config > global config
```

Primary environment variables:

```bash
PI_WEB_PROVIDER=openai-codex
PI_WEB_MODEL='<standalone-search-model>'
PI_WEB_SUMMARY_PROVIDER=anthropic
PI_WEB_SUMMARY_MODEL='<summary-model>'
PI_WEB_SEARCH_MODE=cached

# Exa
PI_WEB_PROVIDER=exa
EXA_API_KEY='<exa-api-key>'
```

Inspect configuration and paths with:

```text
/web-config show
/web-config paths
```

Both OpenAI stages reuse Pi's ModelRegistry authentication. Running `/web-config` switches the active tool description immediately. After editing a configuration file manually, run `/reload`.

## `web_search`

Minimal search:

```json
{
  "search_query": [{ "q": "latest stable release" }]
}
```

Batched search:

```json
{
  "search_query": [
    { "q": "official release notes", "domains": ["github.com"] },
    { "q": "official migration guide", "recency": 30 }
  ],
  "response_length": "medium"
}
```

| Capability | OpenAI Direct | OpenAI + Summary | Exa Direct |
|---|:---:|:---:|:---:|
| `search_query` | ✓ | ✓ | ✓ |
| `image_query` | ✓ | ✓ | — |
| `open` / `click` / `find` | ✓ | ✓ | — |
| `screenshot` | ✓ | ✓ | — |
| `finance` / `weather` / `sports` / `time` | ✓ | ✓ | — |

To inspect an Exa result in depth, pass its direct URL to `web_fetch`. Unsupported operations are rejected instead of being silently mapped to different semantics.

### Exa result limits

`response_length` controls one result budget shared by the **entire tool call**, not a separate budget for each query:

| Value | Maximum results per call |
|---|---:|
| `short` | 5 |
| `medium` (default) | 10 |
| `long` | 15 |

For example, four `long` queries receive quotas of `4 + 4 + 4 + 3 = 15`; they do not return 60 results. Exa requests only query-focused highlights of up to 1,200 characters, never complete page text. Duplicate URLs are removed during aggregation.

### OpenAI + Summary

1. The main Agent emits a structured search command.
2. The extension sends one direct `/alpha/search` POST.
3. The raw output is split into N results.
4. Each result enters an independent, single-message completion, with up to four running concurrently.
5. The extension deterministically appends the exact Source ID and direct URL.
6. Results are merged in their original order, and only the complete summary report is returned to the main Agent.

If one summary request times out, that item becomes `**Error: timeout**` while retaining its Source ID and URL. Every other result continues normally. User cancellation and non-timeout errors still propagate.

### Search artifacts

```text
report.md          # complete report returned to the main Agent
raw-search.txt     # raw OpenAI or Exa response
metadata.json      # pipeline, provider/model, IDs, counts, hashes, and usage
```

`web_search` returns the absolute `report_path`. After historical content is cleared from active context, read the stored report with Pi's built-in `read` tool:

```json
{
  "path": "/absolute/path/to/ws_.../report.md",
  "offset": 81,
  "limit": 80
}
```

No additional search-artifact tool is required.

## `web_fetch`

```json
{
  "url": "https://example.com/resource",
  "offset": 1,
  "limit": 80
}
```

When `format` is omitted, HTML/XHTML responses default to readable Markdown. Other textual formats preserve their source form, while images and binary resources return an absolute local path.

Available explicit `format` values:

- `source` — preserves textual source formats; binary resources return an absolute local path.
- `text` — returns locally extracted text from HTML, PDF, Office, and similar documents.
- `markdown` — returns normalized Markdown when it can be generated.
- `html` — compatibility alias for legacy HTML/source-view calls.

Supported content includes:

- HTML/XHTML, Markdown, plain text, and common source-code or configuration formats.
- JSON/JSONL, XML/RSS/Atom, CSV/TSV, YAML, and TOML.
- PDF, DOCX, PPTX, XLSX, ODT, ODP, ODS, RTF, and EPUB.
- JPEG, PNG, GIF, WebP, BMP, TIFF, SVG, AVIF, HEIC, and other common image formats.
- Any other binary content, which is still downloaded and exposed through a local path.

PDF, Office, and OpenDocument files are parsed locally from downloaded bytes with `officeparser`. Parsing does not make network requests or launch external programs, and both OCR and attachment extraction are explicitly disabled. Images are not OCR-processed. They are stored with their real extension, for example:

```text
~/.pi/agent/cache/pi-web/<session>/wf_.../original.png
```

The main Agent can read the image from that path. Unknown binary content also preserves its exact original bytes and is never misrepresented as text.

Depending on the source, a successful fetch stores:

```text
original.<ext>     # exact bytes returned by Impit
source.<ext>       # decoded source-format text, when available
content.txt        # extracted plain text, when available
content.md         # generated Markdown, when available
metadata.json      # MIME type, format, paths, redirects, sizes, and SHA-256 hashes
```

Continue reading the same immutable snapshot with:

```json
{
  "fetch_id": "wf_...",
  "format": "text",
  "offset": 201,
  "limit": 80
}
```

`web_fetch_read` never performs another network request.

## Pagination limits

`web_fetch` and `web_fetch_read` share the following extension-level pagination limits:

| Limit | Value |
|---|---:|
| Default lines | 80 |
| Maximum lines per call | 200 |
| Hard body limit per page | 8 KiB |
| Maximum virtual-line size | 2 KiB UTF-8 |

When a limit is reached, the tool returns an exact `next_offset` and an example continuation call.

## Fetching and security

`web_fetch` performs the following steps:

1. Accept only credential-free HTTP(S) URLs and validate the initial address.
2. Send a GET through `Impit({ browser: "chrome", vanillaFallback: false })`.
3. Follow at most five redirects manually, repeating DNS and SSRF validation at every hop.
4. Stream at most 10 MiB of response data.
5. Preserve the original bytes, then classify the content from MIME type, filename, and magic bytes.
6. Preserve source views for text, extract documents locally, and return paths for images or unknown binary content.

Additional safeguards:

- Blocks localhost, private networks, metadata endpoints, reserved addresses, IPv4-mapped IPv6 addresses, and URLs containing credentials.
- Resolves and validates every A and AAAA address again after each redirect.
- Uses directory mode `0700` and file mode `0600` for artifacts.
- Retains artifacts for seven days by default, with a 256 MiB per-session limit.
- Gives each summary request exactly one untrusted result, no tools, and no shared context.

## Development

```bash
npm run check
npm pack --dry-run
```

## License

MIT