# Pi Web Core

A focused web search and page-fetching extension for the [Pi coding agent](https://github.com/badlogic/pi-mono).

Pi Web Core exposes two tools:

- `web_search` — search the web through a configured Provider.
- `web_fetch` — fetch HTTP(S) pages as readable Markdown or raw text.

## Why Pi Web Core

- **Minimal tool footprint** — registers exactly two tools and no extra commands, shortcuts, browser UI, or workflow tools.
- **Context efficient** — small tool schemas reduce prompt and tool-selection overhead for the Agent.
- **Focused API** — search and page fetching stay predictable without media, PDF, repository-cloning, or content-cache APIs.
- **Broad Provider support** — keeps a compact public surface while supporting configurable search and fetch routing behind it.
- **Secure fetching** — includes SSRF protection, redirect validation, response limits, domain policies, and credential redaction.

## Installation

```bash
pi install npm:pi-web-core
```

Pi Web Core requires Pi 0.37.3 or newer.

## Tools

### `web_search`

Search the web and return an answer with structured source citations when the selected Provider supplies them.

```ts
web_search({ query: "TypeScript 5.8 release notes" })

web_search({
  queries: [
    "React Server Components performance benchmarks",
    "React Server Components caching guidance",
  ],
  numResults: 10,
  recencyFilter: "month",
  domainFilter: ["react.dev", "github.com", "-example.com"],
})
```

Parameters:

| Parameter | Description |
| --- | --- |
| `query` | A single search query. |
| `queries` | Multiple queries executed sequentially. |
| `numResults` | Requested results per query, from 1 to 20. |
| `recencyFilter` | `day`, `week`, `month`, or `year`. |
| `domainFilter` | Domains to include. Prefix a domain with `-` to exclude it. |
| `proxy` | Optional HTTP(S) proxy URL for this call. |

Search Provider selection is configured in `web-search.json` rather than through the tool schema.

Supported search Providers include:

- OpenAI and Codex Hosted Search
- Exa
- Brave
- Parallel and Parallel MCP
- TinyFish
- Search1API
- Searchinfinity
- Querit
- Tavily
- Firecrawl
- Jina Search (`jina`, using `https://s.jina.ai`)
- SERPdive
- Kagi
- Bocha
- Ollama
- SearXNG
- DuckDuckGo
- Perplexity
- Gemini API and Gemini Web
- Kimi
- AnySearch
- XCrawl
- xAI
- Bright Data SERP
- SerpBase
- Serper
- Valyu

### `web_fetch`

Fetch one or more HTTP(S) URLs.

```ts
web_fetch({ url: "https://example.com/article" })

web_fetch({
  urls: [
    "https://example.com/guide",
    "https://example.com/reference",
  ],
})

web_fetch({
  url: "https://example.com/api/status",
  mode: "raw",
})
```

Parameters:

| Parameter | Description |
| --- | --- |
| `url` | A single absolute HTTP(S) URL. |
| `urls` | Up to 10 absolute HTTP(S) URLs. |
| `mode` | `readable` for Markdown extraction or `raw` for the exact textual response body. |
| `proxy` | Optional HTTP(S) proxy URL for this call. |

Readable mode uses Readability and Turndown to convert HTML into Markdown. Text, JSON, XML, and Markdown responses are returned directly.

Fetch Provider routing supports:

- Direct HTTP
- Firecrawl
- Jina Reader
- TinyFish
- Search1API
- Querit
- Kagi
- Ollama
- Parallel and Parallel MCP
- Bright Data Web Unlocker
- Gemini URL Context and Gemini Web

Remote hosted Fetch Providers are disabled unless `fetchRouting.allowRemoteHostedProviders` is explicitly enabled.

## Oversized Content

The default inline result limit is 30,000 characters. Configure it with `maxInlineContentChars`, up to 200,000 characters.

When a fetched page exceeds the limit, Pi Web Core returns a preview and writes the complete content to a restricted temporary file:

```text
/tmp/pi-web-core-XXXXXX/content.md
```

The tool result includes the path in `details.filePath` for a single URL or `details.tempFiles` for multiple URLs. Temporary directories use mode `0700`, files use mode `0600`, and session shutdown triggers best-effort cleanup.

## Configuration

The default configuration file is:

```text
~/.pi/web-search.json
```

`PI_CODING_AGENT_DIR` takes precedence. When it is unset, `XDG_CONFIG_HOME/pi` is also supported.

Example:

```json
{
  "provider": "openai",
  "openaiApiKey": "$OPENAI_API_KEY",
  "jinaApiKey": "$JINA_API_KEY",
  "openaiResponsesUrl": "https://api.openai.com/v1/responses",
  "searchRouting": {
    "providers": ["openai", "exa", "brave"],
    "useCurrentModel": false,
    "fallbackOn": [
      "transient",
      "quota",
      "network",
      "invalid-response",
      "unsupported"
    ]
  },
  "fetchRouting": {
    "providers": ["http", "firecrawl", "jina", "tinyfish"],
    "allowRemoteHostedProviders": false
  },
  "maxInlineContentChars": 30000,
  "proxy": "http://127.0.0.1:8080",
  "fetchContent": {
    "domainPolicy": {
      "allow": ["example.com"],
      "deny": ["blocked.example.com"]
    }
  },
  "ssrf": {
    "allowRanges": [],
    "trustEnvProxy": false
  }
}
```

### Search Provider Selection

Use one of the following configuration forms.

A single Provider:

```json
{
  "provider": "openai"
}
```

Concurrent aggregation:

```json
{
  "provider": ["openai", "exa"]
}
```

All eligible Providers:

```json
{
  "provider": "all"
}
```

Ordered fallback routing:

```json
{
  "searchRouting": {
    "providers": ["openai", "exa", "brave"],
    "useCurrentModel": false,
    "fallbackOn": ["transient", "quota", "network", "invalid-response"]
  }
}
```

A top-level `provider` or `searchProvider` takes precedence over `searchRouting`.

### Credentials

Provider credential fields support three forms:

```json
{
  "braveApiKey": "literal-key",
  "exaApiKey": "$EXA_API_KEY",
  "openaiApiKey": "!/absolute/path/to/secret-command"
}
```

- A literal value is used directly.
- `$NAME` or `${NAME}` reads a named environment variable.
- `!command` resolves the credential at request time.
- Prefix `$$` or `$!` to escape a literal leading `$` or `!`.

Common environment variables include:

```text
OPENAI_API_KEY
BRAVE_API_KEY
EXA_API_KEY
PARALLEL_API_KEY
TINYFISH_API_KEY
SEARCH1API_KEY
SEARCHINFINITY_API_KEY
QUERIT_API_KEY
TAVILY_API_KEY
JINA_API_KEY
SERPDIVE_API_KEY
KAGI_API_KEY
OLLAMA_API_KEY
SEARXNG_BASE_URL
PERPLEXITY_API_KEY
GEMINI_API_KEY
XAI_API_KEY
BRIGHTDATA_API_KEY
```

## Proxy Support

Set a global proxy in the configuration file:

```json
{
  "proxy": "http://proxy.example.com:8080"
}
```

Or pass `proxy` to an individual tool call. An empty string forces a direct connection.

Proxy requests use a curl-backed transport. `NO_PROXY` is honored, and localhost is never sent through the configured proxy.

## Security

Pi Web Core applies the following protections to page fetching:

- Only absolute HTTP(S) URLs are accepted.
- Loopback, private, link-local, metadata, and reserved addresses are blocked.
- DNS results and every redirect target are validated.
- `fetchContent.domainPolicy` can restrict allowed and denied hostnames.
- Responses are streamed with a 5 MiB limit.
- Hosted Fetch Providers require explicit opt-in.
- Credential headers are removed on cross-origin API redirects.
- Provider credentials are redacted from surfaced errors.
- Inline `data:` URIs are replaced with bounded omission markers.

Networks using TUN or fake-IP ranges can add a narrow exception:

```json
{
  "ssrf": {
    "allowRanges": ["198.18.0.0/15"]
  }
}
```

Avoid broad CIDR exceptions.

## Architecture

```text
src/
├── index.ts
├── config/          shared config, paths, and credentials
├── tools/           web_search, web_fetch, and temporary output
├── search/          search types, routing, and lazy Provider registry
├── fetch/           fetching, HTML extraction, routing, and registry
├── providers/       Provider API adapters
├── net/             proxy, curl, API redirects, CIDR, and SSRF
└── observability/   request activity tracking
```

Search and Fetch Routers depend only on typed registries, not concrete Provider implementations. Provider adapters are loaded lazily and share one signature-aware configuration store.

## Development

```bash
npm install
npm run typecheck
npm test
```

Additional checks used by the project:

```bash
npx tsc --noEmit --noUnusedLocals --noUnusedParameters
npx knip --include files,exports,dependencies
npm pack --dry-run
```

## Fork and Acknowledgements

Pi Web Core is a fork of [nicobailon/pi-web-access](https://github.com/nicobailon/pi-web-access). The original project established the multi-Provider search and extraction foundation that this package builds on.

This fork is maintained as an independent package with a focused two-tool API and a configuration-controlled Provider model.

## License

MIT