# pi-web-search-and-fetch 🌐⚡

[![npm version](https://img.shields.io/npm/v/pi-web-search-and-fetch?color=blue&logo=npm)](https://www.npmjs.com/package/pi-web-search-and-fetch)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
[![pi Extension](https://img.shields.io/badge/pi-extension-purple.svg)](https://pi.dev)
[![pi 1.0.3 Verified](https://img.shields.io/badge/pi-1.0.3%20verified-teal.svg)](https://github.com/xinaps-dev/pi-web-search-and-fetch)

> **Empower your [pi](https://pi.dev) coding agent with real-time web intelligence: instant neural search, clean Markdown page extraction, and autonomous deep research.**

`pi-web-search-and-fetch` transforms your **pi** coding agent into an autonomous web researcher. Say goodbye to outdated training cutoffs and missing documentation — equip your LLM with enterprise-grade web search, high-fidelity webpage retrieval, and multi-query deep synthesis. Built with a modular multi-provider engine, an interactive `/ws` terminal dashboard, and seamless zero-conflict integration with `pi-requesty-provider`.

---

## ⚡ Superpowers & Features

- 🔍 **Real-Time Neural Web Search (`web_search`)** — Search the live web with semantic and category filtering (`news`, `github`, `research paper`, `pdf`, `company`, `tweet`, `financial report`) to break past knowledge cutoffs.
- 📄 **High-Fidelity Web Fetch (`web_fetch`)** — Instantly retrieve live documentation, articles, and repositories, converting raw HTML into clean, token-optimized Markdown.
- 🧠 **Autonomous Deep Research (`web_deep_search`)** — Multi-angle iterative exploration that executes parallel queries and synthesizes multi-source findings for complex inquiries.
- 🎛️ **Interactive TUI Control Hub (`/ws`)** — Full-featured terminal dashboard to toggle tools on the fly, switch provider backends, and configure API keys with smooth keyboard controls.
- 🚀 **Zero-Config Instant Start** — Works immediately out of the box with Exa's public MCP endpoint (no API key required) or plug in your own `EXA_API_KEY` for unlimited bandwidth and deep search.
- 🔌 **Decoupled Multi-Provider Architecture** — Independent capability interfaces (`SearchProvider`, `FetchProvider`, `DeepSearchProvider`). Mix and match providers freely or implement custom engines with minimal boilerplate.
- 🤝 **Smart `pi-requesty-provider` Synergy** — Automatically coordinates with `pi-requesty-provider` to detect native server-side search models, preventing duplicate searches while keeping markdown extraction active.
- 🔒 **Secure Credential Management** — Stores API keys safely in `~/.pi/agent/auth.json` with strict `0o600` file permissions.

---

## ✅ Compatibility

This extension is **compatible with and verified against [pi](https://pi.dev) `1.0.3`** — the latest stable release. The full test suite and the TypeScript typecheck pass against pi `1.0.3`.

The supported peer dependency range is `^0.84 || ^1.0.0`, so it works with both the `0.84.x` and `1.0.x` lines of pi.

---

## 🚀 Quick Installation

### From npm (Recommended)

```bash
pi install npm:pi-web-search-and-fetch
```

### From GitHub

```bash
pi install git:github.com/xinaps-dev/pi-web-search-and-fetch
```

### Local Development

```bash
# Clone and build
git clone https://github.com/xinaps-dev/pi-web-search-and-fetch.git
cd pi-web-search-and-fetch
pnpm install

# Run locally in pi
pi -e ./
```

---

## 🎮 Getting Started & The `/ws` Control Hub

Managing your web tools is effortless with the unified `/ws` command.

### Interactive Hub Dashboard

Run `/ws` inside `pi` to open the interactive TUI Control Panel:

```text
┌───────────────────────────────────────────────────────────┐
│  🌐 Web Search and Fetch - Control Panel                             │
├───────────────────────────────────────────────────────────┤
│  [✓] Search (web_search)          : ON  (Provider: exa)   │
│  [✓] Fetch (web_fetch)           : ON  (Provider: exa)   │
│  [ ] Deep Search (deep)          : OFF (Provider: exa)   │
│                                                           │
│  Actions:                                                 │
│  > Assign Providers (3-tool wizard)                       │
│  > Configure Active Provider (Exa API Key / Mode)         │
│  > View Detailed Status                                   │
│  > Exit                                                   │
└───────────────────────────────────────────────────────────┘
```

- **Navigate:** Use `↑` / `↓` arrow keys to browse options.
- **Toggle / Select:** Press `Space` or `Enter` to toggle tools or trigger configuration wizards.
- **Close:** Press `Esc` or select `Exit`.

### Text-Mode Subcommands

Prefer the keyboard? Every hub action is also available as a direct command:

```text
/ws                            Open the interactive control hub
/ws status                     Show the detailed status report
/ws search on|off              Enable/disable web_search
/ws fetch on|off               Enable/disable web_fetch
/ws deep on|off                Enable/disable web_deep_search
/ws provider <tool> <id|none>  Assign a provider (tool: search|fetch|deep)
/ws provider                   Interactive provider assignment wizard
/ws config [providerId]        Configure a provider (e.g. Exa API key)
/ws help                       Show usage help
```

---

## 🛠️ Standardized LLM Tools

`pi-web-search-and-fetch` registers 3 powerful tools with the Pi agent harness:

### 1. `web_search` *(Enabled by default)*
Performs real-time web searches and returns structured results with titles, URLs, snippets, publication dates, and citations.

| Parameter | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| `query` | `string` | **Yes** | Search keywords or natural language question |
| `numResults` | `number` | No | Number of results to return (default: `10`) |
| `category` | `string` | No | Content category (`company`, `research_paper`, `news`, `pdf`, `github`, `tweet`, `personal_site`) |
| `includeDomains` | `string[]` | No | Restrict results to these domains |
| `excludeDomains` | `string[]` | No | Exclude these domains from the results |
| `startPublishedDate` | `string` | No | Only return results published on or after this ISO date |
| `endPublishedDate` | `string` | No | Only return results published on or before this ISO date |
| `similarUrl` | `string` | No | When provided, returns pages similar to that URL (Exa find-similar) |

### 2. `web_fetch` *(Enabled by default)*
Fetches full web page content from known URLs and converts it into clean, LLM-ready Markdown. Supports batch fetching of multiple pages in a single call.

| Parameter | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| `urls` | `string \| string[]` | Yes* | One or multiple webpage URLs to fetch |
| `url` | `string` | No | Single-URL compatibility alias for `urls` |
| `maxCharacters` | `number` | No | Maximum extracted content length per page (default: `5,000`) |

\* At least one of `urls` or `url` must be provided.

### 3. `web_deep_search` *(Optional — disabled by default)*
Agentic multi-query web research that executes parallel queries and synthesizes multi-source findings, or answers directly via Exa's answer tool when available. Requires your own `EXA_API_KEY`. Enable it from the `/ws` hub or with `/ws deep on`.

| Parameter | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| `query` | `string` | **Yes** | Primary research question |
| `numSources` | `number` | No | Reference sources to consult (default: `5`) |
| `includeText` | `boolean` | No | Include text extracts from cited sources (default: `true`) |
| `numResults` | `number` | No | Results per query; overrides `numSources` (default: `10`) |
| `category` | `string` | No | Content category filter |
| `additionalQueries` | `string[]` | No | Supplementary parallel sub-queries |

---

## ⚙️ Configuration & Key Storage

### Extension Configuration: `~/.pi/agent/pi-web-search-and-fetch.json`

Persistent settings are stored cleanly in your agent directory:

```json
{
  "search": {
    "enabled": true,
    "provider": "exa"
  },
  "fetch": {
    "enabled": true,
    "provider": "exa"
  },
  "deepSearch": {
    "enabled": false,
    "provider": "exa"
  },
  "providers": {
    "exa": {
      "useApiKey": true
    }
  }
}
```

### Authentication & API Keys: `~/.pi/agent/auth.json`

API keys are read and stored using Pi's standard authentication store (`~/.pi/agent/auth.json`) with strict `0o600` file permissions:

```json
{
  "exa": {
    "type": "api_key",
    "key": "your-exa-api-key-here"
  }
}
```

**Key Resolution Hierarchy for Exa:**
1. **Public MCP Tier:** If `useApiKey: false`, uses Exa's public endpoint (free, no key required).
2. **Authenticated Tier:** If `useApiKey: true`:
   - Checks `~/.pi/agent/auth.json` (`exa.key`).
   - Falls back to `EXA_API_KEY` environment variable.

Manage keys from pi with `/ws config exa` (or `/ws` → *Configure Active Provider*). Note that `web_deep_search` always requires its own API key: it is not available in the public free mode.

---

## 🧩 Provider Architecture

`pi-web-search-and-fetch` is designed with a fully decoupled capability model:

```text
┌────────────────────────────────────────────────────────┐
│                   ProviderRegistry                     │
├────────────────────────────────────────────────────────┤
│  SearchProvider      │  FetchProvider   │ DeepSearch   │
│  - exa               │  - exa           │ - exa        │
│  - (custom search)   │  - (custom fetch)│ - (custom)   │
└────────────────────────────────────────────────────────┘
```

### Implementing a Custom Provider

Easily create and register custom providers conforming to `ProviderModule`:

```typescript
import type {
  ProviderModule,
  SearchProvider,
  FetchProvider,
  DeepSearchProvider,
  SearchResponse,
  FetchResponse,
} from "pi-web-search-and-fetch";

export const myCustomProvider: ProviderModule = {
  id: "custom",
  name: "My Custom Provider",
  description: "Custom web search and scraper",
  capabilities: ["search", "fetch"],

  searchProvider: {
    id: "custom",
    name: "My Custom Provider",
    description: "Search via Custom API",
    supportsApiKey: true,
    requiresApiKey: true,
    async search(query, options, signal): Promise<SearchResponse> {
      // Implement search logic...
      return {
        query,
        provider: "custom",
        results: [
          { title: "Example", url: "https://example.com", snippet: "..." }
        ],
      };
    },
  },

  fetchProvider: {
    id: "custom",
    name: "My Custom Provider",
    description: "Fetch via Custom Scraper",
    supportsApiKey: false,
    requiresApiKey: false,
    async fetch(url, options, signal): Promise<FetchResponse> {
      // Implement fetch logic...
      return {
        url,
        provider: "custom",
        content: "# Example Page Content\n\n...",
      };
    },
  },
};
```

---

## 🤝 `pi-requesty-provider` Smart Synergy

When used alongside [`pi-requesty-provider`](https://github.com/xinaps-dev/pi-requesty-provider):
1. `pi-web-search-and-fetch` inspects `~/.pi/agent/pi-requesty.json` and evaluates the active session model on `session_start` and `model_select`.
2. If Requesty has native search enabled (`nativeSearch: true`) in `pi-requesty-provider` and the active model supports server-side search grounding, `pi-web-search-and-fetch` automatically suppresses `web_search` to prevent duplicate web queries and token waste.
3. `web_fetch` remains **fully active**, allowing your agent to extract and inspect complete web pages on demand.

---

## 🧪 Development & Testing

```bash
# Run all unit and integration tests
pnpm test

# Run TypeScript typecheck
pnpm typecheck
```

---

## 📄 License

This project is licensed under the [MIT License](LICENSE) © [xinaps](https://github.com/xinaps-dev).