# pi-anysearch-tools

[![npm](https://img.shields.io/npm/v/pi-anysearch-tools)](https://www.npmjs.com/package/pi-anysearch-tools)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](#license)

`pi-anysearch-tools` is a [Pi Coding Agent](https://pi.dev/) extension with two focused tools: `anysearch` for proactive general or vertical web search, and `anysearch_extract` for reading a selected page. Both are backed by the official AnySearch REST API `/v1` endpoints:

```text
POST https://api.anysearch.com/v1/search      (search, JSON summaries)
GET  https://api.anysearch.com/v1/sub-domains (capability discovery)
POST https://api.anysearch.com/v1/extract     (full-page extraction)
```

Use it proactively when an answer depends on current or externally verifiable information: documentation, news, prices, products, people, comparisons, fact-checking, recommendations, or vertical-domain data such as stocks, papers, legal cases, flights, drugs, code docs, and weather. It is not a replacement for Pi's local file or shell tools.

> 中文说明：这是一个面向 Pi Coding Agent 的 AnySearch 联网搜索扩展，统一使用官方 REST API `/v1`：搜索走 `/v1/search`，能力发现走 `/v1/sub-domains`，抽取走 `/v1/extract`。无需 API key 也可匿名使用，但匿名请求受速率和配额限制。AnySearch Skill 的 v3.1.1 是集成包版本，不是 API 路径版本。

## Core capabilities

- General web search as bounded Markdown (title, URL, and short snippet only — 500 chars per result, 12,000 chars total final cap including notices; use `anysearch_extract` for full page content).
- Structured results without a source URL remain visible without a fabricated citation; empty titles display as `Untitled result`.
- Vertical domain search via `domain` / `sub_domain` / `sub_domain_params` (17 domains), with capability discovery handled by the same `anysearch` entry.
- Parallel batch search: hard limit 1–5 queries per call (2–3 recommended); split larger workloads into multiple calls. Omitted `max_results` defaults to 3 per query; up to 5×10 is supported but heavy — the 12k cap keeps context bounded; a single failure does not block the rest.
- Full-page URL extraction as clean Markdown (server-truncated at 50,000 characters) — escalation path when snippets are insufficient. Explicit `.pdf`/arXiv PDF URLs, and extensionless URLs whose `Content-Type` is PDF, are downloaded directly and converted with the system `/usr/bin/pdftotext` (no OCR); direct PDF downloads are limited to 20 MiB and five redirects.
- Domain-only calls return valid sub-domains and required parameters without executing a search; directory results are cached per session.
- Region (`zone`) and language passthrough; the public tool schema accepts `max_results` only in 1–10 and rejects out-of-range values before execution (default 3 per query). Lower-level helpers retain defensive clamping for direct programmatic calls.
- Anonymous access without initial configuration.
- Optional API-key authentication from an environment variable or Pi agent configuration.
- If the API auto-registers a new key on quota exhaustion, the extension asks once via the TUI and saves it after confirmation.
- Compact Pi TUI rendering (collapsed by default, expandable).

## Installation

Install the published npm package:

```bash
pi install npm:pi-anysearch-tools
```

### Local development and temporary use

From a checkout of this repository:

```bash
# Load the extension for the current Pi invocation only
pi -e .

# Install the local directory as a Pi package
pi install .
```

## API key configuration

Configuration priority is:

```text
ANYSEARCH_API_KEY environment variable
> <agent dir>/anysearch.json
> anonymous mode
```

Set the environment variable in your shell or CI environment:

```bash
export ANYSEARCH_API_KEY="your-api-key"
```

Or create `~/.pi/agent/anysearch.json`:

```json
{
  "anysearchApiKey": "your-api-key"
}
```

`~/.pi/agent` is Pi's default agent directory. If `PI_CODING_AGENT_DIR` is set, the extension reads `anysearch.json` from that directory instead.

Anonymous mode remains usable when no key is configured, but it has stricter rate and quota limits.

### Auto-registered keys

Legacy structured responses containing `auto_registered.api_key` are still detected: the extension asks for confirmation in the TUI and writes the key to the agent configuration file after you confirm. In headless mode it instead appends a notice to the tool result. The current anonymous-quota HTTP 402 response may put generated login credentials in its message; that message is withheld, and the rejected request is not retried automatically. Configure a key with `/anysearch-setup` or `ANYSEARCH_API_KEY` before trying again.

### Pi commands

```text
/anysearch-setup
```

Opens an interactive prompt and saves the key to the Pi agent configuration file. The new key is used by subsequent searches without restarting Pi, unless a non-empty `ANYSEARCH_API_KEY` remains set; the environment variable always takes precedence.

```text
/anysearch-status
```

Shows the configuration path and reports only `已配置` (configured) or `匿名模式` (anonymous mode). It does not display the key or a key prefix.

### `postinstall` behavior

The npm package includes a `postinstall` script:

- In an interactive terminal, it may ask for an API key when none is configured. Press Enter to keep using anonymous mode.
- In a non-interactive installation, it skips the prompt and prints setup guidance; installation still succeeds.
- If npm lifecycle scripts are disabled, for example with `npm install --ignore-scripts`, the script does not run.

You can always configure the extension later with `/anysearch-setup` or `ANYSEARCH_API_KEY`.

## Tool reference

### `anysearch`

| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| `queries` | array (1–5 items) | Yes | Hard limit 5. One item is a single search; several items run in parallel after all domain inputs pass preflight. Split 6+ intents into multiple calls. |
| `queries[].query` | string | Yes | One natural-language search intent. |
| `queries[].domain` | enum (17 values) | No | Omit for general search. Domain alone returns legal sub-domain choices without executing any search. |
| `queries[].sub_domain` | string | No | A routing key returned by an earlier domain-only call, such as `finance.quote`; requires `domain`. |
| `queries[].sub_domain_params` | object | No | String-valued parameters from the domain directory; every parameter marked required must be present. |
| `queries[].max_results` | integer | No | Hard limit 1–10; out-of-range values are rejected by tool validation before execution. Default 3. |
| `queries[].zone` | `cn` \| `intl` | No | Search region. |
| `queries[].language` | string | No | Preferred language, such as `zh-CN` or `en`. |

`anysearch` has one explicit routing contract:

1. Omit `domain` for general web search.
2. Provide only `domain` to receive valid sub-domains and required parameters. No REST search runs for that item; in a mixed batch, domain-only items return a one-line capability summary while other queries execute normally.
3. Call `anysearch` again with `domain`, a returned `sub_domain`, and every required `sub_domain_params` key. The extension validates the selection against the cached directory before searching (bare suffixes like `quote` for `finance.quote` are normalized when unambiguous).

General search:

```json
{
  "queries": [
    { "query": "latest Pi Coding Agent extension documentation", "max_results": 3 }
  ]
}
```

Discover finance capabilities through the same tool:

```json
{
  "queries": [
    { "query": "AAPL price", "domain": "finance" }
  ]
}
```

Then run the selected vertical search:

```json
{
  "queries": [
    {
      "query": "AAPL",
      "domain": "finance",
      "sub_domain": "finance.quote",
      "sub_domain_params": { "type": "stock", "symbol": "AAPL", "cn_code": "" },
      "max_results": 3
    }
  ]
}
```

### `anysearch_extract`

| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| `url` | string | Yes | URL (must start with `http://` or `https://`). HTML uses AnySearch; PDFs use direct `/usr/bin/pdftotext` conversion with no OCR. Direct PDF downloads reject credentials/private, loopback, and link-local targets, limit responses to 20 MiB and five redirects, and truncate output at 50,000 characters. |

Use when search snippets are too short to answer, when the user provides a URL, or to verify a claim against the original source.

## Returned results

Search uses `POST https://api.anysearch.com/v1/search` (JSON summaries) and returns bounded Markdown containing the exact title, URL, and a whitespace-normalized snippet of at most 500 characters. The REST `content` field is discarded before the Pi tool result is built. Each query defaults to 3 results; the public tool accepts only 1–10 and rejects out-of-range values before execution. Each call accepts at most 5 queries; split larger workloads across calls. Batch output is fairly allocated and capped at 12,000 characters including notices. Domain-selection responses come from `GET https://api.anysearch.com/v1/sub-domains`, execute no search, and are rendered as directory Markdown so the returned parameter contract remains usable. Use `anysearch_extract` for full-page content when snippets are insufficient.

Search results are deduplicated before being returned — high-duplication items are filtered out and never reach the agent's context (structured dedupe; batch dedupes globally across queries). An item is dropped when its normalized URL (scheme, `www`, tracking params, trailing slash, case-insensitive) matches an earlier item, OR its normalized title (case/punctuation-insensitive, leading arXiv id stripped, >= 12 chars) is >= 0.85 similar to an earlier title (char-bigram overlap; empirical split: real duplicates >= 0.93, distinct papers <= 0.72). Kept items are renumbered, and the `## Search Results` count is rewritten. `anysearch_extract` converts the `/v1/extract` JSON response to Markdown; domain capability JSON is likewise rendered as directory Markdown.

Each tool returns Markdown plus structured details containing the response `request_id` when present and the auth mode (`anonymous` or `configured`). Search details additionally expose `outcome: "search_results" | "needs_subdomain_selection"`; selection details list the affected zero-based query indexes and legal sub-domains. Details never contain the API key. API and network failures are thrown so Pi can mark the tool result as an error.

Transient failures are absorbed instead of failing the call. Every REST/MCP request is attempted twice: a connection-level failure (undici reports these as `fetch failed` with the real reason in `cause` — `ECONNRESET`, `ETIMEDOUT`, `EAI_AGAIN`, `UND_ERR_SOCKET`, …) or any 5xx response triggers one retry after a 300 ms backoff, honoring `Retry-After` when present but capping it at 2 s. Cancellations and timeouts (`AbortSignal`) and 4xx responses are never retried. When retries are exhausted, the thrown message names the hidden transport reason (for example `AnySearch request failed after 2 attempts: fetch failed (ECONNRESET: read ECONNRESET)`) instead of a bare `fetch failed`, so a flaky proxy or DNS path is distinguishable from a failing service.

In the TUI, both general and vertical calls appear under the same `anysearch` tool. Search responses render as a numbered list; domain selection and extraction render their Markdown directly.

## Coexisting with `pi-web-access`

This package can be installed alongside [`pi-web-access`](https://github.com/nicobailon/pi-web-access). It registers only `anysearch` and `anysearch_extract`, so it does not replace tools such as `web_search`.

Use `anysearch` for routine direct lookup, current facts, and vertical-domain data. Use `web_search` only when the task requires independent multi-provider synthesis or corroboration, or when AnySearch fails. `source_check` and other `pi-web-access` features remain complementary.

## Security

- Direct PDF downloads use native `http`/`https` requests after DNS resolution; each hop rejects private, loopback, and link-local addresses and supplies the verified address through a fixed lookup callback while retaining the hostname for SNI. Redirects are revalidated one hop at a time.
- When local DNS returns only proxy fake-IP addresses (`198.18.0.0/15`), the hostname is queried through Cloudflare DNS-over-HTTPS (`https://cloudflare-dns.com/dns-query`, IPv4 A records). The returned addresses still undergo the public-address check and connection pinning. This sends the document hostname, not its path or query, to Cloudflare. Literal fake-IP URLs remain blocked.
- Direct PDF downloads never send the AnySearch API key. Non-PDF HTML URLs retain the existing upstream `/v1/extract` path; a failed/non-PDF `HEAD` probe falls back to that path.
- Direct PDF responses are capped at 20 MiB and must have both a PDF body signature (`%PDF-`) and a PDF-compatible content type when one is supplied; HTML error pages are not passed to `pdftotext`.
- PDF conversion requires the system `/usr/bin/pdftotext` command, uses a cleaned-up mode-600 temporary file, bounded output, and no OCR. Missing command, conversion failures, and documents without extractable text are reported explicitly.
- Never commit API keys to Git, source files, examples, or issue reports.
- HTTP 402 responses can contain automatically generated login credentials. Their raw messages are withheld from tool results and errors; configure an API key through the existing setup flow instead.
- Prefer `ANYSEARCH_API_KEY` for CI and managed environments.
- The configuration file stores the key locally as plain JSON; protect access to your Pi agent directory.
- Pi extensions execute with the user's permissions. Review the source before installation.
- This extension performs network requests to `https://api.anysearch.com/v1/search`, `https://api.anysearch.com/v1/sub-domains`, and `https://api.anysearch.com/v1/extract` when one of the tools is called. It does not make a network request merely by loading the extension.

## Development and verification

```bash
npm install --ignore-scripts
npm test
ANYSEARCH_E2E_API_KEY=as_sk_... npm run test:e2e  # optional real endpoints
npm pack --dry-run
```

`npm test` runs the deterministic mocked-fetch suite; real network checks are explicit so quota, registration, and connectivity cannot make the default test command flaky:

- `test/anysearch.test.ts` — deterministic mocked-fetch/native-transport coverage for the two-tool registration surface, Pi prompt metadata, domain selection, fail-closed vertical validation, REST request assembly, retry classification (transport codes, 5xx, capped `Retry-After`) and the exhaustion diagnostics, direct PDF parsing, PDF size/redirect/content-type boundaries, HTML fallback, error paths, caching, and deduplication.
- `test/e2e.test.ts` — explicit real-endpoint checks for general search, unified domain selection followed by finance search, batch search, extraction, a five-query `intl` batch in the shape of the reported incident, and fault injection: one synthetic transport failure (and one synthetic 502) per query, retried against the live service. It uses `ANYSEARCH_E2E_API_KEY` when provided and otherwise runs anonymously. Each passing scenario stores raw evidence under `.evidence/`; unreachable networks are skipped with an explicit reason — the fault-injection scenarios deliberately fail instead, so a broken retry can never hide behind a skip.

Final Pi routing acceptance is performed in a fresh Pi session driven through Herdr so the real system prompt, active-tool list, and two-step domain interaction are exercised together.

For a temporary local Pi run:

```bash
pi -e .
```

## Links

- [GitHub repository](https://github.com/XiaYiHann/pi-anysearch-tools)
- [npm package](https://www.npmjs.com/package/pi-anysearch-tools)
- [Pi package page](https://pi.dev/packages/pi-anysearch-tools)
- [AnySearch documentation](https://www.anysearch.com/docs)
- [Pi Coding Agent](https://pi.dev/)

## License

MIT, as declared in [`package.json`](./package.json).