> [!IMPORTANT]
> This extension is **maintained by [Wyna Informatik GmbH](https://wyna-informatik.ch)**.
> The **Wyna search API** (`searchProvider: "wyna"`) uses the same API key as your
> Wyna LLM gateway — no separate credential needed. See
> [Quick start](#quick-start) below.

# pi-web-research

[![CI](https://github.com/wynainfo/pi-web-research/actions/workflows/ci.yml/badge.svg)](https://github.com/wynainfo/pi-web-research/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

Token-saving web research for the [pi](https://pi.dev) coding agent. Not a bare
web-search tool: an advanced pipeline — search (**Kagi** or **Wyna**) → parallel
SSRF-guarded fetch → Readability → Markdown → isolated sub-agents — distills
pages into cited briefings and targeted answers. Raw pages never enter your
context; you spend tokens on findings, not noise.

## What you get

Two tools, backed by specialized, isolated sub-agents that keep raw pages out of
the main context:

- **`web_search`** — search results; in `comprehensive` mode a sub-agent reads
  the top N pages and returns one cited briefing.
- **`web_fetch`** — one page distilled into a summary (concise by default, or
  `thorough`/`full`) or, with a `prompt`, a targeted answer to just that question.

All research output is **cited** — every web-sourced claim carries its source as
a visible, clickable URL with the publish date — and **defended against content
farms and fake news**: no publisher dominates a search, same-publisher pages
never count as corroboration, and uncorroborated single-source "events" are
quarantined as unverified instead of reported as fact.

**Privacy:** only your chosen search provider sees your queries; pages are
fetched and extracted locally and nothing is sent to any other third party.
Fetched text is treated as untrusted, and all outbound fetches are SSRF-guarded.
Bot-protected pages (Reddit, Cloudflare-challenged sites) can be handled by an
opt-in, locally-run headless-browser fallback or a self-hosted reader — no
third-party reader by default.

## Quick start

```bash
pi install npm:pi-web-research
```

That's it — pi installs the package and its dependencies. Other methods:

- git: `pi install git:github.com/wynainfo/pi-web-research`
- local clone (development): `pi install /path/to/pi-web-research`
- manual: clone into `~/.pi/agent/extensions/` (auto-discovered) and run
  `npm install` there yourself

Then provide **one** search API key — either:

- **Kagi:** `export KAGI_API_KEY="<key>"` (key from
  [kagi.com/api/keys](https://kagi.com/api/keys)), or
- **Wyna:** nothing to do if `providers.wyna.apiKey` is already in
  `~/.pi/agent/models.json` — the LLM-gateway key doubles as the search key.
  Otherwise `export WYNA_API_KEY="<key>"`.

Start pi and run `/web-research-status` to verify. If anything is missing, pi prints
a single `⚠ web-research` notice — just **ask pi to finish the setup**; it is
briefed on exactly what's missing and where to store the key.

### Optional: headless-browser fallback (Playwright)

Fetching **bot-protected pages** needs this optional dependency; everything else
works without it. Install it up front — or simply ask pi to do it when a page
comes back blocked:

```bash
cd ~/.pi/agent/git/github.com/wynainfo/pi-web-research
npm install playwright          # the Node client (also installed via optionalDependencies)
npx playwright install chromium # the Chromium browser binary (~once, ~150 MB)
```

The fallback activates automatically once installed (`/web-research-status` shows
the state) and runs Chromium sandboxed in a separate process — details under
[Notes & limitations](#notes--limitations).

## Choosing a provider

| Provider | `searchProvider` value | Auth | Key source |
|---|---|---|---|
| **Kagi v1** (default) | `"kagi"` / `"kagi-v1"` | Bearer token | `KAGI_API_KEY` env / `kagiToken` config |
| **Kagi v0** (legacy/deprecated) | `"kagi-v0"` / `"kagi-legacy"` | Bot token | `KAGI_API_KEY_V0` env / `kagiV0Token` config |
| **Wyna** | `"wyna"` | Wyna API key | `WYNA_API_KEY` env / Pi models.json (`providers.wyna.apiKey`) / `wynaApiKey` config |

The default is **Kagi v1**; with only a Wyna key present, Wyna is auto-detected.
To force a provider, set `searchProvider` in the **global** `config.json` (a
security-sensitive, global-only key — see [Configuration](#configuration-optional)).

## Disabling

- Per run: `pi --no-extensions` (disables all extensions).
- Uninstall: `pi remove git:github.com/wynainfo/pi-web-research`
  (or toggle it in `pi config`).

## Development

Maintained at **https://github.com/wynainfo/pi-web-research** — pull requests and
issues welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the code layout, test
suite, security invariants, and how to add a search backend.

---

# Reference — details for pi (and power users)

Operational detail for pi and power users — not needed to get started. "The
extension directory" below means wherever the package lives — under
`~/.pi/agent/npm/` for the default npm install,
`~/.pi/agent/git/github.com/wynainfo/pi-web-research/` for a git install
(`pi list` shows installed sources).

## Search keys: resolution details

**Either key suffices** — the provider auto-detects whichever is available
(an explicit `searchProvider` in global config wins).

**Kagi** (default): the v1 key is read from, in order: `KAGI_API_KEY` env var →
`.env` in this directory → `kagiToken` in the global `config.json`.

```bash
# alternative to the shell profile: a .env file in the extension directory (git-ignored)
echo 'KAGI_API_KEY=<your-kagi-v1-api-key>' > <extension-dir>/.env
```

**Wyna** ([ai.wyna.info](https://ai.wyna.info), Kagi upstream): resolution order:
`WYNA_API_KEY` env var → `~/.pi/agent/models.json` (`providers.wyna.apiKey`) →
`wynaApiKey` in global config.
API errors: 401 bad key · 402 top up at ai.wyna.info/account · 429 rate-limited
(30/min) · 502 transient upstream (one retry) · 503 not configured server-side.

**Legacy Kagi v0** (deprecated): set `searchProvider: "kagi-v0"` and supply its
separate "Bot" token via `KAGI_API_KEY_V0` (or legacy `KAGI_TOKEN`, or
`kagiV0Token`). Migrating from v0? `KAGI_API_KEY` now holds the **v1** key —
move the old token to `KAGI_API_KEY_V0`.

Real env vars take precedence over `.env`. Global-config values support pi-style
secret resolution: `"$SOME_ENV"` / `"${SOME_ENV}"` (env var), `"!command"`
(command stdout, e.g. a secret manager), otherwise a literal token.

> **Security:** secret resolution and the tokens/endpoints are honored **only** from
> the global config (or env / `.env`) — a project's `.pi/web-research.json` is
> restricted to a tuning-knob allowlist, so opening an untrusted repo cannot execute
> code, leak your token, or redirect it to another server. Full key list under
> [Configuration](#configuration-optional).

## Tools

Extraction chain: **search provider → fetch (parallel, SSRF-guarded) → Mozilla
Readability → HTML→Markdown → capped → sub-agent**.

### `web_search(query, questions?, context?, count?, recency?, recency_strict?)`
The default, context-cheap way to gather information. Searches the configured
provider; the result shape depends on `summaryMode` (below). Raw pages never
enter the calling agent's context.
- `query` — the search string (keywords).
- `questions` — optional; one or more specific questions the briefing must answer.
  Omit for a comprehensive digest of the top pages for the query.
- `context` — optional background on what you're researching, to orient the
  sub-agent.
- `count` — number of top pages to read, 1–10 (default 5).
- `recency` — `day` | `week` | `month` | `year`. Only read results newer than the
  window, newest first.
- `recency_strict` — default `false`. Only with `recency`: also drop undated (and
  suspect future-dated) results — may return fewer or zero results.

**Summary mode (`summaryMode`):**
- `concise` (default) — **no sub-agent, no page reads**: returns the ranked result
  list (title, snippet, date, URL) directly. The main agent then calls
  `web_fetch(url, prompt)` on whichever result it wants — that runs a sub-agent on
  the one fetched page (cached) and answers the specific question. Cheapest per
  search; detail is pulled only on demand.
- `comprehensive` — an **isolated sub-agent reads the top N pages (markdown +
  llms.txt) in its own context** and returns a thorough briefing (retains code,
  signatures, parameters, numbers, caveats) with inline `[n]` citations and a
  Sources list. **Bounded multi-hop:** if the top pages don't fully answer, the
  sub-agent may follow up to `researchMaxHops` cited links (default 3).

Either way, follow a citation with `web_fetch` (optionally with a `prompt`) when you
need a source's full text or a targeted answer.

### `web_fetch(url, mode?, maxChars?, prompt?)`
Fetch one page and get it back distilled, so the raw page never enters your context.
A cheap, configurable model (`fetchModel`, falling back to `subAgentModel`, then the
main model) does the distilling. Four behaviors:
- **`mode: "concise"` (default) — page summarizer:** a short summary of the page.
  Cheapest; best when you just need the gist.
- **`mode: "thorough"` — page summarizer:** a detailed summary that retains
  specifics, code, signatures, parameters, and caveats.
- **`mode: "full"`:** the entire main content as clean markdown (+ the site's
  `/llms.txt`), no sub-agent. Use only when you need a source verbatim.
- **`prompt` set — page researcher:** answers **just that question** about the page
  (no whole-page summary); only the answer comes back. `prompt` overrides `mode`.

The default for the no-prompt case is set by `fetchMode` (default `concise`); a
per-call `mode` always wins. If no sub-agent model is available, summary/research
modes fall back to returning the full page.

Fetched+extracted pages are **cached ~1h**, so a follow-up `web_fetch` on an
already-fetched URL is fast (no re-fetch; only the cheap model re-runs).
- `maxChars` — cap on page content read/returned (default `perPageMaxChars`).
- Only `http`/`https` URLs whose host resolves to a public address (see the SSRF
  note below).

Both tools return a clear error message (not a crash) if the token or a dependency
is missing, or if a page can't be fetched/parsed.

**Live UI:** both tools stream their progress (searching → reading pages →
following links → distilling → the streamed answer) with a transient status while
they run. The TUI shows the call parameters, a header panel (provider, pages read,
links followed / fetch mode), and the full output uncollapsed; without `pi-tui`,
pi's default rendering applies.

## Configuration (optional)

`config.json` in the extension directory (global) and/or
`<project>/.pi/web-research.json` (project, merged over global):

```jsonc
{
  // --- sensitive: GLOBAL config only (ignored in project .pi/web-research.json) ---
  "searchProvider": "kagi",              // "kagi" (= v1, default) | "kagi-v0" (legacy) | "wyna"
  "kagiToken": "$KAGI_API_KEY",          // v1 Bearer key; optional (env / .env take precedence)
  "kagiEndpoint": "https://kagi.com/api/v1/search",  // v1 endpoint (POST); must be https
  "kagiV0Token": "$KAGI_API_KEY_V0",     // legacy v0 "Bot" token (only if using kagi-v0)
  "kagiV0Endpoint": "https://kagi.com/api/v0/search", // legacy v0 endpoint (GET)
  "wynaApiKey": "$WYNA_API_KEY",         // Wyna API key override (normally from models.json or env). Global-only.
  "wynaEndpoint": "https://ai.wyna.info/api/search", // Wyna search endpoint (GET/POST). Global-only.
  "allowPrivateNetwork": false,          // opt out of the SSRF private-IP guard

  // bot-protected-page fallbacks — sensitive, GLOBAL config only
  "browserFallbackEnabled": true,        // escalate bot-blocked fetches to headless Chromium (needs Playwright)
  "browserNoSandbox": false,             // RISKY escape hatch: drop Chromium's sandbox (only if user namespaces can't be enabled)
  "readerEndpoint": "",                  // optional self-hosted reader (Jina-Reader / FlareSolverr); no third-party default
  "readerMode": "auto",                  // "auto" | "jina" | "flaresolverr"

  // --- tuning knobs: may also be set per-project in .pi/web-research.json ---
  "browserTimeoutMs": 20000,             // navigation/challenge-settle budget for the headless render
  "acceptLanguage": "en-US,en;q=0.9",    // Accept-Language header for plain fetches
  "maxResults": 5,                       // default number of top pages web_search reads
  "perPageMaxChars": 12000,              // cap per fetched page
  "totalMaxChars": 40000,                // cap across all pages (sub-agent input budget)
  "fetchTimeoutMs": 12000,
  "concurrency": 5,                      // parallel page fetches
  "userAgent": "Mozilla/5.0 ...",

  // llms.txt (https://llmstxt.org/) injection
  "llmsTxtEnabled": true,                // probe /llms.txt from result domains
  "llmsTxtMaxChars": 8000,               // max chars of llms.txt content per origin
  "llmsTxtFetchFull": false,             // also try /llms-full.txt and walk parent dirs

  // web_search briefing sub-agent + multi-hop
  "summaryMode": "concise",              // "concise" (default: raw results, read on demand via web_fetch) | "comprehensive" (read pages, full briefing)
  "subAgentModel": "",                   // "provider/id" override; default: main model
  "subAgentThinking": "off",             // off|minimal|low|medium|high|xhigh
  "researchMaxHops": 3,                  // extra cited pages the sub-agent may fetch (0 disables; max 5) — comprehensive mode

  // web_fetch distilling sub-agent (page summarizer / page researcher)
  "fetchModel": "",                      // "provider/id" cheap model; falls back to subAgentModel/main
  "fetchMode": "concise"                 // default web_fetch depth w/o a prompt: "concise" | "thorough" | "full" (per-call `mode` overrides)
}
```

A project-level `<project>/.pi/web-research.json` is merged over the global config
but is **restricted to the tuning knobs** (the second half of the block above);
the keys marked `GLOBAL config only` are silently dropped from it, so an untrusted
repo cannot redirect your queries, leak the token, relax the SSRF guard, spawn a
browser, drop its sandbox, or send fetched URLs off-box.

## Notes & limitations

- **Endpoints:** the active provider's endpoint must be **https** (the token rides
  in the Authorization header); override `kagiEndpoint` / `kagiV0Endpoint` /
  `wynaEndpoint` in global config if your account is provisioned differently.
- **Bot-protected / JS-heavy pages:** fetching escalates in tiers — plain
  `fetch()` with browser-like headers (Tier 0) → sandboxed headless Chromium that
  runs the page's JS (Tier 1, needs
  [Playwright](#optional-headless-browser-fallback-playwright)) → optional
  self-hosted reader (Tier 2, `readerEndpoint`). If all tiers fail, the host is
  recorded for the session and results carry `[BLOCKED-DOMAIN]` /
  `[DO-NOT-RETRY]` markers so the agent moves on (warn-only — a later call may
  retry, since the challenge may have cleared). If Playwright is missing, the
  failure text briefs the agent on the install steps and benefits, so you can
  just ask pi to set it up.
  - **Isolation:** the browser runs in a dedicated child process; page JS executes
    only inside Chromium's renderer sandbox and never in the pi process. The
    Chromium sandbox is kept **on** — on WSL2/containers it needs unprivileged user
    namespaces (`sysctl -w kernel.unprivileged_userns_clone=1`, or
    `kernel.apparmor_restrict_unprivileged_userns=0` on newer kernels). If Chromium
    can't launch sandboxed it fails loudly with this guidance rather than silently
    disabling the sandbox; `browserNoSandbox: true` (global config) is the explicit,
    risk-accepting escape hatch. To relocate the browser binary, set
    `PLAYWRIGHT_BROWSERS_PATH` (for both the install and pi's environment).
  - **Self-hosted reader (optional):** set `readerEndpoint` (global config only) to
    a Jina-Reader-style or FlareSolverr endpoint. No third-party default; egress
    goes only where you point it.
- **Page cache:** extracted pages are cached in-process for ~1h (keyed by URL +
  char cap), so repeat reads — per-page RAG follow-ups, multi-hop, or the same URL
  across searches — skip the re-fetch. Failures aren't cached.
- **Content-farm / fake-news defenses:** at most 2 of the results read/shown per
  search come from the same publisher (a domain flooding the ranking can't
  dominate the research), and the sub-agent prompts enforce: same-publisher pages
  count as ONE source for corroboration; a page's self-description is never
  treated as credibility; high-salience events claimed by a single non-official
  source are verified against a primary source or quarantined as
  "Unverified — single source"; and the main agent is instructed to treat
  missing independent coverage as evidence against a claim.
- **Citations:** every result and source ships with a ready-made citation
  assembled by the extension — publish date, age, and the full URL as visible
  link text (`2026-05-03 (2 months ago) [https://…](https://…)`) — and the
  agent is instructed to copy it verbatim next to each claim it relays, so
  sources stay clickable and readable without clicking, and the format can't
  be mangled.
- **Redundant-search guard:** a near-duplicate `web_search` within a session
  (reworded query, or results fully covered by an earlier search) is
  short-circuited to a compact pointer at the prior findings; partially
  overlapping searches read only the genuinely-new pages. Refine the query to
  force a fresh search. (Token-set Jaccard ≥ 0.6, 30-min window.)
- **SSRF guard:** all outbound page/llms.txt fetches are restricted to `http`/
  `https`, the host must resolve to a public address (loopback, RFC-1918,
  link-local `169.254.0.0/16` incl. cloud metadata, ULA, etc. are refused), and
  redirects are followed manually with each hop re-validated. This applies to
  every fetch, including the research sub-agent's multi-hop follow-ups. Set
  `"allowPrivateNetwork": true` in **global** config if you intentionally fetch
  `localhost`/internal docs. Response bodies are size-capped (≈3 MB per page,
  256 KB per llms.txt) to bound memory.
- **Prompt injection:** fetched page text is untrusted. It normally flows into a
  sub-agent (the search summarizer, page summarizer, or page researcher), which is
  instructed to treat source text as data and ignore embedded instructions — so only
  the distilled output reaches the main agent. It enters the main agent's context
  directly only via `web_fetch(url, mode:"full")` (or a summary/research fallback
  when no sub-agent model is available). Treat any actions an agent takes off the
  back of fetched content with care.