# SearchBoost

**Multi-engine web search & evidence synthesis for AI coding agents**  
*One shared core runtime, deeply adapted for MCP, Pi, and DeepSeek Harness*

[![version](https://img.shields.io/badge/version-v0.2.2-orange?style=flat-square)](#)
[![npm version](https://img.shields.io/badge/npm-search--boost-cb3837?style=flat-square&logo=npm)](https://www.npmjs.com/package/search-boost)
[![Node version](https://img.shields.io/badge/node-%3E%3D22.13-339933?style=flat-square&logo=node.js)](https://nodejs.org/)
[![License](https://img.shields.io/badge/license-MIT-blue?style=flat-square)](./LICENSE)
[![Architecture](https://img.shields.io/badge/architecture-unified%20core-8a2be2?style=flat-square)](#)
[![Free tier](https://img.shields.io/badge/free%20tier-zero%20key%20required-success?style=flat-square)](#)

[English](./README.md) · [中文文档](./README_zh.md)

---

> [!NOTE]
> **Release & Branch Notice**: The formerly standalone `pi-search-boost` and `dsh-search-boost` projects are merged into one codebase under `lib/`. Documented commands specifying `@latest` retrieve the official published npm package. To test code that is not in a published release, check out the branch or tag you need and follow [Installation from Source & Development](#installation-from-source--development).

---

> **v0.2.2**: adds AnySearch, correlation-aware fusion scoring and batched Jev target research with approved-result pagination. Includes release-audit fixes for version matching, publication timestamps and cache clearing. See the [release and migration notes](./docs/v0.2.2-release.md). A source merge does not publish npm; `@latest` only includes this version after a separate npm publication.

## Table of Contents

- [Key Features](#key-features)
- [System Architecture](#system-architecture)
- [Supported Hosts Matrix](#supported-hosts-matrix)
- [Quick Start (30 Seconds)](#quick-start-30-seconds)
- [Upgrade & Migration Guide](#upgrade--migration-guide)
- [Interactive Console (TUI)](#interactive-console-tui)
- [Tool Suite & Usage Guide](#tool-suite--usage-guide)
  - [Tool Responsibilities & Boundaries](#tool-responsibilities--boundaries)
  - [1. `fused_search` Multi-Engine Search](#1-fused_search-multi-engine-search)
  - [2. `fetch_page` Smart Content Reader](#2-fetch_page-smart-content-reader)
  - [3. `x_search` X (Twitter) Intelligence](#3-x_search-x-twitter-intelligence)
  - [4. `adaptive_search` Jev Evidence Loop (Experimental)](#4-adaptive_search-jev-evidence-loop-experimental)
  - [Engine Pools & Scoring Presets](#engine-pools--scoring-presets)
- [Parallel Multi-Agent Research Workflows](#parallel-multi-agent-research-workflows)
- [Security](#security)
- [CLI Command Reference (Headless & CI)](#cli-command-reference-headless--ci)
- [Installation from Source & Development](#installation-from-source--development)
- [Friendly Links](#friendly-links)
- [License](#license)

---

## Key Features

- **Multi-Engine Parallel Fusion (`fused_search`)**  
  Queries multiple search providers in parallel. Features an out-of-the-box **keyless free pool** (Bing, DuckDuckGo, Yahoo, Exa-free, AnySearch) and a keyed **API pool** (Tavily, Brave, Exa, AnySearch). Automatically performs cross-engine URL deduplication, domain routing, and relevance re-ranking.
- **Clean Webpage Content Extractor (`fetch_page`)**  
  Fetches the origin first for low latency, with optional same-route curl compatibility fallback and Jina Reader backup. Strips CSS, JS, and ad clutter. Supports focused contextual paragraph extraction via `focus`, backed by in-memory caching and size limits.
- **X / Twitter Community Intelligence (`x_search`)**  
  Retrieves public posts, user timelines, and discussion threads via official xAI API or an anonymous fallback channel. Recovers accurate UTC timestamps from Snowflake post IDs and enforces local author/date filtering without hallucination.
- **Jev Adaptive Evidence Loop (`adaptive_search` · Experimental)**  
  Accepts task context, keyword alternatives and acceptance targets. Each round uses at most three Jev calls: plan, score and coverage. Returns approved URLs, titles and descriptions with cursor pagination and no fixed cumulative result-count cap; execution remains budgeted.
- **Native Multi-Agent Parallel Research**  
  Bundles `search-boost` and `search-boost-parallel-research` skills. In hosts supporting subagents (Cursor, Claude Code, Pi, DSH), tasks can be dispatched to parallel Searchers (gathering evidence) and Summarizers (pure synthesis without tools), supporting both Fast and Complex waves.
- **Unified Core Across All Host Ecosystems**  
  A single, host-neutral core runtime powering standard Model Context Protocol (MCP) servers, alongside native extensions for Pi and Cordis plugin bundles for DeepSeek Harness (DSH).
- **Zero-Config Onboarding & Strict Security**  
  **Requires zero API keys to start** using the free engine pool. Sensitive credentials are stored as plaintext in local private configs with restricted file permissions (POSIX `0600`); this is not encryption at rest. Uses local networking and proxy DNS, with bounded requests and explicit fallback, with zero credential leakage into model prompts.

---

## System Architecture

SearchBoost follows a **"One Core, Three Adapters"** architecture. All search logic, content parsing, deduplication algorithms, and network safety mechanisms reside in the shared core:

```text
                    SearchBoost TUI / CLI
             installation · configuration · updates
                              │
                    Shared SearchBoost Core
                    lib/runtime.mjs facade
              search · fetch · X · adaptive evidence
                              │
              ┌───────────────┼───────────────┐
              │               │               │
         MCP Adapter      Pi Adapter     DSH Adapter
         stdio server     extension      Cordis bundle
              │               │               │
     ┌────────┴────────┐      ▼               ▼
     │ Cursor / Claude │      Pi       DeepSeek Harness
     │ Codex / Grok    │
     │ Antigravity     │
     └─────────────────┘
```

- **Core (`lib/`)**: Host-neutral algorithms, engine orchestration, Jev client protocol, and network safety policies.
- **Adapters (`adapters/`)**: Maps core operations into host-specific protocols (MCP JSON-RPC, Pi Extension API, DSH Cordis lifecycle).
- **Agents (`agents/`)**: Host prompt contracts, workflow templates, and native skill definitions.

---

## Supported Hosts Matrix

| Host | Integration Type | Files / Mechanism | Capabilities & Notes |
| :--- | :--- | :--- | :--- |
| **Cursor / Cursor CLI** | MCP + Skills | `~/.cursor/mcp.json` / `skills/` | Supports CLI auto-approval (`cli-config.json`), session start injection, parallel research skill |
| **Claude Code** | MCP + Prompts | Official `claude` MCP config | Full multi-engine search and page extraction tools with prompt boundaries |
| **Codex** | MCP + Prompts | Host MCP configuration | Seamless access to fused search, X retrieval, and reader tools |
| **Grok Build** | MCP + Plugin | MCP config + bundled plugin | Automatically syncs and installs companion plugin when `grok` CLI is available |
| **Google Antigravity** | MCP + Workspace | Workspace MCP configuration | Per-project workspace guidance and full search tool suite |
| **Pi** | Native Extension | `~/.pi/agent/extensions/` / `prompts/` | Registers native extension, `/fast-parallel`, `/complex-parallel`, searcher/summarizer roles |
| **DeepSeek Harness** | Native Bundle | `cordis.patch.yml` / `dsh plugin` | Integrated with Cordis runtime; provides native `research_parallel` subagents and call cards |

---

## Quick Start (30 Seconds)

### Prerequisites
- **Node.js**: `>= 22.13.0`
- **Package Manager**: `npm` (or `pnpm`)

### 1. Install & Launch the Dashboard

```bash
# Install globally
npm install -g search-boost

# Launch the interactive terminal UI (TUI)
search-boost
```

> [!TIP]
> **Zero API Keys Required to Start**: SearchBoost includes a robust free engine pool (Bing, DuckDuckGo, Yahoo, Exa-free). You can begin searching immediately without signing up for any paid provider!

### 2. 3-Step Setup Wizard
1. Select **`Setup`** in the TUI menu. Configure engine keys (or skip to use free tier) and optional X credentials.
2. Check the agents you want to integrate (Cursor, Claude Code, Pi, etc.), confirming auto-approval and native search replacement.
3. Restart or reload your chosen agents, then ask them to research anything in conversation!

---

## Upgrade & Migration Guide

### 1. Routine Updates: Upgrading via TUI

Whether you run unified `search-boost` or have legacy `pi-search-boost` / `dsh-search-boost` installations, **select `Update` in the TUI to upgrade everything**:

```bash
# Option 1: Open the interactive menu
search-boost
# -> Select "Update"

# Option 2: Run headless update
search-boost upgrade -y
```

> **Update Behavior**: Fetches the latest published release, updates SearchBoost, and refreshes prompt assets across all configured agents while preserving existing keys, layer choices, and permission settings.

---

### 2. Migrating from Legacy `search-boost-mcp`

> [!IMPORTANT]
> If you have the old global package `search-boost-mcp` installed, npm cannot automatically replace the global binary across package renames. **Use the one-line npx migration command**:

```bash
# Execute safe migration via npx
npx --yes --package=search-boost@latest -- search-boost migrate -y

# Once migrated, routine updates only require:
search-boost
# -> Select "Update" (or search-boost upgrade -y)
```

---

## Interactive Console (TUI)

Launch `search-boost` without arguments to access the interactive dashboard built with Clack. Manage installation, updates, and credentials effortlessly:

| Menu Option | Purpose |
| :--- | :--- |
| **Setup** | Complete initial walkthrough: configure engines, search layers, X credentials, and install agents. |
| **Install / update agents** | Install or refresh selected host integrations, preserving existing credentials and layer settings. |
| **Update** | **One-click upgrade**: checks npm for updates, upgrades SearchBoost, and refreshes all installed agents (including legacy Pi/DSH adapters). |
| **API keys / Search layer** | Manage Tavily, Brave, Exa and optional AnySearch credentials, and change the default search layer (`free` / `api`). |
| **X credentials** | Manage X (Twitter) authentication; supports one-click import from local Grok login. |
| **Jev credentials (experimental)** | Configure TypeSafe Jev cognitive engine endpoint and Bearer token. |
| **Native web search** | Enable or disable host-native search for hosts supporting config switches. |
| **Status** | Inspect current engine availability, configuration status, and active integrations. |
| **Print MCP snippet** | Print MCP JSON configuration snippets to stdout for manual setups. |
| **Uninstall** | Safely remove SearchBoost integrations from selected agents, keeping user configurations intact. |

---

## Tool Suite & Usage Guide

When integrated, agents automatically receive standard tool definitions and autonomously determine when to call them.

### Tool Responsibilities & Boundaries

| Tool | Best Used For | Boundary / Non-Goals |
| :--- | :--- | :--- |
| `fused_search` | Parallel multi-engine querying, deduplication, and diversity re-ranking | A single search step; follow-up decisions remain with the parent agent |
| `fetch_page` | Reading clean content from public URLs with optional keyword focus | Not a browser with login state; cannot access internal/private networks |
| `x_search` | Retrieving public X posts, author timelines, or discussion threads | Does not guarantee exhaustive comment threads or total sentiment sampling |
| `adaptive_search` | **Experimental**: Jev-guided autonomous follow-up and evidence evaluation | Approved URLs and descriptions; model approval is not independent verification |
| `search_stats` | Reading engine status, memory cache hits, and recent diagnostic stats | Read-only; configuration readiness does not guarantee active external network reachability |
| `search_layer` | Viewing or switching compatibility search layer in MCP | `show` is read-only; changing layers mutates persistent configuration on disk |

---

### 1. `fused_search` Multi-Engine Search

Dispatches queries across engines concurrently, normalizes URLs, strips redirects, and applies diversity filters.

**Tool Arguments Example (JSON)**:
```json
{
  "query": "Node.js 22 built-in WebSocket API guide",
  "include_domains": ["nodejs.org", "developer.mozilla.org"],
  "engine_pool": "free",
  "ranking": "balanced",
  "complexity": "simple",
  "max_results": 5
}
```

- **`engine_pool`**: `free` (keyless Bing, DuckDuckGo, Yahoo, Exa-free), `api` (configured paid engines only), or `hybrid` (all available engines).
- **`ranking`**: Scoring presets: `balanced` (default), `research` (favors authoritative/documentation sources), or `fresh` (favors recent publications).
- **`complexity`**: `simple` (1 query variant), `medium` (up to 2 variants), `complex` (up to 3 deep variants).
- **`community`**: Boolean (`false` by default). Set to `true` to blend real-time X developer discussions into the final result quota.

---

### 2. `fetch_page` Smart Content Reader

Reads webpage content from search URLs. Reads and cleans the origin first; curl handles transport compatibility when installed, and Jina Reader is the backup.

**Tool Arguments Example**:
```json
{
  "url": "https://nodejs.org/api/globals.html",
  "focus": "AbortSignal.any"
}
```

- **`focus` (Optional)**: Filters and retains paragraphs matching the target keywords.
  > [!TIP]
  > A `focus` miss does **not** prove the information is absent from the page. If in doubt, re-fetch without `focus` to inspect the full context.

---

### 3. `x_search` X (Twitter) Intelligence

Designed for real-time technical tracking and first-party developer updates. Supports keyword search, author timelines, and thread conversations.

**Tool Arguments Example**:
```json
{
  "query": "Claude 3.7 Sonnet hybrid reasoning from:AnthropicAI",
  "mode": "keyword",
  "max_results": 5
}
```

- **Precise Timestamps**: When platform timestamps are missing or inconsistent, recovers true UTC creation times from 64-bit Snowflake IDs.
- **Native Operators**: Full support for `from:username`, `since:YYYY-MM-DD`, and `until:YYYY-MM-DD`.

---

### 4. `adaptive_search` Jev Evidence Loop (Experimental)

**Vercel support**: in TUI → Jev credentials, enter `https://ai-gateway.vercel.sh/v1` and a Vercel AI Gateway key. SearchBoost selects the official SDK evaluation model `typesafe-ai/jev`, not chat completions. The default TypeSafe `/systemone` path remains supported. Both paths use only the canonical user Jev credential, not environment keys, and respect server rate-limit delays.

The caller supplies task context, keyword alternatives and explicit acceptance questions. Jev chooses query strategies and engines, then scores retrieved evidence and judges which targets need another round. Each round has at most three logical requests, not one request per URL.

```json
{
  "tasks": [{
    "context": "ExampleDB 4.2 upgrade impact",
    "targets": [{
      "id": "migration",
      "keywords": ["migration guide", "upgrade guide"],
      "question": "What migration steps are required from 4.1 to 4.2?"
    }]
  }],
  "page_size": 20
}
```

- Up to 6 tasks, 4 targets per task, 12 total targets; legacy `questions` input remains supported.
- `results` is a flat `{url, title, description}` list containing only assessed, approved material, not unassessed/rejected candidates.
- Read more with `{"cursor":"<nextCursor>"}`: no new search or Jev call. Pages default to 20 results (max 50) and have a byte budget, but cumulative approved results have no fixed count cap.
- Inspect `coverageComplete` and warnings. End of pagination is not exhaustive search, and model approval is not independent fact verification.
- Results are process-local, retained for up to 30 minutes / 32 recent result sets; expiry, eviction or restart invalidates cursors.
- Optional task `time_range` distinguishes publication dates from event dates; unknown dates cannot qualify as today's evidence.

See the [adaptive-search contract and limitations](./docs/jev-adaptive-search.md).

---

### Engine Pools & Scoring Presets

`engine_pool` selects engines, `ranking` selects shared cross-pool weights, and `complexity` controls only budget. AnySearch is one logical engine: anonymous in free, key-required in api, and key-preferred in hybrid. Configure `ANYSEARCH_API_KEY` or `config keys --set anysearch=KEY`.

| Engine | balanced | research | fresh |
| --- | ---: | ---: | ---: |
| bing | 0.957 | 0.927 | 1.020 |
| ddg | 0.981 | 0.903 | 0.927 |
| yahoo | 0.957 | 0.877 | 0.902 |
| exa-free | 1.004 | 1.085 | 0.951 |
| tavily | 1.049 | 1.105 | 1.084 |
| brave | 1.004 | 0.951 | 1.125 |
| exa | 1.049 | 1.146 | 1.063 |
| anysearch | 1.004 | 1.042 | 0.951 |

These are uncalibrated cold-start priors, not measured quality rankings. `consensus-v2.1` combines original ranks, related-provider discounts and max+log consensus, with metadata adjustments capped at 20%. Quality and list-selection scores stay separate; zero-weight sources cannot vote. Recalibrate old `min_score` thresholds. See [scoring and migration](docs/fusion-scoring.md) and [pool routing](docs/search-routing.md).

---

## Parallel Multi-Agent Research Workflows

One shared workflow with three host bindings. The parent agent splits a question into independent tracks; each searcher gathers evidence with `fused_search` and `fetch_page`; a tool-less summarizer (or the parent itself) then turns the reports into one answer.

```text
                        [ Parent Agent ]
                   splits tracks · owns the answer
                              │
              ┌───────────────┴───────────────┐
              ▼                               ▼
       [ Searcher A ]                  [ Searcher B ]
   fused_search · fetch_page       fused_search · fetch_page
              │                               │
              └───────────────┬───────────────┘
                              ▼
                      [ Summarizer ]
                   no tools · synthesis only
                              ▼
                     [ Parent Report ]
```

| Host | Entry point | How children run |
| :--- | :--- | :--- |
| **Pi** | `/fast-parallel` and `/complex-parallel` prompts; the `search-parallel-subagent` tool; `searcher` / `summarizer` roles | Each searcher is a child process that explicitly loads the SearchBoost Pi extension (`adapters/pi/index.js`) and may call only `fused_search` and `fetch_page`; summarizers launch with `--no-tools`. The caller chooses the wave size — this runner sets no concurrency cap. |
| **DeepSeek Harness** | Native `research_parallel` tool | Children spawn on the same Cordis context through the host's subagent service. Searchers receive `toolFilter.allow = ['fused_search','fetch_page']`; summarizers receive an empty allowlist. Before a wave, `research_parallel` verifies both tools are registered and refuses to spawn otherwise. Handles are disposed on success and failure, and `maxDepth` stays at 1, so children cannot nest further research. |
| **MCP hosts** (Cursor, Claude Code, Codex, Grok, Antigravity) | Bundled `search-boost-parallel-research` skill | The skill embeds the same roles and workflow and runs them through whatever subagent mechanism the host provides. A host without usable delegation falls back to serial research by the parent. |

Installing into an MCP host also installs the companion **`search-boost`** routing skill, which picks the right tool for an open-ended question.

- **Fast mode**: one searcher wave, then parent synthesis — no summarizer and no second wave.
- **Complex mode**: one to three waves with a gap review in between; another wave starts only when a material gap remains, and the run stops early when the evidence is sufficient. These wave limits are workflow instructions, not global quotas enforced across independent tool calls.
- **Fallback**: if the host cannot delegate, the parent researches serially and says so instead of pretending a wave ran.

> [!IMPORTANT]
> A child returning `ok` means it finished with non-empty text, **not** that its claims are verified. Failed or partial reports stay visible, but their URLs are excluded from the successful aggregate, and final judgment always belongs to the parent.

For Pi/DSH child-tool loading, stale `pi-search-boost` references, and the retired `deep_research` tool, see [Subagent tool setup and diagnosis](docs/subagent-tools.md).

---

## Security

API keys use a credential store this tool owns. They are not written into prompts, tool results or shell profiles:

- **Private file permissions**: keys live in `~/.search-boost/config/keys.json` (redirect the root with `SEARCH_BOOST_HOME`). On POSIX the store directory is `0700` and the file is `0600`; a rewrite is built from a fresh `0600` temporary file rather than written in place, so an existing file is never widened. Backups and upgrade receipts under the same root are `0600` as well. An env-overridden directory keeps its own mode, while the credential file is still created `0600`. Windows has no POSIX mode bits — ACL inheritance applies there.
- **Atomic, locked writes**: replacement uses an `O_EXCL` temporary file plus rename, so a reader sees either the old or the new file and never a partial one; a failed write cleans up after itself and leaves the previous file untouched. Read-modify-write cycles take an exclusive lock, so two writers cannot silently lose each other's update, and a rejected change does not touch the file at all.
- **Never echoed back**: a key is only sent to the engine it belongs to (Tavily, Brave, Exa and AnySearch request parameters/headers). Status output, `search-boost config keys --show` and the doctor report print masked values (`abcd****wxyz`); error messages are tested not to contain credential material, and Jev evaluation payloads deliberately carry no key, masked key or fingerprint. AnySearch uses anonymous quota in free, requires a key in api, and prefers a configured key in hybrid. Its potentially credential-bearing error envelopes are never echoed or persisted.


---

## CLI Command Reference (Headless & CI)

In addition to the interactive TUI, SearchBoost provides a comprehensive CLI for scripting and automation:

```bash
# ----------------- Core & Interactive -----------------
search-boost                                # Launch interactive dashboard (TUI)
search-boost status                         # Print active configuration and host summary
search-boost --help                         # Display full CLI documentation

# ----------------- Headless Installation -----------------
search-boost install -t cursor -y           # Install for Cursor with auto-approval
search-boost install -t claude,codex --keep-native  # Install while preserving host native search
search-boost install -t antigravity --workspace /path/to/project # Target a specific workspace
search-boost install -t pi -y               # Mount Pi extension and prompt templates
search-boost install -t dsh --profile web   # Connect to DeepSeek Harness web profile
search-boost install -t cursor --dry-run    # Preview installation without writing files

# ----------------- Configuration Management -----------------
search-boost config keys                    # Manage API keys from CLI
search-boost config keys --set anysearch=KEY  # Configure the AnySearch key (ANYSEARCH_API_KEY)
search-boost config layer                   # Switch default layer (free / api)
search-boost config x --import-grok         # Import X credentials from local Grok login
search-boost config jev                     # Configure Jev endpoint and token

# ----------------- Diagnostics & Health -----------------
search-boost doctor                         # Run offline diagnostic checks
search-boost doctor --strict                # Strict mode (exits non-zero on warnings)
search-boost doctor --json                  # Output machine-readable JSON report

# ----------------- Uninstall -----------------
search-boost uninstall -t cursor,claude -y  # Remove integrations from selected hosts
```

---

## Installation from Source & Development

Use this path to contribute to SearchBoost, or to test code that is not in a published release yet. Everywhere else, `@latest` refers to the published package.

### Prerequisites

- **Node.js** `>= 22.13.0` (matching `package.json`)
- **git** and **npm**
- Optional: **`curl`**, for the same-route transport-compatibility fallback in `fetch_page`

### Set up a checkout

```bash
# 1. Clone the repository
git clone https://github.com/Mr-remon219/search-boost.git
cd search-boost

# 2. Install dependencies exactly as CI does
npm ci

# 3. Regenerate the bundled Grok plugin assets
npm run plugin:sync-grok
```

The clone lands on the repository's default branch. To test a different branch or tag, check it out before installing dependencies (`git checkout BRANCH_OR_TAG`).

### Run it without a global install

```bash
node cli.mjs                  # interactive TUI straight from the checkout
node cli.mjs status           # one-shot status summary
node cli.mjs install -t pi -y # mount the Pi extension from this checkout
node cli.mjs install -t dsh --profile web
```

`node cli.mjs` accepts the same commands as the installed `search-boost` binary. If you want the bare command in your terminal, link the checkout instead of installing from npm:

```bash
npm link        # or: npm install -g .
search-boost
```

After changing adapter or agent assets, re-run the same install command for that host (`node cli.mjs install -t HOST`) so the host picks up the new files, then restart or reload that host.

### Test gates before a PR

| Command | What it covers |
| :--- | :--- |
| `npm run check` | Syntax check across the CLI, core, adapters and scripts |
| `npm run prepublishOnly` | The full offline suite: plugin sync, syntax, CLI, install, doctor, fusion, X, engines, X auth, Jev, key authority, dry-run, network, search routing, adapters, parallel research, upgrade, MCP and smoke |
| `npm run test:network` | Proxy retries, curl fallback, request bounds and compatibility regressions |
| `npm run test:adapters` | MCP, Pi and DSH adapter protocol suites plus Pi subagent settings migration/diagnosis |
| `npm run test:parallel` | Searcher/summarizer contracts, DSH dispatch preflight, cancellation and tool isolation |
| `npm run test:adaptive` | Jev adaptive evidence loop |
| `npm run smoke` | MCP JSON-RPC protocol smoke test |

These suites run against loopback fixtures and process doubles, so no engine keys are needed; a green `npm run prepublishOnly` is the bar for a PR.

---

## Friendly Links

- [LINUX DO](https://linux.do/)

---

## License

This project is licensed under the [MIT License](./LICENSE).