# codebase-semantic-search

> **Project location:** `/Users/jdobson/developer/150-codebase-semantic-search/`
> — migrated 2026-08-21 from `~/developer/codebase-semantic-search/` to conform
> with the project-folder convention. See [`AGENTS.md`](./AGENTS.md) for the
> remote setup (this repo has two: `github` real, `origin` local mirror).

A local, project-agnostic **semantic search engine** for any codebase. Drop
it into a project, point it at your source dirs, and any agent (Claude
Code, GitHub Copilot Chat, OpenCode, Codex) or human can query it by
meaning rather than by literal text match.

- **Vector DB:** Milvus Standalone (Docker)
- **Embeddings:** Ollama `nomic-embed-text` (768-dim, local, fast)
- **Chunker:** AST-aware for TS/JS, heading-based for markdown, sliding window for the rest
- **Interfaces:** stdio **MCP server** (for agents, **preferred**) + HTTP API (for humans / curl debugging)
- **Live updates:** chokidar file watcher with debounced incremental reindex

> **TL;DR for agents.** Install the package, run `npx codesearch up` once,
> then register the stdio MCP server with your agent runtime. The agent
> gets four tools — `codebase_semantic_search`, `codebase_clip`,
> `codebase_read_file`, `codebase_stats` — and queries them as part of its
> normal tool calls. The HTTP API is a fallback for humans only; agents
> should not shell out to `curl`.

## Quickstart

```bash
# 1. Install as a dev dep
npm install --save-dev codebase-semantic-search

# 2. Bootstrap the engine (writes config + starts Milvus + pulls the
#    embedding model + runs an initial reindex if empty + starts the
#    file watcher). Idempotent. Blocks in the foreground.
npx codesearch up
```

Then **register the MCP server with your agent runtime** (one-time per
machine):

**Claude Code** (`~/.claude/mcp.json` or per-project `.mcp.json`):
```json
{
  "mcpServers": {
    "codebase": {
      "command": "npx",
      "args": ["-y", "codebase-semantic-search", "mcp"]
    }
  }
}
```

**GitHub Copilot Chat** (`.vscode/mcp.json`):
```json
{
  "servers": {
    "codebase-semantic-search": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "codebase-semantic-search", "mcp"]
    }
  }
}
```

**OpenCode** (`opencode.json`):
```json
{
  "mcp": {
    "codebase-semantic-search": {
      "type": "local",
      "command": ["npx", "-y", "codebase-semantic-search", "mcp"]
    }
  }
}
```

Your agent runtime spawns the MCP server on demand over stdio whenever it
needs to query the index. **You do not need to run `codesearch mcp`
yourself** — that's the agent's job. (See "How it fits together" below.)

After registration, the agent has four new tools:

| Tool                       | Returns                                                          |
|----------------------------|------------------------------------------------------------------|
| `codebase_semantic_search` | Ranked code chunks matching a natural-language query             |
| `codebase_clip`            | Full text of a chunk by its short numeric `id` (single or batch) |
| `codebase_read_file`       | File slice by line range (`sed -n 'A,Bp'` semantics)             |
| `codebase_stats`           | Chunk count, collection name, embedding model                    |

> **If `up` fails on first run,** it's almost always a missing dep. Run
> `npx codesearch doctor` and follow the fix it prints — usually "Ollama
> not running" or "Docker compose stack not up."

## Requirements

`codesearch up` shells out to three external things on your machine:
Docker, Ollama, and `curl`. They must already be installed and reachable
**before** you run `up`. The package itself installs cleanly via npm on any
platform Node 20+ runs on — its only native code is the optional C++ grammar,
which npm skips if it cannot build (see [C++ support](#c-support-optional-native-grammar)).

| Need | Why | Where |
|---|---|---|
| **Node.js ≥ 20** | The `engines` requirement | any platform |
| **Docker + Compose v2 plugin** (`docker compose …`) | Milvus Standalone stack (Milvus + etcd + MinIO) | uses ~600 MB of images on first pull |
| **Ollama running** on `http://127.0.0.1:11434` | Provides the embedding endpoint. `up` will *throw* if it's not reachable — `up` does **not** auto-install Ollama (system service requires sudo, out of scope for an npm package) | see install notes below |
| **Embedding model** (`nomic-embed-text`, ~270 MB) | `up` **auto-pulls** this once Ollama is running — but the first pull is slow | managed by `up` |
| **`curl` on your `$PATH`** | `up` uses `curl` to probe Ollama | shipped on macOS / Linux; modern Windows too |

**Verify nothing's missing:** `npx codesearch doctor` (works pre-`init`, prints one line per dep with status and the fix command if anything fails).

### C++ support (optional native grammar)

TypeScript/JavaScript is parsed by `ts-morph`, which is pure JS and always
installed. C++ (`.cpp .h .hpp .cc .cxx .ipp`) needs `tree-sitter` and
`tree-sitter-cpp`, which are **native addons** and therefore ship as
`optionalDependencies`. npm installs them when a build toolchain is present and
skips them when the build fails, so a machine without a C++ compiler still gets
a working install.

You do not need to do anything for the common case. If you never index C++, the
grammar is never loaded.

If the grammar is missing or unloadable, the chunker **warns once** at the first
C++ file and falls back to line-window chunking for C++ only — JS/TS indexing is
unaffected and an index run never aborts:

```
C++ AST chunking unavailable (tree-sitter-cpp not loadable: …). Falling back to
line-window chunking for .cpp/.h/.hpp/.cc/.cxx/.ipp files.
```

The fallback is noticeably worse: line windows cut mid-function, so a search
hit's line range tends to land on a body statement instead of a declaration. If
you index a large C++ codebase, install the grammar explicitly:

```bash
npm i tree-sitter tree-sitter-cpp
```

### X-macro tables (`.macro`)

C/C++ projects that generate code from macros keep their protocol tables in
`*.macro` files — a list of top-level invocations like
`TRANSACTION(ttACCOUNT_SET, 3, AccountSet, …)` that a generator expands. In
rippled these hold the authoritative answer to "which fields does this
transaction accept?" and "what type is this field?".

These are **not** valid C++, so the AST chunker cannot help; the line chunker
is worse still, since one 800-token window covers ~40 unrelated table rows.
`chunkMacro()` splits them on invocation boundaries instead, yielding one chunk
per row, with the entry's doc comment and its macro identifier attached.

No native dependency: this is pure TypeScript.

### Install Ollama (one-time)

| OS | Command |
|---|---|
| macOS | `brew install ollama && brew services start ollama` (or just `ollama serve` in another terminal) |
| Linux | `curl -fsSL https://ollama.com/install.sh \| sh`, then `ollama serve` |
| Windows | Install from [ollama.com/download](https://ollama.com/download), then `ollama serve` |

To prefetch the embedding model (optional — `up` will do it for you):

```bash
ollama pull nomic-embed-text
```

### Footprint

This is a chunky tool. Don't `up` it on a half-full laptop without expecting:

- **Disk** — ~600 MB of Docker images + a few GB per project for the Milvus
  index volume (`<project>_milvus_data`). First time is the biggest.
- **Ports (defaults)** — `19530` Milvus gRPC, `9091` Milvus metrics, `9000`/`9001`
  MinIO API/console, `7700` HTTP API. (etcd + MinIO are internal
  dependencies of Milvus — they're reachable between containers on the
  `search-network` bridge, not from the host.) All overridable via
  `.codesearchrc.json` or env vars (`MILVUS_PORT`, `SEARCH_PORT`, etc.).
- **First-run time** — ~45s on Apple Silicon (the bundled Milvus image is
  multi-arch, so Apple Silicon gets **native** `linux/arm64` execution — no
  QEMU emulation), plus 30–60s for the initial index build depending on
  codebase size.

### Apple Silicon / ARM64

**Native, not emulated.** The bundled Milvus image (`milvusdb/milvus:v2.5.5`)
publishes a multi-arch manifest with both `linux/amd64` and `linux/arm64`
digests. `codesearch up` automatically forwards your host architecture to
Docker via the `UNAME_M` env var, so an M-series Mac pulls the native arm64
image and Docker runs it directly — no Rosetta, no QEMU. Startup on M1/M2/M3
is roughly the same as on `linux/amd64` (~45s for the stack + index build).

If you'd rather pin the platform yourself (e.g. for CI on a mixed-architecture
runner), set `UNAME_M=linux/amd64` (or `linux/arm64`) in the shell before
running `codesearch up` and the bundled compose will respect it.

## How it fits together

The CLI brings up three runtime components — Milvus, Ollama, the file
watcher — and **two query surfaces** sit on top:

- **MCP stdio server** (preferred for agents). The agent runtime spawns
  `npx -y codebase-semantic-search mcp` as a child process and talks to it
  over JSON-RPC. No port, no daemon — the server starts on demand when the
  agent makes its first tool call and exits when the agent quits. Each MCP
  process has its own in-memory clip store (the short numeric ids assigned
  by `codebase_semantic_search` are valid for the lifetime of that
  process only).
- **HTTP API** (fallback for humans / curl debugging). Run
  `npx codesearch serve` (just HTTP) or `npx codesearch serve:watch` (HTTP
  + file watcher). Listens on `http://localhost:7700`. Use this when an
  agent runtime can't speak MCP, or when you want to poke at the index
  with curl / Postman / a browser.

Both query surfaces read the same Milvus collection, so results are
consistent across them. Agents should always prefer the MCP tools — no
port collisions in multi-project setups, no long-running server to manage,
clean process boundaries.

```
┌─────────────────────────────────────────────────────────┐
│             Agent (Claude Code, Copilot, …)             │
└────────────────────────┬────────────────────────────────┘
                         │  stdio MCP (preferred) / HTTP fallback
                         ▼
┌─────────────────────────────────────────────────────────┐
│              codesearch (Node.js, TypeScript)            │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────────┐  │
│  │  Watcher    │  │  HTTP API    │  │  MCP Server    │  │
│  │ (chokidar)  │  │  (express)   │  │ (@modelctx/sdk)│  │
│  └──────┬──────┘  └──────┬───────┘  └────────┬───────┘  │
│         └───────────────┼───────────────────┘           │
│                         ▼                                │
│         ┌────────────────────────────┐                   │
│         │  Indexer (mtime-diff)      │                   │
│         └─────────┬──────────────────┘                   │
│                   ▼                                      │
│         ┌────────────────────────────┐                   │
│         │  Milvus Standalone :19530  │                   │
│         └────────────────────────────┘                   │
│                         ▲                                │
│                         │                                │
│         ┌───────────────┴──────────────┐                 │
│         │  Ollama :11434 (embeddings)  │                 │
│         └──────────────────────────────┘                 │
└─────────────────────────────────────────────────────────┘
```

## MCP Server

Spawns a stdio MCP server (or, with `--http`, one shared HTTP server; see below) exposing four tools:

- `codebase_semantic_search` — query the index by meaning. Args:
  `query` (natural language), `top_k` (1–100, default 100), `module`,
  `language`, `chunk_type`, `min_score` (absolute quality threshold,
  0..1), `min_score_diff` (relative quality threshold, 0..1, **default
  0.1**, mutually exclusive with `min_score`), `include` (opt-in
  metadata fields to add to each result), `ignore` (path patterns to leave
  out; see "`ignore` — leave paths out of a search"), `format` (response format:
  `"markdown"` default or `"json"` opt-in). **Default response is a
  single markdown document** with a `# Search: "..."` title and per-hit
  code fences + metadata captions. Pass `format: "json"` for the
  structured response. When `format: "json"`, default per-hit fields are
  `id`, `filePath`, `symbolName`, `score`, `startLine`, `endLine`,
  `content` (lean); pass `include: ["chunkType", "module", "language"]`
  to opt in to the metadata fields. The response always echoes whichever
  quality filter was applied (default or explicit) so callers can see
  the threshold and `maxScore`. When `include` is set, response also
  includes `includedFields`.
- `codebase_clip` — fetch a clip by its short numeric id from the
  in-memory store. Args: EITHER `id: number` (single) OR `ids: number[]`
  (batch). Ids are assigned by `codebase_semantic_search` and are valid
  for the lifetime of this MCP process (FIFO evict at 10K entries;
  cleared on restart).
- `codebase_read_file` — fetch a slice of a file by line range
  (`sed -n '<start>,<end>p'` semantics). Args: `filePath` (relative to
  workspace root), `startLine` (1-indexed inclusive), `endLine`
  (1-indexed inclusive). 500 lines per call, 25 MB file cap. Use it to
  expand the context around a hit returned by `codebase_semantic_search`
  without re-loading the whole file.
- `codebase_stats` — chunk count, collection name, embedding model, embedding dimensions.

### One shared server for every session (`mcp --http`)

By default each agent session spawns its own stdio server, a separate Node
process of roughly 150–200 MB. With many sessions open on one machine that
adds up (15 sessions ≈ 2.3 GB). `codesearch mcp --http` serves MCP over
Streamable HTTP instead: one long-running process, one MCP server per
session inside it, sharing the Milvus client and clip store. Five sessions
searching at once measured 183 MB in total.

```bash
npx codesearch mcp --http          # 127.0.0.1:7701 (CONFIG.mcpPort; MCP_PORT or rc "mcpPort" to change)
npx codesearch mcp --http 7799     # explicit port
```

Point the agent at it instead of the stdio command, e.g. Claude Code:

```json
{ "mcpServers": { "codebase-semantic-search": { "type": "http", "url": "http://127.0.0.1:7701/mcp" } } }
```

It binds to `127.0.0.1` only, refuses requests whose `Host` header isn't
local (DNS rebinding protection), and has no authentication, the same
trust level as the HTTP search server. Things to know:

- The server reads `.codesearchrc.json` from **its own** working directory,
  so `codebase_read_file` resolves paths against that checkout, not the
  session's. Searches are unaffected: every session already queries the
  same collection.
- Clip ids are shared by all sessions on the server (still FIFO at 10K).
- Something has to keep the process running (a terminal, `nohup`, a
  systemd user unit). If it is down, the agent's MCP connection fails;
  the stdio command still works as a fallback.

Typical workflow:

```text
1. codebase_semantic_search(query="how invoices are created")          → results[] (each has an id)
2a. codebase_clip(ids=[r.id for r in results])                        → full text of each
2b. codebase_read_file(filePath=results[0].filePath, startLine=…, endLine=…)  // expand context around one
3. make your edit in the lines you've now seen in full
```

### Agent templates

`npx codesearch init` (or `up`, if no config exists yet) wires the
canonical MCP + HTTP reference into your project's agent ecosystem:

- **`.github/instructions/codebase-semantic-search.instructions.md`** —
  the canonical reference (full tool list, defaults, response shapes,
  filters, recovery flow, bootstrap). Auto-loaded into every agent
  context via `applyTo: "**"`. Written once per project.
- **A slim dual-pointer block** into every `.github/agents/*.agent.md`
  (marker-bounded so re-runs are idempotent). Points at the canonical
  doc above — does NOT duplicate its content.
- **A slim pointer section** in `.github/copilot-instructions.md`
  (created if missing).

The per-agent blocks are intentionally ~15-20 lines each — they point
at the canonical doc rather than duplicating it. With N agent files,
that keeps total agent-ecosystem bloat at `N × 20 + 1 × ~300` lines
instead of `N × 200`.

## CLI

| Command | What it does |
|---|---|
| `codesearch init` | Scaffold `.codesearchrc.json` + agent file snippets + docker-compose |
| `codesearch init --port=19531` | Same, with a Milvus port offset for running multiple codebases in parallel |
| `codesearch init --search-port=7800` | Same, with a custom HTTP API port |
| `codesearch up` | One-shot bootstrap: init (if needed) → Milvus → Ollama model → index (if empty) → dev loop. Idempotent. |
| `codesearch up --port=19531` | Same, with a Milvus port offset |
| `codesearch up --no-index` | Same, but skip the initial reindex |
| `codesearch up --no-serve` | Same, but stop after bootstrap (no dev loop) |
| `codesearch down` | Stop Milvus (volumes preserved — index survives) |
| `codesearch doctor` | Check Ollama, Milvus, embedding model availability |
| `codesearch index` | Reindex the codebase (incremental by default) |
| `codesearch index --full` | Drop the collection and rebuild from scratch |
| `codesearch index --dry-run` | Show what would change without writing |
| `codesearch serve` | HTTP search server on `:7700` (humans / curl fallback) |
| `codesearch watch` | File watcher only (no HTTP) |
| `codesearch serve:watch` | HTTP server + watcher in one process |
| `codesearch mcp` | stdio MCP server (spawned automatically by agent runtimes; rarely run manually) |
| `codesearch status` | Show collection stats and current config |

All config is overridable via `.codesearchrc.json` or env vars
(`OLLAMA_HOST`, `MILVUS_HOST`, `MILVUS_PORT`, `EMBEDDING_MODEL`,
`SEARCH_PORT`).

`searchIgnore` lists path patterns searches leave out unless a request passes `ignore`
(see "`ignore` — leave paths out of a search"). Default `[]`: search everything.

`indexDirs` entries are paths relative to the project root and may name a directory
(indexed recursively, honouring the root `.gitignore` and `excludePatterns`) or a single
file — useful for top-level docs such as `README.md` or `AGENTS.md`.

### The underlying subcommands

`up` is the recommended entry point, but the granular subcommands are still available for fine-grained control:

```bash
npx codesearch init         # Scaffold .codesearchrc.json + agent file snippets + docker-compose
npx codesearch doctor       # Check Ollama, Milvus, embedding model
npx codesearch index        # Reindex the codebase (incremental by default)
npx codesearch index --full # Drop the collection and rebuild from scratch
npx codesearch serve        # HTTP search server on :7700 (fallback for humans)
npx codesearch watch        # File watcher only (no HTTP)
npx codesearch serve:watch  # HTTP server + watcher in one process
npx codesearch mcp          # stdio MCP server (spawned automatically by agent runtimes)
npx codesearch status       # Show collection stats and current config
npx codesearch up           # One-shot bootstrap (init + Milvus + index + dev loop)
npx codesearch down         # Stop Milvus (volumes preserved — index survives)
```

The `init` command drops:
- `.codesearchrc.json` — project-specific config
- `docker-compose.search.yml` — Milvus + etcd + MinIO
- `.github/instructions/codebase-semantic-search.instructions.md` — the canonical MCP + HTTP reference, auto-loaded into every agent context
- A slim dual-pointer block appended to every `.github/agents/*.agent.md` (with marker comments so re-runs are idempotent)
- A slim pointer section in `.github/copilot-instructions.md` (created if missing)

### Running multiple codebases in parallel

Docker container names are auto-namespaced by the compose project, so two projects no longer collide. To run two stacks side-by-side on the same machine, give the second one a port offset:

```bash
# In project A (default ports)
npx codesearch up

# In project B
npx codesearch up --port=19531 --search-port=7800
```

`up` forwards the offset to `init` (if `.codesearchrc.json` is missing) and sets the matching env vars on the `docker compose` invocation. Each project gets its own Milvus data volume (`<project>_milvus_data`), so indexes don't share or overwrite.

For MCP-only setups (no HTTP server running), multi-project is trivially
isolated — each agent runtime spawns its own MCP child process per
project, with no port to share at all.

## HTTP API (humans / curl fallback)

Base URL: `http://localhost:7700` (port overridable via `SEARCH_PORT`).
The HTTP API exposes the same data as the MCP server; it's intended for
humans debugging with curl / Postman, or for tools that can't speak MCP.

### `POST /search` — semantic search by meaning

Minimum viable call:

```
POST /search
Content-Type: application/json

{
  "query": "how does tenant isolation work in mongoose queries"
}
```

Defaults applied: `top_k: 100` and `min_score_diff: 0.1` (drop anything more
than 10% below the best match). To override or add filters:

```
POST /search
Content-Type: application/json

{
  "query": "how does tenant isolation work in mongoose queries",
  "top_k": 100,                  // optional, default 100, max 100
  "module": "platform",           // optional filter
  "language": "typescript",       // optional filter
  "chunk_type": "function",       // optional filter
  "min_score": 0.7,               // optional absolute quality threshold (0..1)
  "min_score_diff": 0.1,          // optional, default 0.1; mutually exclusive with min_score
  "include": ["chunkType", "module", "language"],  // optional opt-in metadata
  "ignore": ["docs/", "*.md"],    // optional path patterns to leave out; [] = nothing
  "format": "json"                // optional response format; default "markdown"
}
```

**Any other field is refused with a 400** that names it and lists the accepted ones, with a
hint for likely guesses: `{"query": "…", "limit": 5}` answers
`unknown field: "limit" (did you mean "top_k"?)`. There is no `limit`; use `top_k`.

**Default response is markdown** — a single document with a `# Search: "..."`
title and one-line summary at the top, then per-hit fenced code blocks
with a metadata caption line beneath each. Code is the primary matter;
metadata is the caption. See "Response format: markdown by default"
below for the shape.

Pass `"format": "json"` for the structured response. See "JSON response
shape" further down.

#### Response format: markdown by default

The markdown response is a single document:

````markdown
# Search: "how invoices are created"

3 results • top_k: 10 • min_score: 0.7 • clip store: 142

---

```typescript
/**
 * Clawback transaction — reclaim issued tokens or MPTs from a holder's account.
 *
 * @see https://xrpl.org/clawback.html
 */
import type { BaseTransactionFields } from '../types/base.js';
...
```

src/transactions/clawback.ts:1-11 • file-header • score: 0.8146 • id: 1

---

```typescript
export class ClawbackTx extends TokenTransaction {
  override readonly TransactionType = 'Clawback' as const;
  ...
}
```

src/transactions/clawback.ts:18-38 • ClawbackTx • score: 0.7353 • id: 2
````

Structure:
- **Header**: `# Search: "<query>"` followed by a one-line summary
  (`count • top_k • min_score? • min_score_diff? • included? • clip store: N`).
  Each filter only shows when it's actually applied.
- **Per hit**:
  - A code fence with the language hint (from the chunker; falls back to
    `text` if unknown).
  - The full chunk content, pristine.
  - A plain-text caption line: `filePath:startLine-endLine` • `symbolName`
    (when present) • `score: N` • `chunkType: …` (when `include`d) •
    `module: …` (when `include`d) • `id: N`.
- Hits are separated by `---` (horizontal rule).

`Content-Type: text/markdown; charset=utf-8`.

**Why markdown**: agents and humans read code naturally when it's fenced.
Metadata sits below each block as a caption, visually subordinate. JSON
preserves typed structure but every field competes for visual weight;
markdown gives the code the prominence it deserves and parks metadata in
the gutter.

#### JSON response shape

Pass `"format": "json"` to opt in. The response envelope stays the same
as before; only the body is structured instead of rendered.

```json
{
  "success": true,
  "data": {
    "query": "...",
    "count": 10,
    "topK": 10,
    "clipStoreSize": 142,
    "results": [
      {
        "id": 42,
        "filePath": "server/src/modules/billing/routes.ts",
        "symbolName": "createInvoice",
        "score": 0.842,
        "startLine": 42,
        "endLine": 80,
        "content": "export async function createInvoice(req, res) { ... }"
      }
    ],
    "minScore": 0.7,
    "candidatesBeforeFilter": 10,
    "includedFields": ["chunkType", "module", "language"]
  }
}
```

**Default response per hit is lean** — `id`, `filePath`, `symbolName`,
`score`, `startLine`, `endLine`, `content`. The `chunkType`, `module`,
`language` fields are **opt-in** via `include` (see below).

- `score` is 0..1 cosine similarity — ≥0.75 = strong, 0.55–0.75 = review,
  <0.55 = likely noise.
- `id` is a **short numeric handle** assigned at search time. Pass it to
  `GET /clip/:id` (or `/clips`) to fetch the full text without re-encoding
  the path + line range.
- `clipStoreSize` reports the current in-memory clip-store size.
- `minScore` and `candidatesBeforeFilter` are echoed only when `min_score`
  was supplied.
- `includedFields` is echoed only when `include` was supplied.

#### `include` — opt-in metadata fields

```jsonc
{
  "include": ["chunkType", "module", "language"]  // any subset
}
```

Allowed values: `"chunkType"`, `"module"`, `"language"`. Unknown value or
wrong type (e.g. a string instead of an array) returns HTTP 400 with the
allowed list. Omit or pass `[]` for the lean default.

#### `ignore` — leave paths out of a search

A list of path patterns, relative to the project root, whose chunks the search
leaves out. It's applied in Milvus as part of the vector search, so `top_k` still
counts kept results, and the index itself is untouched: changing what is ignored
needs no reindex.

| Pattern | Leaves out |
|---|---|
| `docs/` | the `docs` folder (trailing slash = folder) |
| `docs` | the file `docs`, or a folder of that name |
| `*.md` | every path matching the wildcard (`*` and `**` match any run of characters, `/` included) |
| `server/**/README.md` | every `README.md` under `server/` |
| `server/src/__tests__/` | that folder: `_` and `%` match literally |

At most 20 patterns of up to 200 characters each; anything else returns HTTP 400.

**Project default.** Set `searchIgnore` in `.codesearchrc.json` to choose what is
left out when a request doesn't pass `ignore`, for example to keep prose out of
code-mapping queries:

```json
{ "searchIgnore": ["docs/", "wiki/"] }
```

A request's own `ignore` replaces the default, and `"ignore": []` searches
everything. Whenever something was ignored, the response says so: JSON gets
`ignored` (the patterns) and `ignoreSource` (`"config"` or `"request"`), and the
markdown summary line reads, for example,
`ignored: docs/, wiki/ (project default; pass ignore: [] to include)`. That way a
caller can't miss results without knowing it.

#### `min_score` semantics

Quality filter applied **after** the vector search. The engine asks Milvus
for `top_k` candidates, then drops anything below `min_score`. So the
response may contain fewer than `top_k` results — that's by design (you
asked for "up to `top_k` results that score ≥ `min_score`"). With the
default `top_k: 100` you're already at the maximum candidate pool;
tighten the query (or use `min_score_diff` for a relative cutoff)
instead of pushing `top_k` higher — it can't go above 100.

Recommended bands:
- **0.75+** — strong, relevant matches; safe to act on.
- **0.55–0.75** — worth reviewing; context may help disambiguate.
- **< 0.55** — likely noise; widen the query or add filters instead of
  lowering the threshold.

Validation: `min_score` must be a finite number in `[0, 1]`. Out-of-range
or non-numeric values return HTTP 400.

#### `min_score_diff` semantics (relative threshold)

**Applied by default at `0.1`** when the caller doesn't supply either
filter. To disable the default, set `min_score_diff: 0` explicitly (keeps
only ties with the top hit). To use a different threshold, pass any value
in `[0, 1]`.

The threshold is computed from the best hit in the result set:
`appliedThreshold = max_score - min_score_diff`. Drop any hit whose score
is below that. Useful when you don't know the absolute score distribution
in advance — "everything within 0.1 of the best match" is often more
meaningful than "everything above 0.7".

Mutually exclusive with `min_score` — passing both returns HTTP 400.
The default `min_score_diff` does NOT count as "provided" for this
check; if you set `min_score`, the default is ignored.

Response echo when applied:

```json
{
  "minScoreDiff": 0.1,
  "appliedThreshold": 0.7146,
  "maxScore": 0.8146,
  "candidatesBeforeFilter": 10
}
```

Recommended bands:
- **0.05–0.15** — strict; typically keeps 3–8 hits on a sizeable codebase
  with `top_k: 100`. The default (`0.1`) sits in the middle of this band.
- **0.2–0.3** — lenient; useful when the top match is strong but the
  semantic space drops off sharply below it.

Validation: `min_score_diff` must be a finite number in `[0, 1]`.
Out-of-range or non-numeric values return HTTP 400. Threshold is clamped
at 0 if `max_score` is 0 (pathological).

### `GET /clip/:id` — fetch one clip by its short id

```
GET /clip/42
```

Returns: `{ success, data: { id, filePath, startLine, endLine, totalLines, content } }`.

The id is assigned by `/search`. The underlying `(filePath, startLine,
endLine)` is stored in an in-memory table keyed by id. Use this when
you want the chunk back exactly as the search returned it.

### `GET /clips?ids=1,2,3` — batch fetch (small batches, curl-friendly)

```
GET /clips?ids=42,17,99
```

Accepts both `?ids=1,2,3` (comma-separated) and `?ids=1&ids=2&ids=3`
(repeated param). Max 500 ids per request.

### `POST /clips` — batch fetch (large batches)

```
POST /clips
Content-Type: application/json

{ "ids": [42, 17, 99, 128, 256] }
```

Same batch cap (500 ids). Use this when the query-string approach gets
unwieldy.

**Batch response shape** (both GET and POST):

```json
{
  "success": true,
  "data": {
    "results": [
      { "id": 42, "success": true,  "filePath": "...", "startLine": 1, "endLine": 50, "totalLines": 191, "content": "..." },
      { "id": 99, "success": false, "error": "id not found or expired. Re-run /search to get a fresh id." }
    ],
    "requested": 2,
    "succeeded": 1,
    "failed": 1,
    "clipStoreSize": 142
  }
}
```

Per-item errors do NOT abort the batch — one bad id returns
`success: false` in its slot, the rest proceed.

### `POST /read` — fetch a file slice by raw line range

Use this when you want to expand a chunk's context (e.g. `startLine - 5` to
`endLine + 5` to see the surrounding function/class). Semantics match
`sed -n '<start>,<end>p' <filePath>`.

```
POST /read
Content-Type: application/json

{
  "filePath": "server/src/modules/billing/routes.ts",
  "startLine": 42,
  "endLine": 95
}
```

Response: `{ success, data: { filePath, startLine, endLine, totalLines, rangeRequested, content } }`.

**Limits shared by `/read`, `/clip/:id`, `/clips`:**

- **Range cap:** 500 lines per call. Chain reads to paginate larger ranges.
- **File size cap:** 25 MB (HTTP 413 if exceeded). Protects against OOM
  when an agent points at a huge generated file.
- **Path safety:** `filePath` is resolved relative to the workspace root.
  Absolute paths and `../` escapes are rejected with HTTP 403.
- **Not found:** HTTP 404.

### Clip store: how ids work

The clip store is a per-process, in-memory `Map<id, {filePath, startLine, endLine}>`:

- **Auto-increment numeric ids** (1, 2, 3, …). Short — easier to log,
  curl, and pass through agent context than encoded base64.
- **Dedup** on `(filePath, startLine, endLine)` — the same chunk always
  returns the same id across searches.
- **FIFO eviction at 10K entries** — once full, the oldest inserted id is
  dropped to make room. Long-idle ids may return 404 ("not found or expired").
- **Ephemeral** — server restart clears the table. Agents mid-session can
  just re-search; no data loss.
- **Per-process** — the MCP server has its own table; an HTTP `/search`
  id is not resolvable by an MCP `codebase_clip` call (they're separate
  processes). Run a search in the same process that will resolve the ids.

### Typical workflow

```bash
# 1. search — results carry ids
curl -s http://localhost:7700/search -H "Content-Type: application/json" \
  -d '{"query": "how invoices are created", "chunk_type": "function"}'

# 2a. shortcut — fetch by id (recommended for the common case)
curl -s http://localhost:7700/clip/42

# 2b. expand context — fetch by raw filePath + adjusted range
curl -s -X POST http://localhost:7700/read -H "Content-Type: application/json" \
  -d '{"filePath": "server/src/modules/billing/routes.ts", "startLine": 38, "endLine": 95}'
```

### Other endpoints

`GET /health` — liveness check. `GET /stats` — collection stats (chunk count, etc.).

## Project Layout

```
codebase-semantic-search/
├── package.json             # name: codebase-semantic-search, bin: codesearch
├── docker-compose.search.yml
├── src/
│   ├── cli.ts               # entry: commander dispatch
│   ├── config.ts            # loads .codesearchrc.json + env
│   ├── walker.ts            # file system walker + mtime diff
│   ├── chunker.ts           # AST-aware chunker (ts-morph for TS/JS, tree-sitter for C++)
│   ├── embedder.ts          # Ollama client
│   ├── milvus.ts            # Milvus client (create/upsert/search/delete)
│   ├── search-server.ts     # Express HTTP API (humans / curl fallback)
│   ├── watcher.ts           # Chokidar file watcher
│   ├── mcp-server.ts        # stdio MCP server (preferred for agents)
│   ├── indexer.ts           # reindex library (used by index + watch)
│   ├── clip-store.ts        # in-memory store of clip refs (FIFO evict)
│   ├── read-clip.ts         # shared file-slice helper (path safety, caps)
│   └── commands/
│       ├── init.ts          # scaffold config + agent files + docker-compose
│       ├── doctor.ts        # prereq check
│       ├── index.ts         # reindex (full or incremental)
│       ├── serve.ts         # HTTP server (fallback for humans)
│       ├── watch.ts         # file watcher
│       ├── serve-and-watch.ts
│       ├── mcp.ts           # stdio MCP server
│       ├── up.ts            # one-shot bootstrap (init + Milvus + index + dev loop)
│       ├── down.ts          # stop Milvus (volumes preserved)
│       └── status.ts        # config + stats
└── templates/               # bundled init templates (written into your project)
    ├── codesearchrc.json                       # project config
    ├── codesearch-instructions.md              # canonical MCP + HTTP reference (written to .github/instructions/)
    ├── copilot-instructions-section.md         # slim pointer (appended to .github/copilot-instructions.md)
    └── agent-semantic-search-section.md        # slim dual-pointer (appended to each .github/agents/*.agent.md)
```

## Why this exists

Grep is literal. Agents reinvent the same utilities because they don't know
they exist. Semantic search by meaning fixes that, but the existing OSS
options (Qdrant Cloud, hosted vector DBs, custom RAG frameworks) all add
infrastructure that doesn't fit a single-dev box.

This package is the local-first version: Milvus in Docker, Ollama for
embeddings, chokidar for live updates, MCP for agent integration. ~10 source
files, no cloud dependencies, ~50ms per query after a warm cache.

## Releasing

This package publishes to npm automatically via GitHub Actions — see
[`.github/workflows/publish.yml`](.github/workflows/publish.yml). The
publish is triggered by pushing a `v*` tag. To publish an existing tag again (say,
after a failed run), dispatch **Publish** from `main` in the Actions tab with that tag.

### To cut a release

1. Bump `version` in `package.json` (`npm version 0.2.0` for stable,
   `npm version 0.2.0-beta.1 --preid=beta` for a pre-release — `npm
   version` also commits and tags for you).
2. Add an entry to `CHANGELOG.md` summarising what's in the cut.
3. Push the commit and tag:
   ```bash
   git push --follow-tags
   ```
4. CI publishes the version to npm — the dist-tag is **auto-derived from
   the version**: `v0.2.0` → `latest`, `v0.2.0-beta.1` → `beta`,
   `v0.2.0-rc.2` → `rc`. A GitHub Release is opened with auto-generated
   release notes.

### CI prerequisites (one-time)

Publishing uses npm **trusted publishing**: npm accepts the workflow's GitHub identity (OIDC),
so there is no token to store or renew. It's configured on npmjs.com under the package's
**Settings → Trusted Publisher → GitHub Actions** (owner `Jonathan-Dobson`, repository
`codebase-semantic-search`, workflow `publish.yml`)
([npm docs](https://docs.npmjs.com/trusted-publishers)). The workflow needs
`permissions: id-token: write` and npm 11.5.1 or later, and passes no `NODE_AUTH_TOKEN`.
Provenance is signed automatically. An `NPM_TOKEN` secret is no longer used and can be deleted.

## License

MIT