# @fragments-sdk/context

> **Source-available.** This package is distributed publicly on npm under FSL-1.1-MIT; source stays private. Contribute by filing an issue at https://github.com/fragments-sdk/fragments/issues.

Code intelligence and RAG primitives for codebase indexing and search.

## Modules

### Chunking (`@fragments-sdk/context/chunking`)

Split source files into semantically meaningful chunks for embedding.

- **`chunkFile(content, filePath, language)`** — Auto-selects AST or line-based chunking
- **`chunkByAST(content, filePath, language, grammar)`** — Tree-sitter AST-aware chunking (preserves functions, classes, etc.)
- **`chunkByLines(content, filePath, language)`** — Sliding-window fallback

AST chunking supports: TypeScript, TSX, JavaScript, Python, Go, Rust, Java.

```ts
import { chunkFile } from "@fragments-sdk/context/chunking";

const chunks = await chunkFile(code, "src/auth.ts", "typescript");
// Each chunk has: content, filePath, startLine, endLine, language, symbolName?, scopeChain?
```

### Embeddings (`@fragments-sdk/context/embeddings`)

Generate code-optimized embeddings via Voyage AI.

- **`generateEmbeddings(texts, options)`** — Batch embed with `voyage-code-3`
- **`rerankResults(query, documents, options)`** — Cross-encoder reranking with `rerank-2.5`

```ts
import { generateEmbeddings } from "@fragments-sdk/context/embeddings";

const vectors = await generateEmbeddings(chunks.map(c => c.content), {
  apiKey: process.env.VOYAGE_API_KEY!,
  inputType: "document",
});
```

### Search (`@fragments-sdk/context/search`)

Post-retrieval utilities for combining and deduplicating results.

- **`reciprocalRankFusion(resultSets)`** — Merge multiple ranked lists (e.g. vector + keyword)
- **`deduplicateChunks(chunks)`** — Remove near-duplicate results

### Indexing (`@fragments-sdk/context/indexing`)

File filtering, change detection, and language detection.

- **`shouldIndexFile(path, size)`** — Filter by extension, size, ignored dirs
- **`detectLanguage(path)`** — Map file extension to language name
- **`resolveChanges(oldFiles, newFiles)`** — Diff two file trees for incremental re-indexing
- **`hashContent(content)`** — Content hashing for change detection

## Install

```sh
pnpm add @fragments-sdk/context
```

Requires Node.js 18+. Tree-sitter WASM grammars are included as dependencies.

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=a02beb71-df11-498d-8e1f-39de2e64ce5b" />