# Data Standard: RAG Chunking Strategies

> **Scope:** data-vector-search
> **Layer:** 2
> **Keywords:** chunking, rag, embeddings, vector-search, document-segmentation, overlap, retrieval
> **Load When:** building a RAG pipeline or embedding documents for vector search

**Verified against:** .NET 10 (chunking algorithms — language-level, no external package). Last-verified: 2026-05-20.

---

## Overview
Chunking splits documents into segments for embedding. Chunk quality directly impacts RAG answer quality.

## Chunking Strategies

### 1. Fixed-Size (Baseline)
```csharp
public IEnumerable<Chunk> FixedSizeChunk(string text, int size = 512, int overlap = 50)
{
    for (int i = 0; i < text.Length; i += size - overlap)
    {
        yield return new Chunk
        {
            Content = text[i..Math.Min(i + size, text.Length)],
            StartIndex = i,
            EndIndex = Math.Min(i + size, text.Length)
        };
    }
}
```

### 2. Sentence-Aware (Recommended for prose)
```csharp
public IEnumerable<Chunk> SentenceChunk(string text, int maxTokens = 256)
{
    var sentences = SplitSentences(text);
    var current = new StringBuilder();
    var tokenCount = 0;

    foreach (var sentence in sentences)
    {
        var sentenceTokens = EstimateTokens(sentence);
        if (tokenCount + sentenceTokens > maxTokens && current.Length > 0)
        {
            yield return new Chunk { Content = current.ToString().Trim() };
            current.Clear();
            tokenCount = 0;
        }
        current.Append(sentence).Append(' ');
        tokenCount += sentenceTokens;
    }

    if (current.Length > 0)
        yield return new Chunk { Content = current.ToString().Trim() };
}
```

### 3. Semantic (Best quality, higher cost)
Split at semantic boundaries (paragraphs, headers, sections) rather than character count.

```csharp
public IEnumerable<Chunk> SemanticChunk(string markdown)
{
    // Split at markdown headers
    var sections = Regex.Split(markdown, @"(?=^#{1,3} )", RegexOptions.Multiline);
    foreach (var section in sections.Where(s => s.Trim().Length > 0))
        yield return new Chunk { Content = section.Trim() };
}
```

## Token Estimation
```csharp
// Rough estimate: 1 token ≈ 4 characters (English), ≈ 3 characters (Portuguese)
private static int EstimateTokens(string text) => text.Length / 4;

// More accurate: use tiktoken (if available)
// var encoder = TiktokenSharp.TikToken.EncodingForModel("gpt-4");
// return encoder.Encode(text).Count;
```

## Chunk Metadata
Always store metadata with chunks for filtering and citation:
```csharp
public record Chunk
{
    public string Content { get; init; } = "";
    public string DocumentId { get; init; } = "";
    public string DocumentTitle { get; init; } = "";
    public string SourceUrl { get; init; } = "";
    public int ChunkIndex { get; init; }
    public int StartCharIndex { get; init; }
    public string Section { get; init; } = ""; // H1/H2 header this belongs to
    public DateTime IndexedAt { get; init; } = DateTime.UtcNow;
}
```

## Strategy Selection Guide
| Document Type | Strategy | Chunk Size |
|---------------|----------|------------|
| Legal documents | Sentence-aware | 256 tokens |
| Technical docs | Semantic (headers) | Variable |
| Q&A pairs | Keep as-is | Whole Q+A |
| Code files | Function-boundary | Whole function |
| News articles | Paragraph | 512 tokens |

## Quality Checklist
- [ ] Chunks overlap by 10-20% to avoid cutting context at boundaries
- [ ] Minimum chunk size: 50 tokens (avoid noisy small chunks)
- [ ] Maximum chunk size: ~512 tokens (well under the 8192-token input limit of `text-embedding-3-small`/`-large`)
- [ ] Metadata preserved and searchable
- [ ] Source URL included for citation
- [ ] Re-index when document changes (detect via content hash)