# @classic-homes/chat-indexer

CLI tool for indexing content into the Classic Homes chat system's Vectorize index.

## Features

- **Plugin-based architecture** - Extensible source system for different content types
- **Multiple source types** - Markdown files, web scraping, and GitHub repositories
- **Docusaurus versioning** - Support for versioned documentation
- **Image processing** - Vision LLM processing for marketing site images
- **Intelligent chunking** - Heading-based content chunking with metadata

## Installation

```bash
# In the chat monorepo
npm install

# Build the package
npm run build -w @classic-homes/chat-indexer
```

## Usage

### Quick Start

```bash
# Index a single markdown content directory
npx @classic-homes/chat-indexer index \
  --source ./content/warranty \
  --site docs \
  --site-url https://guide.classichomes.com \
  --guide warranty

# Index a website via sitemap
npx @classic-homes/chat-indexer index \
  --type web \
  --sitemap https://example.com/sitemap_index.xml \
  --site marketing \
  --site-url https://example.com \
  --dry-run \
  --verbose

# Preview chunks without uploading (dry run)
npx @classic-homes/chat-indexer index \
  --source ./content/warranty \
  --site docs \
  --site-url https://guide.classichomes.com \
  --guide warranty \
  --dry-run

# Use a config file
npx @classic-homes/chat-indexer index --config ./chat-index.config.json
```

### Commands

#### `index`

Index content from configured sources into Vectorize.

```bash
chat-indexer index [options]
```

**Options:**

| Option | Description | Required |
|--------|-------------|----------|
| `-c, --config <path>` | Path to config file | No |
| `-s, --source <path>` | Content source directory (for markdown) | Yes* |
| `-t, --type <type>` | Source type: "markdown" (default) or "web" | No |
| `--sitemap <url>` | Sitemap URL (required for web sources) | No |
| `--site <name>` | Site identifier (e.g., "docs") | Yes* |
| `--site-url <url>` | Site base URL | Yes* |
| `--guide <name>` | Guide name (e.g., "warranty") | No |
| `--version <version>` | Content version | No |
| `--index <name>` | Vectorize index name | No |
| `--dry-run` | Preview chunks without uploading | No |
| `--verbose` | Show detailed progress | No |
| `--output <path>` | Write chunks to JSON file | No |

*Required if no config file is provided.

#### `validate`

Validate a configuration file.

```bash
chat-indexer validate --config ./chat-index.config.json
```

## Configuration

### Config File

Create a `chat-index.config.json` or `chat-index.config.js` file.

#### Markdown Source (Local Files)

```json
{
  "site": "docs",
  "siteUrl": "https://guide.classichomes.com",
  "sources": [
    {
      "type": "markdown",
      "path": "./content/warranty",
      "guide": "warranty",
      "categories": {
        "/use-care/": "use-care",
        "/standards/": "standards",
        "/problems/": "problems",
        "/warranty-program/": "warranty-program",
        "/warranty-terms/": "warranty-terms"
      }
    }
  ],
  "versioning": {
    "versionsFile": "./packages/site/versions.json",
    "versionedDocsPath": "./packages/site/versioned_docs"
  },
  "url": {
    "latestVersion": "2026.08.01"
  },
  "vectorize": {
    "indexName": "classic-chat-knowledge"
  }
}
```

#### Web Source (Scraped Content)

Index content from a website by scraping pages via sitemap:

```json
{
  "site": "marketing",
  "siteUrl": "https://classichomes.com",
  "sources": [
    {
      "type": "web",
      "sitemapUrl": "https://classichomes.com/sitemap_index.xml",
      "includeSitemaps": [
        "page-sitemap.xml",
        "project-sitemap.xml",
        "floor_plan-sitemap.xml"
      ],
      "excludePatterns": [
        "/wp-admin/*",
        "/blog/*",
        "/privacy-policy/",
        "/idx-*"
      ],
      "contentSelectors": [
        "#main-content",
        ".et_pb_text_inner",
        ".et_pb_blurb_description"
      ],
      "excludeSelectors": [
        "header",
        "footer",
        "nav",
        ".et_pb_menu",
        "script",
        "style"
      ],
      "categories": {
        "/project/": "communities",
        "/floor_plan/": "floor-plans",
        "/neighborhoods/": "neighborhoods"
      },
      "rateLimit": 2
    }
  ],
  "vectorize": {
    "indexName": "classic-chat-knowledge"
  }
}
```

**Web Source Options:**

| Option | Description | Default |
|--------|-------------|---------|
| `sitemapUrl` | URL to sitemap index or sitemap | Required |
| `includeSitemaps` | Only process these child sitemaps | All |
| `excludePatterns` | URL patterns to exclude (supports `*` wildcards) | None |
| `contentSelectors` | CSS selectors for main content | Divi defaults |
| `excludeSelectors` | CSS selectors to remove before extraction | Divi defaults |
| `categories` | URL pattern to category mappings | None |
| `rateLimit` | Requests per second | 2 |

#### GitHub Source (Repository Content)

Index content directly from a GitHub repository with Docusaurus versioning support:

```json
{
  "site": "docs",
  "siteUrl": "https://guide.classichomes.com",
  "sources": [
    {
      "type": "github",
      "repository": "owner/repo-name",
      "guide": "warranty",
      "routeBasePath": "warranty",
      "docsPath": "docs",
      "versioning": {
        "strategy": "docusaurus",
        "versionsFile": "versions.json",
        "versionedDocsPath": "versioned_docs/version-{version}",
        "currentVersion": "2026.08.01",
        "includeVersions": ["2026.08.01", "2026.04.01"]
      },
      "categories": {
        "/use-care/": "use-care",
        "/standards/": "standards"
      }
    }
  ],
  "vectorize": {
    "indexName": "classic-chat-knowledge"
  }
}
```

**GitHub Source Options:**

| Option | Description | Default |
|--------|-------------|---------|
| `repository` | Repository in "owner/repo" format | Required |
| `guide` | Guide identifier | Required |
| `routeBasePath` | URL route base path | Required |
| `docsPath` | Path to docs directory in repo | `docs` |
| `defaultRef` | Git branch/ref to use | `main` |
| `versioning` | Docusaurus versioning config | None |
| `exclude` | File patterns to exclude | None |
| `categories` | Path pattern to category mappings | None |

### Environment Variables

For uploading to Vectorize, set these environment variables:

```bash
export CLOUDFLARE_ACCOUNT_ID="your-account-id"
export CLOUDFLARE_API_TOKEN="your-api-token"
```

For GitHub source (optional, but recommended to avoid rate limits):

```bash
export GITHUB_TOKEN="ghp_xxxxxxxxxxxx"
```

Or create a `.env` file:

```
CLOUDFLARE_ACCOUNT_ID=your-account-id
CLOUDFLARE_API_TOKEN=your-api-token
GITHUB_TOKEN=ghp_xxxxxxxxxxxx
```

## Features

### Versioning Support

The indexer supports Docusaurus-style versioned documentation:

```json
{
  "versioning": {
    "versionsFile": "./packages/site/versions.json",
    "versionedDocsPath": "./packages/site/versioned_docs"
  }
}
```

This will:
1. Always index the "next" (current) version from the source path
2. Read `versions.json` to find released versions
3. Index each version from `versioned_docs/version-X.Y.Z/`

You can also specify explicit versions:

```json
{
  "versioning": {
    "versions": ["2026.08.01", "2026.04.01"],
    "versionedDocsPath": "./packages/site/versioned_docs"
  }
}
```

### Category Detection

Categories are automatically detected from file paths:

| Path Pattern | Category |
|--------------|----------|
| `/use-care/` | use-care |
| `/standards/` | standards |
| `/problems/` | problems |
| `/warranty-program/` | warranty-program |
| `/warranty-terms/` | warranty-terms |

Custom mappings can be added in the config:

```json
{
  "sources": [
    {
      "path": "./content/warranty",
      "categories": {
        "/custom-path/": "custom-category"
      }
    }
  ]
}
```

### URL Generation

The indexer generates URLs for each chunk with support for:
- Guide name: `/warranty/...`
- Version: `/warranty/2026.08.01/...` (omitted for latest version)
- Anchors: `/warranty/use-care/air-conditioning#maintenance`

Configure the latest version to omit it from URLs:

```json
{
  "url": {
    "latestVersion": "2026.08.01"
  }
}
```

## Output

### Dry Run Statistics

```
Chat Indexer
============

Site: docs (https://guide.classichomes.com)
Index: classic-chat-knowledge

Processing source: ./content/warranty
  Versions to index: next, 2026.08.01, 2026.04.01

Total chunks: 1639

[DRY RUN] No uploads performed

Chunks by category:
  general: 337
  problems: 82
  standards: 428
  use-care: 586
  warranty-program: 98
  warranty-terms: 108

Chunks by guide:
  warranty: 1639

Chunks by version:
  next: 674
  2026.08.01: 674
  2026.04.01: 291
```

### JSON Output

Use `--output` to write chunks to a JSON file for inspection:

```bash
chat-indexer index --dry-run --output chunks.json
```

## Chunk Structure

Each chunk includes:

```json
{
  "id": "docs-next-coverage-lookup-s0",
  "content": "# Coverage Lookup\n\n...",
  "metadata": {
    "title": "Coverage Lookup",
    "path": "/warranty/next/coverage-lookup",
    "section": "Introduction",
    "anchor": null,
    "category": "general",
    "keywords": ["warranty", "coverage"],
    "guide": "warranty",
    "version": "next",
    "site": "docs",
    "siteUrl": "https://guide.classichomes.com",
    "fullUrl": "https://guide.classichomes.com/warranty/next/coverage-lookup",
    "content": "# Coverage Lookup\n\n..."
  }
}
```

## Architecture

The indexer uses a plugin-based architecture for content sources:

```
src/
├── sources/              # Content source plugins
│   ├── types.ts          # Core interfaces (ContentSource, SourceContext)
│   ├── registry.ts       # Source plugin registry
│   ├── base.ts           # Base class with shared utilities
│   ├── web/              # Web scraping source
│   ├── markdown/         # Local markdown files source
│   └── github/           # GitHub repository source
├── orchestrator.ts       # Pipeline coordinator
├── chunking/             # Content chunking strategies
├── embedding/            # Cloudflare Workers AI embeddings
└── vectorstore/          # Cloudflare Vectorize integration
```

### Adding Custom Sources

You can register custom content sources:

```typescript
import { BaseContentSource, sourceRegistry } from '@classic-homes/chat-indexer';

class MySource extends BaseContentSource<MyConfig> {
  readonly type = 'my-source';
  readonly name = 'My Custom Source';

  async process(config: MyConfig, context: SourceContext): Promise<SourceResult> {
    // Fetch and process content
    // Return chunks and stats
  }

  validateConfig(config: unknown): MyConfig {
    // Validate configuration
  }
}

sourceRegistry.register('my-source', () => new MySource());
```

## Development

```bash
# Run tests
npm test -w @classic-homes/chat-indexer

# Type check
npm run typecheck -w @classic-homes/chat-indexer

# Build
npm run build -w @classic-homes/chat-indexer
```

## License

MIT