# Search Module

The search module provides unified search capabilities across all entities in Open Mercato, supporting multiple search strategies including Meilisearch (full-text) and vector embeddings (semantic search).

## Features

- **Multi-strategy search**: Combines full-text search (Meilisearch), vector-based semantic search, and token matching
- **Automatic indexing**: Subscribes to entity events for real-time index updates
- **Queue-based processing**: Supports async batch processing via Redis/BullMQ for high-volume indexing
- **Topbar progress integration**: Fulltext and vector reindex operations create `progress_jobs` entries and stream updates to the global progress bar
- **Configurable embeddings**: Supports OpenAI, Ollama, and other embedding providers
- **Tenant-scoped**: All indexes are scoped by tenant and optionally by organization
- **Admin-configurable**: Global search (Cmd+K) strategies can be configured per-tenant

## Reindex Progress and SSE

Search reindex endpoints now integrate with the Progress module:

- `POST /api/search/reindex` (fulltext) creates/updates a progress job with `jobType = search.reindex.fulltext`
- `POST /api/search/embeddings/reindex` (vector) creates/updates a progress job with `jobType = search.reindex.vector`
- Queue workers increment progress as batches are processed and complete jobs when totals are reached
- The backend topbar (`ProgressTopBar`) receives real-time updates via SSE (`progress.job.*` events), so no periodic polling is required for progress updates
- Search settings sections (`FulltextSearchSection`, `VectorSearchSection`) now refresh logs/settings on `progress.job.*` and `om:bridge:reconnected` events instead of 5s polling loops

Cancellation endpoints (`/api/search/reindex/cancel`, `/api/search/embeddings/reindex/cancel`) now also cancel the corresponding progress job.

## Global Search Settings

The global search dialog (Cmd+K) can be configured by administrators to control which search strategies are used. This configuration is stored per-tenant.

### Admin Configuration

Navigate to **Settings > Search** to configure which search methods are enabled for Cmd+K:

- **Full-Text Search**: Fast, typo-tolerant search powered by Meilisearch
- **Semantic Search (AI)**: AI-powered search that understands meaning (requires embedding provider)
- **Keyword Search**: Exact word matching in the database

### API Endpoint

**`GET /api/search/settings/global-search`**

Returns the currently enabled strategies for global search.

**Response:**
```json
{
  "enabledStrategies": ["fulltext", "vector", "tokens"]
}
```

**`POST /api/search/settings/global-search`**

Updates the enabled strategies.

**Request:**
```json
{
  "enabledStrategies": ["fulltext", "vector"]
}
```

**Permissions:**
- GET requires `search.view`
- POST requires `search.manage`

## Programmatic Integration (DI)

Other modules can use the search functionality by resolving services from the DI container.

### SearchService

The primary service for executing searches and managing indexes:

```typescript
import type { SearchService } from '@open-mercato/search'

// Resolve from DI container
const searchService = container.resolve('searchService') as SearchService

// Execute a search
const results = await searchService.search('john doe', {
  tenantId: 'tenant-123',
  organizationId: 'org-456', // optional
  limit: 20,
  strategies: ['fulltext', 'vector'], // optional - defaults to all available
})

// Index a record
await searchService.index({
  entityId: 'customers:customer_person_profile',
  recordId: 'rec-123',
  tenantId: 'tenant-123',
  organizationId: 'org-456',
  fields: { name: 'John Doe', email: 'john@example.com' },
  presenter: { title: 'John Doe', subtitle: 'Customer' },
  url: '/backend/customers/people/rec-123',
})

// Bulk index multiple records
await searchService.bulkIndex([record1, record2, record3])

// Delete from all indexes
await searchService.delete('customers:customer_person_profile', 'rec-123', 'tenant-123')

// Purge all records for an entity type
await searchService.purge('customers:customer_person_profile', 'tenant-123')

// Check strategy availability
const isAvailable = await searchService.isStrategyAvailable('fulltext')
```

### SearchIndexer (Higher-Level API)

For config-aware indexing with automatic presenter/URL resolution:

```typescript
import type { SearchIndexer } from '@open-mercato/search'

const searchIndexer = container.resolve('searchIndexer') as SearchIndexer

// Index with automatic config-based formatting
await searchIndexer.indexRecord({
  entityId: 'customers:customer_person_profile',
  recordId: 'rec-123',
  tenantId: 'tenant-123',
  organizationId: 'org-456',
  record: { id: 'rec-123', name: 'John Doe', email: 'john@example.com' },
  customFields: { priority: 'high' },
})

// Check if entity is enabled for search
if (searchIndexer.isEntityEnabled('customers:customer_person_profile')) {
  // Entity is configured for indexing
}

// List all search-enabled entities
const entities = searchIndexer.listEnabledEntities()

// Reindex to Meilisearch with queue support
const result = await searchIndexer.reindexEntityToMeilisearch({
  entityId: 'customers:customer_person_profile',
  tenantId: 'tenant-123',
  organizationId: 'org-456',
  recreateIndex: true,
  useQueue: true, // Use async queue if available
})
```

### SearchService Methods

| Method | Description |
|--------|-------------|
| `search(query, options)` | Execute search across strategies |
| `index(record)` | Index a single record |
| `bulkIndex(records)` | Bulk index multiple records |
| `delete(entityId, recordId, tenantId)` | Delete from all strategies |
| `purge(entityId, tenantId)` | Purge all records for entity type |
| `registerStrategy(strategy)` | Add custom strategy at runtime |
| `unregisterStrategy(strategyId)` | Remove a strategy |
| `getRegisteredStrategies()` | List registered strategy IDs |
| `getStrategy(strategyId)` | Get specific strategy instance |
| `isStrategyAvailable(strategyId)` | Check strategy availability |

### SearchIndexer Methods

| Method | Description |
|--------|-------------|
| `indexRecord(params)` | Index with config-based formatting |
| `deleteRecord(params)` | Delete with config handling |
| `bulkIndexRecords(params[])` | Bulk index with formatting |
| `purgeEntity(params)` | Purge entity type from indexes |
| `reindexEntityToMeilisearch(params)` | Reindex single entity |
| `reindexAllToMeilisearch(params)` | Reindex all entities |
| `getEntityConfig(entityId)` | Get entity search configuration |
| `isEntityEnabled(entityId)` | Check if entity is search-enabled |
| `listEnabledEntities()` | List all enabled entities |

## REST API

### Search Endpoint

**`GET /api/search`**

Execute a search query via HTTP.

**Query Parameters:**

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `q` | string | Yes | Search query |
| `limit` | number | No | Max results (default: 50, max: 100) |
| `strategies` | string | No | Comma-separated strategy IDs (e.g., `fulltext,vector`) |

**Headers:**
- Requires authentication (session cookie or bearer token)
- Requires `search.view` feature permission

**Example Request:**

```bash
curl -X GET "https://your-app.com/api/search?q=john%20doe&limit=20" \
  -H "Authorization: Bearer <token>"
```

**Response:**

```json
{
  "results": [
    {
      "entityId": "customers:customer_person_profile",
      "recordId": "rec-123",
      "score": 0.95,
      "source": "fulltext",
      "presenter": {
        "title": "John Doe",
        "subtitle": "Customer",
        "icon": "user"
      },
      "url": "/backend/customers/people/rec-123",
      "links": [
        { "label": "View", "url": "/backend/customers/people/rec-123" }
      ]
    }
  ],
  "strategiesUsed": ["fulltext", "vector"],
  "timing": 45,
  "query": "john doe",
  "limit": 20
}
```

**Error Responses:**

| Status | Description |
|--------|-------------|
| 400 | Missing query parameter |
| 401 | Unauthorized |
| 503 | Search service unavailable |

### Other API Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/search/settings/global-search` | GET | Get global search strategy configuration |
| `/api/search/settings/global-search` | POST | Update global search strategy configuration |
| `/api/search/reindex` | POST | Trigger full-text search reindex |
| `/api/search/embeddings/reindex` | POST | Trigger vector embeddings reindex |
| `/api/search/embeddings/status` | GET | Get vector indexing status |
| `/api/search/embeddings/config` | POST | Update embedding configuration |
| `/api/search/index` | GET | List indexed entries |
| `/api/search/index` | DELETE | Purge vector index |

## Types

### SearchOptions

```typescript
interface SearchOptions {
  tenantId: string
  organizationId?: string | null
  limit?: number
  strategies?: SearchStrategyId[]  // 'fulltext' | 'vector' | 'tokens'
  entityTypes?: string[]           // Filter by entity types
}
```

### SearchResult

```typescript
interface SearchResult {
  entityId: string                 // e.g., 'customers:customer_person_profile'
  recordId: string                 // Primary key of the record
  score: number                    // Relevance score (0-1)
  source: SearchStrategyId         // Which strategy returned this result
  presenter?: {
    title: string                  // Display title
    subtitle?: string              // Secondary text
    icon?: string                  // Icon identifier
  }
  url?: string                     // Link to the record
  links?: Array<{
    label: string
    url: string
  }>
}
```

### IndexableRecord

```typescript
interface IndexableRecord {
  entityId: string
  recordId: string
  tenantId: string
  organizationId?: string | null
  fields: Record<string, unknown>  // Searchable field values
  presenter?: {
    title: string
    subtitle?: string
    icon?: string
  }
  url?: string
  links?: Array<{ label: string; url: string }>
}
```

## CLI Commands

The search module exposes CLI commands via `yarn mercato search <command>`.

### Status

Show search module status and available strategies:

```bash
yarn mercato search status
```

### Query

Execute a search query:

```bash
yarn mercato search query --query "search terms" --tenant <id> [options]
```

Options:
- `--query, -q` - Search query (required)
- `--tenant` - Tenant ID (required)
- `--org` - Organization ID (optional)
- `--entity` - Entity types to search (comma-separated)
- `--strategy` - Strategies to use (comma-separated: fulltext, vector, tokens)
- `--limit` - Max results (default: 20)

### Index

Index a specific record:

```bash
yarn mercato search index --entity <entityId> --record <recordId> --tenant <tenantId>
```

Options:
- `--entity` - Entity ID (e.g., `customers:customer_person_profile`)
- `--record` - Record ID
- `--tenant` - Tenant ID
- `--org` - Organization ID (optional)

### Reindex

Reindex vector embeddings for entities:

```bash
yarn mercato search reindex --tenant <id> [options]
```

Options:
- `--tenant <id>` - Tenant scope (required for purge & coverage)
- `--org <id>` - Organization scope (requires tenant)
- `--entity <module:entity>` - Reindex a single entity (defaults to all enabled entities)
- `--partitions <n>` - Number of partitions to process in parallel
- `--partition <idx>` - Restrict to a specific partition index
- `--batch <n>` - Override batch size per chunk
- `--force` - Force reindex even if another job is running
- `--purgeFirst` - Purge vector rows before reindexing
- `--skipPurge` - Explicitly skip purging vector rows
- `--skipResetCoverage` - Keep existing coverage snapshots

Use `yarn mercato search reindex-help` for detailed options.

### Test Meilisearch

Test the Meilisearch connection:

```bash
yarn mercato search test-meilisearch
```

### Worker

Start a queue worker for processing search indexing jobs:

```bash
yarn mercato search worker <queue-name> [options]
```

Available queues:
- `vector-indexing` - Process vector embedding indexing jobs
- `fulltext-indexing` - Process Meilisearch batch indexing jobs

Options:
- `--concurrency <n>` - Number of concurrent jobs to process (default: 1)

Examples:
```bash
# Start vector indexing worker with 10 concurrent jobs
yarn mercato search worker vector-indexing --concurrency=10

# Start Meilisearch indexing worker with 5 concurrent jobs
yarn mercato search worker fulltext-indexing --concurrency=5
```

**Requirements:**
- `QUEUE_STRATEGY=async` must be set in environment
- Redis must be configured via `REDIS_URL` or `QUEUE_REDIS_URL`

### Help

Show all available commands:

```bash
yarn mercato search help
```

## Queue Configuration

The search module supports two queue strategies:

### Local Queue (Development)

File-based queue stored in `.mercato/queue/` by default (override with `QUEUE_BASE_DIR`).

```env
QUEUE_STRATEGY=local
```

### Async Queue (Production)

Redis-based queue using BullMQ for distributed processing.

```env
QUEUE_STRATEGY=async
REDIS_URL=redis://localhost:6379
```

When using async queues, start workers in separate processes:

```bash
# Terminal 1: Start vector indexing worker
yarn mercato search worker vector-indexing --concurrency=10

# Terminal 2: Start Meilisearch indexing worker
yarn mercato search worker fulltext-indexing --concurrency=5
```

**Note:** If no workers are running when a reindex is triggered, the API will automatically fall back to synchronous processing and display a warning.

## Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `MEILISEARCH_HOST` | Meilisearch server URL | - |
| `MEILISEARCH_API_KEY` | Meilisearch API key | - |
| `OPENAI_API_KEY` | OpenAI API key for embeddings | - |
| `OM_SEARCH_ENABLED` | Enable/disable search module | `true` |
| `OM_SEARCH_DEBUG` | Enable debug logging for search module | `false` |
| `SEARCH_EXCLUDE_ENCRYPTED_FIELDS` | Exclude encrypted fields from Meilisearch indexing | `false` |
| `QUEUE_STRATEGY` | Queue strategy (`local` or `async`) | `local` |
| `REDIS_URL` | Redis connection URL for async queues | - |
| `QUEUE_REDIS_URL` | Alternative Redis URL for queues | - |

### Debug Logging

Set `OM_SEARCH_DEBUG=true` to enable verbose debug logging for the search module. This outputs information about indexing operations, strategy selection, and error handling. Debug output is redacted: it includes field names, token hashes, and counts, but never raw token text, document values, or decrypted PII. Errors are always logged regardless of this flag.

```env
OM_SEARCH_DEBUG=true
```

### Encrypted Field Exclusion

By default, all fields (including decrypted values of encrypted fields) are indexed into Meilisearch for full-text search. For security-sensitive deployments, you can exclude encrypted fields from indexing by setting:

```env
SEARCH_EXCLUDE_ENCRYPTED_FIELDS=true
```

When enabled, fields defined in `encryption_maps` are automatically excluded from the Meilisearch index. This includes fields like:
- Customer PII: `display_name`, `primary_email`, `primary_phone`
- Person profiles: `first_name`, `last_name`, `job_title`
- Company profiles: `legal_name`, `brand_name`, `domain`
- Comments and activities: `body`, `subject`
- And other fields configured in the encryption maps

**Note:** This only affects Meilisearch indexing. Vector search uses its own field configuration via `buildSource`. Consider adjusting your `buildSource` implementation if you also need to exclude sensitive fields from vector embeddings.

## Configuring Entities for Search

Each module can define which entities are searchable by creating a `search.ts` file in the module root.

### Entity Configuration Structure

```typescript
// packages/your-package/src/modules/your-module/search.ts
import type {
  SearchModuleConfig,
  SearchBuildContext,
  SearchIndexSource,
  SearchResultPresenter,
  SearchResultLink,
} from '@open-mercato/shared/modules/search'

export const searchConfig: SearchModuleConfig = {
  entities: [
    {
      entityId: 'your_module:your_entity',  // Must match entity registry
      enabled: true,
      priority: 10,  // Higher = appears first in mixed results

      // FOR VECTOR SEARCH: buildSource generates text for embeddings
      buildSource: async (ctx: SearchBuildContext): Promise<SearchIndexSource | null> => {
        const lines: string[] = []

        // Add text that should be searchable semantically
        lines.push(`Name: ${ctx.record.name}`)
        lines.push(`Description: ${ctx.record.description}`)

        // Include custom fields
        if (ctx.customFields.notes) {
          lines.push(`Notes: ${ctx.customFields.notes}`)
        }

        if (!lines.length) return null

        return {
          text: lines,  // This text gets embedded for vector search
          presenter: {
            title: ctx.record.name,
            subtitle: ctx.record.description,
            icon: 'lucide:file',
          },
          links: [
            { href: `/your-entity/${ctx.record.id}`, label: 'View', kind: 'primary' }
          ],
          checksumSource: { record: ctx.record, customFields: ctx.customFields },
        }
      },

      // FOR MEILISEARCH: fieldPolicy controls full-text indexing
      fieldPolicy: {
        searchable: ['name', 'description', 'notes'],  // Indexed for full-text
        hashOnly: ['email', 'phone'],                   // Hashed, not searchable
        excluded: ['password', 'secret'],               // Never indexed
      },

      // Optional: Custom presenter formatting
      formatResult: async (ctx: SearchBuildContext): Promise<SearchResultPresenter | null> => {
        return {
          title: ctx.record.name,
          subtitle: ctx.record.status,
          icon: 'lucide:user',
        }
      },

      // Optional: Primary URL for the record
      resolveUrl: async (ctx: SearchBuildContext): Promise<string | null> => {
        return `/your-entity/${ctx.record.id}`
      },

      // Optional: Additional action links
      resolveLinks: async (ctx: SearchBuildContext): Promise<SearchResultLink[] | null> => {
        return [
          { href: `/your-entity/${ctx.record.id}/edit`, label: 'Edit', kind: 'secondary' }
        ]
      },
    },
  ],
}

export default searchConfig
```

### Search Strategies Comparison

| Aspect | Full-Text (fulltext) | Vector Search (vector) | Token Search (tokens) |
|--------|----------------------|------------------------|----------------------|
| **Configuration** | `fieldPolicy` | `buildSource` | Automatic |
| **Search Type** | Full-text with typo tolerance | Semantic similarity | Exact token matching |
| **Good For** | Exact matches, filters, facets | Natural language, "find similar" | Simple lookups |
| **Backend** | Meilisearch server | pgvector/Qdrant/ChromaDB + embeddings | PostgreSQL |
| **Requires** | `MEILISEARCH_HOST` | `OPENAI_API_KEY` (or other provider) | Database connection |

### Enabling/Disabling Strategies

You can control which strategies are used at multiple levels:

#### Per Entity

```typescript
// Full-text only (no vector search)
{
  entityId: 'your_module:your_entity',
  enabled: true,
  // NO buildSource = no vector search
  fieldPolicy: {
    searchable: ['name', 'description'],
  },
}

// Vector only (no full-text)
{
  entityId: 'your_module:your_entity',
  enabled: true,
  buildSource: async (ctx) => ({ text: [...], presenter: {...} }),
  // NO fieldPolicy = no full-text search
}

// Both strategies
{
  entityId: 'your_module:your_entity',
  enabled: true,
  buildSource: async (ctx) => ({ text: [...], presenter: {...} }),
  fieldPolicy: { searchable: ['name'] },
}
```

#### Global Level (DI Registration)

In `packages/core/src/bootstrap.ts`:

```typescript
import { registerSearchModule } from '@open-mercato/search'

registerSearchModule(container, {
  moduleConfigs: searchModuleConfigs,
  skipVector: true,    // Disable vector search globally
  skipFulltext: true,  // Disable full-text search globally
  skipTokens: true,    // Disable token search globally
})
```

#### Per Query

```typescript
// Only use full-text search for this query
const results = await searchService.search('query', {
  tenantId: '...',
  strategies: ['fulltext'],
})

// Only use vector search
const results = await searchService.search('query', {
  tenantId: '...',
  strategies: ['vector'],
})

// Use all available strategies (default)
const results = await searchService.search('query', {
  tenantId: '...',
})
```

#### Environment-Based

Strategies automatically become unavailable if their backend is not configured:

| Strategy | Required Environment |
|----------|---------------------|
| fulltext | `MEILISEARCH_HOST` |
| vector | `OPENAI_API_KEY` (or other embedding provider) |
| tokens | Database connection (always available) |

### SearchBuildContext

The context object passed to `buildSource` and other config functions:

```typescript
interface SearchBuildContext {
  record: Record<string, unknown>       // The database record
  customFields: Record<string, unknown> // Custom field values (cf:* fields)
  tenantId?: string | null
  organizationId?: string | null
  queryEngine?: QueryEngine             // For loading related entities
}
```

### SearchIndexSource

The return type from `buildSource`:

```typescript
interface SearchIndexSource {
  text: string | string[]              // Text to embed for vector search
  presenter?: SearchResultPresenter    // Display info for search results
  links?: SearchResultLink[]           // Action links
  checksumSource?: unknown             // Used for change detection
}
```

## Architecture

```
packages/search/
├── src/
│   ├── modules/search/
│   │   ├── api/              # API routes
│   │   ├── cli.ts            # CLI commands
│   │   ├── di.ts             # Dependency injection
│   │   ├── subscribers/      # Event subscribers for auto-indexing
│   │   └── workers/          # Queue job handlers
│   ├── indexer/              # Search indexer implementation
│   ├── queue/                # Queue definitions
│   ├── strategies/           # Search strategy implementations
│   └── vector/               # Vector index service
```