# Automatic Embeddings

Quickback can automatically generate embeddings for your data using Cloudflare Queues and Workers AI. When configured, INSERT and UPDATE operations automatically enqueue embedding jobs that are processed asynchronously.

## Enabling Embeddings

Add an `embeddings` configuration to your resource definition:

```typescript
// quickback/features/jobs/jobs.ts
import { feature, q } from "@quickback/compiler";

export default feature("jobs", {
  columns: {
    id:             q.id(),
    title:          q.text().required(),
    description:    q.text().required(),
    department:     q.text().optional(),
    embedding:      q.text().optional(),
    organizationId: q.scope("organization"),
    ...q.audit(),
    ...q.softDelete(),
  },
  firewall: [{ field: 'organizationId', equals: 'ctx.activeOrgId' }],

  embeddings: {
    fields: ['description'],              // Fields to concatenate and embed
    model: '@cf/baai/bge-base-en-v1.5',   // Embedding model (optional)
    onInsert: true,                       // Auto-embed on create (default: true)
    onUpdate: ['description'],            // Re-embed when these fields change
    embeddingColumn: 'embedding',         // Column to store embedding
    metadata: ['department'],             // Metadata for Vectorize index
  },

  create: { access: { roles: ['recruiter'] } },
  update: { access: { roles: ['recruiter'] } },
});
```

## Configuration Options

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `fields` | `string[]` | **Required** | Fields to concatenate and embed |
| `model` | `string` | `'@cf/baai/bge-base-en-v1.5'` | Workers AI embedding model |
| `onInsert` | `boolean` | `true` | Embed on INSERT operations |
| `onUpdate` | `boolean \| string[]` | `true` | Embed on UPDATE; array limits to specific fields |
| `embeddingColumn` | `string` | `'embedding'` | Column to store the embedding vector |
| `separator` | `string` | `' '` | Separator for joining multiple fields |
| `metadata` | `string[]` | `[]` | Fields to include in Vectorize metadata |

### onUpdate Options

```typescript
// Always re-embed on any update
onUpdate: true

// Never re-embed on update
onUpdate: false

// Only re-embed when specific fields change
onUpdate: ['description', 'title']
```

## How It Works

```
┌─────────────────────────────────────────────────────────────────────┐
│                     Main API Worker                                 │
│                                                                     │
│  ┌─────────────────────┐    ┌────────────────────┐                 │
│  │ POST /jobs           │    │ Queue Consumer     │                 │
│  │                     │    │                    │                 │
│  │ 1. Auth middleware  │    │ 1. Workers AI      │                 │
│  │ 2. Firewall         │    │    embed()         │                 │
│  │ 3. Guards           │    │ 2. D1 update()     │                 │
│  │ 4. Insert to D1     │    │ 3. Vectorize       │                 │
│  │ 5. Enqueue job ─────┼───▶│    upsert()        │                 │
│  └─────────────────────┘    └────────────────────┘                 │
│                                      ▲                              │
│                    ┌─────────────────┴──────────────────┐          │
│                    │      EMBEDDINGS_QUEUE              │          │
│                    └────────────────────────────────────┘          │
└─────────────────────────────────────────────────────────────────────┘
```

1. **API Request** - POST/PATCH arrives and passes through auth, firewall, guards
2. **Database Insert** - Record is created/updated in D1
3. **Enqueue Job** - Embedding job is sent to Cloudflare Queue
4. **Queue Consumer** - Processes job asynchronously:
   - Calls Workers AI to generate embedding
   - Updates D1 with embedding vector
   - Optionally upserts to Vectorize index

## Security Model

**Security is enforced at enqueue time, not consume time:**

- Jobs are only enqueued after passing all security checks (auth, firewall, guards)
- The queue consumer is an internal process that executes pre-validated jobs
- If a user can't create a job posting, they can't trigger an embedding job

## Generic Embeddings API

In addition to automatic embeddings on standard resource operations, Quickback generates a generic embeddings API endpoint that allows you to trigger embeddings on arbitrary content.

### POST /api/v1/embeddings

Generate an embedding for any text content:

```bash
curl -X POST https://your-api.workers.dev/api/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Cookie: better-auth.session_token=..." \
  -d '{
    "content": "Text to embed",
    "model": "@cf/baai/bge-base-en-v1.5",
    "table": "jobs",
    "id": "job_123"
  }'
```

#### Request Body

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `content` | `string` | Yes | Text to generate embedding for |
| `model` | `string` | No | Embedding model (defaults to table config or `@cf/baai/bge-base-en-v1.5`) |
| `table` | `string` | No | Table to store embedding back to |
| `id` | `string` | Conditional | Record ID to update (required if `table` is specified) |

#### Response

```json
{
  "queued": true,
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
  "table": "jobs",
  "id": "job_123",
  "model": "@cf/baai/bge-base-en-v1.5"
}
```

### GET /api/v1/embeddings/tables

List tables that have embeddings configured:

```bash
curl https://your-api.workers.dev/api/v1/embeddings/tables \
  -H "Cookie: better-auth.session_token=..."
```

Response:

```json
{
  "tables": [
    {
      "name": "jobs",
      "embeddingColumn": "embedding",
      "model": "@cf/baai/bge-base-en-v1.5"
    }
  ]
}
```

### Authentication & Authorization

The generic embeddings API requires authentication and uses the `activeOrgId` from the user's context to enforce organization-level isolation. Embedding jobs are scoped to the user's current organization.

### Use Cases

The generic embeddings API is useful for:

- **Batch embedding**: Embed content without going through CRUD routes
- **Re-embedding**: Force re-generation of embeddings for existing records
- **Preview embeddings**: Test embedding generation before persisting
- **External content**: Embed content that doesn't fit your defined schemas

## Generated Files

When embeddings are configured, the compiler generates:

| File | Purpose |
|------|---------|
| `src/queue-consumer.ts` | Queue consumer handler for processing embedding jobs |
| `src/routes/embeddings.ts` | Generic embeddings API routes |
| `wrangler.toml` | Queue producer/consumer bindings, AI binding |
| `src/env.d.ts` | `EMBEDDINGS_QUEUE`, `AI` types |
| `src/index.ts` | Exports `queue` handler, mounts `/api/v1/embeddings` |

### wrangler.toml additions

```toml
# Embeddings Queue
[[queues.producers]]
queue = "your-app-embeddings-queue"
binding = "EMBEDDINGS_QUEUE"

[[queues.consumers]]
queue = "your-app-embeddings-queue"
max_batch_size = 10
max_batch_timeout = 30
max_retries = 3

# Workers AI
[ai]
binding = "AI"
```

## Multiple Fields

Embed multiple fields by concatenating them:

```typescript
embeddings: {
  fields: ['title', 'description', 'department'],  // Joined with spaces by default
  // ...
}
```

Generated embedding text: `"${title} ${description} ${department}"`

### Custom Separator

Use `separator` to control how fields are joined. This is useful for sentence boundary detection:

```typescript
embeddings: {
  fields: ['title', 'description'],
  separator: '. ',  // Join with period + space
  // ...
}
```

Generated code: `[result[0].title, result[0].description].filter(Boolean).join('. ')`

The `filter(Boolean)` ensures null or empty fields are excluded cleanly — no trailing separator when a field is absent.

## Vectorize Integration

If you have a Vectorize index configured, embeddings are automatically upserted:

```typescript
// quickback.config.ts
providers: {
  database: {
    config: {
      vectorizeIndexName: 'jobs-embeddings',  // Your Vectorize index
      vectorizeBinding: 'VECTORIZE',
    },
  },
},
```

The queue consumer will:
1. Generate the embedding via Workers AI
2. Store the vector in D1 (JSON string)
3. Upsert to Vectorize with metadata

### Vectorize Metadata

Include fields in Vectorize metadata for filtering:

```typescript
embeddings: {
  fields: ['description'],
  metadata: ['department', 'organizationId', 'status'],
}
```

Enables queries like:
```typescript
const results = await env.VECTORIZE.query(vector, {
  topK: 10,
  filter: { department: 'engineering' }
});
```

## Schema Requirements

Your schema must include the embedding column:

```typescript
// quickback/features/jobs/jobs.ts
import { feature, q } from "@quickback/compiler";

export default feature("jobs", {
  columns: {
    id:             q.id(),
    title:          q.text().required(),
    description:    q.text().required(),
    department:     q.text().optional(),
    status:         q.text().required(),
    organizationId: q.scope("organization"),

    // Embedding column — stores JSON array of floats
    embedding:      q.text().optional(),
    ...q.audit(),
    ...q.softDelete(),
  },
  firewall: [{ field: 'organizationId', equals: 'ctx.activeOrgId' }],
  read: {},
  create: {},
  update: {},
  delete: {},
});
```

## Supported Models

Any Workers AI embedding model can be used:

| Model | Dimensions | Notes |
|-------|------------|-------|
| `@cf/baai/bge-base-en-v1.5` | 768 | Default, good general-purpose |
| `@cf/baai/bge-small-en-v1.5` | 384 | Faster, smaller |
| `@cf/baai/bge-large-en-v1.5` | 1024 | Higher quality |

See [Workers AI Models](https://developers.cloudflare.com/workers-ai/models/) for the full list.

## Deployment

After compiling with embeddings:

1. **Create the queue:**
   ```bash
   wrangler queues create your-app-embeddings-queue
   ```

2. **Deploy the worker:**
   ```bash
   wrangler deploy
   ```

The single worker handles both HTTP requests and queue consumption.

## Monitoring

Queue metrics are available in the Cloudflare dashboard:
- Messages enqueued
- Messages processed
- Retry count
- Consumer lag

Check queue health:
```bash
wrangler queues list
wrangler queues consumer your-app-embeddings-queue
```

## Error Handling

Failed jobs are automatically retried (up to `max_retries`):

```typescript
// In queue consumer
try {
  const embedding = await env.AI.run(job.model, { text: job.content });
  // ...
  message.ack();
} catch (error) {
  console.error('[Queue] Embedding job failed:', error);
  message.retry();  // Will retry up to 3 times
}
```

After max retries, the message is dead-lettered (if configured) or dropped.

## Similarity Search Service

For applications that need typed similarity search with classification, you can define embedding search configurations using `defineEmbedding`. This generates a service layer with typed search functions.

### Defining Search Configurations

Create a file in `services/embeddings/`:

```typescript
// services/embeddings/job-similarity.ts
import { defineEmbedding } from '@quickback/compiler';

export default defineEmbedding({
  name: 'job-similarity',
  description: 'Find similar job postings by description content',

  // Source configuration
  source: 'jobs',                // Table name
  vectorIndex: 'VECTORIZE',      // Binding name
  model: '@cf/baai/bge-base-en-v1.5',

  // Search configuration
  search: {
    threshold: 0.60,             // Minimum similarity (default: 0.60)
    limit: 10,                   // Max results (default: 10)
    classify: {
      DUPLICATE: 0.90,           // Score >= 0.90 = DUPLICATE
      CONFIRMS: 0.85,            // Score >= 0.85 = CONFIRMS
      RELATED: 0.75,             // Score >= 0.75 = RELATED
    },
    filters: ['department', 'organizationId'],  // Filterable fields
  },

  // Generation triggers (beyond CRUD)
  triggers: {
    onQueueMessage: 'embed_job',  // Listen for queue messages
  },
});
```

### Generated Service Layer

After compilation, a `createEmbeddings()` helper is generated in `src/lib/embeddings.ts`:

```typescript
import { createEmbeddings } from '../lib/embeddings';

export const execute: ActionExecutor = async ({ ctx, input }) => {
  const embeddings = createEmbeddings(ctx.env);

  // Search with automatic classification
  const similar = await embeddings.jobSimilarity.search(
    'Senior Full-Stack Engineer with React and Node.js experience',
    {
      department: 'engineering',
      limit: 5,
      threshold: 0.70,
    }
  );

  // Returns: [{ id, score: 0.87, classification: 'CONFIRMS', metadata }]
  for (const match of similar) {
    console.log(`${match.classification}: ${match.id} (${match.score})`);
  }

  return { similar };
};
```

### Classification Thresholds

Results are automatically classified based on similarity score. The classification
names are **fixed** — `DUPLICATE`, `CONFIRMS`, `RELATED`, and the implicit `NEW`
floor. What you configure is the cosine-similarity cutoff each band starts at:

| Classification | Default Threshold | Meaning |
|----------------|-------------------|---------|
| `DUPLICATE` | >= 0.90 | Near-identical to an existing record |
| `CONFIRMS` | >= 0.85 | Strongly similar — corroborates the existing record |
| `RELATED` | >= 0.75 | Topically related |
| `NEW` | < 0.75 | No significant match |

> **The three keys are the only keys**
>
> `classify` accepts exactly `DUPLICATE`, `CONFIRMS`, and `RELATED`. `NEW` is the
> implicit floor and is not configurable. Any other key is **silently ignored** —
> the parser matches these three names literally, so an invented name like
> `SIMILAR_ROLE: 0.85` leaves the corresponding band at its default instead of
> raising an error.


Customize the cutoffs per use case:

```typescript
search: {
  classify: {
    DUPLICATE: 0.95,  // Stricter duplicate detection
    CONFIRMS: 0.88,
    RELATED: 0.70,    // Broader "related" category
  },
}
```

### Gray Zone Detection

For cases where automatic classification isn't sufficient, use gray zone detection to get matches that need semantic evaluation:

```typescript
const results = await embeddings.jobSimilarity.findWithGrayZone(
  'Some job description text',
  { min: 0.60, max: 0.85 }
);

// Returns structured results:
// {
//   high_confidence: [...],  // Score >= 0.85 (auto-classified)
//   gray_zone: [...]         // 0.60 <= score < 0.85 (needs review)
// }

// Process high confidence matches automatically
for (const match of results.high_confidence) {
  await markAsDuplicate(match.id);
}

// Queue gray zone for manual review
for (const match of results.gray_zone) {
  await queueForReview(match.id, match.score);
}
```

### Generate Embeddings Directly

Generate embeddings without searching:

```typescript
const embeddings = createEmbeddings(ctx.env);

// Get raw embedding vector
const vector = await embeddings.jobSimilarity.embed(
  'Text to embed'
);
// Returns: number[] (768 dimensions for bge-base)
```

### Multiple Search Configurations

Define different configurations for different use cases:

```typescript
// services/embeddings/candidate-similarity.ts
export default defineEmbedding({
  name: 'candidate-similarity',
  description: 'Match candidates by resume content',
  source: 'candidates',
  search: {
    threshold: 0.65,
    limit: 20,
    classify: {
      DUPLICATE: 0.92,
      CONFIRMS: 0.80,
      RELATED: 0.65,
    },
    filters: ['source'],
  },
});

// services/embeddings/application-match.ts
export default defineEmbedding({
  name: 'application-match',
  description: 'Match applications to similar job postings',
  source: 'applications',
  search: {
    threshold: 0.70,
    limit: 5,
    classify: {
      DUPLICATE: 0.95,
      CONFIRMS: 0.90,
      RELATED: 0.80,
    },
    filters: ['jobId', 'organizationId'],
  },
});
```

Usage:
```typescript
const embeddings = createEmbeddings(ctx.env);

// Different search behaviors for different content types
const similarJobs = await embeddings.jobSimilarity.search(text, opts);
const similarCandidates = await embeddings.candidateSimilarity.search(text, opts);
const matchingApplications = await embeddings.applicationMatch.search(text, opts);
```

### Table-Level vs Service-Level

| Feature | Table-level (`defineTable`) | Service-level (`defineEmbedding`) |
|---------|----------------------------|----------------------------------|
| Auto-embed on INSERT | ✅ | ❌ |
| Auto-embed on UPDATE | ✅ | ❌ |
| Custom search functions | ❌ | ✅ |
| Classification thresholds | ❌ | ✅ |
| Gray zone detection | ❌ | ✅ |
| Filterable searches | ❌ | ✅ |
| Queue message triggers | ❌ | ✅ |

**Use both together:**
- `defineTable` with `embeddings` config for automatic embedding generation
- `defineEmbedding` for typed search functions with classification