# Media & asset search — local index plan

## Implementation status (2026-06)

**Implemented** in `@se-studio/contentful-cms`:

- `cms-edit index sync` / `index status` (Preview API draft default + `--published`)
- SQLite index at `~/.contentful-cms/index/<spaceId>/<environment>/media.db` via `node:sqlite`
- Delivery token resolution: `deliveryAccessToken` → `deliveryApiKeyName` → environment match → optional `autoCreateDeliveryApiKey`
- **Index required** (no CMA fallback) for: `asset search`, `asset audit`, `list --type media` with `--asset-id` / `--asset-filename*`, upload `--if-exists-by-filename`
- **Reference catalog** (index-preferred, CMA fallback): `template`, `articleType`, `tagType`, `tag` in `reference_entries` table (schema v2)
- `list --type template|articleType|tagType|tag`, `resolve --type … --slug`, taxonomy `--if-not-exists`, and `tagTypeSlug` shorthands use the catalog when the index exists
- Bulk CMA asset pagination and N+1 `links_to_asset` scans removed from those paths

Help: [help/index-sync.md](./help/index-sync.md).

---

## Problem

Today, cms-edit discovers assets and Media entries via the **Contentful Management API (CMA)**:

| Operation | CMA approach | Pain |
|-----------|--------------|------|
| Exact `fileName` | `fields.file.fileName` filter | Fast |
| Regex `fileName` | Paginate assets (+ optional `[match]` prefix), filter client-side | Slow on large spaces |
| `list --type media --asset-filename*` | Find assets, then **N×** `links_to_asset` queries for Media | Very slow at scale |
| Title search | `fields.title[match]` | OK for small result sets |

CMA is rate-limited, optimized for writes, and regex scans can mean thousands of round trips. Publication-import and deduplication workflows need **fast repeated lookups** by `fileName`, title, MIME type, and “which Media wraps this asset?”.

## Goal

A **local, per-space index** built from the **Content Delivery API (CDA)** (read-only, cache-friendly) that powers:

- Sub-second filename / regex / title search over all assets
- Reverse lookup: asset ID → Media entry IDs
- Optional freshness via incremental sync or scheduled rebuild

CMA remains the source of truth for **creates/updates** (`asset upload`, `create media`, taxonomy).

## Non-goals (v1)

- Full-text search across entry bodies
- Replacing CMA for asset upload or processing
- Cross-space search
- Publishing or preview-mode mixing in one index (pick one: published CDA or preview CPA — see below)

## Proposed architecture

```mermaid
flowchart LR
  subgraph sync [Index sync]
    CMA_keys[CMA API keys]
    CDA[Contentful CDA]
    SyncCmd["cms-edit index sync"]
    Store[(Local index)]
    CMA_keys -->|"resolve accessToken"| SyncCmd
    SyncCmd --> CDA
    CDA --> Store
  end
  subgraph query [Search]
    CLI[cms-edit asset search]
    List[cms-edit list --type media]
    CLI --> Store
    List --> Store
  end
  subgraph write [Writes unchanged]
    CMA[Contentful CMA]
    Upload[asset upload]
    Upload --> CMA
  end
```

### Index store

**Location:** `~/.contentful-cms/index/<spaceId>/<environment>/` (or project-local `.contentful-cms/index/` when `--config` is set).

**Format (v1):** SQLite via `better-sqlite3` or embedded JSONL + in-memory Map for zero native deps. Prefer **SQLite** for regex scans and stable pagination on 10k+ assets.

**Tables (conceptual):**

| Table | Key fields | Purpose |
|-------|------------|---------|
| `assets` | `id`, `fileName`, `title`, `description`, `contentType`, `url`, `width`, `height`, `size`, `updatedAt` | Asset search |
| `media` | `id`, `name`, `assetId`, `updatedAt` | Media wrapper lookup |
| `meta` | `syncedAt`, `assetCount`, `mediaCount`, `cdaTokenHash` | Freshness |

Indexes: `assets.fileName`, `assets.title`; `media.assetId`.

### Sync command

```bash
# Full rebuild (paginate CDA assets + media entries)
cms-edit index sync

# Incremental (if sync token / updatedSince supported)
cms-edit index sync --since <iso>

# Status
cms-edit index status
```

**CDA pagination:** Use `contentful` JS SDK or fetch `https://cdn.contentful.com/.../assets` and `.../entries?content_type=media` with `limit=1000`, following `skip` until complete.

**Preview vs published:** Default index = **Preview API** (draft content, matches CMA uploads). Optional `--published` uses Delivery API for published assets only (separate DB file suffix `-published`).

### Delivery token resolution (CMA → CDA)

The **management token is not a delivery token** — it cannot call `cdn.contentful.com`. For index sync we only need a **read-only** CDA (or CPA) access token. That token can be obtained without asking operators to copy a second secret, by using the **same CMA client** cms-edit already uses.

Contentful exposes delivery keys on the space via the [Management API — API keys](https://www.contentful.com/developers/docs/references/content-management-api/#/reference/api-keys): list/create delivery keys; each key includes an `accessToken` for CDA. Creating a delivery key also creates a linked preview key (resolve separately for `--preview` sync).

**Resolution order** (first match wins):

| Priority | Source | Notes |
|----------|--------|--------|
| 1 | Explicit override | Space config `deliveryAccessToken` or `${CONTENTFUL_ACCESS_TOKEN}` / `${CONTENTFUL_DELIVERY_TOKEN}` — same vars as marketing apps |
| 2 | Named key via CMA | `deliveryApiKeyName` in `.contentful-cms.json` → `space.getApiKeys()` → match `name` |
| 3 | Environment match via CMA | Pick first delivery key whose `environments` includes the configured `environment` (default `master`) |
| 4 | Auto-create (optional) | If PAT has API-key manage rights and no key matches: create `cms-edit-index` delivery key scoped to that environment |

Implement as `resolveDeliveryAccessToken(space, { preview?: boolean })` in `src/contentful/delivery-token.ts` (or similar), called at the start of `index sync` only — not on every search (cache resolved token in `meta` as a hash + key id, not the raw token).

**SDK sketch (contentful-management, already a dependency):**

```typescript
const client = createClient({ accessToken: space.managementToken }, { type: 'plain' });
const spaceApi = await client.getSpace(space.spaceId);
const { items: keys } = await spaceApi.getApiKeys();
const key = pickDeliveryKey(keys, { name: space.deliveryApiKeyName, environment: space.environment });
const deliveryToken = key.accessToken; // CDA
// preview: await spaceApi.getPreviewApiKey(key.preview_api_key.sys.id) → accessToken
```

**What does not work:**

- Passing `managementToken` directly to the CDA client.
- Assuming any single PAT can read API keys (editor-only tokens may lack `ContentDelivery` / key read rights — fail with a clear error and point to explicit `deliveryAccessToken`).

**Caveats:**

| Topic | Guidance |
|-------|----------|
| Permissions | PAT needs rights to read (and optionally create) space API keys; document in hosted/CLI setup guides |
| Multiple keys | Never “first key in list” without environment/name filter |
| Propagation | New keys may 401 for a few seconds after create — retry with backoff |
| Secrets | Do not log `accessToken`; store only `cdaTokenHash` + `apiKeyId` in index `meta` |
| Preview index | Resolve linked preview key when `index sync --preview` |

**Config additions** (`.contentful-cms.json` per space, all optional):

```json
{
  "deliveryAccessToken": "${CONTENTFUL_ACCESS_TOKEN}",
  "deliveryApiKeyName": "Production",
  "autoCreateDeliveryApiKey": false
}
```

If only `managementToken` is configured, `index sync` should still work when CMA can list a suitable delivery key — aligning with agent setups that already have cms-edit credentials.

### Query integration (implemented)

1. **`asset search`** — Requires index; queries SQLite (title `LIKE`, filename exact/regex, MIME filter).
2. **`list --type media --asset-filename*`** — Single index query (assets + joined media rows); no CMA.
3. **`--if-exists-by-filename`** — Index exact `fileName` lookup; optional post-upload row upsert.

Stale index: warn when `syncedAt` > 24h; no `--no-index` CMA fallback.

### Freshness

| Strategy | When |
|----------|------|
| Manual `index sync` | Before large import jobs |
| `index sync` in CI nightly | Keeps agents fast |
| Post-upload hook (optional) | After `asset upload`, upsert one row into index |

Webhook-driven incremental sync is out of scope for v1 but fits the same schema.

## Implementation phases

### Phase 1 — Foundation (MVP)

- [x] Add `contentful` delivery client dependency
- [x] `resolveDeliveryAccessToken()` — explicit config, then CMA `getApiKeys()` (+ optional `createApiKey`)
- [x] Config fields: `deliveryAccessToken`, `deliveryApiKeyName`, `autoCreateDeliveryApiKey`
- [x] `cms-edit index sync` full rebuild → SQLite (CDA pagination after token resolve)
- [x] `cms-edit index status`
- [x] Unit tests: delivery token, index queries

### Phase 2 — Wire search

- [x] `asset search --filename` / `--filename-match` uses index (required)
- [x] `findAssetByExactFilename` replaced by index lookup
- [x] README / help document index requirement

### Phase 3 — Media discovery

- [x] Index `media` entries with `assetId` and denormalized `assetFileName`
- [x] `list --type media --asset-id` / `--asset-filename*` without N+1 CMA
- [x] JSON output for indexed media list

### Phase 3b — Reference catalog

- [x] Index `template`, `articleType`, `tagType`, `tag` in `reference_entries` (schema v2)
- [x] `list --type` for reference types from index (`--force-cma` escape hatch)
- [x] `resolve --type … --slug` and taxonomy `--if-not-exists` prefer index
- [x] Post-create upsert for taxonomy and template creates
- [x] `index dump --reference-only` / `--reference-type`

### Phase 3c — Auto-sync and incremental refresh

- [x] `ensureContentIndex` — auto full rebuild when missing or schema mismatch
- [x] Incremental upsert when only stale (>24h), same schema (`sys.updatedAt[gte]` last `syncedAt`)
- [x] Manual `index sync` always full rebuild
- [x] File lock — single writer per index database
- [x] Duplicate slug/label detection in reference catalog (index + CMA)

### Phase 4 — Operator UX

- [x] Auto-sync on index-backed commands (see [help/index-sync.md](./help/index-sync.md))
- [ ] Optional `index sync` before bulk upload scripts (skill note — partial)
- [ ] MCP tool exposure if hosted MCP should search assets

## Dependencies & package boundaries

- Keep index code inside `packages/contentful-cms` (CLI owns agent workflows).
- **Do not** pull `@se-studio/contentful-rest-api` into cms-edit unless we need shared cache tags; a thin CDA client keeps the CLI package self-contained.
- SQLite: evaluate `node:sqlite` (Node 22+) vs `better-sqlite3` for install size in npx usage.

## Risks

| Risk | Mitigation |
|------|------------|
| Index stale vs CMA | Show `syncedAt` in search output; exact match confirm via CMA before skip-upload |
| Draft assets missing from CDA | Default index uses Preview API; use `--published` only when you need published-only discovery |
| Disk size | 10k assets ≈ few MB in SQLite |
| Token sprawl | Prefer CMA-resolve when only `managementToken` is set; optional explicit `deliveryAccessToken` for CI |
| CMA cannot read API keys | Clear error; require `deliveryAccessToken` in config |
| Leaked delivery token in logs | Never log; `meta.cdaTokenHash` only |
| New API key 401 | Retry CDA requests after create (Contentful propagation delay) |

## Success criteria

- Regex search over 5k assets completes in **< 2s** locally after sync
- `list --type media --asset-filename-match` does **not** scale with asset count via N+1 CMA calls
- No change to write paths; tests cover index-off fallback

## References

- Current CMA helpers: `src/contentful/assets.ts`, `src/contentful/media.ts`
- CMA client: `src/contentful/client.ts` (`managementToken` per space)
- Contentful: [API keys (CMA)](https://www.contentful.com/developers/docs/references/content-management-api/#/reference/api-keys), [Authentication](https://www.contentful.com/developers/docs/references/authentication/)
- App delivery env vars (override): `CONTENTFUL_ACCESS_TOKEN`, `CONTENTFUL_PREVIEW_ACCESS_TOKEN` in `contentful-rest-api`
- Issue context: #42 publication import (GITHUB_ISSUES_IMPLEMENTATION_STATUS.md)