# Search

> Keep an external search index (Typesense / Meilisearch / Algolia) in sync with your tables via the ChangeEvent tap, query it tenant-scoped through a typed action + hook.



---

<!-- source: en/plugins/search.md -->
## Search

_Keep an external search index (Typesense / Meilisearch / Algolia) in sync with your tables via the ChangeEvent tap, query it tenant-scoped through a typed action + hook._

`@voltro/plugin-search` mirrors your tables into an external search engine. It rides the **ChangeEvent tap** — every insert/update/delete on a configured table maps into the index automatically; a typed `search.query` action reads it back, tenant-scoped, with a `useSearch` hook on the client.

## Wiring

```ts
// app.config.ts
import { searchPlugin } from '@voltro/plugin-search'

export default {
  type: 'api' as const,
  name: 'api',
  plugins: [
    searchPlugin({
      // backend: defaults to memory — dev + tests only. Under NODE_ENV=production
      // the memory backend REFUSES TO BOOT (see "The memory backend refuses to
      // boot in production" below). Name a durable engine for a deployment:
      // backend: { engine: 'typesense', url, apiKey } | { engine: 'meilisearch', … } | { engine: 'algolia', appId, apiKey }
      indexes: {
        posts: {
          index: 'posts',
          tenantField: 'tenantId',                       // index documents carry the tenant → query scopes by it
          queryableFields: ['title', 'body', 'status'],  // optional: fields a caller may filter/facet/highlight on
          map: (row) => ({ id: String(row.id), title: String(row.title), body: String(row.body) }),
        },
      },
      // allowedEngineParams: ['query_by'],               // optional: widen the engineParams allowlist (see below)
    }),
  ],
}
```

## How sync works

The plugin declares `onChangeEvent`. On every committed write to a configured table, `applyChange` maps the row via `map(row)` and upserts (insert/update) or removes (delete) the index document. No `*.subscribe.ts`, no manual indexing calls — the tap is the single sync path, and it runs under both `voltro dev` and `voltro serve`.

For pre-existing rows, `backfillIndex(backend, spec, rows)` indexes the rows you supply (call it from a `*.startup.tsx` or a one-off script) — or use the **Reindex** button / `POST /reindex` inspect endpoint, which reads the table's current rows for you. The endpoint **streams** the table (keyset-paginated) and upserts one bounded page at a time — `sync.reindexBatchSize`, default 1000 rows — so memory stays flat no matter how large the table is. `backfillIndex` itself stays a plain array API for small explicit seeds.

## When the engine is down

The database commit has already happened by the time the tap runs, so a failed index write is **drift**: the row is in your database and missing from — or stale in — your index. That is not left to a log line.

1. **Retry.** A failure the backend marks *transient* (network, engine unavailable, 5xx, rate limited) is retried inside the tap's own Effect with capped exponential backoff. A *permanent* failure — an unsupported query shape, a `map(row)` that throws on one row's shape — is **not** retried: repeating it cannot succeed, and it lands in step 2 immediately.
2. **Record.** A change that outlives the retry is written to the framework-owned `_voltro_search_drift` table, one row per `(index, source row)`. Nothing is lost at that point: the entry lives in your own database — the thing that just committed successfully — while the search engine is what is down.
3. **Repair.** A cluster-coordinated sweep re-reads each recorded row **from the database** and re-derives its document. It never replays the failed event, and that is what makes repair order-free and idempotent: a row updated three times during an outage converges in one pass, and a row deleted since the failure converges to a removal.

The one case that is still a loss is the engine failing **and** the ledger write failing — and that one fails the tap loudly rather than reporting success.

Every number here is yours to set (defaults shown):

```ts
searchPlugin({
  indexes: { /* … */ },
  sync: {
    retries: 5,               // retries after the first attempt; transient failures only. 0 disables retry
    retryBaseDelayMs: 200,    // first backoff — doubles per attempt
    retryMaxDelayMs: 10_000,  // ceiling for any single wait
    resyncIntervalMs: 60_000, // repair-sweep interval. 0 turns the sweep off (POST /resync still repairs on demand)
    resyncBatchSize: 200,     // max ledger entries repaired per sweep
    reindexBatchSize: 1000,   // rows per page for POST /reindex (streamed keyset walk — memory stays flat)
    statsFlushIntervalMs: 5_000, // buffered sync-counter flush cadence. 0 = one durable write per event
    statsFlushMaxBuffered: 1000, // flush early once this many counts are pending
  },
})
```

Drift is visible **without reading logs**:

| Endpoint | Shows |
|---|---|
| `GET /_voltro/inspect/plugins/search/indexes` | per index: `dropped`, `pendingDrift`, `lastDriftAt`, `drifted` — plus the retry policy actually in force |
| `GET …/search/drift` | the failing rows themselves, oldest first, with their last error and attempt count |
| `POST …/search/resync` | runs a repair pass now (`{ scanned, repaired, failed }`) |

An entry whose `attempts` keeps climbing is telling you something a retry cannot fix — a `map(row)` that throws on that row, a document the engine rejects. That is the case to look at by hand; everything else drains on its own.

`_voltro_search_drift` is applied by the declarative differ on `voltro db apply` and on a `voltro dev` boot, on every dialect — there is nothing to migrate.

### Under multiple replicas

Index **writes** and index **counters** are deliberately treated differently when your store's change scope is `fleet` (every replica receives every change — postgres `LISTEN/NOTIFY` CDC, for instance):

- **The write runs on every replica.** An `upsert`/`remove` of the same document is idempotent, so a duplicate costs write amplification — while electing a single writer would cost a *lost* update whenever that replica dies mid-change, and would import the leadership-gap window with it.
- **The count runs everywhere too, but is aggregated as a maximum.** Each replica's stats row is already a fleet-wide count of the same changes, so `/indexes` takes the highest rather than the sum — the panel reports one sync per change, not one per replica. Under `local` scope only the replica that actually made the write counts (peers see an echo), and the rows are summed. Neither path needs a leader, so neither has a window in which counting stops.

## Querying

`search.query` is a typed **action** (not a streaming query — search results aren't reactive):

```tsx
import { useSearch } from '@voltro/plugin-search/web'

const { results, facets, run, pending } = useSearch('posts')
run('voltro effect')                       // → results scoped to the caller's tenant
run('voltro effect', { limit: 20, offset: 20 })   // offset paging — page 2
```

`results` are the hit documents; `facets` are the facet counts from the last run. The query is automatically filtered to the caller's `tenantId` (from the resolved Subject) against the index's `tenantField` — no cross-tenant leak.

`run(q, { limit, offset })` pages the results: `offset` skips leading hits, `limit` bounds the window. It maps to each engine's native paging (memory slice, Typesense/Meilisearch `offset`, Algolia `offset`+`length`).

## Query features

Every option is plain JSON and round-trips through the action's Schema:

```ts
import { memoryBackend } from '@voltro/plugin-search'

const backend = memoryBackend()
const res = await backend.query('posts', {
  q: 'kubernetis',                                   // matches the typo below
  limit: 20,
  offset: 20,                                        // offset paging
  fuzziness: 'auto',                                 // typo tolerance (or a number: max edits, 0 = exact)
  filters: [
    { field: 'status', op: 'neq', value: 'draft' },  // negation
    { field: 'score', op: 'gte', value: 50 },        // numeric range
    { field: 'publishedAt', op: 'lt', value: '2026-06-01T00:00:00Z' }, // date range
    { field: 'tag', op: 'in', value: ['a', 'b'] },   // set membership
  ],
  facets: ['status', 'tag'],                         // facet counts per field
  highlight: { fields: ['title'], preTag: '<mark>', postTag: '</mark>' },
  engineParams: { num_typos: 1 },                    // escape hatch → forwarded verbatim to the engine
})
// res.hits: [{ doc, highlights? }, …]   res.facets: { status: { open: 12, done: 3 }, … }
```

- **filters** — a list of clauses ANDed together, each `{ field, op, value }` with `op ∈ eq | neq | gt | gte | lt | lte | in | nin`. Range (`gt`/`lte`/…) and negation (`neq`/`nin`), not equality-only.
- **facets** — per-value counts for the named fields (over the full matched set, before paging).
- **highlight** — matched-term snippets per field, returned as `hit.highlights[field]`.
- **fuzziness** — a max edit distance (`0` = exact) or `'auto'`.
- **engineParams** — the engine's **presentation-only** params (see the next section). Ignored by the memory backend.

Native support degrades honestly: memory, Typesense, Meilisearch and Algolia all do filters/facets/highlighting; Typesense honors a numeric typo count (`num_typos` 0–2) while Meilisearch and Algolia only toggle typo tolerance on/off (`fuzziness: 0` disables it, other values keep their built-in tolerance). A backend op fails with a typed `SearchBackendError`.

## What the server validates

`search.query` is a public wire surface, and three of its inputs — the index name, the field names, and `engineParams` — become the search engine's **control plane**. The plugin validates all three server-side, before the engine sees them, and refuses with a typed error rather than answering a query it cannot scope:

| Input | Rule | On violation |
|---|---|---|
| `index` | must be one of the indexes you declared in `searchPlugin({ indexes })` | `SearchIndexNotFound` |
| `filters[].field`, `facets[]`, `highlight.fields[]` | a plain field path (`^[A-Za-z_][A-Za-z0-9_.]*$`), and — if the index declares `queryableFields` — one of those | `SearchFieldRejected` |
| `engineParams` | only the engine's presentation-only keys (paging, ordering, typo tolerance, highlight shaping) | the key is dropped and logged |

```ts
import { SearchFieldRejected, SearchIndexNotFound } from '@voltro/plugin-search'
```

Why each one, because the reasoning is what tells you whether your own declaration is tight enough:

- **An unknown index has no spec, therefore no `tenantField`, therefore no tenant clause.** Answering it would run the caller's query unscoped against whatever collection of that name exists on your engine — and a Typesense / Meilisearch / Algolia instance is usually shared with indexes this app never declared. It is refused, not answered empty.
- **A field name is spliced into the engine's filter DSL.** Typesense's `filter_by` is one flat string that supports `||`, so a crafted name can re-group the boolean tree around the tenant clause appended after it. The identifier pattern is the floor every index gets; `queryableFields` narrows it further to what your UI actually needs. Your `tenantField` does **not** belong in that list — the tenant clause is injected after this check, by the server, and is never a caller's to name.
- **`engineParams` is not a filter hatch.** It used to be merged last into the engine's params, so a caller could set `filter_by` (Typesense), `filter` (Meilisearch) or `facetFilters` (Algolia) and overwrite the tenant clause. Keys that could select a different document set — those, plus `query_by`, `restrictSearchableAttributes`, `preset`, `pinned_hits`, `enableRules`, … — are dropped. Express a filter as a `filters[]` clause instead: those are validated *and* ANDed with the tenant scope rather than replacing it.

If your app genuinely needs one more engine key, widen the allowlist **server-side**, where it is a deliberate decision instead of a caller's:

```ts
searchPlugin({
  indexes: { /* … */ },
  allowedEngineParams: ['query_by'],   // this app trusts callers with this key
})
```

Document-selecting keys stay refused even when listed there — that is the authority the tenant clause holds.

## Backends

| Backend | Notes |
|---|---|
| `memoryBackend` (default) | Fully in-process; dev + tests. **Refuses to boot under `NODE_ENV=production`** — see below. |
| `typesenseBackend` | Optional dep `typesense`. Lazy-loaded. |
| `meilisearchBackend` | Optional dep `meilisearch`. Lazy-loaded. |
| `algoliaBackend` | Optional dep `algoliasearch`. Lazy-loaded. |

A backend is the `SearchBackend` interface (`upsert` / `remove` / `query`) — bring your own (OpenSearch, Elastic, …).

### The memory backend refuses to boot in production

This is a **refusal, not a warning, and not a scale caveat**. Under
`NODE_ENV=production` on the in-memory backend, `onActivate` throws
`SearchBackendNotDurable` and the process does not start:

```text
plugin-search refuses to boot in production on the in-memory backend.

The memory index lives in THIS process's heap. Two consequences, both silent:
  • every replica holds a different index, so a result depends on which replica served you;
  • the index starts EMPTY after every restart/deploy, and nothing re-seeds it automatically.

Configure a durable engine in app.config.ts:
  searchPlugin({ backend: { engine: 'typesense',    url: …, apiKey: … }, indexes })
  searchPlugin({ backend: { engine: 'meilisearch',  url: …, apiKey: … }, indexes })
  searchPlugin({ backend: { engine: 'algolia',      appId: …, apiKey: … }, indexes })
or pass your own `SearchBackend` implementation.
```

**Running one replica does not make it correct**, which is why the refusal is
not conditional on detecting a cluster. Two independent things are wrong with an
in-process index in a deployment and only one of them is the multi-replica
story:

1. **Per-process.** N replicas hold N divergent indexes. Which results you get
   depends on which replica served the request — including *zero hits* for a
   document that demonstrably exists.
2. **Non-durable.** The index lives in the heap, so every restart and every
   deploy starts EMPTY and nothing re-seeds it: `backfillIndex` is a function
   your app calls, not something the plugin does at boot.

(2) is what a single replica does not fix. It only removes one of the two ways
the backend is wrong.

Dev is **silent** — the memory backend is exactly right there, and a warning
that fires on every `voltro dev` boot is a warning nobody reads.

#### The escape hatch — `singleProcessMemoryIndex`

If this deployment genuinely is ONE process that re-seeds its index at startup,
say so and the boot proceeds:

```ts
searchPlugin({
  singleProcessMemoryIndex: true,   // exactly one process, and it calls backfillIndex at startup
  indexes: { /* … */ },
})
```

Two things to be clear about before you reach for it:

- **It is a claim about your topology, not a mute switch.** The plugin holds you
  to both halves: exactly one process serves search, and a `*.startup.tsx`
  calls `backfillIndex` for every index — because the index *is* empty after
  each restart until something fills it. The boot logs a `note` at info
  restating what you signed up for.
- **The plugin checks the claim against reality.** When the instance-membership
  registry reports that a peer replica joined, the plugin logs that the
  declaration is now false — in *any* environment, because a peer announcing
  itself is an observation rather than a guess about `NODE_ENV`:

  ```text
  search: replica "<id>" joined, but this app declared `singleProcessMemoryIndex: true`.
  That declaration is now false: each replica has its own in-memory index, so search
  results depend on which one serves the request. Configure a durable backend
  (typesense / meilisearch / algolia) or run one process.
  ```

  Without the declaration, a peer joining while search is served from a heap
  index warns too — same observation, different wording.

## Dashboard panel

Both dashboards ship a **Search** panel (api apps): the configured indexes with per-index sync stats (docs synced/removed/dropped, last reindex) + the resolved backend, per-index **drift badges** (`pendingDrift` / `drifted` / last drift time), the **repair queue itself** (the drift-ledger rows, oldest first, with attempt counts and the engine's last error), and two actions — **Reindex** per index (streams the table's current rows back in) and **Resync now** (repairs the drifted rows on demand). Reindex gates on the `canReindexSearch` capability, resync on `canResyncSearch`. Backed by `/_voltro/inspect/plugins/search/{indexes,drift,reindex,resync}`.

The sync stats behind this panel are **durable and aggregated across replicas** — and counted **in memory first**: each replica buffers its per-event counts and flushes them to the stats table once per window (`sync.statsFlushIntervalMs`, default 5 s; early once `statsFlushMaxBuffered` counts are pending), so an indexed-table write never pays a per-write read+CAS against your primary. A graceful shutdown flushes the tail; a hard crash loses at most the current window of *counters* (never a change — the drift ledger, not these counters, is the durable record of what did not reach the engine). `GET /indexes` drains the buffer before reading, so the panel is always current. `statsFlushIntervalMs: 0` restores one durable write per event. They live in a framework-owned `_voltro_search_stats` table (contributed via `extendSchema.tables`; the plugin declares `store:write`), one row per `(index, replica)`, each bumped with an atomic compare-and-set. `/indexes` aggregates every replica's row — summed under `local` change scope, maxed under `fleet` (see [Under multiple replicas](#under-multiple-replicas)) — and takes the most-recent reindex, so the counts are truthful under multiple instances and survive a restart. Zero-infra dev/tests use an in-process stats store; `bindDataStore` swaps in the durable one at boot, along with the drift ledger.

## Permissions

`store:changes:read` (the ChangeEvent tap) + `store:write` (the durable `_voltro_search_stats` counters and the `_voltro_search_drift` ledger) + `inspect:read` / `inspect:write` (dashboard panel, incl. reindex + resync) + `network:outbound:<host>` (for a remote backend).