# Release 0.3.8 — Indexed FTS key deletes via rowid mappings (schema 16)

**Version analyzed:** DS4 Context Engine `0.3.8`
**Commit:** `18f36e1`
**Coordinated packages:** `ds4-context-core` 0.3.8, `ds4-context-reference-adapter` 0.3.8, `ds4-context-engine` 0.3.8

## Summary

Bugfix release: FTS deletes were full virtual-table scans (FTS5 `UNINDEXED`
key columns), making session-index rebuilds on large sessions stall for
hours. The FTS tables are now rebuilt over rowid mapping tables, turning
every keyed delete into an indexed lookup. No behavior, configuration or
search-surface change: the fix is transparent and addresses the
fork/resume stall on large sessions.

## Root cause

`DELETE FROM entries_fts WHERE entry_key = ?` cannot use the FTS5
vocabulary because `entry_key` (and `project_snippets_fts.snippet_id`) are
declared `UNINDEXED`. Every row removal scanned the entire FTS index.
A rebuild of a session with ~11k entries performed thousands of those
scans: measured 65k `pread64`/s against a 615MB database and ~100 full
index passes per ~140s, with an estimated **hours** to completion —
surfacing as an indefinite hang of the agent session after a session
fork/resume. The behavior class pre-dates 0.3.7; the 0.3.7 work sessions
made it visible by pushing the database into the hundreds of MB.

Indexing the keys directly inside FTS5 was benchmarked and **rejected**:
it would make key text part of the `MATCH` vocabulary (false positives)
and shift the `bm25` column weights. See ADR 063.

## Changes

- Migration **16** `fts-rowid-key-mappings`: two derived mapping tables
  (`entries_fts_keys`, `project_snippets_fts_keys`) backfilled from the
  live FTS rows in one scan, inside a single transaction.
- `SessionIndexRepository`: per-row deletes go through the mapping
  (`DELETE … WHERE rowid = ?`); the rebuild stale-row cleanup joins the
  mapping with `entries` and uses the session index; the mapping is
  upserted from the FTS `last_insert_rowid` in the same transaction.
- `ProjectKnowledgeRepository`: same pattern for per-snippet deletes
  (`replaceFile`) and per-project cleanup (`clearProject`).
- The FTS schemas, the `MATCH` queries and the `bm25` weights are
  byte-identical to 0.3.7; search results do not change.
- Tests: `tests/integration/fts-rowid-keys.test.ts` covering the
  v15 → v16 upgrade (data preservation, key ↔ rowid correctness, search
  surface unchanged), rebuild/append/stale-cleanup consistency and the
  snippet `replaceFile`/`clearProject` paths; version-pinned tests moved
  to `CURRENT_SCHEMA_VERSION`.

## Compatibility and migration

- **Non-destructive by design:** the FTS indexes are derived state; the
  migration is transactional, so on any failure the previous schema
  remains intact and the extension keeps working with the old behavior.
- **No data stored in the mapping tables:** the base tables remain the
  single source of truth.
- Upgrade runs automatically on first open after the extension update
  (measured 467 ms on a 615MB database with 31k entries / 62k snippets;
  no user-visible step required).
- The mapping backfill is one scan; a very large database adds only
  seconds to the first open after update.
- Existing guarantees (fail-open, atomicity, privacy, pins, compaction,
  hard limits) are unchanged.