# Changelog All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com), and this project adheres to [Semantic Versioning](https://semver.org). ## [Unreleased] ## [0.2.0] - 2026-05-30 ### Added - **Multi-writer mode (`TermLogOptions.agentId`)** — multiple processes can now open the same termlog directory concurrently without contending on `.lock`. When `agentId` is provided, the manager uses per-agent file naming: `manifest-.json`, `.lock-`, `seg--NNNNNN.seg`, `docids-.snap/.log`. Each agent owns its own writer state; the reader view at open() is the union of own + every other agent's committed manifest in the same directory. Tombstones written by one agent target docs in any agent's segments (cross-agent updates / removes work correctly via the existing per-segment tombstone mechanism). Compaction stays per-agent. - **Slot registry (`slots.json`)** — per-agent uint32 numId partitioning via a `slots.json` file at the index root. Each agent claims a unique 12-bit slot on first open (4096 slots × 1,048,576 docs per slot). Opening two agents in parallel serializes through `slots.json.lock`. Re-opening with an existing `agentId` returns the same slot (idempotent). Exported helpers: `acquireSlot`, `readSlotsRegistry`, `slotRange`, `SlotRegistryError`, `SLOT_BITS`, `LOCAL_BITS`, `MAX_SLOTS`, `SLOT_SIZE`. `SlotExhaustedError` thrown when a slot's 1M-doc range is exhausted. - **`TermLog.refresh()`** — in multi-writer mode, re-scans the directory for newly committed external manifests and external `docids-*.snap/.log` files, picking up writes from other agents without reopening. No-op in legacy single-writer mode. - **`TermLog.migrateLegacy(dir, agentId)`** — one-time, idempotent conversion of a legacy single-writer layout (`manifest.json` + `seg-NNNNNN.seg` + `docids.snap/.log`) to multi-writer (`manifest-.json` + `seg--NNNNNN.seg` + `docids-.snap/.log`). The migrating agent is assigned slot 0. Refuses when another per-agent manifest already exists (mixed-state guard). - **`LegacyIndexError`** — thrown by `TermLog.open({ agentId })` when the directory still contains a legacy `manifest.json` and no per-agent manifests. Carries the directory path; callers must invoke `TermLog.migrateLegacy()` once before retrying. - **`SegmentManager.refresh()` and federated readers** — `SegmentManager.segments()` now returns the union of own + external readers; `indexTotalDocs` / `indexTotalLen` sum across all manifests. New getters `ownTotalDocs` / `ownTotalLen` for cases that need the writer-local figures only. `recoverOrphans` only touches own-agent segments in multi-writer mode; external agents' segments stay untouched even if the local view of their manifest is stale. - **15 new tests in `tests/multi-writer.test.ts`** — concurrent open without lock contention, federated read via `refresh()`, `docCount` as the federated total, cross-agent update / remove tombstoning, slot allocation idempotency and partitioning, slot bounds, and legacy migration (happy path, idempotency, refusal on mixed state). ### Compatibility - Pure additive at the API level: omitting `agentId` preserves byte-for-byte legacy behavior (same file names, same lock semantics, same docid layout). All 286 pre-existing tests continue to pass unchanged. - Multi-writer mode is opt-in. Existing 0.1.x callers experience no behavioral change. - Legacy → multi-writer is an explicit one-time migration via `TermLog.migrateLegacy()` so a downstream upgrade cannot silently mutate file layouts. ## [0.1.2] - 2026-05-05 ### Changed - Republish of 0.1.1 with corrected CHANGELOG date in the published tarball (the 0.1.1 tarball shipped with the entry dated 2026-05-04; identical code). 0.1.1 deprecated on npm with a pointer to 0.1.2. ## [0.1.1] - 2026-05-05 ### Added - `TermLog.estimatedBytes()` — approximate in-memory footprint for memory-budget callers. Sums postings region buffers, doc-length and tombstone `Uint32Array`s, TermDict entry objects, write-buffer term arrays, pending tombstone Set entries, and strToNum/numToStr Map entries. ## [0.1.0] - 2026-05-05 ### Added - **Posting list codec** — VByte + delta-encoded posting lists (`vbyte(count) | [vbyte(delta) vbyte(tf)]*`); lazy `postingIterator` for the query path; handles doc IDs up to 2^49-1. - **Segment format v2** — self-contained binary `.seg` file: postings region, packed `Uint32Array` doc-length sidecar (`[docId, len, ...]` interleaved, 8 bytes/doc), tombstones region, term dictionary, 64-byte footer with CRC32 over each region. `SegmentWriter.writeTerm` API enforces strictly ascending lex order (RangeError on violation); sidecar sort uses a `Uint32Array` index array (no tuple allocation). `SegmentReader.open` verifies all four CRC32s and throws `SegmentCorruptionError(region)` on mismatch or missing file. - **Term dictionary** — sorted on-disk binary format; `add` (in lex order), `lookup` (binary search, O(log n)), `serialize`/`deserialize`, `fromSortedEntries` (zero-copy for already-sorted entries). - **StorageBackend abstraction** — `StorageBackend` interface: `readBlob`, `writeBlob`, `listBlobs`, `deleteBlob`, optional `appendBlob`, `createWriteStream`. `BlobWriteStream` interface (`write`/`end`/`abort`). `WriteStreamError` typed error for missing multipart commands. - **FsBackend** — atomic `writeBlob` (unique `.tmp` nonce + fsync data + rename + fsync parent dir); `appendBlob` via `O_APPEND|O_CREAT`; `createWriteStream` with exception-safe `end()` (on fsync/close/rename failure: closes handle, unlinks `.tmp`, sets `done=true`, re-throws); idempotent `deleteBlob`; nested directory creation. - **S3 support** — available via the companion package `@backloghq/termlog-s3` (peerDeps on termlog). - **SegmentManager + manifest** — write buffer with configurable flush threshold; atomic manifest (`manifest.tmp` → rename); immutable reader snapshots (reads are lock-free); `SegmentReader.open` called before `writeManifest` to prevent in-memory desync on open failure; monotonic `generation` counter; `onBeforeManifest` hook. - **Size-tiered compaction** — fanout-based LSM merge (default fanout=4, configurable); `chooseCompactionTargets` picks the lowest tier with `>= fanout` segments and cascades until no tier is eligible; write amplification bounded at `O(N log_{fanout} N)`; original doc IDs preserved through merge (sparse uint32 natively supported); tombstones targeting docs in unmerged segments are carried forward on the merged output; streaming k-way heap merge (`MinHeap`); compaction memory bounded via `tombstonedMergedDocs` intersection (vs full `mergedDocIds` Set) and running `mergedTotalLen` accumulator. - **Query execution** — `andQuery` (zigzag merge), `orQuery` (k-way union); `MultiSegmentIter` with `MinHeap` (O(1) peek, O(K log K) seek); `buildTombstoneSet`. - **BM25 ranking** — `bm25Score` pure function; `BM25Ranker` with configurable `k1`/`b`; top-k `MinHeap` when `limit` is set (O(hits × log(limit))); score-desc tie-broken by numeric docId ascending; AND/OR mode. - **TermLog facade** — `TermLog.open/add/remove/search/flush/compact/close`; string↔number docId mapping persisted in `docids.snap` + `docids.log` (append-only journal, collapsed to snapshot on `compact()`/`close()`); write mutex via promise-chain `serialize()`; `add()` on an existing docId tombstones the old numeric ID and allocates a fresh one. - **Tokenizer abstraction** — `Tokenizer` interface; `UnicodeTokenizer` (NFD normalization, configurable `minLen`); kind + minLen persisted in manifest; `TokenizerMismatchError` on reopen with different config. - **Cross-process advisory lockfile** — `O_EXCL` open; stale-PID auto-claim; `IndexLockedError` on live-process conflict; `close()` releases. - **Crash recovery** — manifest is source of truth; orphan `.seg.tmp` and orphan `.seg` cleaned on open; stale `manifest.tmp` deleted; missing segment file throws `SegmentCorruptionError(region="footer")`; `recoverOrphans` uses prefix-scoped `listBlobs` (no full-bucket scan). - **Public errors** — `ManifestCorruptionError`, `ManifestVersionError` (with `found`/`expected` fields), `SegmentCorruptionError` (with `region`), `MappingCorruptionError`, `TokenizerMismatchError` (with `persisted`/`runtime`), `IndexLockedError`, `WriteStreamError`. - **1M-doc stress target** — tiered cascade peaks under 1024 MB RSS in normal CI (10k docs); `STRESS=1 STRESS_TIERED=1` runs full 1M-doc cascade. - **282 tests** — posting list codec (19), term dict (20), storage/FsBackend (23, including `createWriteStream` atomicity, abort, double-end/abort idempotence, write-error propagation, end-failure cleanup), Crc32Stream parity (5), segment fuzz round-trip (1000 iterations), 1M-doc sidecar, writeTerm RangeError guard, tiered compaction suite (cascade, recovery, fanout, write-amplification bound), crash recovery, concurrency (reader isolation, 100 concurrent adds, racing flush+compact, racing add+remove), manifest-before-reader-open invariant, writeManifest atomicity (flush and compact — no totals/tombstones inflation on failure, onBeforeManifest failure preserves buffer), sidecar non-ascending guard, BM25 parity vs reference implementation, TermLog facade integration. - **GitHub Actions CI** — Node 22/24/25 matrix; lint + build + test on push/PR to main. - Source maps and declaration maps (`sourceMap: true`, `declarationMap: true`). - `"sideEffects": false` in package.json for bundler tree-shaking. - `DEFAULT_FLUSH_THRESHOLD` exported constant (1000). - `VERSION` constant sourced from `package.json` via JSON import attribute.