# Feature request: `@x12i/xronox-store` — cache, retrieval, and tombstone visibility (generic data tier)

**Audience:** maintainers of `@x12i/xronox-store` and `@x12i/xronox`  
**Origin:** `@x12i/activix` today layers some of this behavior in application code; this document asks the **store** to own the same guarantees using **only generic primitives**—the same patterns any document store client would implement.

---

## 0. Design principle: stay generic; these are native data-tier concerns

`xronox-store` should remain **domain-agnostic**: no activity logs, sessions, identities, or product-specific method names in core APIs. Everything below is **standard data-layer capability**:

| Concern | Why it belongs in a data tier |
|--------|-------------------------------|
| Per-key **write-through L1 cache** with LRU / TTL | Classic read-your-writes and latency control in-process. |
| **Stable contract** for PK reads vs **scan/query** reads | Every ODM/store documents whether secondary reads see the same view as primary-key reads. |
| Optional **tombstone / soft-delete field** (caller-chosen name) | Ubiquitous pattern (`deletedAt`, `purgedAt`, `archived`, `isDeleted`); not tied to one product. |
| **Bulk updates** that stay coherent with the same cache | Normal expectation when `updateMany` touches rows that `getByKey` may serve. |
| Optional **query ∪ cache** merge | Generic fix for “scan path lags PK path” in a single process—not an Activix idea. |

**Activix** (or any other package) only **configures** field names and filter shapes; it does not require the store to know what those fields *mean*.

---

## 1. Problem statement

Application layers (including Activix) need:

1. **Predictable cache-first behavior** for **primary-key** reads and writes in-process.
2. **Consistent retrieval semantics** so callers do not need a second database driver or parallel code paths.
3. **Optional document visibility rules** (e.g. soft-purge / tombstone fields) applied uniformly on **both** cache and query paths, without every consumer re-implementing `$and` / `$or` filter wrapping.

Today `XronoxCollection` already provides a per-key `Map` cache and write-through behavior for `insert` / `update` / `patchByKey` / `getByKey`, but:

- **`readMany` does not participate** in that cache, so **query results can lag** what was just written in the same process.
- There is **no first-class delete** API (Activix avoided importing `mongodb` directly by using soft-purge + `updateMany`).
- **Visibility** (hide “purged” documents) is enforced in Activix by rewriting filters and post-filtering `getByKey` results—logic that belongs in one place if multiple products need the same pattern.

---

## 2. Reference semantics (validated by Activix today)

The following is **normative in generic terms**: `xronox-store` should expose these behaviors via **optional collection config** and **documented method contracts**. Activix is cited only as a **concrete consumer** that already relies on equivalent behavior (field names like `purgedAt` are **examples**, not store vocabulary).

### 2.1 Collection cache defaults

When the parent `XronoxStore` is constructed, a caller may merge defaults (Activix currently merges):

- `cache.maxSize` default **10000**
- `cache.ttlMs` default **0** (meaning: **no age-based eviction**; entries remain valid until explicitly invalidated or LRU evicted under `maxSize`)

**Requirement:** `xronox-store` MUST document this default merge behavior explicitly (or provide `resolveCacheConfig(def)` helper) so all collections get a defined cache policy even if the caller omits `cache`.

### 2.2 Primary-key write path (write-through)

For `insert`, `update`, `patchByKey`:

1. Compute the **final document** that will be persisted (including PK, `createdAt` / `updatedAt` as today).
2. **Update the in-memory cache** for that PK **before** calling the async persistence path (`ensureInsert` / `ensureUpdate`), matching current `XronoxCollection` behavior.
3. On persistence failure, respect existing `errorHandling` (throw / queue / silent).

**Requirement:** Preserve this ordering; it is relied upon for “read your writes” in the same Node process.

### 2.3 Primary-key read path (`getByKey`)

1. If cache entry exists and TTL allows, return **cloned** document from cache.
2. Otherwise load from DB via xronox read path, then populate cache.

**Requirement:** Support optional **post-read hooks** (see §3.3 “visibility”) so a document can be treated as **missing** even if present in storage/cache.

### 2.4 Query path (`readMany`)

Today: **always DB**, no cache merge.

**Requirement (generic):** Provide one of:

- **Option A (documented limitation):** Keep DB-only `readMany`, but expose a **clear, stable** API contract in docs: “queries never see unflushed cache-only rows.”
- **Option B (preferred for product parity):** `readMany` gains optional mode:
  - `readMany(filter, { ..., visibility?, mergeCache?: boolean })`
  - When `mergeCache: true`, results **union** matching in-memory entries **with** DB results, **deduped by PK**, with defined precedence (cache wins on PK conflict for fields? or merge strategy documented).

Option B is larger work; if deferred, Option A MUST remain explicit.

### 2.5 Tombstone / soft-delete visibility (caller-defined field)

A **tombstone** is a document that remains in storage but must be **hidden** from normal reads. The **field name and meaning** are chosen by the application (`purgedAt`, `deletedAt`, `archivedAt`, `voidedAt`, …). The store only implements the **predicate**, not the business reason.

**Rules (generic; Activix maps its soft-purge to this today):**

1. **Definition:** A document is **visible** iff `purgeField` is **missing** or **`null`**. Any other value ⇒ **hidden** for product reads.
2. **`getByKey`:** if resolved document is hidden ⇒ return **`null`** (as if missing).
3. **`readMany`:** all filters MUST implicitly AND:
   - `(purgeField missing OR purgeField == null)`
4. **`updateMany` / bulk operations** that should only affect **visible** rows MUST use the same “not tombstoned” predicate unless explicitly opted out (e.g. admin / repair).

**Requirement:** Implement as optional **`CollectionDefinition.visibility`** (or equivalent name such as `tombstoneField` / `softDelete`—naming is for store maintainers; behavior stays generic):

```ts
visibility?: {
  field: string;          // caller-defined, e.g. "deletedAt", "purgedAt"
  hiddenIf?: 'truthy' | 'nonNull'; // typical: nonNull for timestamp tombstones
};
```

Default: `undefined` (no visibility filtering).

### 2.6 Nested-document filters (stay in the app unless a generic helper is justified)

**Domain-specific** helpers (e.g. “find by session + identity + status”) belong in **product packages**. The store only needs:

- Arbitrary **Mongo-shaped** `readMany` filters, and
- If tombstone support exists, **automatic composition** of the “visible only” predicate with caller filters (so apps do not hand-roll `$and` / `$exists` for every query).

Optional **generic** helper (only if it reduces foot-guns across teams), e.g. matching on a nested object under a configurable path—not “identity” as a special case:

- `readMany` + docs, **or**
- A small helper like `filterNestedEquals(baseField, partial)` that returns a filter object—still **no** session/activity vocabulary in `xronox-store`.

---

## 3. Proposed `xronox-store` API additions (all optional / generic)

Naming below is illustrative; the bar is **behavior** and **composability**, not coupling to any one consumer.

### 3.1 Configuration

Extend `CollectionDefinition`:

```ts
export interface CollectionDefinition {
  name: string;
  primaryKey: string;
  primaryKeyPrefix?: string;
  indexes?: IndexSpec[];
  cache?: { maxSize?: number; ttlMs?: number };

  /** NEW: optional visibility / tombstone rules */
  visibility?: {
    field: string;
    mode?: 'hiddenIfNonNull' | 'hiddenIfTruthy';
  };
}
```

**Semantics:**

- `hiddenIfNonNull` (default if mode omitted): hidden when `doc[field] != null && doc[field] !== undefined`.
- `hiddenIfTruthy`: hidden when Boolean(doc[field]) is true.

### 3.2 `XronoxCollection` methods

#### `getByKey(key)`

- Apply visibility: if hidden ⇒ return `null`.
- If cache returns hidden doc (because visibility was set after insert) ⇒ treat as miss OR purge cache entry (pick one; document choice).

#### `readMany(filter, options?)`

- Automatically wrap filter with visibility predicate unless `options.includeHidden === true` (for admin tools).

#### `updateMany(filter, set, options?)`

- Default: automatically AND visibility “not hidden” into `filter` unless `options.includeHidden === true`.

#### `purgeSoftByStartTime` (optional convenience — likely unnecessary)

Not required if `updateMany` + tombstone field suffice; any app can express “set tombstone where `startTime` < threshold” with a normal filter. If a convenience exists, it should remain **parameterized by field names**, not hard-coded to “activity” semantics:

```ts
purgeSoftByStartTime(params: {
  startField: string;
  olderThanMs: number;
  at: number; // timestamp written to visibility.field
}): Promise<number>;
```

### 3.3 Cache invalidation rules

When visibility field transitions hidden ⇒ visible (rare) or visible ⇒ hidden:

- **`update` / `patchByKey`:** cache must store post-update doc; if hidden, subsequent `getByKey` returns null.
- **`updateMany`:** for each touched PK, update cache entry to merged doc **or** evict those keys from cache.

**Requirement:** Define deterministic behavior in tests (Activix currently depends on `updateMany` loop updating cache via `update`).

---

## 4. Interaction with xronox write queue / errors

No change to queue semantics beyond:

- If an operation is queued, cache behavior must remain consistent with today (insert may return PK while queued—documented behavior).

---

## 5. Acceptance criteria (tests to add in `xronox-store`)

1. **Write-through:** After `insert`, `getByKey` returns inserted fields without requiring DB round-trip (when xronox ready + cache hit), with visibility applied.
2. **Visibility + readMany:** Insert row with `purgedAt: 1`; `readMany({})` must not return it; `readMany({}, { includeHidden: true })` must return it.
3. **Visibility + getByKey:** Row with non-null purge field returns `null` from `getByKey`.
4. **updateMany + cache:** `updateMany` that sets purge field must result in `getByKey` returning `null` immediately after.
5. **Default cache merge:** Documented effective defaults when `cache` is omitted (Activix currently merges `{ maxSize: 10000, ttlMs: 0 }`; the store may adopt the same or document its own single source of truth).

---

## 6. Non-goals (for this FR)

- Cross-process distributed cache (Redis, etc.).
- Hard `deleteMany` in xronox public API (can be a separate FR to xronox core + engine).
- Automatic TTL index management in MongoDB.

---

## 7. Consumer migration (example: Activix)

With `xronox-store` ≥ 1.2, Activix maps `purgeAtField` / `purgeVisibilityMode` → `CollectionDefinition.visibility`, uses `resolveCacheConfig` for cache defaults, and relies on the store for tombstone filtering on `getByKey` / `readMany` / `updateMany`. Activix still strips the tombstone field from caller-supplied patches/updates (`stripPurgeAtField`) so clients cannot accidentally un-purge or pre-purge via lifecycle APIs.

---

## 8. Versioning

Breaking changes in `CollectionDefinition` should be **semver-minor** if only additive (`visibility?`).  
If default `readMany` behavior changes (always wraps filters), consider **semver-major** or gate behind `visibility` being set.

---

## 9. References (this repo)

- Activix implementation: [`src/Activix.ts`](../src/Activix.ts)
- Types: [`src/types.ts`](../src/types.ts)
- Public docs: [`README.md`](../README.md)