# Memory & Recall

Pi Fabric's `memory` provider searches Pi session JSONL files. Session JSONL
forms the source of truth. The memory index holds derived, disposable state.

Structural extraction is the only indexing method. Regexes never classify
goals, preferences, errors, or other prose concepts. Roles, tool names,
timestamps, entry IDs, operation addresses, exact `ref`/`provider`/`action`
identities, execution outcomes, tool errors, and tool argument paths all come
from typed session fields.

## Retrieval workflow

Treat recall as navigation and expansion as reading:

1. `memory.recall` locates ranked, bounded evidence.
2. Inspect entry snippets directly. Every hit has one copy-ready `follow` call.
3. Run `await tools.call(hit.follow)` to expand an entry or resolve a cold session.
4. Continue any paged result with `await tools.call(result.next)` until `next` is `null`.
5. Use guest-local `memory.walk` when TypeScript should scan, filter, reduce, join,
   or traverse complete normalized records.

These calls are the supported session-data API. Consumers should not open,
scan, or parse Pi session files themselves. Follow calls carry source hashes and
selected lineages without duplicating that integrity metadata on each hit;
single-session page continuations preserve the same bindings. Coverage explains
when indexed absence is not authoritative.

## Active branches

`memory.recall`, `memory.sessions`, and `memory.expand` accept
`branches: "active" | "all"`. Every scope defaults to `"active"`, including
`project`, `global`, and explicit `session:<id-or-path>` scopes. Only records
carried on each session's active parent-linked path contribute to hot text,
cold vocabulary, structural addresses, hits, and entry counts, so an
abandoned sibling cannot match by default. `branches: "all"` asks explicitly
for every branch. Expansion results and session rows identify their branch
mode; recall follow calls preserve it.

For the current live session, every memory action calls the extension
context's live `SessionManager.getBranch()` and `getLeafId()` getters.
These live reads observe `/tree` navigation even when no new record has
been appended. For another persisted session, Fabric follows Pi 0.80.6's
persisted semantics. The last persisted non-header entry serves as the
leaf, duplicate IDs resolve to the last record in the ID map, and Fabric
walks `parentId` links from that leaf to a root. Append order never counts as one transcript. When a parent
cycle occurs, Fabric stops defensively and marks coverage incomplete with
`invalid_parent_graph`.

Each derived record carries `branches` and a SHA-256 `lineageFingerprint`
over the selected branch mode, leaf, and active path IDs. Active and all
caches use separate filenames. Branch mode, lineage fingerprint, and privacy
settings all feed cache policy, so navigation, an append, or a policy change
rebuilds the relevant derived record without contaminating the other mode.
`sourceHash` still covers the complete JSONL source, including off-lineage
records.

## Cache V6

Cache records carry `cacheVersion: 6`. Fabric removes older or malformed
records and rebuilds them from source. Rebuilding from source replaces any
migration of old records into V6. Refresh also clears orphan records,
records whose encoded cache path fails to match their source identity, and
records for deleted source sessions.

Every cache record stores the exact session file path, branch mode, lineage
fingerprint, privacy policy, and a SHA-256 `sourceHash`, along with source
mtime and size. A same-size rewrite that preserves mtime still invalidates
the record. Fabric creates cache directories with `0700` permissions and
cache files with `0600` permissions on a best-effort basis.

A hot shard holds bounded normalized entry text plus `indexCoverage`. Each
cold digest contains:

```ts
{
  cacheVersion: 6,
  kind: "digest",
  sessionId, file, cwd,
  mtime, size, sourceHash,
  branches, lineageFingerprint, policy,
  firstTs, lastTs, entryCount,
  filesTouched, toolHistogram, errorCount,
  vocabulary,   // sorted unique canonical strings, no posting lists
  addresses,    // structural identity + ref/provider/action/outcome postings
  indexCoverage,
  cacheBytes, cacheSourceRatio
}
```

Cold vocabulary maps each exact lexical term only to the session that
contains it. The digest keeps no per-term entry indices. Structural address
tuples separately keep exact entry identity, role/tool/time, and persisted
`ref`/`provider`/`action`/`outcome` fields. A cold lexical result remains a
compact session candidate whose `follow` call carries its exact source and
hash bindings. It never appears as an inferred lexical entry range. Exact
lexical entry matches come back only after following that call.

`maxColdVocabularyBytes` bounds vocabulary construction for each session.
`maxColdCacheBytes` is a hard per-session cap on the persisted cache.
Reaching either cap sets `indexCoverage.complete` to false with an explicit
reason. The cache-size cap can force structural addresses or vocabulary to
persist as exact prefixes only. Fabric always reports this state as
`max_cold_cache_bytes` and never treats it as complete. `cacheSourceRatio`
divides persisted cache bytes by source bytes.

## Capability heads and exact structural retrieval

Fabric separates capability navigation from memory evidence.
`tools.catalog()` returns a deterministic current tree:

```text
Fabric capabilities
└── provider head
    └── action head
```

Provider/action names, descriptions, and descriptor hashes describe the
currently registered catalog. Full schemas stay available through
`tools.search()` and `tools.describe()`. A caller can use this metadata to
choose an action ref. Fabric never copies it into session entries.
Historical evidence comes from session records alone. Catalog descriptor
changes can shift discovery ranking. Historical structural membership stays
intact.

`memory.recall` accepts exact structural filters:

- `ref`, for example `pi.grep`.
- `provider`, for example `pi`.
- `action`, for example `grep`.
- `outcome`: one of `succeeded | failed | aborted | timed_out`.
- the existing `role`, `tool`, `since`, and `until` filters.

With no `query` present, persisted typed fields alone decide membership. A
`query` value constrains lexical or explicit-regex search with the same exact
filters. The result does not echo request diagnostics; the call already records
them. Catalog description text never becomes a lexical match.

```ts
const heads = await tools.search({ query: "search source files" });
const history = await memory.recall({
  scope: "project",
  ref: heads[0]?.ref,
});
const failures = await memory.recall({
  scope: "project",
  query: "timeout",
  ref: "agents.run",
  outcome: "failed",
});
```

A complete cold structural posting proves that the selected session contains
a matching typed entry. The cold response remains an integrity-bound session
candidate. A combined cold lexical + structural candidate cannot prove that
both conditions occur on the same entry, because cold vocabulary has no
posting lists. Coverage reports
`cold_structural_filter_requires_hydration`; follow the candidate before
claiming entry-level co-location.

## Exact lexical queries

Callers choose `queryMode` explicitly:

- `"literal"` is the default token mode.
- `"phrase"` performs a case-insensitive contiguous text search after Unicode
  normalization.
- `"regex"` requires an explicit opt-in.

Literal mode does not inspect punctuation to guess whether input is a regular
expression. It never compiles input with `RegExp`; a path such as `src/foo.ts`
stays literal text. Quotes are punctuation only in literal mode; quoted words
still become independent canonical terms and never turn into regex syntax.

`tokenize.ts` is the single canonical tokenizer for literal queries, hot BM25
scoring, and cold vocabulary creation. It applies Unicode NFKC normalization,
extracts Unicode letters, numbers, and `_` characters, then lowercases them.
Literal terms match through exact canonical-token equality. Fabric applies no
stemming, synonym expansion, or semantic regex classification.

Multi-term literal queries default to `queryMatch: "any"` so recall does not
silently lose an answer whose wording differs from the query. BM25 relevance
is weighted by typed provenance: assistant and user prose ranks ahead of
tool-call plumbing unless structural filters explicitly request operations.
Every exact entry is ranked individually; context-segment grouping cannot give
thousands of neighbors one shared score.

The response is still hard-bounded, so broad discovery cannot flood model
context. Request `queryMatch: "all"` when every canonical term must occur in
one indexed entry:

```ts
memory.recall({
  query: '"Use CEF" "Stage 1" browser Heddlework',
  queryMatch: "all"
})
```

Use `queryMode: "phrase"` when word order and adjacency are material. The
request is not echoed in the compact result.

In a cold session with complete coverage, every unique canonical token of the
normalized source text occurs exactly once in the sorted vocabulary. An
`all` query requires every term in that vocabulary, but the vocabulary cannot
prove same-entry co-location. Phrase adjacency cannot be proved either. Both
cases return an explicit incomplete cold-coverage reason until the session is
resolved. Rare terms stay exactly discoverable as
long as the configured vocabulary and cache bounds hold. Exceeding a bound
makes an empty result explicitly non-authoritative.

Unicode scalar count sets the hot-text limit. Raw UTF-16 code units play no
part. A cut cannot split a surrogate pair, and the shard text remains valid
UTF-8. Hot shards retain no separate complete tail vocabulary, so truncating
any normalized entry sets shard `indexCoverage.complete: false` with reason
`max_entry_chars`. A token that occurs only after the cut cannot yield an
authoritative no-match. Recall returns no hit and reports that reason through
`coverage`. Expansion still re-reads the complete source record.

## Bounded regular expressions

Regex mode runs JavaScript regex inside a disposable worker thread. The host
never evaluates an untrusted pattern. Fabric terminates the worker forcibly
at the hard timeout, so catastrophic backtracking cannot continue on the
host thread. Four limits bound regex execution:

- UTF-8 pattern byte count.
- haystack item count.
- aggregate UTF-8 haystack bytes.
- wall-clock worker timeout.

A hot haystack is normalized entry text. A cold haystack is one bounded
canonical vocabulary term. Transcript prose never serves as a cold
haystack. Invalid patterns, oversized patterns, haystack truncation,
worker failures, and timeouts each return structured query coverage. A timeout, for example, returns `coverage.complete: false`, the
reason `regex_timeout`, and a structured `coverage.error`. An incomplete
regex result never counts as an authoritative no-match.

## Tiers, refresh, and work budgets

The `memory.hotSessions` most recently modified sessions stay hot. Every
older session is cold. Once a session crosses the boundary, Fabric drops
the old derived tier record after building the replacement. Explicit
hydration re-reads source without promoting a cold session.

Session count and aggregate source bytes bound cache synchronization.
Cache cleanup follows budgets on inspected cache files and aggregate
cache bytes. The cleanup byte budget is shared with
`maxSyncSourceBytes`. Reaching a work budget stops additional indexing
and sets `coverage.complete: false`. All eligible sessions remain
counted. Fabric bounds every background job, and the index needs no
database dependency.

Query mode and no-query browse mode both discover every eligible session.
`memory.maxSessions` limits session listing only. Search materialization
works under explicit deterministic per-call budgets: 50,000 filtered hot
entry candidates, 10,000 cold digest candidates, and 10,000 ranked result
candidates. Hitting one of these marks coverage incomplete with
`candidate_entry_budget`, `candidate_digest_budget`, or
`candidate_item_budget`. Public totals count the retained exact entry hits
and cold session candidates after filtering. Unknown omitted candidates stay
outside those totals.
Coverage reports:

```ts
coverage: {
  complete: boolean,
  indexedSessions: number,
  eligibleSessions: number,
  staleSessions: number,
  incompleteSessions: number,
  reasons: string[],
  error?: { code: string, message: string }
}
```

An empty `hits` array is authoritative only when cache/index coverage and
query execution coverage are both complete. Otherwise `coverage.reasons` names
causes such as source unavailability, `max_entry_chars`, duplicate identities,
vocabulary, file-metadata, or cache caps, candidate or synchronization budgets,
or regex limits.

## Scopes

| Scope | Meaning |
| --- | --- |
| `session` | The current session, or the newest session for the current cwd. |
| `project` | All sessions in the current cwd's Pi session directory. |
| `global` | Sessions under the agent directory. This scope requires an explicit request and can never be the default. |
| `session:<id-or-path>` | One source session, resolved explicitly without promotion. |

Duplicate session IDs are ambiguous. `session:<id>` and `memory.expand`
reject an ambiguous ID with `ambiguous_session` and list the candidate paths.
Normal recall navigation does not require extracting a path: copy the hit
`follow` call unchanged.
Duplicate normalized entry IDs and operation addresses also mark index
coverage incomplete, with `duplicate_entry_id` or
`duplicate_operation_address`. Stable-address expansion demands exactly one
record. Zero matches return `address_not_found`. More than one match
returns `ambiguous_address`. Fabric returns no source records in either case.

## Compact results and call tokens

`memory.recall` returns one flat, bounded hit stream:

```ts
type FabricCall = {
  ref: "memory.recall" | "memory.expand";
  args: Record<string, unknown>;
};

type RecallPage = {
  total: number;
  hits: MemoryHit[];
  next: FabricCall | null;
  coverage: MemoryCoverage;
  error?: MemoryError;
};
```

`total` is the retained pre-page count of exact entry hits plus cold session
candidates. `hits.length` is the returned count. The request already records
query semantics, so results do not echo query terms, modes, filters, offsets,
or redundant counts and prose.

A hot or explicitly resolved entry hit carries evidence plus one action:

```ts
{
  kind: "entry",
  sessionId,
  tier,
  index,
  entryId,
  parentId,
  operationAddress,
  type,
  role,
  tool,
  ref,
  provider,
  action,
  outcome,
  timestamp,
  isError,
  score,
  snippet,
  truncated,
  follow: { ref: "memory.expand", args: { /* integrity-bound selector */ } }
}
```

A cold candidate cannot claim an exact source entry. It carries only
session-level evidence and a recall action:

```ts
{
  kind: "session",
  sessionId,
  tier: "cold",
  cwd,
  lastTimestamp,
  score,
  matchedTerms,
  matchedStructuralEntries,
  follow: { ref: "memory.recall", args: { /* integrity-bound request */ } }
}
```

The integrity arguments live only inside `follow`; they are not duplicated in
separate source or action-specific pointer objects. Dispatch either hit without
branch-specific plumbing:

```ts
const detail = await tools.call(hit.follow);
```

When a statically typed result is useful, discriminate the hit and call the
known provider method:

```ts
const detail = hit.kind === "entry"
  ? await memory.expand(hit.follow.args)
  : await memory.recall(hit.follow.args);
```

Recall ranks one mixed stream and then paginates it. `pageSize` is a request
ceiling, not permission to exceed the hard 30,000-character JSON envelope.
If more retained items remain, `next` is another complete call token:

```ts
let page = await memory.recall({ query: "timeout", scope: "project" });
while (page.next !== null) {
  page = await memory.recall(page.next.args);
}
```

A single-session continuation includes exact source-hash and lineage bindings;
an append or branch change returns `stale_pointer`, preventing a silent page
shift. Each project or global call takes a fresh, nontransactional snapshot.
Every snippet is independently bounded and reports only whether it was
truncated.

An optional inclusive `entryRange` can bound explicit single-session recall.
Both endpoints must be valid normalized indices. Out-of-range or negative
addresses return `index_out_of_bounds`; Fabric never clamps them.

## Expansion and guest-local computation

`memory.expand` re-reads full, untruncated normalized records. It accepts
indices, stable entry IDs, operation addresses, or an inclusive range.
`before` and `after` add adjacent entries around exactly one selected anchor:

```ts
const exact = await memory.expand(hit.follow.args);
const around = await memory.expand({
  ...hit.follow.args,
  before: 2,
  after: 3,
});
const operation = await memory.expand({
  session,
  operationAddresses: ["entry-uuid/7"],
});
```

Expansion results use `entries`. Recall and expansion both call the capability
field `tool`. Expanded records include Pi's `parentId`, so code can reconstruct
branch relationships under `branches: "all"`; `parentEntryId` remains the
separate carrier link for extracted Fabric child records.

Expansion never silently slices away a long record. Every returned chunk has
`textRange: { start, end, total, complete }`. A partial final entry returns a
non-null `next`; follow it to reconstruct the exact text:

```ts
let chunk = await memory.expand(args);
while (chunk.next !== null && chunk.next !== undefined) {
  chunk = await memory.expand(chunk.next.args);
}
```

The default aggregate text budget is 20,000 characters and `maxChars` can
request a smaller chunk. Valid typed operation and branch-fact payloads stay
available as bounded structured fields. Oversized structured payloads set
`structuredTruncated` and stay out of the envelope.

For arbitrary filtering, projection, aggregation, joins, and parent traversal,
the query language is the surrounding TypeScript. `memory.walk` is a
guest-only combinator, not a fourth host action and not a serialized callback:

```ts
const failedFiles = new Set<string>();
const parents = new Map<string, string | null>();

const walk = await memory.walk(
  { session, branches: "all" },
  async (entry) => {
    if (entry.entryId) parents.set(entry.entryId, entry.parentId);
    if (entry.outcome === "failed") {
      for (const file of entry.filesTouched ?? []) {
        if (file) failedFiles.add(file);
      }
    }
    return failedFiles.size < 20; // exactly false stops early
  },
);

return { files: [...failedFiles], visited: walk.visited };
```

With no selector, `memory.walk` obtains an integrity-bound session range and
visits the whole normalized session. With a selector, it visits only that
selection and optional context. Start it from an original selection, not a
continuation with a nonzero `textOffset`, because every callback receives a
complete entry. The helper follows all expansion pages, validates continuity,
and reassembles each `textRange` fully before invoking the visitor,
awaits async visitors and nested tool calls, and returns `{visited, stopped}`.
A provider error is returned as `walk.error`. Functions remain inside the guest
runtime, isolated by default; no predicate source or closure crosses the host
bridge.

This is the Python replacement boundary: the host supplies bounded search and
lossless normalized records; the fabric program supplies callbacks, maps,
sets, joins, reductions, and the small final `return`. Do not add JSONPath,
SQL, projection, grouping, or host-evaluated predicate fields, and do not open
Pi session JSONL directly.

During recall follow calls and expansion, Fabric compares the expected source
hash and lineage fingerprint with the selected live or persisted session. A
rewrite, append, or active-leaf navigation that changes a binding returns
`stale_pointer` and no source content. Under active mode, an off-lineage stable
address returns `address_not_found`; `branches: "all"` is the explicit way to
read it.

A valid `FabricExecutionTraceV1` on an outer `fabric_exec` result emits one
child record per operation, placed immediately after the outer normalized
entry. Each child keeps `parentEntryId`, `operationAddress`, the exact
`tool`, `ref`, `provider`, `action`, typed `filesTouched`, `outcome`, and a
bounded structured `operation` object. Expansion re-reads and
re-normalizes source. Fabric never reconstructs operations from output
prose.

Valid `FabricBranchSummaryDetailsV1` and V2 envelopes emit typed child
records for user, phase, and operation facts. Under V2, they also emit
children for named `fabricRun` facts. Run children keep the bounded declared
name and description, the paired aggregate outcome, and the original call
address. They index as `fabric_exec` and expand by that operation address.
Other children keep the original fact address, the
ref/provider/action/tool/outcome/arguments values, and the structurally
derived paths, plus `carrierEntryId`, `carrierParentId`, and
`carrierFromId`. Operation facts expand by their original operation address.
User and phase facts use that address as their stable entry ID. Fabric
deduplicates repeated nested summaries by exact fact address in source
order, which keeps the earliest carrier deterministic. Inside each consumed
details envelope, addresses must be unique. Fabric rejects an envelope with
a duplicate address and marks coverage incomplete. Unknown or malformed
details and all branch-summary prose remain non-semantic.

## Local-cache privacy and deletion

Treat the memory index as local derived state. It sets no encryption or
semantic-privacy boundary. Depending on configuration and tier, cache JSON
can retain plaintext lexical vocabulary, cwd and file paths, source
pointers and hashes, structural tool and capability metadata, selected
user/assistant/custom-message content, tool arguments, and selected tool
output when the default `indexToolOutput: true` applies. Cold vocabulary
holds no posting lists. The exact words still appear in it as plaintext.
Thinking text stays excluded by default. Enabling it explicitly stores it
as plaintext. Fabric runs no secret scanning. Privacy here comes from
structural inclusion or exclusion, and regex classification plays no
part.

Fabric requests `0700` for cache directories and `0600` for cache files.
These permissions are best effort and inherit the host filesystem, account,
backup, and administrative trust model. The cache stays unencrypted. Fabric
reads project configuration only for a project that Pi has marked trusted.
Otherwise, only global Fabric configuration applies. A `global` memory
search is an explicit scope, never the default. Under that explicit scope,
global indexing still creates local derived records for the sessions that
the call selects.

Deleting a Pi session removes the source of truth. A later memory refresh
removes the orphaned cache records on a best-effort basis. When immediate
cache removal is required, delete the configured `memory.indexDir` as
well.
Removing that directory is safe, because every record is disposable and
rebuilt from the remaining session JSONL. Cache deletion leaves source
sessions, filesystem backups, and copies outside the index directory
untouched.

## Benchmarking capability-head retrieval

After changes to discovery, structural postings, ranking, or cache layout,
run the deterministic synthetic benchmark:

```sh
bun run benchmark:memory-heads
```

The benchmark reports catalog head selection separately from source
retrieval, hot exact operation-address recall, cold session recall, combined
lexical + structural retrieval, negative controls, the cold digest/source
ratio, and p50/p95/p99 search latency. The command fails when catalog
description text leaks into lexical matches, structural provenance is lost,
cold combined candidates omit the hydration requirement, or a nonexistent
ref returns history. Repository tests cover source JSONL, branches, staleness, and cache generation.
The synthetic timing corpus covers the measured search paths.

## Configuration

```json
{
  "memory": {
    "enabled": true,
    "indexDir": "~/.pi/agent/fabric/memory-index",
    "maxSessions": 500,
    "maxEntryChars": 2000,
    "indexThinking": false,
    "indexToolOutput": true,
    "hotSessions": 50,
    "maxColdVocabularyBytes": 524288,
    "maxColdCacheBytes": 1048576,
    "maxSyncSessions": 10000,
    "maxSyncSourceBytes": 536870912,
    "maxCacheCleanupFiles": 100000,
    "regexMaxPatternBytes": 1024,
    "regexMaxHaystackTerms": 20000,
    "regexMaxHaystackBytes": 2097152,
    "regexTimeoutMs": 250
  }
}
```

- `maxSessions`: limits session-list discovery only. Candidate and indexing
  budgets control recall.
- `maxEntryChars`: Unicode-scalar limit for persisted hot entry text. Any
  cut marks lexical coverage incomplete, and expand still re-reads full
  source.
- `indexThinking`: when true, assistant thinking blocks enter normalized
  text and lexical vocabulary. Default: false.
- `indexToolOutput`: when true, tool-result bodies, bash output, and typed
  Fabric operation results enter derived text. Default: true. Coding recall then includes tool outputs. When false, typed tool name/ref/action,
  error/outcome, and structurally extracted path metadata stay searchable.
  Output bodies do not.
- `hotSessions`: count of globally newest sessions that keep hot shards.
- `maxColdVocabularyBytes`: per-session bound on the canonical vocabulary.
- `maxColdCacheBytes`: hard per-session bound on the cold cache file.
- `maxSyncSessions` / `maxSyncSourceBytes`: work budgets for synchronous
  indexing.
- `maxCacheCleanupFiles`: count budget for synchronous cache-file cleanup.
  Cleanup bytes draw on `maxSyncSourceBytes`.
- `regexMaxPatternBytes`, `regexMaxHaystackTerms`,
  `regexMaxHaystackBytes`, `regexTimeoutMs`: bounds on isolated regex
  execution.