# @skepsun/pi-loom —— Long-Term Memory for AI Agents  [![ci](https://github.com/skepsun/pi-loom/actions/workflows/ci.yml/badge.svg)](https://github.com/skepsun/pi-loom/actions/workflows/ci.yml)

A **long-term memory plugin** for AI coding agents. Remembers what happened across sessions, surfaces relevant context when needed, and discovers patterns via Dream Engine insight generation.

> **v0.5.0** — Native agent hooks · Operator CLI · 5-signal RRF retrieval

## What problem does it solve?

AI coding agents forget. Close the terminal, start a new session, and the agent has no memory of yesterday's work — what was decided, what failed, why the architecture ended up this way. @skepsun/pi-loom solves this by:

- **Storing memories** — explicitly (`loom_store`) or automatically (tool event capture)
- **Recalling relevant context** — via 5-signal RRF fusion: FTS5 + vector KNN + recency + entity graph + cross-session boost
- **Generating insights** — Dream Engine finds patterns across memories
- **Injecting context** — `[PI_LOOM]` block in the system prompt before every turn

## Where it fits

```
┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
│  context-mode    │  │    @skepsun/pi-loom       │  │     pi-esr       │
│  "don't flood"   │  │  "don't forget"   │  │  "don't drift"   │
├──────────────────┤  ├──────────────────┤  ├──────────────────┤
│ Context window   │  │ Long-term        │  │ Structured       │
│ optimization     │  │ memory           │  │ state graph      │
├──────────────────┤  ├──────────────────┤  ├──────────────────┤
│ 16 platforms     │  │ Pi + MCP (5)     │  │                  │
│ HN #1            │  │ 88.4% judge      │  │                  │
│                  │  │ 49% tokens of FC │  │                  │
└──────────────────┘  └──────────────────┘  └──────────────────┘
```

**Layers of memory:**

| Layer | Plugin | What it remembers | Time horizon |
|-------|--------|-------------------|-------------|
| **Session** | context-mode | Tool calls, file edits, errors, decisions — this conversation | Current session |
| **Long-term** | @skepsun/pi-loom | Facts, patterns, architecture decisions — cross-session | Weeks to months |
| **Structural** | pi-esr | Tasks, artifacts, evaluations, constraints — the state graph | Project lifetime |

---

## Private Coding Benchmark

Evaluated on 5 real coding scenarios with a strict LLM-judge (LoCoMo-Refined prompt). Each scenario simulates multi-session development work with ground-truth facts and questions.

### Aggregate Results (Phase 9, v4-pro judge)

| Baseline | Precision@10 | Recall@10 | Judge Score | Avg Tokens |
|----------|:---:|:---:|:---:|:---:|
| **@skepsun/pi-loom** | 49.3% | 100% | **88.4%** | 4,712 |
| full-context | 36.5% | 98.3% | 73.3% | 9,713 |
| grep | 6.8% | 42.5% | 22.7% | 3,522 |

**@skepsun/pi-loom beats full-context by +15.1pp on judge score while using 49% of the tokens.**

### Per-Scenario Results (Phase 10, v4-flash)

| Scenario | Type | FC Judge | @skepsun/pi-loom Judge | Δ | @skepsun/pi-loom Tok |
|---|---|---|---|---|---|
| dedup-tags | 2 sessions, 7 Q | 71.4% | 86.2% | +14.8pp | 4,075 |
| bug-fix | 2 sessions, 4 Q | 80.0% | 90.0% | +10.0pp | 1,884 |
| dream-evo | 3 sessions, 4 Q | 87.5% | 95.0% | +7.5pp | 2,729 |
| perf-tuning | 6 sessions, 12 Q | 77.0% | 63.9% | -13.1pp | 13,062 |
| recmem | 3 sessions, 2 Q | 63.3% | 100.0% | +36.7pp | 1,925 |

### Ablation: 4-Run Evolution on perf-tuning

| Run | @skepsun/pi-loom | FC | Δ | Key change |
|---|---|---|---|---|
| R1 base | 85.6% | 79.2% | +6.4 | Compact ingest + prompt compression |
| R2 +graph | 86.1% | 75.8% | +10.3 | Tool output inline, graph de-tuned (0.12) |
| R3 +fact-entity | 86.5% | 79.5% | +7.0 | Required_fact entity_ids in entity recall |
| R4 +edge-prompt | 88.4% | 73.3% | +15.1 | Entity edges serialized in answer prompt |
| **R5 +graph-boost** | **63.9%** | 75.5% | -11.6 | **Hop-aware graph boost, dynamic token budget** |

> R5's -11.6pp gap is a model effect — v4-flash underperforms v4-pro on multi-hop answer synthesis. With v4-pro, perf-tuning hit 70.3% (-4.1pp) in R4. R5's precision jumped from 26.7% to 45.0% (+68%) — the retrieval pipeline is working, but v4-flash answer quality lags.

## Implementation Phases

| Phase | Description | Status |
|---|---|---|
| **0** | Private Benchmark: 5 scenarios, 3 baselines, 95% CI | ✅ |
| **1.1** | Three-way Hybrid Retrieval: FTS5 + sqlite-vec + recency RRF | ✅ |
| **1.2** | Temporal KG + ESR: episodes, entity_edges, BFS graph traversal | ✅ |
| **2.1** | Raw Event Log + Session Summary (two-layer, 30d TTL) | ✅ |
| **2.2** | Cross-Session Recurrence Boost (Signal 5) | ✅ |
| **3** | Symbolic Injection + Progressive Disclosure (~520 tok index) | ✅ |
| **4** | RecMem: subconscious layer + consolidation engine | ✅ |
| **5** | Dream→ESR reverse edge generation | ✅ |
| **6** | Benchmark CI: 95% confidence interval reporting | ✅ |
| **7** | Token Efficiency + Judge Gap: compact ingest, fact-indexed | ✅ |
| **8** | Entity Graph Boost: hop-aware BFS for multi-hop retrieval | ✅ |
| **9** | Per-Question Dynamic Token Budget: multi-hop gets 1000 chars | ✅ |

---

## Install

### Prerequisites

Node.js ≥ 18. On Windows: [VS Build Tools](https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2022). macOS: `xcode-select --install`. Linux: `build-essential`.

### A. Pi (recommended)

```bash
git clone <repo> && cd @skepsun/pi-loom
npm install && npx tsc
pi install .                      # register as local extension
# /reload in pi to activate
```

### B. Codex / Claude Code / Cursor (standalone MCP)

```bash
git clone <repo> && cd @skepsun/pi-loom
npm install && npx tsc

codex mcp add @skepsun/pi-loom -- node "$(pwd)/dist/mcp-server.js"
claude mcp add @skepsun/pi-loom -- node "$(pwd)/dist/mcp-server.js"
```

Cursor: add to `.cursor/mcp.json`:
```json
{ "mcpServers": { "@skepsun/pi-loom": { "command": "node", "args": ["dist/mcp-server.js"] } } }
```

### C. Native Agent Hooks

Codex and Claude Code can also receive Loom context without asking the model to call MCP first. The bundled hook bridges are intentionally small: `SessionStart` injects the compact `[PI_LOOM]` block, `PostToolUse` stores deterministic low-cost observations, and `Stop` records the final assistant message as a raw event.

```bash
npm install -g @skepsun/pi-loom
codex plugin marketplace add "$(npm root -g)/@skepsun/pi-loom"
# Then install pi-loom from /plugins and restart Codex.
```

Claude Code uses the bundled `.claude-plugin` manifest and `plugin/hooks/hooks.json`. It follows the same direct SQLite path as Codex; no worker daemon is started by default.

For local development, build first and use the bundled plugin directory:

```bash
npm install && npm run build
codex plugin marketplace add "$(pwd)"
```

The hook bridge is non-blocking: if Loom cannot open its local DB, the agent continues without memory injection. Set `PI_LOOM_CODEX_HOOK_DISABLE=true`, `PI_LOOM_CLAUDE_HOOK_DISABLE=true`, or `PI_LOOM_HOOK_DISABLE=true` to disable native hooks while keeping MCP available.

### D. Library (npm)

```bash
npm install @skepsun/pi-loom
```

```ts
import { LoomStore, openDb, buildLoomContext } from "@skepsun/pi-loom";
const db = openDb();
const store = new LoomStore(db);
store.store({ content: "hello world", importance: 0.8 });
```

### E. Operator CLI

`pi-loom` remains the MCP stdio server for backward compatibility. Use `pi-loom-cli` for local inspection and lightweight operations:

```bash
pi-loom-cli doctor
pi-loom-cli status --json
pi-loom-cli context --max-total-chars 1200
pi-loom-cli search "release procedure" --limit 5
pi-loom-cli store "Run retrieval precision before publishing" --importance 0.8 --tags procedure
pi-loom-cli install codex
```

The CLI does not start a worker daemon and does not mutate `~/.claude` / `~/.codex` settings automatically. Install commands print the minimal steps for the selected agent.

### F. Minimal HTTP/HTTPS Service

Run pi-loom as a small tenant-isolated service when the extension process should not own the memory database directly.

```bash
npm run build
PI_LOOM_DATA_DIR=.pi-loom-service \
PI_LOOM_SERVICE_TOKEN=dev-token \
npm run service
```

For multiple tenants, map bearer tokens to tenant IDs:

```bash
PI_LOOM_DATA_DIR=.pi-loom-service \
PI_LOOM_SERVICE_PORT=8787 \
PI_LOOM_SERVICE_TOKENS='{"alice-token":"alice","bob-token":"bob"}' \
npm run service
```

To start HTTPS directly, provide a key and certificate:

```bash
PI_LOOM_SERVICE_TOKEN=dev-token \
PI_LOOM_SERVICE_TLS_KEY=/path/to/tls.key \
PI_LOOM_SERVICE_TLS_CERT=/path/to/tls.crt \
npm run service
```

Each tenant gets a separate SQLite store under `<PI_LOOM_DATA_DIR>/<tenant>/loom.db`. The service intentionally exposes only the core memory path:

```bash
curl -H "Authorization: Bearer dev-token" \
  -H "Content-Type: application/json" \
  -d '{"content":"Use SQLite as source of truth","importance":0.8}' \
  http://localhost:8787/v1/tools/loom_store

curl -H "Authorization: Bearer dev-token" \
  -H "Content-Type: application/json" \
  -d '{"query":"SQLite","limit":5}' \
  http://localhost:8787/v1/tools/loom_recall
```

Available service tools: `loom_store`, `loom_recall`, `loom_search`, `loom_context`, `loom_review`, `loom_apply`, `loom_status`, `loom_insights`, `loom_profile`, and `loom_evidence`.

### Environment Variables

| Variable | Purpose |
|----------|---------|
| `PI_LOOM_DIR` | Data directory (default: `.pi-loom/`). Set to a shared path for cross-project memory. |
| `PI_LOOM_DATA_DIR` | Service data root for per-tenant stores (default: `.pi-loom-service/`). |
| `PI_LOOM_SERVICE_HOST` / `PI_LOOM_SERVICE_PORT` | HTTP/HTTPS service bind address (defaults: `0.0.0.0:8787`). |
| `PI_LOOM_SERVICE_TOKEN` | Single-tenant bearer token, mapped to tenant `default`. |
| `PI_LOOM_SERVICE_TOKENS` | JSON object mapping bearer tokens to tenant IDs, e.g. `{"token":"alice"}`. |
| `PI_LOOM_SERVICE_ALLOW_ANON` | Set `true` only for trusted local development; uses `X-Loom-Tenant` or `default`. |
| `PI_LOOM_SERVICE_TLS_KEY` / `PI_LOOM_SERVICE_TLS_CERT` | Enable direct HTTPS service with the given private key and certificate paths. |
| `PI_LOOM_SERVICE_TLS_CA` | Optional CA bundle path for HTTPS. |
| `PI_LOOM_CODEX_HOOK_DISABLE` | Set `true` to disable native Codex hooks while keeping MCP/service modes available. |
| `PI_LOOM_CLAUDE_HOOK_DISABLE` | Set `true` to disable native Claude Code hooks while keeping MCP/service modes available. |
| `PI_LOOM_HOOK_DISABLE` | Set `true` to disable all native agent hooks while keeping MCP/service modes available. |
| `PI_LOOM_CODEX_MAX_TOKENS` / `PI_LOOM_CODEX_MAX_TOTAL_CHARS` | Budget controls for Codex hook context injection. |
| `PI_LOOM_CLAUDE_MAX_TOKENS` / `PI_LOOM_CLAUDE_MAX_TOTAL_CHARS` | Budget controls for Claude Code hook context injection. |
| `PI_LOOM_HOOK_MAX_TOKENS` / `PI_LOOM_HOOK_MAX_TOTAL_CHARS` | Shared budget fallback for native hook context injection. |
| `PI_LOOM_CODEX_SCOPE_TYPE` / `PI_LOOM_CODEX_SCOPE_ID` | Optional scope filter for Codex hook context injection. |
| `PI_LOOM_CLAUDE_SCOPE_TYPE` / `PI_LOOM_CLAUDE_SCOPE_ID` | Optional scope filter for Claude Code hook context injection. |
| `PI_LOOM_HOOK_SCOPE_TYPE` / `PI_LOOM_HOOK_SCOPE_ID` | Shared scope fallback for native hook context injection. |
| `EMBED_API_KEY` | API key for semantic embeddings (OpenAI-compatible). Falls back to local n-gram TF if unset. |
| `EMBED_API_BASE` | Embedding API base URL |
| `EMBED_MODEL` | Embedding model (default: `text-embedding-3-small`) |
| `EMBED_LOCAL_MODEL` | HF model for local embedding (requires `@xenova/transformers`) |
| `PI_DREAM_MODEL` | Model for Dream Engine, e.g. `deepseek/deepseek-v3.1` |
| `PI_FACT_MODEL` | Model for fact extraction |
| `RAW_EVENT_TTL_DAYS` | Raw event retention (default: 30) |
| `BENCHMARK_RECMEM` | Enable RecMem consolidation mode in eval |

### Security Boundary

pi-loom's `scope` and `visibility` fields are **agent-governance controls**, not a hard multi-user security boundary. By default, recall, context, Dream sampling, views, and apply paths exclude `visibility="private"` unless explicitly requested, which prevents accidental leakage inside a trusted local workspace.

For real multi-user isolation, run separate stores per user/project, for example by setting a per-user `PI_LOOM_DIR`, or place pi-loom behind a trusted service that enforces identity, authorization, filesystem isolation, audit logs, and encryption. A local pi extension shares the caller's process and filesystem permissions, so any user or tool with direct access to `.pi-loom/loom.db` can bypass in-process filters.

---

## Tools

| Tool | Description |
|------|-------------|
| `loom_store` | Store a memory with importance, expiration, entity anchoring |
| `loom_recall` | Recall memories by entity, text search, or list all active |
| `loom_search` | 5-signal hybrid search: keyword / semantic / hybrid modes |
| `loom_detail` | Full content of a memory by ID |
| `loom_timeline` | Chronological timeline for an entity |
| `loom_episode` | Full session summary (decisions, errors, changes, unfinished) |
| `loom_dream` | Run Dream Engine for insight generation |
| `loom_insights` | View generated insights, optionally filtered by entity |
| `loom_manage_insight` | Update or delete an insight |
| `loom_extract` | Extract atomic facts from memories |
| `loom_consolidate` | Consolidate recurring subconscious memories |
| `loom_link` | Create typed relation between two entities |
| `loom_related` | List all entities related to an entity |
| `loom_graph` | BFS graph traversal from an entity |
| `loom_audit` | View raw tool event logs |
| `loom_summarize_session` | Generate LLM summary of a session |
| `loom_stats` | Memory statistics |
| `loom_status` | Lightweight session status check |
| `loom_constrain` | Create a path-conditioned constraint |
| `loom_check_path` | Check runtime guardrail violations |
| `loom_offload` | Offload large text to external file, return [REF:node_id] |
| `loom_offload_recall` | Retrieve offloaded content by node_id |
| `loom_mermaid` | Generate Mermaid task graph from offloaded session refs |
| `loom_context` | Build context injection block [PI_LOOM] |
| `loom_profile` | Generate or view entity profiles |
| `loom_setup` | Setup assistant: verifies DB + embedding config |
| `loom_evidence` | Query-time evidence distillation with source citations |
| `loom_views` | Export read-only Markdown views from MemoryNode rows |

Recommended default path: `loom_setup` → `loom_recall`/`loom_context` → `loom_review` → `loom_apply`. Use the other tools only when the session needs specialized extraction, consolidation, profiles, offloading, or graph traversal.

---

## Dream Engine

1. **Weighted sampling** — importance (70%) × recency (30%) with exponential decay
2. **Conflict detection** — contradictory memory pairs on same entity
3. **Shuffle** — randomize to eliminate positional bias
4. **LLM generation** — concise insights with supporting memory references
5. **Auto edge suggestion** — co-mentioned entities get RELATES_TO edges

---

## Design Principles

- **Long-term memory layer** — between context-mode (session) and ESR (state graph)
- **Unified memory graph** — facts, decisions, procedures, handoffs, profiles, and insights are all MemoryNodes; relationships are MemoryEdges
- **Independent but collaborative** — entity_id is opaque string, no cross-plugin schema knowledge
- **Restrained by design** — expiration as core mechanism against unbounded growth
- **Dual intake** — explicit `loom_store` + implicit auto-capture, two layers no duplication
- **Format-stable context** — deterministic `[PI_LOOM]` block for LLM prefix-cache efficiency
- **No echo loops** — insights are output-only; Dream Engine only samples raw memories
- **Hop-aware retrieval** — entity graph BFS ensures multi-hop facts aren't buried by keyword ranking

### Memory Graph Runtime

pi-loom keeps the architecture small: one `memories` table for MemoryNodes, one generic `memory_edges` table for relations, and view-based recall. Specialized behavior is metadata, not a separate subsystem:

- `kind`: `memory`, `fact`, `decision`, `procedure`, `profile`, `insight`, `constraint`, `handoff`
- `scope`: `user`, `repo`, `task`, `session`, `entity`
- `visibility`: `private`, `project`, `shared`
- `relation`: `derives_from`, `supports`, `contradicts`, `supersedes`, `applies_to`, `mentions`

`visibility` is intentionally simple: default reads see project/shared memory, while explicit `visibility="private"` reads only private memory. It is a reliable local policy for trusted agents, not authentication or tenant isolation.

`loom_review` uses the same graph model to produce maintenance proposals such as merge, supersede, contradiction-linking, and promote-to-procedure. It is proposal-only by design; memory is changed only by explicit `loom_apply` follow-up actions.

`loom_recall(view=...)` provides specialized views without specialized storage: `procedures`, `handoffs`, `profiles`, `insights`, and `decisions` all read from the same MemoryNode table.

`loom_context(scope_type, scope_id)` is a deterministic context planner. Within the token budget it prioritizes handoffs, procedures, profiles, insights, decisions/errors, then recent high-signal facts for the requested scope.

`loom_views(scope_type, scope_id)` exports deterministic Markdown projections under `.pi-loom/views/` (`index.md`, `procedures.md`, `handoffs.md`, `decisions.md`, `profiles.md`, `insights.md`, `edges.md`). These files are for inspection and handoff only; SQLite remains the source of truth.

See [Memory Graph Runtime Roadmap](./docs/memory-graph-runtime.md) for the current improvement boundary and next steps.

---

## Collaboration with pi-esr

```
Agent calls esr_promote_task("loom-arch-eval", "stable")
  → ESR: updates entity state, validates closure
  → @skepsun/pi-loom (auto-capture): stores "ESR task → stable"
  → @skepsun/pi-loom: auto-link entities from co-occurrence

Agent calls loom_search("multi-hop entity chain")
  → @skepsun/pi-loom: FTS5 + vec + recency + graph BFS → top-10
  → Entity edges serialized in answer context

Agent calls esr_get_context()
  → ESR: returns entity graph
  → @skepsun/pi-loom: [PI_LOOM] block shows insights anchored to active entities
```

---

## License

MIT