---
name: hipocampus-compaction
description: "Build 5-level compaction tree (daily/weekly/monthly/root) with smart thresholds and fixed/tentative lifecycle. Run at session start when triggers are met, or via external scheduler."
---

# Memory Compaction Tree

5-level hierarchical index over raw memory logs. Compaction nodes are **search indices** — originals are never deleted.

## Hierarchy

```
memory/
  ROOT.md                      <- root node (topic index, ~3K tokens, Layer 1)
  2026-03-15.md                <- raw daily log (permanent, append-only)
  daily/2026-03-15.md          <- daily compaction node
  weekly/2026-W11.md           <- weekly compaction node
  monthly/2026-03.md           <- monthly compaction node
```

**Compaction chain:** Raw → Daily → Weekly → Monthly → Root

**Tree traversal (search):** Root → Monthly → Weekly → Daily → Raw

## Fixed vs Tentative Nodes

Every compaction node has a status:
- **tentative** — period is still ongoing, regenerated when new data arrives
- **fixed** — period ended, never updated again

```yaml
# Indicated in YAML frontmatter
---
type: weekly
status: tentative
period: 2026-W11
---
```

**Key: tentative nodes are created immediately — ROOT.md is usable from day one.**

## When to Run

Called from hipocampus:core Session Start step 7, or directly by an external scheduler (e.g., OpenClaw heartbeat). Check trigger conditions below.

## Trigger Conditions

| Level | Tentative Create/Update | Fixed Transition |
|-------|------------------------|-----------------|
| Raw → Daily | On each new raw addition | Date changes |
| Daily → Weekly | On daily add/change | ISO week ended + 7 days elapsed |
| Weekly → Monthly | On weekly add/change | Month ended + 7 days elapsed |
| Monthly → Root | On monthly add/change | Never (root accumulates forever) |

## Smart Thresholds

Below threshold: copy/concat verbatim (no information loss).
Above threshold: generate LLM keyword-dense summary.

| Level | Threshold | Above | Below |
|-------|-----------|-------|-------|
| Raw → Daily | ~200 lines | LLM keyword-dense summary | Copy raw verbatim |
| Daily → Weekly | ~300 lines combined | LLM keyword-dense summary | Concat dailies |
| Weekly → Monthly | ~500 lines combined | LLM keyword-dense summary | Concat weeklies |
| Monthly → Root | Always | Recursive recompaction | (N/A) |

## Type-Aware Compaction

### Memory Types

Entries in daily logs are tagged: `## Topic [type]`. Four types exist:

| Type | Compaction behavior |
|------|-------------------|
| `user` | Always preserve core content. Never compress to Historical Summary. |
| `feedback` | Always preserve rule + why + how-to-apply structure. Never compress to Historical Summary. |
| `project` | Completed → compress to Historical Summary. Active → keep in Active Context. |
| `reference` | Preserve pointer + 1-line description. Mark `[?]` if >30 days unverified. |

**Backward compat:** Untagged entries (no `[type]` in heading) → treat as `[project]`.

### Topics Keyword Extraction

At every compaction level, extract topic keywords from content and write to frontmatter `topics` field:
- Scan headings (`## Topic [type]`) for keywords
- Scan Key Decisions for decision keywords
- Include type tag: `topics: [hipocampus [project], terse-responses [feedback]]`

### Exclusion Filtering

When generating LLM summaries, strip:
- Code blocks (triple backtick) → replace with `→ filepath:lines`
- Stack traces → 1-line error message
- Entries containing "임시", "테스트 중", "나중에 삭제", "temporary", "test run", "delete later" → remove entirely

## Algorithm

**CRITICAL — STRICT CHAIN ORDER: Steps 2→3→4→5 MUST execute in sequence. NEVER skip a level.**

Each step feeds the next. Root reads from monthly. Monthly reads from weekly. Weekly reads from daily. If you skip a level, the chain breaks and data is lost or corrupted.

```
Raw → [Step 2] → Daily → [Step 3] → Weekly → [Step 4] → Monthly → [Step 5] → Root
         ↑                    ↑                     ↑                    ↑
     reads raw           reads daily            reads weekly        reads monthly
     writes daily/       writes weekly/         writes monthly/     writes ROOT.md
```

**NEVER:**
- Modify ROOT.md based on daily or weekly data (root reads ONLY from monthly)
- Modify monthly based on daily data (monthly reads ONLY from weekly)
- Skip Step 2 or 3 because "there's nothing new" — always verify by checking files
- Touch ROOT.md directly without going through the full chain

### Step 0: Pre-Compaction Snapshot

Before starting the compaction chain, preserve current working state:

1. Read `WORKING.md`
2. If it contains active task content (not just the empty template):
   - Append a snapshot to today's daily log (`memory/YYYY-MM-DD.md`):
     ```
     ## Working State Snapshot [project]
     - context: pre-compaction automatic snapshot
     - state: [copy WORKING.md content]
     ```
   - This ensures in-progress work is captured before any context compression
3. If WORKING.md is empty or contains only the template, skip this step

### Step 1: Discover Candidates

Scan `memory/` for raw files. Group by date, ISO week, and month. Check each group against trigger conditions.

### Step 2: Daily Compaction (max 1 per cycle)

**Input:** raw files (`memory/YYYY-MM-DD.md`)
**Output:** daily nodes (`memory/daily/YYYY-MM-DD.md`)

For each date where raw exists and daily needs create/update:

1. Read raw file `memory/YYYY-MM-DD.md`
2. Count lines — compare against ~200 line threshold
3. Below threshold: copy raw verbatim to `memory/daily/YYYY-MM-DD.md`
4. Above threshold: generate keyword-dense summary
5. Write with frontmatter:

```markdown
---
type: daily
status: tentative
period: YYYY-MM-DD
source-files: [memory/YYYY-MM-DD.md]
topics: [keyword1, keyword2, keyword3]
---

## Topics
## Key Decisions
## Tasks Completed
## Lessons Learned
## Open Items
```

6. If date has changed (raw is from a past date): set `status: fixed`

**Secret scanning:** The mechanical compaction (`hipocampus compact`) automatically redacts secrets in compaction nodes using regex patterns. When generating LLM summaries for above-threshold nodes, also avoid reproducing any API keys, tokens, passwords, or credentials from the source material. If you encounter a secret in the source, write `[REDACTED]` in its place.

**CHECKPOINT:** Verify `memory/daily/` has the updated file before proceeding to Step 3.

### Step 3: Weekly Compaction (max 1 per cycle)

**Input:** daily nodes (`memory/daily/YYYY-MM-DD.md`) — NEVER raw files
**Output:** weekly nodes (`memory/weekly/YYYY-WNN.md`)

**STOP-CHECK:** Did Step 2 produce or update a daily node? If not, skip Steps 3-5 entirely — there's nothing new to propagate.

For each ISO week where dailies exist and weekly needs create/update:

1. Read all daily compaction files for that week (from `memory/daily/`, NOT from `memory/`)
2. Count combined lines — compare against ~300 line threshold
3. Below threshold: concat all dailies
4. Above threshold: generate keyword-dense weekly summary
5. Write to `memory/weekly/YYYY-WNN.md` with frontmatter
6. If ISO week ended + 7 days elapsed: set `status: fixed`

**CHECKPOINT:** Verify `memory/weekly/` has the updated file before proceeding to Step 4.

### Step 4: Monthly Compaction (max 1 per cycle)

**Input:** weekly nodes (`memory/weekly/YYYY-WNN.md`) — NEVER daily or raw files
**Output:** monthly nodes (`memory/monthly/YYYY-MM.md`)

**STOP-CHECK:** Did Step 3 produce or update a weekly node? If not, skip Steps 4-5 — there's nothing new to propagate.

For each month where weeklies exist and monthly needs create/update:

1. Read all weekly compaction files for that month (from `memory/weekly/`, NOT from `memory/daily/`)
2. Count combined lines — compare against ~500 line threshold
3. Below threshold: concat all weeklies
4. Above threshold: generate keyword-dense monthly summary
5. Write to `memory/monthly/YYYY-MM.md` with frontmatter
6. If month ended + 7 days elapsed: set `status: fixed`

**CHECKPOINT:** Verify `memory/monthly/` has the updated file before proceeding to Step 5.

### Step 5: Root Compaction

**Input:** monthly nodes (`memory/monthly/YYYY-MM.md`) — NEVER weekly, daily, or raw files
**Output:** `memory/ROOT.md`

**STOP-CHECK:** Did Step 4 produce or update a monthly node? If not, DO NOT touch ROOT.md.

When a monthly node is created or updated:

1. Read existing `memory/ROOT.md` (if exists)
2. Read the new/updated monthly node (from `memory/monthly/`, NOT from any other directory)
3. Recursive compaction: `root = recompact(existing_root + monthly_changes)`
   - **Active Context**: replace with current week's highlights — what's in progress, immediate priorities
   - **Recent Patterns**: update with newly emerged cross-cutting insights
   - **Historical Summary**: append/compress older context — merge periods, keep brief summaries
   - **Topics Index**: merge new topics, update existing entries with new sub-keywords and references
4. Write to `memory/ROOT.md`
5. If root exceeds size cap (`compaction.rootMaxTokens` in config, default 3000 tokens / ~100 lines): self-compress — compress Historical Summary first, keep Active Context and Topics Index intact

```markdown
---
type: root
status: tentative
last-updated: YYYY-MM-DD
---

## Active Context (recent ~7 days)
- topic: current state, what's happening now

## Recent Patterns
- pattern: cross-cutting insight that emerged recently

## Historical Summary
- YYYY-MM~MM: high-level summary of that period
- YYYY-MM: key events

## Topics Index
- topic-keyword [type, Nd]: sub-keywords, references → knowledge/file.md
- topic-keyword [type]: sub-keywords
```

**Age calculation:** For each topic, find the most recent source-file date that mentions it. Compute days since that date. Write as `Nd` (e.g., `2d`, `30d`).

**Type-specific root rules:**
- `user`/`feedback` topics: always in Topics Index, never in Historical Summary only
- `project` topics: active → Active Context + Topics Index; completed >90d → Historical Summary only (remove from Topics Index if root exceeds size cap)
- `reference` topics: mark `[?]` if >30 days since last mention

### Step 6: OpenClaw ROOT.md Sync

**OpenClaw only:** Sync ROOT.md content into the "Compaction Root" section of MEMORY.md:
- Read MEMORY.md, find `## Compaction Root` section
- Replace everything between `## Compaction Root` and the next `##` heading (or EOF) with the Active Context, Recent Patterns, and Topics Index sections from ROOT.md
- This keeps the auto-loaded MEMORY.md in sync with the canonical ROOT.md

### Step 7: Re-index

After writing any compaction files:

```bash
qmd update
```

If vector search is enabled (`search.vector: true` in `hipocampus.config.json`):

```bash
qmd embed
```

## Guards

- **CHAIN ORDER IS MANDATORY:** Daily→Weekly→Monthly→Root. Never skip a level. Never read from a wrong source directory.
- **Each level reads ONLY from its immediate predecessor:** Root←Monthly←Weekly←Daily←Raw
- Raw files: **never delete** (permanent leaf nodes)
- Max 1 daily + 1 weekly + 1 monthly + 1 root per compaction cycle
- No empty summaries (minimum 50 bytes)
- Skip failed file reads — never abort entire compaction
- qmd update failure: warning only, not fatal
- Root self-compresses when exceeding size cap (shrink older topics first)
- Keyword-dense format only — no prose, no narrative. Optimized for BM25 recall.
- **If you feel tempted to "just update ROOT.md quickly" — STOP. Run the full chain.**

## Agent Memory (Optional)

If `memory/agents/compaction/AGENT.md` exists, read it before starting compaction. Use learned patterns to inform decisions (e.g., typical raw log size, common threshold behavior).

After compaction completes, if you observed a new pattern worth remembering:
- Append to `memory/agents/compaction/AGENT.md`
- Keep under ~30 lines
- Example patterns: "this project averages ~80 raw lines/day — daily threshold rarely hit", "weekly nodes frequently need LLM summary (>300 lines)"

If no new patterns were observed, skip the update.

## Edge Cases

- **Empty days:** No daily compaction node is generated for days without raw logs. Weekly naturally skips those days.
- **First day:** Create the full tentative tree immediately (daily → weekly → monthly → root). ROOT.md is usable from day one.
- **Lifecycle example:**
  - Day 1: raw created → daily(tentative) → weekly(tentative) → monthly(tentative) → ROOT
  - Day 2: daily(tentative) updated, weekly(tentative) updated, monthly(tentative) updated, ROOT updated
  - Week ends + 7 days: weekly → fixed, new weekly(tentative) starts
  - Month ends + 7 days: monthly → fixed, new monthly(tentative) starts