# Design notes ## Goal The main goal is keeping every agent session (pi, Claude Code, Codex, cursor-agent) safely backed up in one place. Search is secondary and rare, and an agent does it using a skill. ## Decisions - **Store = plain folders under rsync.** Pi session files are append-only JSONL files with UUID names, so `rsync --append-verify` without `--delete` makes a correct, incremental, never-deleting copy. - **Per-host folders instead of merging.** Each machine owns `//`, so writes can never race. A project worked on from several machines appears under several host folders, and a glob such as `*/*project*` covers them all. - **A turn index, built by the search, for conversation text only.** At 1.3 GB and 1,800 sessions `rg` over the store took about a second; at 4.2 GB and 7,200 sessions `find` took 2.5 to 4s, three `rg` passes over every byte plus a walk of the store, and on a laptop with endpoint scanning a cold run took 15s. Only 67 MB of those 4.2 GB is user and assistant text: tool output (1.5 GB), thinking (430 MB) and tool-call arguments (370 MB) are the rest. So `find` keeps that text in `.museum-turns.sqlite` at the store's root, one row per turn, in an SQLite FTS5 table with the trigram tokenizer. A trigram phrase query matches any substring of three or more characters, so on lowercased text it is the same literal match the scan does, and on the 4.2 GB store it gave the same sessions for every term tried. The conversation tier comes from the index; `find` reads the files only for the tool tier, and only when the conversation tier leaves room in the listing, since tool-only sessions rank after every conversation one. Otherwise the header says `tool output not searched`. The ranking weight changes with it: the scan counts a pi session's user and assistant lines that contain the term anywhere, tool-call arguments and thinking included, so a session that ran a tool by that name in 40 calls outranked one that discussed it in 12 turns; the index counts turns whose text says it, the `N turns` each listed session shows, so the listing now goes down that number. `museum-sync` updates it after each successful sync to a local store, under its lock so only one sync writes the index at a time, which on the store host covers what the other machines rsynced in too; an ssh client cannot, as that would cost a second connection. A search and a sync can update it at once, so each write transaction is `BEGIN IMMEDIATE` and a file's offset is read again inside it; reading it beforehand let two updates add the same new turns twice. Each search also brings the index up to date first, so the index is never stale: it stats every session, reads a grown pi file on from where it stopped (a changed hash of its first 4 KB means a replaced file, read again), and reads a changed claude, codex or cursor-agent file again whole, because its turns are merged across lines. The index is 280 MB (6.6% of the store), builds in 30 to 80s on first use, and answers in 1 to 3 ms; `find` drops to 0.7s, the walk and the summaries of the 20 sessions shown. A term under three characters, a Python without `sqlite3` or SQLite older than 3.34 (no trigram tokenizer), a store the search cannot write, or `MUSEUM_NO_INDEX=1` all fall back to the scan. The search runs on the store host, so SQLite never locks across rsync or a network filesystem; the index is a cache, and deleting it costs one rebuild. The tool tier stays a scan: a trigram index over 2.3 GB of tool text would be about 6 GB with positions, and without positions (`detail=none`) it matched 231 of 234 sessions for a word none of them held. The other exception is start dates: a claude session's name holds none, and a codex rollout's is in local time, so a session's start comes from its first record. A first record never changes, so `museum-search` keeps the ones it has read in `.museum-index.json` at the store's root and writes it when it has learnt new ones. Losing it costs one read of each file's first lines; a store the search cannot write is only slower. - **`museum-search` over hand-written recipes.** Recipes in the skill had to be quoted twice to run over ssh, and broke on the store host's login shell. ssh starts `python3 -` on the store host and the script arrives on stdin, so the remote login shell only parses a few quoted words and the store host needs no museum install. `rg` picks the matching files when the store host has it (about 0.8s over the store, against 2.8s for a pure-Python scan), and Python parses only those. It ranks by where the term appears: a user or assistant turn's text first, then anything else (tool output, thinking, tool-call arguments), and it drops sessions where only the recorded system prompt mentions it. Every session in a project records the same AGENTS.md and docs, so a word in them otherwise matches every session in the project. Within a tier, project folders come before mu workspace folders, because an orchestrator session holds the decisions its workers carry out, and then the session with the most candidate turns comes first. Date order put a newer passing mention above the session that debated the topic: for "herdr", the 46-turn session that designed mu's multiplexer layer ranked 9th of 24. `--newest` keeps date order for "what is the latest on X". Three `rg` passes do the work: mention counts, system-prompt mention counts (a session where the two are equal is dropped, with no file read), and lines that may be a turn saying the term. Python parses candidate lines only until one per session confirms, because the regex cannot tell a turn's text from its thinking or tool calls. That takes 0.3 to 2.5s on the 1.3 GB store, against about 6s to classify every mention in Python. The searching session (`$PI_SESSION_ID`, passed as `--skip`) is left out. - **TOML config, with the store split.** The first config was `KEY=value` lines, once sourced as shell, with the store as one rsync-style `host:/path` string that the scripts split on the first colon. `config.toml` names the parts: `machine`, then `[store]` with `host`, `path` and `ssh_mux_only`. The scripts read it with `tomllib` and reject unknown keys, a relative path, a `host` with a colon in it and `ssh_mux_only` without a `host`, so a typo fails in the footer instead of quietly syncing to the wrong place. `host` alone marks an ssh store, rather than a `[store.ssh]` / `[store.local]` pair: the one mistake that shape guards against, a forgotten `host` line, is caught instead by requiring a local store's folder to exist. The sync creates only `//`, and `museum init` creates the store folder once, so a stray `path` meant for the store host fails rather than filling a local copy and reporting success. The config file carries these rules as comments; `museum init` and the legacy converter both write it from one template. `tomllib` needs Python 3.11, which drops macOS's Command Line Tools Python 3.9; macOS clients already need Homebrew for rsync 3. An older `python3` gets that message in the footer, not an ImportError. The store host's side of a search is handed the folder with `--store`, so it reads no config and runs on any `python3`. Both scripts carry the same config code, since each must stay a single file, and a test fails if the two copies differ. - **Python for the scripts, not sh.** Both scripts started as POSIX sh. The search half was jq programs inside sh strings, and the sync needed `flock` on Linux and `lockf` on macOS. Stdlib Python has JSON, argparse and one `fcntl.flock` for both platforms, and it brings tests (`unittest`), linting (ruff) and type checks (ty). The scripts stay single files with no dependencies, so installing means one `npm i -g` and `museum init`. `bin/museum` stayed sh longest, as a dispatcher cheap enough for a status line, until it held the tick's change check and the timer's units: rules `museum-sync` also needed (the state folders, each harness's sessions folder), kept in two languages and tested only through subprocesses, and a plist written with `sed` escaping. Since 0.6 those live in `museum-sync` (`tick`, `timer`), and `bin/museum` is a Python launcher that maps the verbs and execs it. A tick inside the interval costs about 90 ms instead of 10, twice Python's startup and compiling `museum-sync`, which is nothing for a 5-minute timer or a status line refreshing every 15 seconds. The launcher runs on any python3, so an old one still reaches `museum-sync`, which records that it needs 3.11. The migration stays sh, because it runs on the store host inside rsync's `--rsync-path`. - **Repo metadata beside the session, for every harness.** Until 0.6 pi's extension wrote a `museum` custom entry into the session (`pi.appendEntry`), while the other harnesses' hooks wrote a meta record beside theirs. Two implementations of the same record drift, and only pi's could be written into the transcript (museum never writes into another harness's). So pi's extension now runs `museum hook pi` with the payload the others send, and every harness's record goes to `/meta/.json`: git common dir (the same for every worktree of a repo), remote, branch, commit, and mu identity from the anchor or the environment. It has to be taken at session start, inside the session's process tree: by sync time the branch may have moved, and only an ancestor walk from the hook finds mu's anchor. Sessions from before have the entry inside, which `museum-search` still reads, or nothing, and it falls back to the folder name. - **No daemon, no server-side binary.** The store host needs only ssh and rsync. - **Only `museum tick` syncs.** Until 0.6 pi's extension synced (at `session_start`, `agent_settled` and `session_shutdown`), harness hooks left a flush stamp at `SessionEnd`, and the timer synced every 10 minutes whether or not anything changed. Three triggers for one job, and the timer's made an ssh connection 144 times a day on an idle machine. Hooks cannot sync themselves: they may run in a sandbox that refuses ssh and rsync. So the tick, which the timer and tmux run outside every sandbox, is the one trigger, and it syncs only when a session file is newer than `last-ok`, whose mtime is when the last sync that worked started: a `find` once per interval, and no connection when nothing changed. A failed sync leaves `last-ok` alone, so what it missed stays changed. The cost: a session's last turn waits up to one interval plus one timer period (15 minutes), not until its `SessionEnd`; and a machine whose timer is not running backs up nothing, pi included, which pi's footer and `museum doctor` now say. The timer gets `PI_SESSIONS_DIR` with the other harnesses' folder variables, since pi no longer syncs from its own environment. - **The sync decides the failing label; every pi only shows it.** The label used to be computed by each pi from `state` at its own `session_start` and `agent_settled`. With many agents on a node, an idle one kept showing "ok" through an outage another agent's sync had hit, and kept showing "failing" after another agent's sync recovered. Now `museum-sync` writes the label (empty while backups work) as the `warning` field of `state.json`, next to `status`, `time` and `message`, atomically, and each pi re-reads it every 5 seconds and only calls `setStatus` when it changes. The rule lives in one place, under test. pi decides only what no sync can write: `no backup yet` (no `state.json`) and `cannot run museum-sync` (spawn failed, or exit 126/127). A crash inside the script is caught and recorded like a failed sync. Since 0.6 pi no longer triggers syncs, so a timer that is not running shows as nothing failing. pi therefore also decides `backup stale`: a change to its own session file has waited over 30 minutes (or `MUSEUM_INTERVAL` plus 10 minutes, if longer) for a sync that worked. It is keyed to how long a change has waited, not to the age of the last sync, because a machine with nothing to sync legitimately goes hours without one, and the first turn after a break would otherwise show stale until the next tick. The wait counts from the earlier of the file's mtime and when this pi first saw it unsynced, so a session that keeps writing is caught too. `museum doctor` applies the same rule to the oldest unsynced file's mtime. State moved from `~/.cache/museum` to `$XDG_STATE_HOME/museum` (`~/.local/state/museum`), because a cache may be wiped and this is state, and `state` (`key=value` lines) became `state.json`, which the extension parses without a regex. The old folder is not migrated: it holds no sessions, only sync bookkeeping, and the first sync starts afresh. - **Concurrency lives in `museum-sync`, not the extension.** A node can run 10+ agents. The script takes a non-blocking kernel lock and exits if another sync holds it; the due check happens under the lock. A flush (`--now`) first writes a `pending` mark, and the holder loops until no mark is left, re-checking after it unlocks. The lock is `flock(2)` on an fd, released on death; the fd is not inheritable, so ssh, rsync and any orphan they leave cannot keep it; rsync and ssh timeouts bound a hung connection. - **No automatic retry over a fresh ssh connection.** `BatchMode=yes` blocks password and passphrase prompts, but not a security-key agent asking for a touch. A retry could cost the user a touch, so a failed sync waits for the next due one. Sessions stay on local disk, and `--append-verify` resumes, so waiting loses nothing. One ssh connection per sync: the remote `mkdir` rides rsync's connection through `--rsync-path`. - **`ssh_mux_only` for store hosts that need a human to log in.** The sync runs only over an existing ControlMaster, checked with `ssh -O check`, which never connects. With no master, it records `no ssh master for ` and skips, so the footer asks the user to open one. All ssh auth methods are also disabled, so a master that dies between the check and rsync fails instead of prompting. Here a retry costs nothing, so a failure retries once after `MUSEUM_RETRY_DELAY` (30s) while the master is up. Opting in is explicit because the script cannot tell ahead of time whether a fresh connection would prompt. - **Other harnesses: a filtered copy, synced by the same rsync.** Claude Code, Codex and cursor-agent sessions are backed up too, to `////`. cursor-agent's live SQLite chats are unsafe to copy, so museum stages its greppable `~/.cursor/projects/*/agent-transcripts/` JSONL instead. claude and codex transcripts carry bulk that is neither conversation nor safe to spread (claude's `api-request*` copies, one of which holds the whole system prompt, and file snapshots, codex's `encrypted_content` and a full `replacement_history` and `guardian_history` at every compaction), so `museum-sync` keeps a filtered copy under the state folder and syncs that; cursor-agent's filter drops `turn_ended` and caps oversized tool inputs. The copy grows from a byte cursor, one complete line at a time, so it stays append-only and keeps `--append-verify`; the cursor also records the copy's length, so a crash between writing and saving the cursor is cut back rather than duplicated. A line the filter cannot need (short, and naming nothing it drops) is copied without parsing. A file the filter fails on is skipped, so it cannot stop the other files' cursors or pi's copy. The staged folder and pi's sessions go in one rsync, so a sync still opens one ssh connection. - **Hooks write records; `museum tick` syncs.** Harness hooks may run in a sandbox that refuses rsync and ssh, and Muse Code clears their environment. So `museum hook` only writes the meta record at `SessionStart`, in `~/.local/state/museum` whatever `XDG_STATE_HOME` says, because the hook, the timer and pi each see a different environment. `museum tick`, which usually reads two small files and exits, is run by things outside every sandbox: a tmux status line and a launchd or systemd timer every 5 minutes. It keeps its own `last-tick` stamp, so its walk of the session folders runs once per interval however often a status line calls, and a sync that cannot start (no config, an old Python) is not retried on every refresh. Because a timer and tmux bring their own `PATH`, the config pins `rsync_path` at install. - **One layout: `//`.** pi's folders sat at the top of each machine's folder, beside `claude/`, `codex/` and `cursor-agent/`, because pi came first, and `museum-search` told them apart by depth. 0.6 moves pi into `pi/` like the others. pi's sessions are still copied as they are, not staged: the sync sends a link to pi's sessions folder, `/pi-link/pi`, with `--relative`, so they land in `pi/` in the same rsync as the staged folder. The store is append-only, so changing the destination alone would leave every old copy beside a new one. Each sync therefore first moves what is left at the top of its own machine's folder into `pi/`, as a shell script that rides rsync's own ssh connection (`--rsync-path`), so a machine migrates on its first sync with 0.6, never touches another's, and needs no coordination; once moved, its globs match nothing. A file in both places can only be one an older museum on the same machine grew after the move: the longer copy is kept and the other removed only when it is a prefix of it, the one deletion museum makes, of bytes the store still holds. `museum-search` reads both layouts, since a machine that has not updated keeps the old one. - **`museum doctor`, read-only.** With one trigger, "why is my backup stale" needs one answer: config, rsync, store, timer (and linger), the last sync and its age, hooks, pi and PATH, each with its fix. It runs init's own planning (`plan_hooks`, the pi and PATH checks) without writing, so the two cannot disagree, and it has no `--fix`, because nearly every fix is `museum init`, which already leaves what is right alone. - **Search reads every format through a normaliser to pi's shape.** `museum-search` grew up on pi's serialisation: its rg passes match `"message":{"role":"user"`, and `stats` reads pi's `toolCall` blocks and `toolResult` messages. claude, codex and cursor-agent write the same conversation differently, so the script walks the store, tells formats apart by depth, and turns each into pi's entries: a `session` entry with the cwd, a `museum` entry from the meta record, and `message` entries with text, thinking and `toolCall` blocks. Everything downstream (`stats`, `calls`, `show`, the ranking) is unchanged, and a pi session is still read line by line as it is. Counting must not depend on the format, so merging happens in the normaliser: claude's lines of one reply share a `message.id` (a parallel call's result can come between them), and codex's records of one call share a `call_id`. A codex reply is the run of items between two user messages or tool outputs, as pi records it. cursor-agent's transcript lines are already one turn each (`Shell` → `bash`). `find` keeps its three rg passes for pi; for the others, rg picks the files that mention the term and Python classifies only the lines that do, on normalised text: a line is a turn saying it, the system prompt (codex's instructions, claude's `isMeta` additions), an echo of another line (codex's `event_msg`, claude's queued prompts, skipped), or tool output. - **Released to npm, like the rest of mu-crew; versions reported through the store.** Each machine used to run whatever its own checkout held, with no version anywhere: one devserver was three releases behind, and nothing showed it. museum is now `@mu-crew/museum` on npm, like mu, murmur, mule and muzzle. The package carries plain Python and shell, so a release builds nothing: a `v*` tag runs `make check` and `npm publish` with trusted publishing, as mu's release does. A checkout still installs, for development. To see who runs what, each sync leaves `/.museum/.json` in the store, the one place every machine writes, and `museum status` reads them. One file per version, never rewritten, because the copy is `--append-verify` and would not update a file whose length stayed the same; a second rsync for a status file would cost a second ssh connection. Mixed versions are expected; the one thing they share is the turn index, and a museum leaves an index from a newer one alone instead of rebuilding it, which would make the newer one rebuild it in turn. Rejected: a pi git package (`pi update` would move unpinned machines to whatever main holds, with no release to point at), a self-updating checkout (a dev tree with local edits must never pull, and update logic would live in museum), and shipping code through the store (a bad push breaks every machine's backup at once). - **Installed like the rest of mu-crew: `npm i -g`, then `museum init`.** 0.4.0 installed through pi (`pi install`, then the package's `install.sh` from inside pi's npm folder, which also linked `~/.local/bin/museum`), and the harness hooks were JSON pasted by hand. mu, murmur and muzzle each have a global CLI and an `init` that writes their hook entries, so museum does the same: npm's bin link puts `museum` on PATH, at a path that survives `npm update -g`, and `museum init` writes the config, the hooks, the skill links, the first sync and the timer. pi's package list stays the user's, as with mu: `pi install npm:@mu-crew/museum` once, and `init` only says when pi needs a `pi install` or a `pi remove`. A machine whose pi cannot fetch the package itself installs it with its own tooling. It owns the hook handlers whose command runs museum (`museum …`, or a path to museum or museum-sync, which older setups wrote) and nothing else in those shared files, as murmur owns `murmur`'s, and it never rewrites a file it cannot parse. It also removes what older installs left: a `~/.local/bin/museum` link to another museum, the links into pi's discovery folders, the pre-0.2 cache folder. That leaves two copies on a machine, the global one and pi's package copy, as for mu; two copies at different versions would sync with different filters into the same staged folder, so pi's extension runs the museum-sync beside the `museum` on PATH, and its own only when there is none. `install.sh` stays, as `museum init` for a checkout. The verbs follow the family's: `museum status` for this machine and every machine's version (`muzzle status`, `murmur status`), and `museum-search machines` for the list, in museum's own noun; not `fleet`, which no other tool uses as a verb, nor `peer` (murmur's and mu's for a machine they exchange state with) or `host` (mule's and mu's for an ssh target, and museum's own `store.host`). ## Rejected or deferred - **A central SQLite FTS index over everything.** SQLite locking does not work across rsync copies or network filesystems, and a trigram index of tool output would outgrow the store. The turn index works because `museum-search` already runs on the store host, from stdin, and indexes only conversation text. - **Per-host index shards plus a merged cache on each client.** This works, but it adds complexity that occasional grep searches don't need. - **Hashed file names or content-addressed chunks.** They aren't needed, because per-host folders already rule out collisions and readable paths make it easy to narrow a search. - **Compressing session files (`.jsonl.zst`).** Search can handle it: `rg -z` reads plain and `.zst` files in one pass, and on the store `find` took about 1.2s instead of 0.4s. The cost is in the sync. rsync appends only to plain files, so a session the store has compressed gets uploaded whole on every sync. Stopping that means sending only recently changed files (`--files-from`), and then the sync no longer repairs the store by itself. The store host would also need `zstd`. Put the store on a compressing filesystem instead: on btrfs with `compress-force=zstd:3`, the 865M store takes 272M on disk. Compressing each file with zstd would bring it to 229M. - **Existing tools.** chronicle (git transport; rewrites session files) doesn't fit. adobe/pi-session-search (scans local files) is the fork candidate if `museum-search` proves too limited. ## Possible later additions - **A failure log.** `state.json` holds only the last result, so a failure that a later sync fixes leaves no trace. - **A tool-output index,** if the tool tier gets slow. A word-level FTS5 index (unicode61) would be far smaller than trigrams, but matches whole words, not the substrings `find` promises.