# distill

`@stravica-ai/engineering-pa-core` › `distill`; the librarian distillation
sweep runner.

Launches a non-interactive Claude Code session as the Librarian, hands it
the repo-wide distillation prompt, and writes its outputs into the working
tree. **It does not commit, pull, fetch, or push.** Git is owned by the
per-host `sync` CLI (`packages/cli/src/sync/`); distill is a pure file
writer.

Designed for cron: quiet on success, loud on failure, exit-code accurate
so cron mail / monitoring can catch breakage.

## Status

Implemented: `run`, `run --dry-run`, `--help`.

## Install / invoke

Via the core package's bin shim; from an operator repo:

```
pnpm exec distill run
pnpm exec distill run --dry-run
pnpm exec distill --help
```

(or directly in this repo: `node packages/core/bin/distill run`.)

Scheduled by the operator repo's `schedules/distillation-sweep.json`
(cadence is operator-configured). The scheduler invokes it from the
operator repo root via cron. The companion sync schedule (e.g.
`schedules/dave-sync.json`, every 30 minutes on the same host) picks up
any file changes distill produced.

## Commands

### `distill run`

1. Resolve prereqs; repo root (walks up to `.git`), `paName` from
   `operator/config/preferences.json`, librarian definition per the
   resolution order below, `claude` on `$PATH`, log dir
   `<repoRoot>/.cache/distill/logs/` (created if missing; gitignored per
   tooling standard §16).
2. Count markdown observations under `agents/*/staging/` (depth ≤ 2,
   excluding `_archived/`). If zero, append a `noop-empty-staging`
   audit record and exit 0.
3. Invoke `claude` with an explicit `--model` (see Model selection below),
   the librarian definition as an appended system prompt, and the
   distillation instructions as the positional prompt.
   Output is written to `<repoRoot>/.cache/distill/logs/<ts>.log`.
4. Append a one-line JSON audit record to
   `<repoRoot>/.cache/distill/logs/runs.jsonl` (schema below).
5. Exit 0 (ok) or 3 (claude transport failure).

### Model selection

The sweep passes `--model claude-opus-5-5` explicitly (operator ruling
2026-07-29 for the pin). Before the pin, no `--model` was passed and the Librarian
inherited whatever the host CLI defaulted to -- so the model running the
sweep was an accident of host configuration, and a host-side default bump
silently changed sweep behaviour across the estate. The pin makes the
choice deliberate. The pinned model follows the estate model ladder:
`claude-opus-5-5` is the base model from the 2026-10-16 operator ruling.
It replaced `claude-opus-4-7` (now deprecated), which was first picked
because it followed briefs better than 4.8/5 in estate experience, and
brief-following matters more for a scripted sweep than raw capability.

Override per host or per run with the `DISTILL_MODEL` environment
variable:

```
DISTILL_MODEL=claude-opus-4-8 pnpm exec distill run
```

An empty or whitespace-only `DISTILL_MODEL` is treated as unset. The
dry-run plan reports the effective model and whether it came from the
pinned default or the override.

### Librarian definition resolution

The system prompt appended to the sweep session is resolved in this
order (highest precedence first):

1. **Explicit `librarianDef` option** (tests pass fixture paths here).
   The caller controls the whole prompt; no supplement is applied.
2. **Full operator override** at
   `operator/extensions/roles/librarian/definition.md` (under the
   discovered repo root). The operator has taken ownership of the whole
   prompt: the file is used byte-for-byte and **no supplement is
   appended, even if `overrides.md` also exists**. This preserves exact
   back-compat with pre-supplement operator repos.
3. **Bundled definition + operator supplement.** When there is no full
   override but `operator/extensions/roles/librarian/overrides.md`
   exists, the composed prompt is the bundled definition followed by the
   supplement, joined by an explicit precedence statement: wherever the
   supplement conflicts with the bundled text, **the supplement wins**;
   where it is silent, the bundled definition applies unchanged. This is
   the recommended way to customise routing (e.g. a repo-specific
   operational-rules destination) without forking; and silently
   diverging from; the whole bundled definition.
4. **Bundled definition alone**
   (`engineering-pa-core/roles/librarian/definition.md` inside the
   package tree).

A supplement file that exists but is empty or whitespace-only is treated
as absent (outcome 4): appending an empty supplement under an
operator-wins banner would claim a customisation that does not exist.
The dry-run plan reports which outcome applied; the resolved definition
path and byte size, the supplement path and byte size (or `none`, or
`empty; ignored`), and the composed total.

**Git changes the librarian produces are committed by sync, not here.**
Distill stops at the audit-record write; the sync CLI's next tick on
the same host picks up the staging-area changes, the deletions from
`agents/*/staging/`, the additions/updates under `operator/knowledge/`
and `operator/profile/`, etc.

### Why the change (2026-05-11)

Previously distill did `git status / fetch / pull --rebase` as a pre-flight
and `git add / commit / push` as a post-flight. This raced with the other
cron writers on the same host (`wsd-ingest` every 15m, `upstream-triage`
every 30m), each doing the same thing on their own clocks. A ~2-second
window between `isTreeClean` and `pull --rebase` was enough to surface
`fail-rebase` audit records; including the 2026-05-11 03:00 BST alert.
The fix is structural: cron tools stop touching git; a single per-host
writer (the `sync` CLI, `packages/cli/src/sync/`) handles all of it.

### `runs.jsonl` audit-record schema

One JSON object per line, camelCase keys (§11 of the tooling standard):

| Field | Type | Description |
|---|---|---|
| `ts` | string | RFC-3339 / ISO-8601 UTC timestamp (e.g. `2026-04-16T02:00:06Z`). Round-trips cleanly through `new Date()`. |
| `status` | string | Semantic status, closed vocabulary (see below). |
| `exitCode` | number | Process exit code (0 / 2 / 3). Consumers that only need pass/fail can ignore `status`. |
| `durationSec` | number | Seconds elapsed. Typically 0 for pre-claude early exits. |
| `stagingCount` | number | Result of `countStagedObservations`. 0 on `noop-empty-staging`. |
| `log` | string \| null | Basename of the per-run log file, or `null` when no log file was created (pre-claude early exits). |
| `reason` | string | Human-readable one-liner. Present on every non-`ok` record; omitted on `ok`. |

Status vocabulary (closed set):

| Status | exitCode | When |
|---|---|---|
| `ok` | 0 | Claude ran successfully (changes, if any, will be committed by sync). |
| `noop-empty-staging` | 0 | No observations in `agents/*/staging/`; healthy no-op. |
| `fail-claude` | 3 | Claude exited non-zero. |
| `fail-cli-bootstrap` | 1 / 2 | parseArgs / startup failure before runDistill was reached. |

Statuses that USED to exist but no longer can (the git pre/post-flight was removed):
`fail-dirty-tree`, `fail-rebase`, `fail-git`. Historical records carrying
those statuses remain valid and consumers should still recognise them.
Git is now owned end-to-end by the per-host `sync` CLI
(`packages/cli/src/sync/`).

Consumer contract: monitors should treat `status == 'ok'` and any
`noop-*` as healthy; any `fail-*` as failing. For backwards compatibility
with historical entries (pre-2026-04-22) that lack the `status` field,
derive from `exitCode`: 0 ⇒ healthy, non-zero ⇒ failing.

### `distill run --dry-run`

Same as step 1–2, then print what *would* be invoked (including the
explicit "git: none; sync (core/tools/sync) picks up changes" line) and
exit 0. Never spawns `claude`, never writes `runs.jsonl`.

## Exit codes

| Code | Meaning |
|---|---|
| `0` | OK; nothing to distill, or sweep completed. |
| `1` | Usage error (unknown verb, bad flag). |
| `2` | Prerequisite missing (no preferences, no paName, no librarian definition, no `claude` on PATH). |
| `3` | Transport / IO failure (claude exited non-zero). |

## Operational discipline rules

1. **One claude invocation per run.** The runner has no "invoke claude
   multiple times" API. Bigger sweeps go in the prompt; they do not
   become more spawns.
2. **No shell interpolation.** Every `spawn()` call passes arguments as
   an array with `shell:false`. User-supplied data (observation filenames,
   paths with spaces) flows through untouched; a malicious name cannot
   become a shell command.
3. **Chokepoint for every external effect.** `src/runner.mjs` is the only
   module that spawns processes or reads wall-clock time. Tests replace
   it with a scripted fake to exercise every flow without touching the
   filesystem, the network, or claude.
4. **No git, ever.** Distill must not spawn `git` of any kind; that
   discipline is the structural fix for the racy pre/post-flight pattern.
   The regression test `regression: distill spawns NO git commands at any
   point` exists specifically to enforce this; if a future maintainer
   re-introduces a git call, that test will fail.

## Project layout

```
packages/core/
  bin/distill              Node shebang shim. One line.
  docs/tools/distill.md    This file. The contract.
  src/distill/
    cli.mjs                Arg parsing (node:util.parseArgs), verb routing,
                           exit-code translation.
    run.mjs                Orchestrator. One function per step of the sweep.
    prereqs.mjs            Repo root, paName, librarian def, claude-on-PATH.
    staging.mjs            Counts agents/*/staging/*.md (depth ≤ 2, no _archived).
    prompt.mjs             DISTILL_PROMPT + claudeArgs(). Pure strings.
    runner.mjs             Chokepoint; the only module that spawns processes
                           and reads the clock. Takes a spawner factory so the
                           tests can inject a fake.
  test/distill/
    fixtures/              Synthetic repo trees (clean, empty, missing-pieces).
    *.test.mjs             node:test.
```

(`src/git.mjs` and `test/git.test.mjs` were removed in the 2026-05-11
sync refactor; they tested porcelain-Z parsing that is no longer
needed.)

## Tests

```
node --test packages/core/test/distill/*.test.mjs
```

Tests use dependency injection over mocking libraries. Real git, real
claude, and real filesystem mutations (outside of a temp dir) never
appear in the suite.

Coverage:

- `staging.test.mjs`; staging counter with and without `_archived/`.
- `prereqs.test.mjs`; every error path on the prerequisite ladder, plus
  the four-step librarian-definition resolution order (explicit opt >
  full override > bundled + supplement > bundled alone).
- `run.test.mjs`; empty staging (audits `noop-empty-staging`),
  dry-run, happy path, claude failure (audits `fail-claude`), two
  regression guards that fail if a future change re-introduces any
  `git` invocation, and the librarian-definition composition contract:
  full-override back-compat (byte-for-byte, supplement ignored),
  bundled + supplement composition with the operator-wins precedence
  statement, whitespace-only supplements treated as absent, and dry-run
  reporting of source paths + byte sizes under each resolution outcome.
- `cli.test.mjs`; `--help`, `-h`, no-args, unknown verb, unknown flag.

## The tooling standard

This tool follows [the tooling standard](../standards/tooling.md)
(§1 identity, §2 Node-only, §3 zero deps, §4 structure, §5 CLI,
§6 exit codes, §7 chokepoint, §8 testing, §11 camelCase JSON, §15 audit-record,
§16 gitignored operational artefacts).