---
name: archive-crawler
description: "Walks a folder of files the owner has accumulated over time (Downloads, Dropbox, an old archive), classifies each candidate, presents a filtered list for confirmation, and routes the approved set to the right ingest skill. Triggers when the owner says 'scan my Downloads for X', 'mine my old files', 'crawl my Dropbox archive'. Compresses at ingest; never writes without explicit operator filter selection."
---

# Archive crawler

Invoked from `specialists:librarian`.

This skill imports volume in a controlled way. It walks a folder, classifies each file, asks the owner to filter, and routes only the approved set through the existing ingest skills. It does not invent a new write path; it sits in front of `document-ingest` and `conversation-archive` and gates them.

**Default parent.** This skill performs no graph writes itself. The downstream ingest skill it dispatches to (`document-ingest`, `conversation-archive`, or a per-source archive importer) is responsible for naming the parent in the canonical hierarchy `LocalBusiness → Project → KnowledgeDocument`. The operator's filter selection answers the parent question implicitly — "scan my Downloads for pricing notes" names the topical scope that the downstream skill resolves into a `:Project` anchor before any write.

## When to run

Run when the owner names a folder and a kind of content to find inside it. Examples:

- "scan my Downloads for articles from this year"
- "mine my Dropbox archive for old meeting notes"
- "find every PDF in this folder about pricing"
- "crawl this folder and import the conversations"

Do not run when the owner uploads a single file (use `document-ingest` directly) or a single archive in a known format (use the matching per-source skill).

## Required inputs

| Input | Meaning |
|---|---|
| `folder` | Absolute path to the folder to walk. Must be inside `$ACCOUNT_DIR` or a path the owner explicitly authorised this turn. |
| `filter` | One or more of: file type (extension), date range, keyword in filename, size band. At least one filter is required. |
| `scope` | `admin` or `public`. Where the resulting graph nodes live. |

If any required input is missing, call `AskUserQuestion` for the missing one. Never guess.

## Method

1. **Walk the folder.** Use `Bash` with `find` against the named folder, applying the filter as `find` predicates (`-type f`, `-name`, `-newer`, `-size`). Cap the candidate count at 1000 per run. If the count exceeds the cap, surface the count and ask the owner to tighten the filter.
2. **Classify each candidate.** Inspect the file's extension and first kilobyte:
   - Narrative document (`.pdf`, `.md`, `.txt`, `.docx`, `.html`, `.rtf`) → routes to `document-ingest`.
   - Conversation transcript (`_chat.txt` from WhatsApp, `.json` from Telegram export, `.vtt` from Zoom, similar) → routes to `conversation-archive`.
   - Flat dataset (`.csv` matching a known per-source archive shape such as LinkedIn `Connections.csv`) → routes to the named per-source skill.
   - Unknown → set aside under "needs operator decision".
3. **Present the candidate list.** Group by classification. Show the count per group and a sample of three file names per group. Ask the owner to confirm each group with **wire all**, **wire some** (the owner names which), or **skip group**.
4. **Route the approved set.** For each approved group, invoke the matching ingest skill, passing the file path and the operator-confirmed anchor (the document subject for documents, the archive owner for conversations). The ingest skill is the only path that touches the graph; this skill never writes directly.

## What it does not do

- It does not classify file content beyond extension and first kilobyte. The downstream ingest skill (`memory-classify` inside `document-ingest`) is the classifier. This skill is the gate.
- It does not compress files. It selects which files to ingest, and downstream `memory-write` writes the compressed representation.
- It does not flatten subdirectories silently. If the folder has subdirectories, list them and ask whether to recurse.
- It does not create placeholder graph nodes for files in the "needs operator decision" group.

## Output to the operator

A short report:

- `<N>` candidate files found across `<G>` classifications.
- `<K>` approved, `<S>` skipped, `<U>` left as "needs operator decision".
- For each approved group: the count routed to the named downstream skill, with the count of writes the downstream skill returned.

## Failure modes

- **Folder does not exist or is outside `$ACCOUNT_DIR`.** Refuse with the path and the reason. Do not walk.
- **Filter matches zero files.** Report the filter and ask whether to widen it.
- **Downstream ingest skill fails mid-batch.** Stop. Report which files completed and which did not. Do not retry inline.

## Why the filter is mandatory

A folder of 30,000 accumulated files exceeds the operator's ability to review case-by-case. The filter is not a convenience; it is the precondition for ingest. Without it, the owner cannot meaningfully consent to what enters the graph, and the graph becomes a landfill. The compress-at-ingest doctrine is enforced at this skill, not in `memory-write`: if a file gets here without an operator-confirmed filter, the skill refuses.