# Seeding — authored source, compiled artifact, loaded rows

You decide how curated data reaches the database: where it is authored, how it is checked and compiled, and how it is loaded. What table it lands in, its keys and the deterministic UUIDv5 seed ids are `database.md`; this skill never restates them. Apply the sections below to the pipeline in hand; when reviewing, report each violation with the section that names it.

## The pipeline

Curated data that ships with the product — vocabularies, catalogs, reference profiles — is **authored in a tree, compiled to SQL, and loaded**. The compiled artifact is build output, not source.

| Stage | Where | Rule |
| --- | --- | --- |
| Author | `seed/<data>/**` | committed — the only thing a human edits |
| Compile | emits `seed/sql/*.seed.sql` | mechanical — parse, validate, emit; no judgment, no network. Output is gitignored and rebuilt on every run |
| Load | the engine's own client — `wrangler d1 execute`, `psql -f` | idempotent |

- **Never commit the compiled artifact.** Rebuilt seconds before it is applied, it cannot go stale, so it owes no drift check and no freshness test; a committed `.sql` would owe both. **Never hand-edit seed SQL** — regenerate it.
- **The load is idempotent**: UUIDv5 ids derived from the natural key plus `ON CONFLICT(slug) DO UPDATE`, preserving live counts and admin-tuned columns. Re-seeding is a no-op, never a duplicate.
- **A build step that reads the artifact runs the compiler first** — wire `pretest` / `prebuild`. Bundlers resolve imports at *build* time, so a gitignored artifact breaks a fresh clone with a resolution error rather than a test failure.

## Choosing the authoring format

**Choose by what the payload is, not by where it lands.** A relational destination argues nothing about the source: rows routinely carry document-valued columns, and the compiler exists to bridge the two shapes. The litmus is one question per field — **is this a scalar a human scans in a table, or prose a human reads in paragraphs?**

| Payload | Format | Why |
| --- | --- | --- |
| Flat, uniform records — one scalar or short list per cell | **CSV** | Reviews as a table; the diff is one line per record |
| Nested structures — list-of-objects, maps of maps | **JSON / YAML block** | The only formats that express nesting at all |
| Authored prose that is itself read or served | **Markdown body** + frontmatter, with a fenced block for the structured half | Renders, diffs by line, survives review |

- **Mixed is normal — never force one format on a heterogeneous corpus.** A flat `index.csv` of per-entry metadata beside one `.md` per entry carrying the body is the *correct* shape when the corpus has both halves, not a failure to standardize.
- **One row per file** when bodies are large and reviewed as prose; **one file of many rows** when records are flat and reviewed as a table.
- **CSV cannot hold multi-line text.** Quoting a paragraph into a cell yields an undiffable blob and breaks most naive parsers, which split on newlines *before* parsing quotes. Prose in a CSV cell is a defect, not a tradeoff.
- **Check the delimiter against the data before choosing CSV.** A corpus whose values contain commas (`oklch(0.8 0.16 84 / 0.3)`) pays quoting tax on every row.

## Rigidity comes from validators, never from containers

**No file format enforces a shape.** CSV checks no types, no required fields, no nesting — not even column count; YAML and JSON check only that the syntax parses. The shape is enforced by a schema or not at all, so a format migration undertaken to gain rigidity buys none.

- **Validate at the compile boundary and fail the build**, naming the file and the reason. A compiler that checks only *parseability* ships a typo'd key or a missing map straight to production, where it is served with full authority.
- **Declare the shape as a schema and share it** between the compiler that writes the column and the code that reads it — one definition, so compile-time and read-time cannot disagree. A `values_json`-style column is opaque to the database; that schema is the only thing standing between it and `unknown` at the read site.
- **Types are code, data is seed.** The shape lives in source and is reviewed as code; the content lives in the authored tree and is reviewed as data. Moving content into source (a 1,700-line constant) puts data behind a deploy; moving shape into the tree puts the contract beyond the typechecker's reach.
- **Distrust forgiving parsers.** Padding a short row with empty strings, or defaulting a missing field, converts a malformed record into a plausible wrong one — the exact failure a validator exists to prevent.

## Canonicalize, never repair

Two operations get called "cleaning the data"; only one is safe:

| Operation | Verdict | Why |
| --- | --- | --- |
| **Canonicalization** — deterministic and information-preserving: slug folding, diacritic normalization, key and unit ordering | ✅ belongs in the compiler | The same input always yields the same output, and nothing is invented |
| **Repair** — filling, clamping, defaulting, or dropping content that failed validation | ❌ reject loudly instead | Deciding what the author meant *is* judgment; a repaired row is then served indistinguishably from a curated one |

**Never guess at malformed input** — a guessing pipeline swaps a loud rejection for a quiet wrong answer, and for curated data, whose whole value is that a human judged it, that is corruption. Reject with a report: the producer retries, or a human triages.

## Ingesting scraped or generated data

Automated sources produce **candidate raw material, not rows.** Publish the compile-boundary schema as the ingestion contract and run scraped output through it unchanged — validate, reject loudly on mismatch, queue the survivors for curation. A curator promotes a candidate by adding the judgment the source could not supply: the rationale, the caveat, the editorial line. **Never** let an unreviewed extract land directly as a catalog row — that swaps a curated corpus for a scraped one.