# Pi Reads architecture

Status: accepted for Phase 0

Date: 2026-08-20

This document defines the dependency boundaries for the roadmap. It does not claim that every component exists yet.

## Design goals

1. Preserve faithful source capture independently from AI generation.
2. Keep core behavior usable from CLI scripts, Pi, tests, and future applications.
3. Keep package installation files separate from user library data.
4. Make every external destination an explicit adapter with testable side effects.
5. Preserve deterministic Astro/Shiki print output and text-hash verification.

## Layers

```text
┌──────────────────────────────────────────────────────────────┐
│ Interfaces                                                   │
│ scripts/*       extensions/pi-reads/*       skills/pi-reads │
└──────────────────────────────┬───────────────────────────────┘
                               │
┌──────────────────────────────▼───────────────────────────────┐
│ Application services                                         │
│ ingest source · create article · verify archive · export     │
└───────────────┬──────────────────────────────┬───────────────┘
                │                              │
┌───────────────▼────────────────┐  ┌──────────▼──────────────┐
│ Core domain                    │  │ Ports                   │
│ records · validation · hashes │  │ storage · render · send │
└────────────────────────────────┘  └──────────┬──────────────┘
                                               │
                                  ┌────────────▼──────────────┐
                                  │ Adapters                  │
                                  │ web · file · Astro/PDF    │
                                  │ Obsidian · EPUB · Kindle  │
                                  └───────────────────────────┘
```

Dependencies point downward. Core domain modules must not import Pi, Astro, Playwright, Obsidian, SMTP, or process-global configuration.

## Planned source layout

The roadmap may refine file names, but it must retain these boundaries:

```text
src/
  core/
    domain/             # schema-backed records and invariants
    hashing/            # exact-byte and normalized-text hashes
    validation/         # record and cross-record validation
  application/
    ingest/             # use cases over source adapters
    articles/           # archive/digest/synthesis persistence
    exports/            # render and delivery orchestration
  adapters/
    sources/            # URL, text, Markdown, and file input
    storage/            # filesystem library implementation
    render/             # Markdown, Astro HTML, PDF, and EPUB
    destinations/       # local, Obsidian, and Kindle
  layouts/              # existing Astro presentation
  pages/                # existing Astro routes
extensions/
  pi-reads/             # Pi tools and slash commands
skills/
  pi-reads/             # distributable end-user workflow
scripts/                # thin compatibility and CLI entry points
schemas/                # versioned persisted-record contracts
tests/
  fixtures/             # deterministic local inputs
```

## Core domain

The core owns:

- `Source`, `Article`, `Citation`, ordered `ReadingCollection`, `Export`, and separate mutable `ArticleUserState` semantics;
- mode and identifier rules;
- path-independent record validation;
- content and visible-text hashing;
- archive/generated separation;
- cross-record provenance checks.

The core receives clocks, ID generators, paths, and I/O through function arguments or ports. It does not read environment variables or infer the package root.

## Application services

Application services coordinate domain rules and ports. Expected use cases are:

- `ingestSource(input, options)`
- `createArchive(sourceId, options)`
- `saveGeneratedArticle(draft, generationContext)`
- `createReadingCollection(articleIds, provenance)`
- `prepareExport(articleOrCollectionId, format)`
- `deliverExport(exportId, destination, confirmation)`
- `listLibrary(query)`

Names here describe responsibilities, not a frozen TypeScript API. Pi-facing tool names are frozen separately in the naming contract.

Application services must be abort-aware where adapters perform network, browser, filesystem, or delivery work.

## Ports and adapters

### Source adapters

A source adapter turns one external input into a capture candidate. It may fetch or read bytes, but it cannot persist a canonical record directly. The ingestion service validates, hashes, and writes the source. URL extraction performs one bounded HTML fetch; every destination and redirect must resolve only to public, routable addresses, embedded credentials are rejected, and timeout/redirect/byte limits apply before buffering. Optional serif/sans-serif presentation is inferred deterministically from inline or embedded CSS in those captured bytes and safely defaults to serif, without browser probing or a second navigation.

Phase ordering:

- Phase 1: URL, text, Markdown, file
- Phase 6: RSS, social threads, transcripts, and other sources

### Storage adapter

The filesystem adapter implements the layout in the product contract. It must:

- resolve all record paths beneath a configured library root;
- reject traversal and absolute record paths;
- write atomically and create-only;
- treat manifests as canonical and `indexes/library.json` as a rebuildable metadata cache;
- derive versioned per-source heading/paragraph indexes from exact `content.md` bytes, with content-hash binding and independently verifiable UTF-8 byte ranges;
- build `indexes/search-v1.json` locally from canonical article/source metadata, separate user state, weighted lexical terms, and exact block ranges; verify corpus/index hashes and rebuild deterministically after staleness, deletion, or corruption;
- keep revisioned mutable reading state under `state/articles/`, serialized by a cross-process atomic lock, with compare-and-swap updates that never touch immutable manifests;
- update the index with synced temporary files and atomic renames while an interruption marker protects canonical record/index consistency;
- detect stale indexes from constant-time source/mode directory stamps and rebuild from immutable manifests when the index is missing, malformed, stale, or interrupted;
- serialize in-process index mutations in addition to the Pi-facing file mutation queue;
- reject a Git working tree as a production library root by default.

### Render adapters

Renderers consume a validated article and resolved source/assets. They do not mutate source or article records.

- Markdown is the canonical readable body format.
- Astro HTML and PDF retain the existing light print behavior.
- EPUB is a separate reflowable renderer with validated ZIP/container/package/spine structure and embedded assets. Collection EPUBs use one navigation entry and spine XHTML item per ordered article while preserving chapter-level mode, sources, and citations.

Archive rendering must run visible-text fidelity verification before an export is reported as prepared.

### Destination adapters

Destinations receive a prepared export; they do not generate article prose.

- Local destination retains the artifact in the library.
- Obsidian renders destination frontmatter, copies/downloads assets, rewrites relative links, and writes only conflict-approved targets in a configured vault. Its deterministic graph builder derives fixed library/topic/status/queue views from delivered-export metadata and separate user state, adds source links only to managed synthesis notes, never rewrites archive notes, refuses unmanaged collisions, and hash-checks targets again at write time.
- Kindle dry-runs retain an immutable local EPUB/PDF and return its export ID and hash; a later SMTP delivery verifies and sends those exact bytes only after explicit interactive confirmation, then records evidence by reference without copying the artifact. The scheduler-facing digest preparation service has no destination/mail port, so unattended jobs can create only local collection EPUBs.

Every external side effect returns delivery evidence suitable for a non-secret export manifest.

## Pi integration boundary

The Pi extension is an interface adapter over application services. Extension registration is lightweight: library, collection/digest preparation, renderer, EPUB, Obsidian, Kindle, SMTP, browser, syntax-highlighting, and credential-store implementations load only when their corresponding workflow first needs them. Text ingestion and metadata listing do not initialize destination or renderer adapters.

It owns:

- TypeBox tool argument schemas;
- `reads_*` tool registration;
- `/reads` and configuration commands;
- active Pi model/session provenance;
- TUI prompts and explicit delivery confirmation;
- compact, truncated tool results.

It must not own extraction, storage, rendering, or destination business logic. The `reads_export` adapter dispatches to independently tested local, Obsidian, and Kindle handlers behind one result contract; confirmation and overwrite decisions remain centralized safety operations. Destination configuration keeps TUI answer collection separate from pure normalization/validation and atomic persistence. Headless Pi modes must be able to use tools without TUI-only APIs; interactive commands must guard UI calls with the documented Pi context capabilities.

Model-visible success text contains only the identifiers, warnings, cursors, and artifact paths needed for the next operation; verbose manifests and paths stay in structured result details. Source reads/searches suggest deterministic compact citation IDs and stable fragment locators. The distributable skill remains the detailed workflow authority while persistent tool snippets retain one concise statement of each mandatory safety rule.

## Main workflows

### Archive

```text
input
  → source adapter
  → deterministic extraction/cleanup
  → canonical-URL/content-hash match under the serialized library transaction
  → reuse an exact Source/archive, report changed content without writing, or explicitly recapture
  → new immutable Source/archive with predecessor lineage when recapturing
  → render
  → fidelity verification
  → Export
```

No model-authored prose enters this path.

### Batch capture

`BatchIngestionService` bounds independent source acquisitions and returns ordered per-item outcomes. The Pi adapter extends `reads_ingest` rather than registering another tool, lazy-loads the service, and holds the existing file-mutation queue around the workflow. The application layer remains Pi-independent.

Each `LibraryService.capture` prepares the source/archive pair and derived source index before publication. `ImmutableRecordGroup` compensates only directories that operation created; an optional catalog-transaction rollback runs under the same queue if index publication fails. Duplicate matching stays serialized. Cancellation aborts network work and pending publication, but an item already publishing finishes or compensates. This is per-item compensation, not a whole-batch or crash-atomic filesystem transaction. See [batch ingestion](batch-ingestion.md).

### Reading-pack collections

`ReadingCollectionService` validates 2–50 explicit unique article IDs, snapshots their ordered immutable mode/content hash/source/citation/generation provenance, and writes one create-only `col_…` manifest. `EpubService` reloads and verifies every snapshot before building a table of contents plus one XHTML spine item per article. `DigestPreparationService` composes only collection persistence and local EPUB preparation; the scheduler CLI deliberately has no Kindle service or mail transport. A later `reads_export` collection send reloads the exact prepared export and passes through the existing interactive confirmation gate. See [reading packs](reading-packs.md).

### Feed and newsletter collections

`parseFeed` and `ingestNewsletterFile` are deterministic adapters that return canonical `IngestedSourceDraft` values. Remote feeds pass through the public-network/redirect/media-type/byte/timeout policy; newsletters are bounded local `.eml` files parsed without mailbox credentials. Embedded HTML uses the existing cleanup and Markdown pipeline, while raw entry or email evidence remains separate.

`CollectionIngestionService` resolves a collection, asks `LibraryService` for read-only duplicate matches, and returns bounded metadata plus a hash-bound token. It retains no server-side preview state. Capture re-resolves the collection, verifies that token, validates explicit unique indexes, and passes only those drafts to `BatchIngestionService`/`LibraryService.captureDraft`. The Pi tool and `/reads` are thin preview/selection adapters; no record is created before selection. See [feed and newsletter ingestion](feed-and-newsletter-ingestion.md).

### Clipboard and transcripts

The core clipboard adapter receives content but has no OS access. Only interactive `/reads`, after a dedicated confirmation, executes one platform clipboard read; generated workflows capture first and refer to the immutable source through bounded retrieval. The transcript adapter parses bounded local SRT/WebVTT into deterministic timestamp-heading/segment paragraphs, so the existing source index and citation-grounding pipeline resolve exact timestamp sections without a parallel locator system. Raw subtitle bytes remain separate evidence. See [clipboard and transcripts](clipboard-and-transcripts.md).

### Digest or synthesis

```text
one or more immutable Sources
  → Pi requests bounded outlines, exact locator ranges, or lexical excerpts
  → delimited source data enters model context under an explicit byte budget
  → continuation cursors traverse long outlines/ranges without unbounded results
  → model authors a cited draft and declares complete or targeted coverage evidence
  → application verifies source hashes, coverage/paragraph locators, markers, and exact quoted source text
  → multi-source synthesis returns a no-write citation-distribution/unused-source review bound to the exact draft
  → the exact reviewed draft and active generation identity are resubmitted with its review token
  → immutable generated Article with bounded coverage and citation-grounding diagnostics
  → render
  → Export
```

The active Pi model is the generator. The extension does not silently invoke a second model. `/reads` resolves a versioned structured generation template, then builds an ordered, hash-bound plan for 2–20 selected captured sources without loading their prose. `LibraryService` owns Pi-independent plan validation and the mandatory two-step multi-source persistence gate; the extension supplies active provider/model/thinking/session provenance. Every non-empty multi-source synthesis section must contain a registered citation marker, a conservative structural rule rather than semantic fact-checking. Review tokens exclude generation time but bind all draft, ordered-source, citation, coverage, template, and active generation-identity fields, so any meaningful change requires another review. Template definitions contain only allowlisted modes, section roles, coverage policy, word bounds, and citation budgets—never executable prompt prose from captures. Articles snapshot template origin/version and bounded budget warnings. See [multi-source cited synthesis](multi-source-synthesis.md) and [generation templates](generation-templates.md).

### Delivery

```text
validated Article
  → prepared local Export
  → destination-specific preview with prepared export ID/hash
  → optional later reload and integrity verification
  → explicit confirmation when required
  → destination adapter using the verified prepared bytes
  → delivery status/evidence referencing the prepared export
```

A failed delivery never deletes the prepared local artifact.

### Offline library maintenance

`MaintenanceService` coordinates read-only canonical verification, derived-index rebuilding, portable directory backup, and create-only restore. `library-verification.ts` scans canonical records independently of caches; core helpers validate installed versioned schemas, hashes, bounded inventories, and safe filesystem paths. The thin `scripts/maintain-library.ts` interface owns CLI/config resolution, not storage behavior.

Backups include only verified manifests, referenced source/article/export files, separate user-state records, and explicitly allowlisted portable defaults. Restore reserves a nonexistent destination exclusively, copies exact bytes, rebuilds derived indexes, and verifies the result. Neither operation invokes destination adapters or reads credentials. They require stopped writers rather than claiming an atomic live snapshot. See [library maintenance](library-maintenance.md) for collision, privacy, and resource limits.

## Configuration and secrets

Path resolution is defined in the product contract. Core services receive a resolved configuration object.

`pi-reads.json` may hold non-secret preferences, including SMTP host/port/TLS mode, a credential backend/profile, and the names of environment-variable overrides used for Kindle delivery. Interactive desktop setup writes protected credential entries to macOS Keychain, Windows Credential Manager, or Linux Secret Service. SMTP usernames/passwords, tokens, full Kindle/sender addresses, and other credentials are never copied into JSON configuration.

Logs, persisted records, and tool results must redact credential material and recipient addresses. The interactive Kindle confirmation may display the full recipient transiently, but it is not persisted.

## Error behavior

- Invalid or unsupported records fail closed with an actionable message.
- Existing immutable record paths produce a collision error, never overwrite.
- Partial writes are cleaned up or left only under a documented temporary name.
- Cancellation propagates to fetch, browser, rendering, and delivery adapters.
- Export or delivery failure does not mutate source/article records.
- A fidelity mismatch blocks archive export.

## Testing strategy

- Core: unit tests for invariants, hashing, validation, and path handling.
- Extraction: local HTML fixtures; no required live network.
- Storage: temporary-directory integration tests, including collisions and atomicity.
- Rendering: existing fidelity verifier plus fixture snapshots where stable.
- Pi extension: extension-load and tool-contract smoke tests.
- Obsidian: temporary fake vault with local/remote image fixtures, conflict handling, unrelated-file checks, and symlink escape rejection.
- Kindle: fake SMTP transport and mandatory confirmation tests.

Each phase adds only the tests needed for its layer. The baseline eventually remains:

```sh
pnpm article:check
pnpm test
pnpm build
```

## Explicitly deferred

Phase 0 does not implement:

- core modules or migrations;
- the Pi package manifest or extension;
- Obsidian writes;
- EPUB rendering;
- SMTP/Kindle delivery;
- RSS, social, or transcript adapters.

Those remain owned by their roadmap phases.