# Pi Reads product contract Status: accepted for schema version 1 Date: 2026-08-20 This document defines the Phase 0 domain and storage contract. The JSON Schemas under [`../schemas/v1/`](../schemas/v1/) are its machine-readable companion. Normative terms such as **must**, **must not**, and **should** describe behavior that future phases need to preserve. ## Product boundary Pi Reads captures source material, creates optional AI-authored reading documents, renders those documents, and delivers exports through adapters. The product has five persistent domain records: - **Source** — an immutable URL/feed/newsletter/clipboard/transcript/text/Markdown/file capture. - **Article** — a reading document in `archive`, `digest`, or `synthesis` mode. - **Citation** — a reference from generated prose to a captured source. - **Collection** — an ordered immutable reading pack of article snapshots. - **Export** — a reproducible artifact prepared for local, Obsidian, or Kindle delivery. Source capture and AI-authored output are separate operations and separate files. ## Modes | Mode | Meaning | Authorship | Source requirements | |---|---|---|---| | `archive` | Faithful readable representation after deterministic cleanup | Source author | Exactly one source; visible-text hash must match the source capture | | `digest` | Condensed or reorganized reading version | AI-authored | One or more sources; generation metadata and citations are required | | `synthesis` | New article combining multiple inputs | AI-authored | One or more sources; generation metadata and citations are required | An `archive` article must never contain AI rewriting. Extraction fixes belong in deterministic extraction code and result in a new immutable source capture when source content changes. A `digest` or `synthesis` article must never replace the source capture or an archive article. Generated prose must be labeled through the article mode and `generatedBy` metadata. ## Identifiers and immutability Identifiers are opaque and prefixed by record type: - `src_…` for sources - `art_…` for articles - `cite_…` for citations - `col_…` for reading collections - `exp_…` for exports Slugs are presentation values and are not identifiers. Duplicate slugs are allowed across record IDs and must be disambiguated at export time without overwriting an existing artifact. Completed source, article, collection, and export records are immutable. A correction or regeneration creates a new ID. Recaptured sources record predecessor/root lineage, and their archive articles point to the prior archive with `supersedesArticleId`; successors are derived by reversing those immutable edges. Writers must use create-only, atomic writes: 1. prepare content in a temporary sibling path; 2. calculate and verify hashes; 3. fail if the final record path already exists; 4. atomically rename the temporary path into place. ## Hashes Hashes use lowercase SHA-256 with a `sha256:` prefix. - `contentHash` hashes the exact stored bytes. - `textHash` hashes normalized visible text using the fidelity verifier's whitespace and block-boundary rules. For an archive article: - `article.archiveVerification.sourceId` must equal its only `sourceIds` entry; - `article.archiveVerification.sourceTextHash` must equal the source's `content.textHash`; - the article body's `textHash` must equal that same source text hash. These cross-record rules are enforced by the application layer because JSON Schema cannot compare values in separate records. ## Source contract A source records: - input kind and original locator; - canonical URL for URL input; - capture time and adapter provenance; - immutable readable Markdown; - exact-byte and normalized-text hashes; - optional raw capture and assets. Raw HTML is evidence, not working prose. It must never be edited in place. Raw captures and copied article content are local library data and must not be committed to this repository. Capture matching is deterministic. For URL input, cleaned canonical-URL matches take precedence: equal content hashes reuse the newest matching source/archive, while changed content is reported without persistence. Without a canonical match, equal content hashes also reuse the newest exact record. `recapture: true` is accepted only as an explicit user decision; it creates new source/archive IDs with `content-changed` or `explicit-duplicate` lineage and never edits predecessor bytes. Each source may have a deterministic `markdown-blocks-v1` structure index derived from `content.md`. The index records the source content/text hashes, UTF-8 byte length, stable content-derived heading and paragraph IDs, exact byte ranges, per-range hashes, heading ancestry, character counts, and approximate token counts. It contains no generated timestamp, so identical source bytes and manifests produce identical index bytes. The derived index may be verified or rebuilt atomically without modifying the source manifest, source prose, or archive article. `reads_library` can expose that index as a bounded outline, read an inclusive heading/paragraph locator range, or search a single source lexically. Retrieval defaults to an 8 KiB result budget and accepts explicit 1–32 KiB budgets. Returned excerpts are exact source substrings, include the source ID and stable locators, and are enclosed in `PI_READS_SOURCE_DATA` records marked as untrusted data rather than instructions. Budget exhaustion omits whole records or clips an exact UTF-8-safe prefix and reports `nextLocator` or `nextByte` continuation cursors. Local file paths may be retained in a local source manifest but must not be presented as public citations. URL citations use the source's canonical URL. Feed and newsletter adapters produce the same canonical draft contract before persistence. Feed entries retain cleaned canonical entry URLs and embedded entry payload evidence. Local newsletter sources retain the original `.eml` envelope as `message/rfc822` evidence while readable MIME body content passes through deterministic cleanup. Remote mailbox access and credentials are not part of source capture. Clipboard access is an interactive extension concern: core accepts only caller-supplied bytes, and `/reads` must obtain a separate confirmation immediately before one platform clipboard read. It must never poll or read in the background. Transcript sources are bounded local SRT/WebVTT files transformed into immutable Markdown sections headed by normalized start/end timestamps plus a deterministic segment ordinal. Timestamp headings and their stable source-index locators resolve exact citation ranges; raw subtitle bytes remain evidence. Live media/provider access is out of scope. See [clipboard and transcripts](clipboard-and-transcripts.md). ## Batch capture Batch capture accepts at most 50 independent inputs and preserves input order in its bounded outcome list. Every newly captured item gets a separate immutable source/archive pair and structure index; duplicates reuse verified IDs, while changed canonical content creates nothing and requires individual user-approved recapture. Batch capture never performs generation or delivery. Successful siblings remain after another item fails or is cancelled. Ordinary publication failures compensate only that item's new directories, including catalog-publication failures, and never overwrite/delete existing collisions. Cancellation propagates to active HTTP requests and is checked before publishing; a committed capture must not be reported as cancelled. Failed compensation is explicit and stops pending work. Cross-directory crash atomicity and unsynchronized external writers are not supported. Limits and error semantics are specified in [batch ingestion](batch-ingestion.md). Feed/newsletter collection capture adds a mandatory no-write preview. Bounded metadata clearly distinguishes library duplicates, changed canonical content, and duplicate/changed entries within the preview. The exact collection bytes and ordered entry identities are hash-bound to a preview token. Only explicit, unique selected indexes submitted with that token reach batch persistence; changed collections fail before writing, and recapture remains forbidden. See [feed and newsletter ingestion](feed-and-newsletter-ingestion.md). ## Private full-text search `indexes/search-v1.json` is a derived, local `bm25-lexical-v1` index over archive/generated prose plus titles, authors, canonical URLs, and forward-compatible tags/status. It stores weighted term frequencies and exact block byte ranges/hashes, not embeddings or a remote-service identifier. Queries run offline and may filter by mode, ISO date range, author, source ID, tag, and status. Results identify article mode, article/source IDs, field, stable locator, score, and an exact UTF-8 excerpt capped at 320 bytes. Archive body hits use source `h_…`/`p_…` locators; generated prose uses deterministic `b_…` locators. Tool output remains within the caller's 1–32 KiB budget and delimits excerpts as untrusted library data. Canonical manifests/content remain authoritative. Corpus and index hashes detect stale/corrupt indexes; search rebuilds atomically on demand or automatically after deletion, corruption, state change, or corpus change. The index contains no timestamp, so rebuilding unchanged records yields identical bytes. ## Reading state and queues Mutable article state is stored separately at `state/articles/.json`; source/article manifests and content never contain or change for reading progress. An absent state is virtual revision 0 with `unread`, no tags, and priority 0. Persisted records carry a monotonically increasing revision, `updatedAt`, status (`unread`, `reading`, `completed`, `archived`), canonical unique tags, optional rating 1–5, priority 0–5, and optional normalized due/read-later timestamps. Updates use an expected-revision compare-and-swap under a cross-process atomic lock, then atomically replace only the state record. Stale revisions fail closed. The default queue contains unread/reading articles ordered by descending priority, earliest due date, oldest article, then ID; explicit filters/sorts remain deterministic. Library list/metadata search and private full-text search can filter state without opening immutable manifests for mutation. A `user-state-snapshot` contains canonical persisted state records sorted by article ID. Restore first validates every referenced article and every collision, writes only absent records, accepts identical canonical state as unchanged, and refuses differing existing state before writing. Full portable library backups preserve those same persisted state records byte-for-byte alongside canonical records; credentials remain out of scope. ## Article contract An article records: - mode, title, slug, and source IDs; - an optional immutable versioned generation-template snapshot and bounded budget diagnostics; - immutable Markdown body and hashes; - citations; - creation time; - optional print presentation settings; - either archive verification or AI generation metadata; - verified source-coverage policy and per-source diagnostics for newly generated articles. `archiveVerification` is mutually exclusive with `generatedBy` and `sourceCoverage`. Generated articles record the active provider, model, generation time, and—when available—the Pi session ID and thinking level. Prompts and credentials are not persisted in the article manifest. Generation templates are structured metadata, not executable prompt text. Version 1 definitions select a digest/synthesis mode, 100–10,000 target-word range, 1–10 allowlisted section roles, complete/targeted coverage, and citation minima of 1–20 per section/from selected sources. User templates use `custom-` IDs, safe display labels, and a maximum of ten definitions. Unknown fields—including free-form instructions—fail closed. Built-in and user choices resolve by mode, snapshot origin/version into generated provenance, and produce bounded deterministic warnings for length, missing sections, citation density, and source use. Mode or coverage mismatch blocks persistence; warnings never bypass existing source, archive, coverage, or citation rules. See [generation templates](generation-templates.md). The existing presentation behavior maps into `article.presentation`: - `sourceFontStyle` - `bodyFontSizeAdjustment` - `imageScalePercent` ## Citation contract Generated Markdown references citation IDs using standard Markdown footnotes, for example `[^cite_example]`. Each referenced ID must have exactly one matching `Citation` in the article manifest. Every citation must reference one of the article's `sourceIds`. A citation should include the most precise available locator and may include a short supporting quote. Bounded source reads and searches suggest compact deterministic citation IDs plus stable locator fragments; generated articles may use those suggestions instead of inventing verbose identifiers. Stable `h_…`/`p_…` fragments, inclusive fragment ranges, one-based paragraph numbers, and exact heading labels resolve through the source index. Unknown, reversed, ambiguous, or mutually inconsistent locator fields fail closed. Exporters resolve source metadata from the source manifest and render destination-appropriate footnotes or endnotes. Before generated content is persisted, every provided quote must occur exactly in the immutable source text and, when an indexed locator is present, within that resolved source range. Unsupported or empty quotes fail closed. Missing locators remain allowed but are explicitly counted with bounded citation-ID examples. Persisted `citation-grounding-v1` diagnostics report citation counts per source/indexed section, verified quote counts, and generated article sections with and without citation markers. Every variable diagnostic list is capped at 20 with a truncation flag; diagnostics identify uncited sections but never rewrite or fact-check prose. A synthesis selecting 2–20 captured sources preserves explicit source order and requires a pre-persistence review. Duplicate source IDs are rejected. Every non-empty Markdown section must contain at least one registered citation marker; this structural gate is deliberately conservative and does not claim semantic fact-checking. The no-write review reports citation counts in selection order and all selected sources with zero citations. Persistence requires the review token for the exact mode/title/body/source order/citations/coverage and active provider/model/thinking/session identity; any change requires another review. Generation time is excluded from the token and recorded only on the final immutable article. See [multi-source cited synthesis](multi-source-synthesis.md). Generated articles choose one source-coverage policy. `complete` requires coverage evidence for every heading and paragraph locator in every source index and is required for `digest`. `targeted` records only selected sections, is limited to `synthesis`, and persists a warning that the result is not comprehensive. Coverage evidence is bound to each source content hash and index locator-set hash. Unknown, duplicate, stale, or incomplete evidence fails before article persistence. Targeted manifests retain missing-section counts and at most 20 stable missing locators plus a truncation flag. Coverage policy and warnings remain visible in generated exports; citations continue to point to original captured sources. For `digest` and `synthesis` modes: - at least one citation is required; - factual claims derived from a source should have a nearby citation marker; - invented or unresolved citations are validation failures. Archive articles retain source provenance through their sole source record and do not need per-paragraph citations. ## Reading collection contract A reading collection records 2–50 unique article IDs in explicit chapter order. Each ordered entry snapshots the immutable article's mode, title, body content hash, source IDs, citations, and generation provenance when present. Loading, EPUB rendering, verification, backup, and delivery fail closed if a snapshot no longer matches its referenced article. Collection EPUBs contain a navigation table of contents and exactly one XHTML spine item per ordered article. Each chapter displays its mode and source attribution, resolves its own citation markers/endnotes, and carries its required offline assets. Collections never merge article bodies or become source evidence. Interactive and scheduler-triggered preparation create only immutable collection and local EPUB records. The scheduler-safe CLI accepts explicit ordered article IDs and has no mail transport or send option. See [reading packs and scheduled Kindle preparation](reading-packs.md). ## Export contract An export records: - exactly one source article or collection ID; - format and local artifact; - optional copied assets needed by the artifact; - destination; - preparation or delivery status; - exact artifact hash; - non-secret delivery evidence. Exports are derived and may be regenerated under new export IDs. They never become the source of truth for article content. EPUB artifacts contain validated package/navigation/spine documents and embed the image bytes needed for offline reading. The Obsidian reading graph considers only delivered Markdown exports for the configured vault whose current note retains its `piReadsArticleId` frontmatter. It deterministically derives four fixed `Pi Reads/` views for the managed library, topics, reading status, and reading queue. Exported syntheses may receive a derived source-note link section so Obsidian exposes backlinks from exported archive notes; archive notes and immutable export artifacts are never changed. Rebuilds are byte-idempotent. Differing managed targets require exact-path approval, unmanaged path collisions are never overwritten, and write-time expected hashes reject edits made after preview. Kindle delivery is an external side effect. A successful Kindle export record must contain the prepared local export ID, interactive confirmation timestamp, and delivery timestamp. Dry-runs retain an immutable local EPUB or PDF and expose its export ID and content hash with only a redacted recipient outside the confirmation dialog. A later send verifies the requested article or collection, format, path, byte length, and hash, sends those exact prepared bytes, and records delivery evidence by reference rather than copying the attachment. SMTP credentials and full Kindle/sender addresses must come from the operating-system credential store or environment overrides and must not be written to JSON configuration, manifests, logs, article metadata, Pi tool results, or Git. ## Portable library maintenance Offline verification scans canonical manifests, referenced bytes, state, citations, lineage, archive fidelity, and derived indexes without repairing records or returning prose. Findings are bounded; corrupt canonical records block backup and rebuilding, while derived-cache problems can be rebuilt explicitly. A `portable-snapshot` is a versioned directory inventory of canonical relative file paths, exact hashes, and byte lengths plus allowlisted default mode/export-format preferences. Backups include referenced raw captures, assets, and export artifacts, not credentials, external destination configuration, locks, or arbitrary adjacent files. They are private plaintext reading data, not encrypted or authenticated storage. Restore requires an absent, separate destination outside Git. It never merges, overwrites, reassigns IDs, edits archived prose, or activates destination configuration. It validates the entire inventory and canonical relationships before writing, preserves record bytes and state revisions, then rebuilds local derived indexes. Portable config is inert until explicitly adopted. Maintenance requires stopped writers; interrupted newly reserved outputs may need manual removal before retrying. See [operational details and limits](library-maintenance.md). ## Library location resolution The application resolves the library root in this order: 1. an explicit API or CLI argument; 2. `PI_READS_LIBRARY_DIR`; 3. `libraryDir` in the resolved `pi-reads.json` configuration; 4. `~/Documents/pi-reads`. Configuration is resolved in this order: 1. an explicit config path; 2. `PI_READS_CONFIG`; 3. `$XDG_CONFIG_HOME/pi-reads/pi-reads.json` when `XDG_CONFIG_HOME` is set; 4. `~/.config/pi-reads/pi-reads.json`. Tilde expansion occurs only at the start of a configured path. Persisted record paths always use forward-slash, library-relative paths. Core services receive resolved paths and do not depend on Pi APIs or the process working directory. Configuration contains preferences, never credentials. Obsidian configuration may contain a vault path/name, inbox and attachment folders, note naming template, tags, custom scalar or string-array frontmatter, and an open-after-export preference. Kindle configuration may contain a device label, default format, SMTP host/port/TLS mode, credential backend/profile, and environment-variable names for recipient, username, password, and approved sender. Actual addresses and credentials belong in the operating-system credential store or environment overrides. Vault files are conflict-checked by hash and require confirmation before a differing target is overwritten. ## Library layout ```text / sources/ / manifest.json content.md raw/ # optional original bytes assets/ # optional captured assets articles/ archive/ / manifest.json content.md digest/ / manifest.json content.md synthesis/ / manifest.json content.md collections/ / manifest.json exports/ / / manifest.json indexes/ library.json # derived and rebuildable library metadata sources/ / structure-v1.json # deterministic heading/paragraph ranges ``` Physical mode directories make archive/generated path collisions impossible. The schema requires each article body path to match its mode directory. `indexes/library.json` is a cache, not canonical data. It contains source/article metadata needed for listing, search, canonical URL/hash lookup, date/mode filtering, and unique slug allocation. It may be deleted and rebuilt solely by scanning immutable manifests. Normal reads validate constant-time source and article-mode directory stamps rather than reopening each manifest. Index replacement is atomic, and an `indexes/dirty` interruption marker forces a rebuild after an incomplete canonical-record/index update. The application must refuse to use a Git working tree as a library root by default. An explicit unsafe override may be added for development fixtures, but production content must remain outside the package checkout. ## Schema evolution All records currently use `schemaVersion: 1` and validate against `schemas/v1`. - Adding an optional field may extend version 1 when old readers safely ignore it. - Changing required meaning, hashes, path semantics, or mode invariants requires a new schema version. - Migrations read an old record and create a validated new record or backup; they never silently mutate archived prose. - Unknown schema versions fail closed with an actionable error. ## Legacy compatibility After the Phase 5 rename: - existing `pnpm article:*` commands remain available; - `skills/irakli-reads` remains a compatibility entry point; - current `articles/.md` output remains supported by thin legacy wrappers; - core APIs, Pi tools, configuration, manifests, and package resources use `pi-reads` or stable `reads_*` names. Legacy Markdown can be imported as a version 1 source plus archive article. Import must preserve the existing `source`, `sourceTextHash`, and presentation frontmatter and must not modify the original file. ## Phase 0 acceptance decisions The following are fixed for version 1: 1. The default library is `~/Documents/pi-reads`, outside the installed package. 2. Sources, archive articles, generated articles, collections, and exports occupy separate immutable paths. 3. The only article modes are `archive`, `digest`, and `synthesis`. 4. Stable Pi tool names use the `reads_` prefix and the primary command is `/reads`. 5. User configuration is named `pi-reads.json` and stores no credentials. 6. Existing Irakli Reads print commands and skill remain documented compatibility surfaces after the Pi Reads rename.