# Dialects — design & build plan

*Status: SHIPPED in 1.2.0 (WP1–WP7; user docs live in
[SOURCES.md → Dialect lenses](SOURCES.md#dialect-lenses-read-a-repos-own-idiom)). This file
remains the design record, and the stance below is what it implements.*

## The stance

The stance, stated once (it governs everything below): **ztrack is to task lists what a
gradual typechecker is to untyped code.** TypeScript won by checking the JavaScript that
already existed — inference first, annotations as an incremental tightening, never a rewrite
as the entry price. Repos that have never heard of ztrack already write real task lists in a
handful of recurring *dialects* (checkbox rosters `- [x] **WS-A: …**`, status-emoji registers
`**Status**: 🟢/🟡/🔴`, numbered workstream sections `## 2. WS1 — title` with `**Acceptance:**`
prose bars, decision-log tables keyed `#N`, pre-registered experiment runbooks). The concepts
map one-to-one onto issues/status/ACs/relations; only the surface grammar differs. So:

- **Read first, rewrite never (by default).** A dialect parse is a read-only *lens*;
  `import` (materializing into native grammar) stays an opt-in migration step, not the
  price of admission. The gradient is lens → adopted (dialect + id aliases saved) →
  materialized (full rigor: evidence, ACs, gating) — value at every rung, each step opt-in.
- **Inference over annotation.** Absence of `status:` is an inference site, not an error —
  checkbox state, emoji vocabulary, and `**Acceptance:**` prose infer into status/ACs in the
  default profiles (simple-sdlc / simple-gh-sdlc); strict rigor levels may still demand declarations.
  Findings degrade gracefully: an inferred doc gets structural checks (unique ids, resolving
  relations, status consistency), not evidence discipline it never claimed.
- **The ids belong to the repo.** Never renumber someone's namespace: adopt native ids
  (`WS1`, `KQ3`) via per-source aliases so prose cross-references keep resolving. Stamping
  fresh ids over an existing namespace is data loss.
- **Dialects are data, not code.** A dialect is a declared per-source profile (id pattern,
  status vocabulary, AC marker, hierarchy convention) persisted on the source entry in
  tracker-config — auto-DETECTED and offered (never silently applied, same philosophy as
  `--register`), deterministic once saved.
- **Not everything is an issue.** Decision logs, macros, and tuning tables are what issues
  *cite* — dialect detection must recognize them to exclude and link them, not import them.
- **The conformance corpus is the definition of done.** Each supported dialect lives in the
  unit tests as a fixture pair (real-world-shaped input file → expected issues JSON), the way
  a compiler carries a conformance suite. The bar it operationalizes — what "instantly
  useful" means: point ztrack at a repo that never heard of it, and one command says
  something true and helpful about its real task lists while mutating nothing.

## The acceptance demo (what "instantly useful" means, operationally)

The reference case is a real repo that has never heard of ztrack (the shapes below are
distilled from an actual game-design repo). It keeps kill-questions in an emoji register
(`### KQ3 — …` + `**Status**: 🟢/🟡/🔴` bullets), build state in a checkbox roster
(`- [x] **WS-A: server core** — …`), and an append-only decision-log table (`| #17 | … |`)
that must NOT be treated as issues. The bar:

```
$ npx ztrack init --team AS
$ npx ztrack check
note: PREPRODUCTION.md parses under the 'emoji-register' dialect: 5 issues (KQ1 … KQ5).
      To track it as-is (read-only, file untouched):
        ztrack import PREPRODUCTION.md --register --dialect emoji-register
note: S1_BUILD.md parses under the 'checkbox-roster' dialect: 4 issues (WS-A … WS-D). …
$ npx ztrack import PREPRODUCTION.md --register --dialect emoji-register
$ npx ztrack import S1_BUILD.md --register --dialect checkbox-roster
$ npx ztrack issue list        # KQ3 done · KQ2 ready · WS-A done · … — true statuses
$ npx ztrack check             # structural truth, exit 0
$ git status                   # ONLY .volter/tracker-config.json changed. No file rewritten.
```

Two commands from first contact to a working tracker view, zero mutations to the repo's own
files. That demo ships as an e2e test over distilled fixtures.

## Vocabulary (three nouns, one invariant)

- **Dialect** — a declarative description (pure data) of how one file surface encodes
  issues: boundary, id, title, status vocabulary, hierarchy. Built-ins are named
  (`emoji-register`, `checkbox-roster`); a custom one is the same object written inline in
  the source entry. A dialect can never contain code.
- **Engine** — the ONE interpreter in core that applies a dialect to a file and emits
  ordinary `IssueRecord`s. Zero per-dialect branches, ever: adding a dialect adds data and
  fixtures, not an `if`.
- **Fixtures** — per dialect, an input file + the exact expected records JSON (plus
  *negative* fixtures: shapes that must NOT detect). The conformance suite is both the spec
  and the contribution interface.

The invariant that keeps the two-parser architecture clean: **dialects may only produce what
a backend could have produced; only presets decide what it means.** Dialects live entirely in
parser 1 (file → record). Parser 2 (the preset's body grammar — ACs, evidence, PR metadata)
never learns dialects exist.

## Why this is cheaper than it looks (verified leverage, all in the code today)

- `markdownDocument.ts#parseMarkdownDocument` already yields sections with level, parent,
  body, raw, line spans, AND parsed `checkboxItems` — the engine's structural substrate
  exists; extractors are a thin layer over it.
- **`GrammarPack` is the precedent** (`markdownDocument.ts`): a named registry of
  vocabulary-as-data (`markdown-ac`, `github-flavored`), config-extensible
  (`organization.grammar.extends` + `slotAliases`), unknown names ERROR. Dialects are the
  same pattern one level up — GrammarPack maps section headings→slots inside an issue;
  a dialect maps file surface→records. (Later, AC desugar reuses GrammarPack directly:
  `github-flavored` already maps "Tasks" → the acceptanceCriteria slot.)
- `readonly` already exists on the source config schema and is enforced by
  `DocumentSource` — the lens tier's write-refusal is plumbed.
- **The waiver post-filter is the leniency mechanism's precedent**: core already downgrades
  preset findings after rules run (error → acknowledged). Lens leniency is the same shape.
- `SAFE_ID` (`markdownBackend.ts`) accepts hyphenless ids — `KQ3`/`WS1` are valid record
  ids as-is. Only the *native document heading grammar* requires a hyphen, and lens sources
  don't use it. Native ids survive untouched until materialization.
- The config schema is strict Zod (`configSchema.ts`) — `dialect` lands in exactly one
  place, fail-closed on typos.
- `MarkdownBackend` already dispatches source format → class (`DocumentSource` vs
  `MarkdownSource`); a `DialectSource` is one more arm keyed on the source entry.
- Detection surfaces already exist: the file-target document-grammar note, the
  dark-sibling sweep, and `import --dry-run` (all shipped in 1.1.0).

## Work packages, in order

**WP1 — config + types.** `dialect?: string | InlineDialect` on the source entry
(configSchema, strict; `resolveSources` passes it through). Legal only with
`format: "document"`; v1 *implies* `readonly` (declaring `dialect` on a writable source is
a config error — one honest failure, no half-writable lens).

**WP2 — the engine + two dialects + the conformance corpus.** `src/dialects.ts`: the
extractor set (issue boundary: `heading@depth` | `checkbox-list-item`; id: pattern at
`heading-start` | `bold-lead`; title separator; status: `field-bullet` label + vocabulary |
`checkbox` | `header-block`; hierarchy: `heading-nesting` | `flat`; body verbatim), the
built-in registry (`emoji-register`, `checkbox-roster`), records stamped
`provenance: { status: 'inferred', dialect }`, diagnostics in the existing finding shapes
(`status_unrecognized`, duplicate id, id-less sections counted-and-skipped). Fixture pairs
including negatives (decision-log table, prose README, the `## Follow-up items` trap). A
registry-driven test asserts the engine has no dialect-specific branches.

**WP3 — `DialectSource`.** Same interface as `DocumentSource`, constructed by the backend
dispatch when the entry carries `dialect`; forced read-only (mutations answer: *"<file> is a
read-only dialect lens — edit the file directly, or materialize it with `ztrack import
<file>` to manage it through ztrack"*). `issue list` / `check` / `--source` work unchanged.

**WP4 — lens leniency as a core post-filter.** After preset rules run, `error` findings on
issues from a lens source downgrade to `warning` (annotated, waiver-style). Zero preset
changes — every existing user's forked preset gets correct lens behavior for free, and
loop/gate safety is automatic (warnings never fail the gate). This is deliberately chosen
over a provenance `if` inside preset templates: templates are copied into user repos, so a
preset-side fix would strand every existing copy.

**WP5 — detection + the offer.** Where the native-grammar note fires today (file-target
check, `import --dry-run`; full-check sweep only where document sources already exist —
same gating as the sibling sweep), also try each built-in dialect speculatively. Match
floor: ≥2 items that yield BOTH an id and a status; ties or below-floor → silence (guessing
wrong is worse than quiet). The offer is `ztrack import <file> --register --dialect <name>`
— a register-only mode that writes the config entry and NEVER touches the file (pinned by
the same config-untouched/file-untouched test pattern as the 1.1.0 registration offer).

**WP6 — materialize (the opt-in climb).** `ztrack import <lens-file>` (no `--dialect`)
converts to native grammar in place: ids kept verbatim when already grammar-legal (`WS-A`),
normalized when hyphenless (`KQ3` → `KQ-3`) with the alias recorded on the source entry
(`aliases: {"KQ3": "KQ-3"}`) so references resolve; statuses become `status:` lines;
`dialect`/`readonly` drop from the entry. Prose cross-references elsewhere in the repo are
never rewritten.

**WP7 — capstone.** The acceptance-demo e2e over distilled fixtures; docs (SOURCES.md
dialect section, README, the skill's authoring guidance, CHANGELOG); a manual smoke against
the real reference repo.

## Explicitly deferred

- **Relations** (`Blocks:` grammars vary too much) — structural checks only until a real
  corpus justifies a vocabulary.
- **AC desugar** (`**Acceptance:**` prose → AC grammar) — arrives as an explicit `desugar`
  block on a dialect, emitting native AC grammar into the body for parser 2 to judge;
  builds on GrammarPack unification.
- **`workstream-sections`** (the `## 2. WS1 — title` + `Tasks:`/`**Acceptance:**` shape) —
  the third dialect; needs section-number stripping and benefits from desugar. Decide after
  WP2 proves the extractor set.
- ~~**Dialect-aware dark-sibling sweep** — v1 sweep stays native-grammar-only.~~ Un-deferred
  during WP5: the sweep turned out to be one `detectDialect` call on the shared floor
  (`unregistered_dialect_sibling`, documentDiagnostics.ts), cheaper to ship than to fence off.

## Open questions (with recommendations)

- **Status vocabulary targets.** Built-in dialects map to the simple-sdlc state names
  (draft/ready/in-progress/in-review/done). That couples built-in *data* (not the engine) to
  the stock presets — acceptable, documented; a team whose preset renames states uses an
  inline dialect. Revisit only if it bites.
- **Emoji ambiguity** (someone's 🟡 means *blocked*): fixed per named dialect; the inline
  form is the remap. Do not make named dialects configurable per-key — that's an inline
  dialect with extra steps.
- **Provenance field shape** on `IssueRecord`: `provenance?: { status?: 'inferred' |
  'declared'; dialect?: string }` — additive, optional, absent everywhere today's paths run.