# OKF v0.2 Interoperability Design

**Status:** Approved umbrella design; non-normative

**Date:** 2026-08-02

**Package:** `@zosmaai/pi-llm-wiki`

> This document records program-level goals and approved direction. Child specs own normative implementation requirements. If this document conflicts with a child spec, the child spec wins.

## Specification Map

1. [OKF Foundation](./2026-08-02-okf-foundation-design.md) — shared document model, dual-read behavior, OKF-canonical writes, vault mode, indexes, logs, and metadata integration. This is the only child currently ready for implementation planning.
2. **OKF Interchange** — import, review, transactions, export, and explicit migration. Write and review this child spec after Foundation is planned.
3. **OKF Intelligence** — trust-aware recall and expanded linting. Write and review this child spec after Interchange boundaries are stable.

MCP and Pi tools must call the same shared service operations defined by each child spec; no interface may implement separate business rules.

## Summary

pi-llm-wiki will become an Open Knowledge Format (OKF) v0.2 producer and consumer without forcing existing users to migrate their vaults. The extension will read both legacy pi-llm-wiki pages and OKF pages, write new pages in an OKF-canonical form, import external bundles through a review gate, export portable bundles, and offer an explicit migration command.

The first release focuses on interoperability, provenance, trust, freshness, stronger linting, and trust-aware recall. Graph visualization, git-backed snapshots, frozen reads, advanced removal/versioning, broader document extraction, and execution of attested computations remain future work.

## Goals

1. Make `.llm-wiki/wiki/` a conformant OKF v0.2 knowledge bundle for new and migrated vaults.
2. Preserve existing vault behavior without automatic migration.
3. Safely import, review, approve, and export external OKF bundles.
4. Preserve unknown types, producer fields, document bodies, and safe concept paths during round trips.
5. Make provenance, verification, lifecycle, and freshness visible during recall.
6. Extend `wiki_lint` into a deterministic OKF and wiki-health validator.
7. Keep extension and MCP behavior equivalent.

## Non-goals for the First Release

- Executing code referenced by an `Attested Computation` document.
- Defining a universal category taxonomy or registered type vocabulary.
- Automatically resolving semantic contradictions.
- Automatically committing wiki changes to git.
- Importing zip or tar archives. First-release imports accept directories; archive transport can be added later without changing the bundle model.
- Building a graph viewer or hosted service.
- Replacing the existing recall system with a new retrieval engine.

## Research and Competitive Findings

### OKF v0.2

OKF is a format, not a runtime or platform. A bundle is a directory tree of Markdown concept documents with YAML frontmatter. A document's path without `.md` is its concept ID. `index.md` and `log.md` are reserved. The only always-required concept field is `type`; consumers must tolerate unknown types and unknown producer fields.

Version 0.2 adds optional first-class metadata for:

- provenance through `sources`
- authorship through `generated`
- verification through `verified`
- lifecycle through `status`
- freshness through `stale_after`
- sanctioned, verifiable calculations through `Attested Computation`

OKF relationships use ordinary Markdown links. Directories provide hierarchy; links provide graph edges. Index files provide progressive disclosure.

### Pi package competitors

| Package | Strong points | Main limitations relative to this design |
|---|---|---|
| `pi-okf-wiki` | Deterministic intake of conformant Markdown, broad extraction support, TF-IDF query injection, generated hierarchical indexes, collision-safe archive, deterministic removal | Targets OKF v0.1, has no layered personal/project recall, no MCP, limited YAML subset, no v0.2 trust model |
| `llm-wiki-okf` | Query-first discipline, immutable raw sources, git-native workflow, strict linting, global/project tiers, source tracing | Skill and Python-script workflow rather than a Pi-native extension; no semantic retrieval; strict custom schema exceeds base OKF requirements |
| `@d1g1tlprim8/pi-okf-wiki` | Atomic single-file writes, git commits, graph checks, tag harmonization | Small skill utility set, custom required schema, no advanced retrieval, no safe bundle exchange workflow |

### Broader projects

- Google's reference implementation demonstrates two-pass enrichment, hierarchical indexes, sample bundles, and a self-contained graph viewer.
- `llm-wiki-compiler` demonstrates review-gated import, trusted import, preservation of foreign fields and paths, and OKF export.
- OKF Harness demonstrates bounded evidence retrieval, deterministic JSON tool output, workspace checks, and a local graph report.
- `wiki-as-an-mcp` demonstrates separate read/manage capabilities, frozen git-backed reads, snapshots, and multi-topic isolation.

The competitive advantage for pi-llm-wiki is combining OKF interoperability with its existing immutable source packets, layered personal/project recall, background ingestion, hybrid lexical/semantic retrieval, guardrails, observations, retrospectives, trajectories, and MCP access.

## Design Decisions

- Deliver both OKF interoperability and Pi-native wiki advantages in phases.
- Use dual-read, OKF-canonical write behavior.
- Never force automatic migration.
- Stage untrusted imports by default; support dry-run and explicit trusted mode.
- Keep recall relevance-first. Trust and freshness may rerank relevant candidates and add warnings, but never silently hide knowledge.
- Keep `category`, `domain`, `aliases`, and `recall_triggers` as optional producer extensions.
- Preserve and validate Attested Computation metadata but do not execute it.
- Restrict automatic lint fixes to deterministic repairs; never fabricate knowledge pages.

## Vault Architecture

```text
.llm-wiki/
├── config.json
├── WIKI_SCHEMA.md
├── wiki/                       # canonical OKF bundle
│   ├── index.md                # okf_version: "0.2"
│   ├── log.md
│   ├── sources/
│   ├── concepts/
│   ├── entities/
│   ├── syntheses/
│   ├── analyses/
│   └── ...                     # requirements, skills, cases, foreign paths
├── raw/                        # immutable pi-llm-wiki source packets
├── imports/                    # extension-owned import staging and audit records
│   ├── pending/<import-id>/
│   └── applied/<import-id>/manifest.json
├── meta/                       # durable local events + generated internal projections
└── outputs/                    # reports and explicit exports
```

`.llm-wiki/wiki/` is the distributable OKF bundle. Product-specific raw packets, staging state, and generated search metadata remain outside it.

`meta/events.jsonl` is durable local pi-llm-wiki state but is not part of the base OKF bundle. A full-vault backup preserves it; an OKF-only export does not. The exported `wiki/log.md` is therefore a readable history snapshot, not a lossless or resumable event source.

Source pages inside the bundle provide stable provenance targets for canonical pages. Raw packet paths may remain pi-llm-wiki extension metadata on source pages, but portable provenance references should resolve to source pages or external resources rather than escaping the bundle.

`imports/**` is extension-owned and protected by the same tool-call guardrail model as `raw/**` and `meta/**`. Pending imports never participate in recall or metadata indexing.

## Knowledge Model

### Terminology

In OKF, every non-reserved Markdown knowledge document is a concept. In the pi-llm-wiki profile, `type: concept` remains one specific page class beside `entity`, `source`, `analysis`, and other existing classes.

### Standard and profile fields

| Field | Role |
|---|---|
| `type` | Primary semantic kind; only always-required OKF field |
| File path | Stable concept identity and broad hierarchy |
| `title` | Human-readable display name |
| `description` | One-sentence search and index summary |
| `resource` | Canonical URI for the described asset, when applicable |
| `tags` | Cross-cutting labels |
| `sources` | Structured provenance and source credibility signals |
| `generated` | Who or what produced the current content and when |
| `verified` | Independent verification events |
| `status` | `draft`, `stable`, or `deprecated`; absent means stable |
| `stale_after` | Absolute date after which content is stale |
| `category` | Optional pi-llm-wiki subtype such as `architecture`, `person`, or `library` |
| `domain` | Optional knowledge area such as `security` or `ai-engineering` |
| `aliases` | Optional alternate names |
| `recall_triggers` | Optional phrases likely to be used when retrieving the page |

`category`, `domain`, `aliases`, and `recall_triggers` remain top-level producer extensions. They are optional, have no central vocabulary, and must be preserved by round-tripping consumers. The profile does not add subjective credibility or confidence scores; trust is derived from objective OKF signals.

Existing type values remain valid:

- `source`
- `entity`
- `concept`
- `synthesis`
- `analysis`
- `requirement`
- `trajectory`
- `case`
- `skill`

Unknown imported types remain valid and are treated as generic concepts by code that lacks type-specific behavior.

### Profile conventions

The root index declares only the base format:

```yaml
---
okf_version: "0.2"
---
```

OKF v0.2 is ambiguous about additional root-index frontmatter keys, so the bundle does not claim `profile: pi-llm-wiki/1` there. The package documentation still defines pi-llm-wiki extension fields and page-type conventions, but plain OKF bundles do not need them and imported documents never require them. A portable profile-discovery mechanism is deferred until OKF specifies one or a later child spec defines a conformant approach.

### Example

```markdown
---
type: concept
title: Retrieval-Augmented Generation
description: Grounds generation using retrieved evidence.
category: architecture
domain: ai-engineering
tags: [rag, retrieval]
aliases: [RAG]
recall_triggers: [grounded generation, document retrieval]
status: stable
generated:
  by: pi-llm-wiki/model
  at: 2026-08-02T10:00:00Z
sources:
  - id: SRC-2026-08-02-001
    resource: /sources/SRC-2026-08-02-001.md
---

# Retrieval-Augmented Generation

RAG retrieves evidence before generation.[^SRC-2026-08-02-001]

[^SRC-2026-08-02-001]: Source summary
```

## Shared Document Layer

A small shared format module becomes the only parser and serializer used by ingestion, metadata generation, recall, lint, migration, import, and export.

Conceptual API:

- `parseKnowledgeDocument(content, path)`
- `serializeKnowledgeDocument(document)`
- `parseBundleIndex(content)`
- `validateKnowledgeDocument(document)`
- `validateOkfBundle(bundle)`
- `resolveKnowledgeLinks(document, bundle)`
- `convertLegacyDocument(document)`

The internal model separates standard OKF fields, pi-llm-wiki extension fields, unknown producer fields, and the Markdown body. Unknown fields must survive semantic round trips. Exact YAML formatting, comments, quoting style, and key order are not guaranteed to survive migration or serialization.

### YAML handling

The current dependency-free parser cannot represent OKF v0.2 nested mappings and lists. The implementation will use a maintained YAML parser rather than grow a bespoke general YAML implementation.

Security requirements:

- disable aliases and alias expansion
- disable custom executable tags
- reject multiple YAML documents
- cap frontmatter bytes and nesting depth
- return structured parse errors rather than throwing through tool boundaries
- preserve unknown ordinary mappings, lists, and scalar values

Parsing is permissive; conformance validation is separate. A parseable document with unknown fields or type is accepted. A malformed document produces a file- and field-specific validation result.

## Link Model

New pages use ordinary Markdown links, preferably bundle-root-relative links such as `/concepts/retrieval.md`. Legacy `[[wikilinks]]` remain readable during the compatibility period.

Metadata and linting resolve both forms:

- standard Markdown links between concept documents
- bundle-root-relative and relative paths
- legacy folder-qualified wikilinks
- body footnotes whose labels join to `sources[].id`

Migration converts resolvable wikilinks into Markdown links while preserving display labels. Unresolvable links remain unchanged and are reported; migration must not guess a target.

## Generated Indexes and Logs

The metadata rebuild writes:

- a root `wiki/index.md`
- one `index.md` in each directory containing concepts directly or transitively
- a root `wiki/log.md`
- existing machine-oriented files under `meta/`

Each directory index lists only direct concepts and immediate child directories. This preserves progressive disclosure and avoids loading a recursive catalog into context.

Reserved `index.md` and `log.md` files are not concept documents and are excluded from ordinary concept recall. Imported reserved files are validated and recorded in the import manifest, but the live bundle regenerates its own indexes and log. Foreign concept paths and document-level metadata are preserved; arbitrary foreign index prose is not merged into the live generated index. Before Interchange implementation, its normative child spec must define whether an imported `log.md` is archived, retained as a separate historical baseline, or replaced when a new local event stream begins. It must not imply that Markdown prose can reconstruct the originating JSONL stream.

## Import Design

### Tool

`wiki_okf_import` accepts:

- a local directory path
- `dry_run`, defaulting to `true`
- `trusted`, defaulting to `false`
- collision policy, defaulting to `error`

First-release directory imports avoid archive extraction and decompression risks. A future archive adapter can feed the same validated staging pipeline.

### Flow

```text
external directory
→ bounded path and filesystem scan
→ parse all concept and reserved documents
→ validate OKF version and conformance
→ calculate content hashes and collision report
→ dry-run report OR immutable staged copy
→ explicit review
→ recoverable live-wiki transaction
→ metadata rebuild
```

### Safety constraints

- Reject paths that escape the selected bundle root.
- Reject symlinks and non-regular files in the first release.
- Cap document count, individual file size, total bytes, frontmatter bytes, and nesting depth with conservative defaults.
- Ignore non-Markdown files unless a future adapter explicitly supports them.
- Never overwrite a live concept silently.
- Treat identical path and content hash as a no-op.
- Treat same path with different content as a conflict.
- Validate trusted imports and enforce collision rules; trusted means bypassing review, not bypassing safety.
- Copy staged bytes and hashes so approval applies to the reviewed content even if the source directory later changes.

### Review

`wiki_okf_review` supports listing imports, showing a manifest and candidate diff, approving selected or all non-conflicting documents, rejecting an import, and explicitly resolving collisions as `skip` or `replace`.

`replace` requires an explicit choice and records the replaced content in the transaction backup. Rejected imports are removed only through this tool.

Applied import manifests retain source bundle metadata, file hashes, decisions, and timestamps under `imports/applied/`. They provide auditability but do not enter recall.

### Transaction behavior

Cross-file filesystem writes cannot be truly atomic. The implementation therefore uses:

1. validation before mutation
2. temporary files for each changed document
3. a transaction journal listing old and new hashes
4. backups of replaced files
5. atomic rename for each individual file
6. startup/tool-entry recovery when an incomplete journal exists
7. metadata rebuild only after the transaction commits

Any normal error rolls back changed files. An interrupted process leaves a journal that can deterministically finish or roll back on the next operation.

## Export Design

`wiki_okf_export` writes to a new or empty output directory and refuses to overwrite an existing non-empty destination.

Flow:

```text
live legacy + OKF pages
→ shared parser
→ in-memory legacy conversion
→ full bundle validation
→ deterministic hierarchical indexes and log
→ portable OKF v0.2 output
→ export report
```

Export never mutates the live vault. It preserves safe concept paths, bodies, unknown document fields, standard metadata, and pi-llm-wiki extension fields. Legacy pages are converted in memory. Generated `meta/**`, pending imports, source packet internals, and embeddings are not exported.

Because `meta/events.jsonl` is excluded, exported `log.md` is a deterministic snapshot rather than a resumable event ledger. Export must use only bundle-safe projected fields. Portable event continuity or a machine-readable event sidecar requires a separately reviewed Interchange decision and is not implied by Foundation.

Raw evidence remains represented through portable source concept pages and their provenance links. A later option may package selected original artifacts under an OKF `references/` convention.

## Migration Design

`wiki_okf_migrate` defaults to dry-run. Applying migration requires an explicit `apply` flag.

Migration performs these deterministic conversions:

- add or update root OKF version metadata
- generate hierarchical indexes and root log
- convert resolvable wikilinks to Markdown links
- convert scalar legacy source references into structured `sources` entries
- preserve existing `type`, `category`, `domain`, aliases, recall triggers, timestamps, bodies, and unknown fields
- derive `description` from an existing concise summary only when no description exists; preserve the original summary field
- leave optional `generated` and `verified` absent when authorship or verification is unknown rather than inventing trust

Dry-run reports every changed file, unresolved link, invalid source reference, and conformance issue. Apply creates a dated backup of changed files, writes through the transaction journal, validates the result, and rebuilds metadata. Re-running migration after success is a no-op.

## Recall and Trust

Existing lexical, chunk-level, pseudo-relevance-feedback, and optional semantic retrieval remain the candidate generators. Trust metadata does not create candidates.

For relevant candidates, recall computes and displays:

- human-reviewed
- machine-confirmed
- unverified
- verification-outdated when the latest verification predates the current generation timestamp
- stale when `today >= stale_after`
- deprecated when `status: deprecated`

A bounded trust factor may adjust a relevant candidate's base score within `0.90` to `1.10`:

- human-reviewed: positive adjustment
- machine-confirmed: smaller positive adjustment
- unverified: neutral
- stale: negative adjustment
- deprecated: larger negative adjustment
- verification-outdated: no verification boost and a warning

The final combined factor is clamped to the stated range. Tests must prove that trust cannot introduce a zero-relevance candidate or remove a relevant candidate from the result set solely because it is unverified, stale, or deprecated.

Recall output includes trust state, lifecycle warnings, and source links. Two-stage links-first behavior remains unchanged.

## Lint Design

`wiki_lint` becomes the single health and conformance entry point. It reports machine-readable findings with severity, stable code, file path, field or line where available, and human guidance.

Checks include:

- OKF v0.2 document and bundle conformance
- missing or empty `type`
- invalid nested `sources`, `generated`, `verified`, lifecycle, or computation fields
- invalid actor and timestamp shapes
- stale content and outdated verification
- body footnote IDs missing from `sources[].id`
- duplicate source IDs within a document
- escaping or unsafe path-valued fields
- broken Markdown links and legacy wikilinks
- missing or stale directory indexes
- malformed logs
- duplicate concept paths and import collisions
- existing orphan, gap, and explicit contradiction-marker checks

Unknown types, absent optional trust metadata, and unknown extension fields are not errors.

Automatic fixes are limited to deterministic operations:

- regenerate indexes and log
- rebuild registry and backlinks
- normalize metadata shapes when conversion is unambiguous

Lint no longer auto-creates stub concept pages. Missing knowledge remains a reported gap until a human or ingestion flow supplies evidence.

## Error Model

New OKF operations return stable result codes rather than relying on prose. Representative groups:

- `parse_invalid_frontmatter`
- `conformance_missing_type`
- `conformance_invalid_reserved_file`
- `path_escape`
- `path_symlink`
- `limit_file_count`
- `limit_file_size`
- `limit_total_bytes`
- `collision_changed`
- `transaction_incomplete`
- `transaction_rollback_failed`
- `migration_unresolved_link`

Tool responses provide a concise summary and store full reports under `outputs/` or import manifests. A failed validation, import, export, or migration does not leave partially indexed knowledge.

## Tool and MCP Surface

New Pi tools:

- `wiki_okf_import`
- `wiki_okf_review`
- `wiki_okf_export`
- `wiki_okf_migrate`

Existing tools updated:

- `wiki_lint`
- `wiki_recall`
- `wiki_search`
- `wiki_status`
- `wiki_rebuild_meta`
- all page-producing ingestion, observation, retro, requirement, and trajectory flows

The MCP server exposes equivalent OKF operations and structured result codes. Slash commands may wrap tools for interactive use, but business logic remains in shared library functions rather than command handlers.

## Testing Strategy

### Unit tests

- nested OKF v0.2 frontmatter parsing and serialization
- alias/custom-tag/multiple-document rejection
- preservation of unknown mappings, lists, and scalar fields
- Markdown and legacy link extraction/resolution
- trust-tier and freshness derivation
- deterministic index and log generation
- legacy source and link conversion

### Fixture tests

- conformant Google/reference-style v0.2 bundles
- current pi-llm-wiki legacy vaults
- unknown foreign types and producer extensions
- Attested Computation documents
- malformed frontmatter and reserved files
- path traversal, symlink, oversized file, excessive file count, deep nesting, and collision cases

### Integration tests

- dry-run import makes no writes
- staged content is independent of later source-directory changes
- approval indexes content only after commit
- rejection removes staged content and never affects live recall
- trusted import still validates and detects collisions
- export converts legacy pages without modifying them
- migration apply is idempotent and recoverable
- simulated interruption recovers or rolls back through the journal
- extension and MCP produce equivalent structured outcomes

### Recall tests

- existing lexical, chunk, and semantic tests remain green
- human-reviewed relevant pages receive only a bounded boost
- stale/deprecated relevant pages remain returned with warnings
- unverified pages are never silently hidden
- outdated verification is detected from timestamps
- pending imports never appear in recall

### Release gates

- Reference/sample OKF v0.2 bundles validate.
- Parse/serialize preserves document bodies, paths, unknown field values, and nested metadata semantically.
- Existing vaults work without migration.
- Migration dry-run is deterministic and apply is idempotent.
- Invalid imports leave live concept files unchanged.
- Security fixtures fail with expected stable codes.
- Existing package tests, typecheck, lint, and coverage gates pass.
- Pi extension and MCP operations remain behaviorally aligned.

## Rollout

1. Introduce shared document model and hardened YAML parsing without changing writes.
2. Switch all readers to dual legacy/OKF support.
3. Switch new page producers to OKF-canonical output.
4. Generate hierarchical bundle indexes and expand lint.
5. Add safe import, review staging, and export.
6. Add explicit migration.
7. Add trust/freshness-aware recall.
8. Publish profile documentation, migration guidance, and a competitor feature matrix.

Each stage is independently releasable. No stage forces migration.

## Future Roadmap

1. Self-contained graph viewer with backlinks and filters.
2. Git snapshots, frozen read versions, rollback, and separate read/manage capabilities.
3. Safe concept removal, deprecation workflows, and version history.
4. DOCX, PPTX, XLSX, EPUB, image, and other broader extraction adapters.
5. Archive import/export adapters with decompression-bomb protections.
6. Bundle registry and multi-topic isolation.
7. Optional Attested Computation executors and deterministic attesters behind explicit security boundaries.
8. Retrieval benchmark suite, competitive evaluations, and larger-scale indexing.
9. Versioned evolution of the `pi-llm-wiki/1` profile.

## Sources

- [Google Cloud: How the Open Knowledge Format can improve data sharing](https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing)
- [Open Knowledge Format v0.2 specification](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)
- [Google knowledge-catalog OKF reference implementation](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf)
- [Pi package: pi-okf-wiki](https://pi.dev/packages/pi-okf-wiki)
- [Pi package: llm-wiki-okf](https://pi.dev/packages/llm-wiki-okf)
- [Pi package: @d1g1tlprim8/pi-okf-wiki](https://pi.dev/packages/@d1g1tlprim8/pi-okf-wiki)
- [llm-wiki-compiler OKF round-trip guide](https://github.com/atomicstrata/llm-wiki-compiler/blob/main/docs/guides/open-knowledge-format.mdx)
- [OKF Harness](https://github.com/pumblus/okf-harness)
- [wiki-as-an-mcp](https://github.com/taikunudel/wiki-as-an-mcp)