# Knowledge System

Dave's knowledge system provides structured, queryable storage for everything your PA learns. It is domain-based, supports linking between documents, and maintains indexes for fast retrieval.


## How it works

Knowledge is organised into domains -- logical groupings of related information. Each domain is a directory under `operator/knowledge/` containing markdown files. The domain registry at `operator/knowledge/domains.json` (seeded from `core/knowledge/scaffold/domains.json`) tracks all active domains.

### Domain structure

A domain directory contains:
- Markdown files (`.md`) -- the actual knowledge content
- An optional `index.json` -- domain-specific metadata and file listing
- Subdirectories for sub-topics as needed

### Example

```
operator/knowledge/
  cooking/
    index.json
    basics.md
    recipes.md
    techniques/
      fermentation.md
  finance/
    index.json
    budgeting.md
    investments.md
```

Each domain entry in `domains.json` looks like:

```json
{
  "id": "cooking",
  "name": "Cooking",
  "path": "operator/knowledge/cooking",
  "description": "Recipes, techniques, and culinary knowledge",
  "fileCount": 3,
  "lastUpdated": "2026-04-12T10:00:00Z"
}
```


## Linking conventions

Dave supports two linking formats for connecting knowledge across documents:

### Wiki-links

Use double-bracket syntax to link to other documents:

```
See [[fermentation]] for details on lacto-fermentation.
```

Wiki-links are resolved against the real file set, in this order:

1. A target containing a `/` (`[[../wespa/spi-no-staging]]`, `[[engineering/standards/secrets]]`) is tried relative to the linking document first, then repo-root-relative, then relative to each knowledge root. An over-deep `../` chain is salvaged when exactly one file can be meant.
2. A bare target (`[[fermentation]]`) matches on filename across all knowledge domains -- exact case first, case-insensitive second.
3. If several files share the name, the one in the linking document's own directory wins, then the one in its own domain. If neither disambiguates, the link is reported as ambiguous and produces no edge -- the builder does not guess, because a guess silently changes what `related`, `path` and `orphans` answer.

A wiki-link that resolves to nothing is reported as dangling and produces no edge. It is never turned into a node: a graph that invents a document for every broken reference inflates itself and corrupts its own orphan statistics.

### Standard markdown links

Use standard markdown links for explicit paths:

```
See [fermentation](techniques/fermentation.md) for details.
```

Markdown links resolve relative to the linking document. Unlike wiki-links, a markdown link to a file that is not present still produces a node -- that is deliberate, and is how a document the build host is not entitled to stays visible in the graph.

### Machine-readable hubs (`index.json`)

A directory can carry an `index.json` listing its documents with curated descriptions:

```json
{
  "version": "1.0",
  "patterns": [
    { "id": "some-pattern", "path": "knowledge/engineering/patterns/some-pattern.md",
      "description": "What it is and when it bites" }
  ]
}
```

Every `.md` path in such a file becomes an edge from the hub to the document, so a document listed in its domain's index is indexed for retrieval without also needing a prose link. Entry paths may be canonical-repo-relative (`knowledge/...`), repo-root-relative, or relative to the index file. A `path` naming a directory resolves to that directory's `README.md`. Hubs nest: each `index.json` is linked from the nearest `index.json` above it.

`index.json` files are hand-maintained source, not build output -- no builder writes them.


## Backlink indexing

The retrieval system maintains a backlink index -- for every document, it knows which other documents link to it. This enables:
- "What references this topic?" queries
- Adjacency graph traversal
- Orphan detection (documents with no inbound or outbound links)

The backlink index is rebuilt during knowledge reindexing (`npm run reindex`).


## Adjacency graph

The retrieval system builds a graph of document relationships based on links. This graph supports:
- Finding related documents (neighbours in the graph)
- Shortest-path queries between topics
- Cluster detection (groups of tightly linked documents)


## Querying knowledge

The retrieval tools at `core/knowledge/retrieval/` provide:

- **index-builder.mjs** -- Rebuilds the domain registry (`operator/knowledge/domains.json`)
- **backlink-indexer.mjs** / **graph-builder.mjs** -- Rebuild the per-device retrieval artefacts (`.backlinks.json`, `.graph.json`). These two are untracked and device-local: each host builds its own over the tree it can actually see, and `pa knowledge` rebuilds them on demand if they are missing. Do not commit them.
- **search.mjs** -- Full-text search across all knowledge
- **graph.mjs** -- Graph queries (neighbours, paths, clusters)

Your PA uses these tools automatically when answering questions. You can also trigger a manual reindex:

```bash
npm run reindex
```


## Orphan policy (path-class exclusions)

`pa knowledge orphans` lists documents with no inbound edges. Some document classes are operator+agent tier by design (local TODO, memory and note documents); a linkless document in one of those trees is correctly scoped, not an orphan to repair. The estate can declare those classes in an explicit policy file:

```
operator/knowledge/orphan-policy.json
```

```json
{
  "version": 1,
  "excludedPathClasses": [
    {
      "pathPrefix": "operator/knowledge/memory/",
      "reason": "One line explaining why this class is operator-local."
    }
  ]
}
```

Every orphan-counting surface honours the same policy: the `orphans` verb, anything that shells it for a count (nightly housekeeping), and the retrieval-eval orphan gauge and its alarms. The rules, all deliberate:

- **Explicit prefixes only.** Each entry is a listed path prefix with a mandatory one-line `reason`. Blanket ignores (the whole knowledge tree, wildcards) are rejected as malformed. The file is the audit trail for the exclusion.
- **The excluded amount is always rendered** (for example `orphans: 139 (87 excluded by policy: operator/knowledge/memory/ 44, ...)`). The number never silently shrinks.
- **Named trade-off:** a document misfiled INTO an excluded tree stops appearing as an orphan. That is the price of the exclusion, and it is why the policy is a human-edited config rather than a hardcoded ignore.
- **Counting only.** The policy does not touch librarian sweep judgement; genuine knowledge is still routed out of excluded trees.
- A missing policy file means no exclusions. A malformed one fails loudly rather than being ignored, because silently dropping the policy would snap counts back up and re-fire alarms.


## Adding new knowledge

1. Create a domain directory under `operator/knowledge/` if it does not exist.
2. Add markdown files with your content.
3. Use wiki-links or markdown links to connect related documents.
4. Run `npm run reindex` to update indexes, or let the PA handle it during normal operation.

The PA's librarian role can also be delegated to organise and structure knowledge from raw observations.


## How knowledge flows

1. **Observation** -- Your PA notices something relevant during a conversation and stages it as a timestamped markdown file.
2. **Staging** -- Staged observations accumulate in the PA's staging directory.
3. **Distillation** -- The librarian role processes staged observations, extracting durable knowledge and updating the operator profile.
4. **Knowledge** -- Distilled content is filed into the appropriate domain under `operator/knowledge/`.
5. **Indexing** -- The retrieval system rebuilds indexes to include the new content.

This cycle runs continuously. Every interaction is a potential source of new knowledge.