# @yolk-sdk/knowledge

Domain-free knowledge document/source/file/context, ingestion, hybrid search, and lookup/manage tool helpers for agents.

## Install

```bash
pnpm add @yolk-sdk/knowledge@canary @yolk-sdk/agent@canary effect@4.0.5
```

Canary APIs are unstable. Keep all `@yolk-sdk/*` packages on the same version.
Use the SDK's matching Effect version (`4.0.5`) in host code.
Published package metadata requires Node.js 22+.

## Model

```txt
KnowledgeDocument
  id
  slug
  title
  purpose
  origin
  content
  summary?
  errorMessage?
  reviewedAt?
  metadata?
  status: processing | ready | error
  availability: pinned | searchable | archived
  createdAt
  updatedAt
```

- `pinned`: host may inject into startup context and search.
- `searchable`: host may expose through search tools.
- `archived`: retained, normally omitted from prompt and search.

Optional files and chunks are modeled as `KnowledgeFile` and `KnowledgeChunk`. File records carry
metadata and host-owned blob keys; `KnowledgeFileBlobStore` stores the bytes separately. Chunks are
search-index records. Hosts own blob storage, chunk persistence, transactions, and lifecycle.

`KnowledgeSource` is a separate ingestion ADT. Build variants with `Schema.TaggedStruct` constructors
from `@yolk-sdk/knowledge/documents`:

- `KnowledgeFileSource.make({ ref, name?, mediaType? })` — host-owned `ref` plus optional name/media type
- `KnowledgeUrlSource.make({ url })` — host-owned URL string
- `KnowledgeTextSource.make({ label? })` — optional label; bytes/text arrive through `LoadedKnowledgeSource.content`

`IndexedKnowledgeDocument` is the search-index record. It carries `source`, indexing `status`, and
optional title/summary/error/hash/count metadata. It is distinct from the content-oriented
`KnowledgeDocument` managed through `KnowledgeStore`.

Status and availability are independent:

- `processing`: saved/indexing, not ready for prompt or search
- `ready`: usable when availability allows
- `error`: retained with an ingestion/indexing failure
- `pinned`: host may include in startup context and search
- `searchable`: host may expose through search
- `archived`: retained but normally omitted from prompt and search

These are contracts, not automatic policy. `buildKnowledgeContext` formats exactly the documents
you pass; `searchKnowledge` delegates candidate filtering to `SearchIndexStore`. Hosts must select
ready/pinned documents and exclude processing, error, or archived records as appropriate.

`KnowledgeScope` and `KnowledgeSearchScope` are opaque, caller-provided routing metadata. Hosts
construct and enforce them; the package does not infer identity, tenancy, or permissions from scope
ids or kinds. The **value** `KnowledgeSearchScope` is the single-id constructor
(`KnowledgeSearchScope.make({ id })`, tag `KnowledgeScope`). `KnowledgeSearchScopes.make({ ids })`
is the non-empty-ids variant. The **type** `KnowledgeSearchScope` remains the union of both.
Scope and document references use the branded `KnowledgeScopeId` and `KnowledgeDocumentId`
(non-empty trimmed strings, same wire representation). Hosts Effect-decode raw external/persisted
ids at their boundary and use `.make` for trusted constants and minted values.

## Subpaths

| Subpath                             | Purpose                                                                |
| ----------------------------------- | ---------------------------------------------------------------------- |
| `@yolk-sdk/knowledge`               | Root context/store/file/error helpers                                  |
| `@yolk-sdk/knowledge/documents`     | Document, source, file, chunk, status, availability, and scope schemas |
| `@yolk-sdk/knowledge/files`         | File blob-store contract                                               |
| `@yolk-sdk/knowledge/store`         | Document store and search-index store contracts                        |
| `@yolk-sdk/knowledge/context`       | Pinned context builder                                                 |
| `@yolk-sdk/knowledge/chunking`      | Text chunker contracts and defaults                                    |
| `@yolk-sdk/knowledge/embeddings`    | Embedder contract and vector types                                     |
| `@yolk-sdk/knowledge/extraction`    | Source/extractor contract                                              |
| `@yolk-sdk/knowledge/ingestion`     | Extract/chunk/embed/summarize/index pipeline                           |
| `@yolk-sdk/knowledge/search`        | Vector/hybrid chunk search helpers                                     |
| `@yolk-sdk/knowledge/summarization` | Optional title/summary service contract                                |
| `@yolk-sdk/knowledge/agent`         | `knowledge_lookup` and `knowledge_manage` tool factories               |
| `@yolk-sdk/knowledge/errors`        | Shared error types                                                     |

## Imports

```ts
import { buildKnowledgeContext, KnowledgeStore } from '@yolk-sdk/knowledge'
import {
  KnowledgeFileSource,
  KnowledgeSearchScope,
  KnowledgeSearchScopes,
  KnowledgeTextSource,
  KnowledgeUrlSource,
  type KnowledgeDocument
} from '@yolk-sdk/knowledge/documents'
import { KnowledgeFileBlobStore } from '@yolk-sdk/knowledge/files'
import { makeKnowledgeLookupTool, makeKnowledgeManageTool } from '@yolk-sdk/knowledge/agent'
```

## Use pinned knowledge as context

```ts
import { buildKnowledgeContext } from '@yolk-sdk/knowledge/context'

const context = buildKnowledgeContext({
  documents,
  maxCharacters: 6000
})
```

Add the resulting text to your system prompt. Your app chooses which documents are pinned.

## Agent tools

`makeKnowledgeLookupTool` creates the read-only `knowledge_lookup` tool with `search` and `get`
operations. `makeKnowledgeManageTool` creates the write-classified `knowledge_manage` tool with
`upsert`, `set_availability`, `rename_slug`, and `delete` operations.

```ts
import { makeKnowledgeLookupTool, makeKnowledgeManageTool } from '@yolk-sdk/knowledge/agent'

const lookupTool = makeKnowledgeLookupTool({ search: searchKnowledge, get: getKnowledge })
const manageTool = makeKnowledgeManageTool({
  upsert: upsertKnowledge,
  setAvailability: setKnowledgeAvailability,
  renameSlug: renameKnowledgeSlug,
  delete: deleteKnowledge
})
```

The callbacks are host adapters and receive the agent tool context. Hosts own authorization,
storage, scope enforcement, and safe `ToolError` mapping.

Both tools declare an output (`KnowledgeLookupOutput`, `KnowledgeManageOutput`) and return its JSON
encoding as `structuredContent` (dates as ISO strings) next to the text, so code mode scripts get
`{ operation: 'search', results }`, `{ operation: 'get', document }`, or
`{ operation, document: { id, slug, title } }`. A handler value that does not encode fails the call
with an `execution` `ToolError`. Unknown argument keys are model-visible validation errors with a
hint naming the allowed keys, not stripped.

## Ingestion semantics

`ingestKnowledgeDocument` upserts a `processing` index record, extracts, chunks, embeds and
summarizes concurrently, checks embedding cardinality, calls `replaceDocumentChunks`, then marks
the index record `ready`.

`DefaultKnowledgeChunkerLive` uses the bundled `gpt-tokenizer` `o200k_base` encoding for default
chunk boundaries and token counts. Hosts can provide another `KnowledgeChunker` layer when they
need model-specific tokenization.

Failures are returned as `KnowledgeIngestionError` with a `store`, `extract`, `chunk`, `embed`, or
`summarize` stage. The pipeline then best-effort calls `markDocumentError` with the error message;
failure to mark the error is suppressed so the original failure survives.

The pipeline is not a transaction and does not update `KnowledgeStore`. The host
`SearchIndexStore` adapter owns atomic chunk replacement, rollback/transaction behavior, ready/error
filtering, and any coordination with the content document store.

## Host responsibilities

- Own users, workspaces, permissions, routing, and product policy.
- Implement `KnowledgeStore` with app storage.
- Implement `KnowledgeFileBlobStore` with R2/S3/blob storage when files are needed.
- When using ingestion, provide `KnowledgeExtractor`, `KnowledgeChunker`, `KnowledgeEmbedder`,
  `KnowledgeSummarizer`, and `SearchIndexStore` layers. Summary/title values may be absent, but the
  current pipeline still requires the summarizer service.
- Own concrete extraction, embeddings, search SQL/vector storage, and upload URLs.
  `@yolk-sdk/extractors/knowledge` provides a `KnowledgeExtractor` for PDF, DOCX, XLSX, PPTX, and
  text files (`FileKnowledgeExtractorLayer`) on top of its Node `FileExtractor`.
- Keep slug uniqueness scoped to the host boundary.

## Boundaries

- No app auth, DB drivers, object storage SDKs, React, Next.js, or provider SDKs.
- Package owns simple schemas, contracts, context formatting, tool factories, chunking, and search helpers.
- Host owns all IO and policy.