# SkyKoi Runtime
> The engine behind a **koi**: a personal AI agent that lives on a device, talks to its
> owner across every messaging channel, runs real work with tools, and remembers.
This repository is the npm package **`skykoi`** (CLI binaries `koi` and `skykoi`). One
install bundles three things into one process: an **agent**, a **gateway**, and a
**multi-channel front door**.
```mermaid
flowchart LR
A["One npm install: skykoi"] --> B["Agent
reasons, calls tools, remembers"]
A --> C["Gateway
always-on WS / RPC server"]
A --> D["Front door
WhatsApp, iMessage, Telegram..."]
```
---
## Table of contents
- [What it is](#what-it-is)
- [System architecture](#system-architecture)
- [Core vocabulary](#core-vocabulary)
- [Quick start](#quick-start)
- [How the program starts](#how-the-program-starts)
- [Repository map](#repository-map)
- [Core subsystems](#core-subsystems)
- [The koi agent loop](#the-koi-agent-loop)
- [The gateway](#the-gateway)
- [Channels, routing and pairing](#channels-routing-and-pairing)
- [Providers and models](#providers-and-models)
- [Configuration](#configuration)
- [Memory](#memory)
- [Cron, infra and self-update](#cron-infra-and-self-update)
- [Device and browser](#device-and-browser)
- [Plugins, extensions and native apps](#plugins-extensions-and-native-apps)
- [Lifecycle of an inbound message](#lifecycle-of-an-inbound-message)
- [Where to add things](#where-to-add-things)
- [Build, test and release](#build-test-and-release)
- [Contributor conventions](#contributor-conventions)
- [Security model](#security-model)
- [Further reading](#further-reading)
> **How to read this doc.** It is the onboarding map, not the full manual. It is written
> to stay correct as the code moves, so it follows three rules. Please keep them when you
> edit: **point to the source of truth, do not duplicate it** (counts, versions and line
> numbers are deliberately avoided); **key on directories, not files** (directories are
> stable, files churn); and **explain the why**. The exhaustive manual is
> [docs.skykoi.com](https://docs.skykoi.com); contributor rules live in
> [`AGENTS.md`](./AGENTS.md). This README bridges the two.
---
## What it is
SkyKoi Runtime is the program that *is* a koi. It is published to npm and runs **wherever
the koi lives**: today that is primarily the **user's own device** (the macOS menubar app,
or any Mac/Linux/Windows host), reachable from the cloud through a tunnel. It can also run
on a server or VM. (It is host-agnostic, so do not assume EC2; an earlier managed fleet has
been retired.)
A single koi is an LLM-driven loop with a persistent **workspace** and **memory**. The
runtime wraps that loop in an always-on **gateway** and connects it to the outside world
through pluggable **channels**, the **CLI/TUI**, and native **apps**.
---
## System architecture
The whole runtime at a glance: every way a message can arrive, funneled into one agent loop
that is backed by a handful of subsystems.
```mermaid
flowchart TB
owner(["Owner"])
subgraph ways["Ways in"]
direction LR
CH["Channels
WhatsApp - iMessage
Telegram - Discord - Slack..."]
GWc["Gateway clients
web - mobile / desktop apps
platform connect"]
CLI["CLI / TUI
local terminal"]
end
owner --> CH & GWc & CLI
CH & GWc & CLI --> RP["Routing + Pairing
which koi? who is this?"]
RP --> KOI["The koi agent loop
prompt then tool then result"]
KOI --> PROV["LLM providers"]
KOI --> TOOLS["Tools
exec - browser - files - device"]
KOI --> MEM["Memory
sqlite + vectors"]
KOI --> WSP["Workspace
KOI / SOUL / USER / MEMORY.md"]
KOI --> CRON["Cron
scheduled turns"]
classDef hot fill:#0aa,stroke:#066,color:#fff;
class KOI hot;
```
Viewed as layers, the same system stacks like this. Surfaces depend on the gateway, which
hosts the agent, which rests on the foundational services:
```mermaid
flowchart TB
subgraph L4["Surfaces"]
direction LR
s1["CLI / TUI"]
s2["Native apps"]
s3["Channels"]
end
subgraph L3["Gateway · src/gateway"]
g1["WebSocket / RPC · sessions · heartbeat · control UI"]
end
subgraph L2["Agent · src/koi"]
k1["agent loop · tools · prompt · sessions"]
end
subgraph L1["Foundations"]
direction LR
f1["providers"]
f2["config"]
f3["memory"]
f4["infra"]
end
L4 --> L3 --> L2 --> L1
```
---
## Core vocabulary
These seven terms unlock the codebase.
| Term | Meaning |
|---|---|
| **koi** | One agent identity. Has a workspace, memory, model config, and sessions. A host can run several. |
| **gateway** | The always-on process. Hosts the WebSocket/RPC server, channels, cron, and the koi runtime. |
| **session** | One conversation thread with a koi, keyed by `(koi, channel, account, peer)`. Persisted as a transcript. |
| **channel** | A messaging surface (Telegram, iMessage, web...). A pluggable adapter translating native to internal messages. |
| **command** vs **tool** | A **command** is owner-typed chat text (`/model`, `/reset`) intercepted *before* the LLM. A **tool** is an action the LLM itself calls (`exec`, `browser`, `email`). |
| **the "Pi" runtime** | The embedded agent core (`@mariozechner/pi-*`). The koi loop wraps it; see `src/koi/pi-embedded*`. |
| **workspace** | `~/.skykoi/...`: the koi's editable brain on disk (`KOI.md`, `SOUL.md`, `USER.md`, `MEMORY.md`, sessions). |
---
## Quick start
> **Requirements:** Node `>= 22`, and **pnpm** (the version is pinned in `package.json`).
```bash
pnpm install
pnpm build # compile src/ to dist/ (tsdown). The CLI always runs dist/.
# Run the CLI from source during development:
pnpm dev # node scripts/run-node.mjs (rebuilds stale dist, then runs)
pnpm skykoi --help # list every command
# Common loops:
pnpm gateway:dev # run the gateway locally (channels skipped, fast)
pnpm gateway:watch # gateway with auto-reload on change
pnpm tui:dev # the terminal chat UI against a dev profile
```
Quality gates, before pushing anything with logic changes:
```bash
pnpm check # tsgo (typecheck) + lint + format
pnpm test # vitest suite (colocated *.test.ts)
```
> [!IMPORTANT]
> **`dist/` vs `src/`.** The `koi` binary (`skykoi.mjs`) imports `dist/entry.js`, so a code
> change is not live on the CLI until it is built. `pnpm dev` and `pnpm gateway:watch`
> rebuild for you; a bare `koi ...` runs whatever is already in `dist/`.
---
## How the program starts
Follow this chain once and the entire CLI makes sense.
```mermaid
flowchart TB
bin["skykoi.mjs
npm bin"] --> entry["src/entry.ts
title, env, re-spawn, profiles"]
entry --> main["src/cli/run-main.ts
dotenv, first-run, parse argv"]
main --> reg["src/cli/program/command-registry.ts
catalog of subcommands"]
reg --> cmd["src/commands/*
the command implementation"]
cmd --> deps["createDefaultDeps()
injected config + helpers"]
```
| Piece | Role |
|---|---|
| `src/cli/` | Commander-based CLI: program construction, argv helpers, help/version, and a **fast-path router** (`src/cli/route.ts`) that shortcuts common read-only commands past full parsing. |
| `src/cli/deps.ts` | `createDefaultDeps()`: the dependency-injection container (config, workspace, koi helpers) passed into commands so they are testable. |
| `src/runtime.ts` | The injectable `log` / `error` / `exit` surface, so tests can capture output. |
> The default action: `koi` with no args checks login, then drops into the **TUI** chat, or
> shows the first-run welcome on a fresh machine.
---
## Repository map
The product is a **pnpm workspace**: the root package (`skykoi`) plus `extensions/*` (plugins)
and `apps/*` (native apps). Start with `src/`.
| Path | What lives here | Start here |
|---|---|---|
| `src/entry.ts`, `src/index.ts`, `src/runtime.ts` | Process bootstrap + public library entry | `src/entry.ts` |
| `src/cli/`, `src/commands/` | The CLI: command registry + every subcommand | `command-registry.ts` |
| **`src/koi/`** | **The agent.** The Pi-embedded loop, tools, prompt assembly, sessions, models, sandbox | `pi-embedded-runner/run.ts` |
| `src/auto-reply/` | Inbound message to command-gating to koi turn to streamed reply | `reply.ts` |
| **`src/gateway/`** | **The server.** WebSocket + RPC, sessions, heartbeat, control UI, OpenAI-compatible HTTP | `server.impl.ts` |
| `src/gateway-client.ts` | The WS client apps/devices use to connect | `gateway-client.ts` |
| `src/channels/` | Channel **plugin system**: adapter interfaces, discovery, allowlists, gating | `plugins/types.core.ts` |
| `src/telegram/ discord/ slack/ signal/ imessage/ web/ whatsapp/ line/` | Per-channel adapters (native API to internal messages) | each dir entry |
| `src/routing/`, `src/pairing/` | Map `(channel, account, peer)` to `(koi, session)`; DM pairing and allowlists | `routing/resolve-route.ts` |
| `src/providers/`, `src/koi/models-config*.ts` | LLM provider adapters and model selection | `models-config.providers.ts` |
| `src/config/` | Config schema, load/validate `~/.skykoi/skykoi.json`, hot-reload, migrations | `zod-schema.ts`, `paths.ts` |
| `src/memory/` | Long-term memory: SQLite + vectors (`sqlite-vec`) + hybrid search | `manager.ts` |
| `src/world-model/` | The per-koi context block injected into the system prompt | `renderer.ts` |
| `src/cron/` | Scheduled, isolated koi turns | `service.ts` |
| `src/infra/` | Backoff/retry, update/self-update, exec-approval policy, heartbeat | by filename |
| `src/node-host/`, `src/browser/` | Device capabilities + Chrome automation (the `nodes` and `browser` tools) | `node-host/runner.ts`, `browser/client.ts` |
| `src/security/`, `src/gateway/device-auth.ts` | Auth tokens, device pairing, exec policy | `device-auth.ts` |
| `src/plugins/`, `src/plugin-sdk/` | The plugin loader + the public SDK `extensions/*` build against | `plugin-sdk/index.ts` |
| `src/tui/` | The terminal chat UI | `tui/` |
| `extensions/*` | Workspace plugins: extra channels and memory backends | each `package.json` |
| `apps/macos ios android shared` | Native clients (Swift/Kotlin) that speak the gateway protocol | per-app folder |
| `docs/` | The Mintlify docs site (docs.skykoi.com) | `docs/index.md` |
| `dist/` | **Build output. Generated, never edit.** | - |
> Some directories hold hundreds of files. The "start here" file is the door; the imports
> take you the rest of the way. A directory's own `README.md` or doc page outranks this table.
---
## Core subsystems
### The koi agent loop
`src/koi/` is the heart. A koi "turn" builds context, calls the LLM, runs any tools it asks
for, feeds results back, and repeats until done. It wraps the embedded **Pi** agent core
(`@mariozechner/pi-*`; local tweaks live in `patches/`).
```mermaid
flowchart LR
A["Build context
system prompt + world model
+ workspace + tools"] --> B["Call the LLM"]
B --> C{"Tool calls?"}
C -->|yes| D["Run tools
exec - browser - files - email..."]
D --> E["Feed results back"]
E --> B
C -->|no| F["Persist session
stream the final reply"]
classDef done fill:#0a6,stroke:#063,color:#fff;
class F done;
```
| Area | Where | Notes |
|---|---|---|
| Orchestration | `pi-embedded-runner/` | `run.ts` drives the loop; `attempt.ts` is one step; `compact.ts` trims history; `model.ts` picks the model and fails over. |
| System prompt | `system-prompt.ts` + `src/world-model/` | Assembled from runtime metadata, the world model, workspace files, tools, and channel rules. |
| Tools | `src/koi/tools/` | One file per capability: `exec`, `browser-tool`, file read/write, `email-tool`, `memory-tool`, `nodes-tool`, `cron-tool`, sub-agents, per-channel `*-actions`. |
| Sessions | `src/koi/` | Persisted per `(koi, sessionKey)`, guarded against concurrent writes and corruption. |
| Auth and models | `auth-profiles/`, `model-selection.ts`, `models-config*.ts` | Resolve provider/key/model and fail over when one is unavailable. |
| Sandbox | `src/koi/sandbox/` | Docker isolation for non-main (group/channel) sessions and risky tool runs. |
**The koi's brain on disk** is plain Markdown it reads each turn and can edit:
```mermaid
flowchart LR
subgraph ws["~/.skykoi workspace"]
direction TB
SOUL["SOUL.md
behavior + rules"]
KOIf["KOI.md
config / personality"]
IDN["IDENTITY.md
name / avatar"]
USR["USER.md + profile.json
who the owner is"]
MEMf["MEMORY.md + memory/*
long-term memory"]
HB["HEARTBEAT.md
live state / tasks"]
end
ws --> sp["into every system prompt"]
```
### The gateway
`src/gateway/` is the always-on process that exposes the koi to the world. `server.impl.ts`
boots the WebSocket server, an HTTP surface (control UI plus an OpenAI-compatible
`/v1/chat/completions`), the channel manager, cron, heartbeat, and **config hot-reload**.
```mermaid
flowchart TB
subgraph clients["Clients"]
direction LR
c1["macOS / iOS / Android apps"]
c2["skykoi-live web surface"]
c3["CLI / TUI"]
c4["platform connect"]
end
clients -->|"WebSocket + RPC"| gw
subgraph gw["Gateway process - src/gateway"]
direction TB
ws["WS server + RPC handlers
server-methods*.ts"]
chat["event fan-out
server-chat.ts"]
cm["channel manager"]
http["HTTP: control UI + OpenAI-compatible API"]
end
gw --> koi["koi runtime - src/koi"]
cm --> chans["Channels"]
```
> [!NOTE]
> The **wire protocol** schema is defined in TypeScript and generated to
> `dist/protocol.schema.json` (plus Swift models) via `scripts/protocol-gen*.ts`. Breaking
> changes require updating all clients together: run `pnpm protocol:check`.
### Channels, routing and pairing
Every messaging surface implements the **same adapter interfaces**
(`src/channels/plugins/types.core.ts`: messaging / auth / outbound / group / command /
setup). Channels are **discovered and lazy-loaded**; a configured channel that is not
installed is logged, not fatal. Built-ins live in their own top-level dirs; extra channels
are `extensions/*` packages built on the `plugin-sdk`.
```mermaid
flowchart TB
M["Inbound native message"] --> ENV["Envelope
normalized"]
ENV --> RES["resolve-route.ts"]
RES --> SK["session-key.ts
channel + account + peer + scope"]
RES --> PAIR{"known sender?"}
PAIR -->|no| P["pairing + allowlist
src/pairing"]
PAIR -->|yes| T["target koi + session"]
P --> T
T --> KOI["koi loop"]
```
> [!WARNING]
> When you change shared **routing / allowlist / pairing / gating** logic, change it for
> **all** channels at once. That is a recurring source of bugs (see `AGENTS.md`).
### Providers and models
Adapters for Anthropic, OpenAI, AWS Bedrock, Google Gemini, Ollama, and others, plus the
custom **"skykoi" provider** that proxies through the platform so a shared key never lands
on the device. The actual LLM calls go through the Pi core.
```mermaid
flowchart LR
sel["model-selection.ts
capability + cost + availability"] --> p1{"primary
available?"}
p1 -->|yes| use["use it"]
p1 -->|no| fb["fallback chain"]
fb --> use
use --> pi["Pi core makes the call"]
cfg["models-config.providers.ts
+ ~/.skykoi/models.json"] -.-> sel
```
### Configuration
Config is `~/.skykoi/skykoi.json` (JSON5: comments and trailing commas allowed, with
`${ENV}` substitution). `paths.ts` resolves locations; **`zod-schema.ts` is the runtime
source of truth** for valid shape; TypeScript types mirror it in `types*.ts`. Old formats
are upgraded by `legacy-migrate.ts`, and the gateway hot-reloads most changes.
> [!NOTE]
> The root `.env.example` is a stale leftover, ignore it. Real configuration is
> `~/.skykoi/skykoi.json` plus a handful of provider env vars; see the docs config pages.
### Memory
Long-term recall beyond the live context window: messages and notes are embedded and stored
in SQLite (`sqlite-vec`); retrieval fuses keyword (BM25) and vector similarity.
```mermaid
flowchart LR
msg["messages + notes"] --> emb["embeddings"]
emb --> db[("SQLite + sqlite-vec")]
q["koi memory search"] --> hyb["hybrid: BM25 + vectors"]
db --> hyb
hyb --> ctx["into the prompt"]
```
> Embeddings index asynchronously, so the newest items can lag the index slightly.
### Cron, infra and self-update
`src/cron/` runs scheduled, **isolated** koi turns (a reminder, a daily digest) with no live
channel input, delivering the result to a target. `src/infra/` is the cross-cutting glue:
backoff, the exec-approval policy engine, heartbeat visibility, and the **self-update** path
that pulls the latest npm version and restarts the gateway in place (how the fleet rolls
forward).
### Device and browser
`src/node-host/` exposes the local machine's capabilities (screenshot, screen frames,
location) that the koi drives via the `nodes` tool. `src/browser/` is full Chrome automation
over CDP plus Playwright, powering the `browser` tool and web-page capture.
### Plugins, extensions and native apps
The plugin loader (`src/plugins/`) discovers packages declaring a `skykoi.extensions` entry
and built against `skykoi/plugin-sdk`. This is how you add a channel or feature **without
touching core**. `apps/` holds the native clients (`apps/macos` is the menubar app that
hosts the gateway today); they are clients of the gateway protocol.
---
## Lifecycle of an inbound message
End to end, what happens when a message arrives from a channel:
```mermaid
sequenceDiagram
actor User
participant Ch as Channel adapter
participant AR as auto-reply
participant Rt as routing
participant Koi as koi loop
participant LLM as LLM provider
participant Tool as Tools
User->>Ch: native message
Ch->>AR: normalize to Envelope
AR->>AR: command? authorized?
AR->>Rt: resolve koi + session
Rt->>Koi: run turn
loop until done
Koi->>LLM: messages + tools
LLM-->>Koi: text or tool call
Koi->>Tool: execute tool
Tool-->>Koi: result
end
Koi-->>AR: streamed events
AR-->>Ch: final reply
Ch-->>User: delivered
```
The gateway/app path is the same picture, except the entry point is a `chat.send` RPC
instead of a channel adapter.
---
## Where to add things
| You want to... | Go to | Notes |
|---|---|---|
| Add a **tool** the koi can call | `src/koi/tools/` | One file per tool; register it in the runner's toolset. Mind the tool-schema guardrails in `AGENTS.md`. |
| Add a **CLI command** | `src/commands/` + register in `command-registry.ts` | Take `deps` from `createDefaultDeps()`. |
| Add a **gateway RPC method** | `src/gateway/server-methods*.ts` + protocol schema | Run `pnpm protocol:gen` and update clients. |
| Add a **channel** | `extensions//` against `plugin-sdk` (or a `src//` adapter) | Implement the `types.core.ts` interfaces; wire routing/allowlist/gating. |
| Add an **LLM provider/model** | `models-config.providers.ts` | Add the client init + model defs; selection picks it up. |
| Change the **system prompt / context** | `system-prompt.ts`, `src/world-model/` | Affects every koi; test broadly. |
| Change **config shape** | `zod-schema.ts` (+ `types*.ts`, add a migration) | Schema is the runtime source of truth. |
---
## Build, test and release
Everything is in `package.json` scripts (`pnpm run` lists them). The ones you will use most:
```bash
pnpm build # tsdown: src/ to dist/ (config: tsdown.config.ts)
pnpm check # typecheck + lint + format (oxlint / oxfmt)
pnpm test # vitest; pnpm test:coverage for V8 coverage
pnpm test:live # tests that hit real provider keys (SKYKOI_LIVE_TEST=1)
pnpm test:docker:* # end-to-end onboarding / gateway / install flows in Docker
pnpm protocol:check # fail if the generated wire protocol is out of date
```
Release and native-version details
- **Tests** are colocated `*.test.ts` (e2e: `*.e2e.test.ts`); the full kit is documented in
`docs/help/testing.md`.
- **Releases** are gated and manual: read `docs/reference/RELEASING.md` and
`docs/platforms/mac/release.md` first, and never bump versions or `npm publish` without
explicit owner sign-off (see `AGENTS.md`).
- **Native version bumps** touch several files at once: see `AGENTS.md` -> "Version
locations".
---
## Contributor conventions
[`AGENTS.md`](./AGENTS.md) is the detailed playbook (commit/PR flow, docs/i18n rules,
multi-koi git safety, channel-refactor checklists, release guardrails). The essentials:
- **TypeScript, ESM, strict.** Avoid `any`. Keep files reasonably small (~500 LOC guideline;
`pnpm check:loc`). Run `pnpm check` before you commit.
- **Add brief comments for non-obvious logic.** Match the surrounding style.
- **Touch one channel, consider them all** when the change is to shared routing / allowlist /
pairing / gating / onboarding logic.
- **Never edit `node_modules` or `dist/`.** Both are generated.
- **Dependency patches/overrides need explicit approval** and must be pinned to an exact version.
---
## Security model
```mermaid
flowchart TB
subgraph trusted["Host - full access"]
main["main session
tools run on the host"]
end
subgraph sandboxed["Docker sandbox"]
grp["group / channel sessions
+ risky tool runs"]
end
unknown(["Unknown sender"]) --> pair["DM pairing required"]
pair --> main
sensitive["Sensitive command"] --> approve["exec-approval system"]
approve --> main
device(["Gateway client"]) --> auth["device auth"] --> main
creds["provider keys"] --> proxy["platform proxy
keys stay off device"]
```
In one breath: the **main** session runs tools on the host with full access; **non-main**
sessions can be **Docker-sandboxed**; unknown senders must pass **DM pairing**; sensitive
commands go through **exec-approval**; **device auth** guards the gateway; and the shared-key
**platform proxy** keeps provider credentials off the device. Report vulnerabilities per
[`docs/SECURITY.md`](./docs/SECURITY.md).
---
## Further reading
- **Full docs:** [docs.skykoi.com](https://docs.skykoi.com) (sources in `docs/`: channels,
gateway, tools, providers, platforms, config, releasing).
- **Contributor playbook:** [`AGENTS.md`](./AGENTS.md).
- **The web/voice surface that talks to this runtime:** the `skykoi-live` repo (it connects
to this gateway for deep work).
## License
BUSL-1.1: see [`LICENSE`](./LICENSE) and
[`docs/THIRD_PARTY_LICENSES.txt`](./docs/THIRD_PARTY_LICENSES.txt).