pi-spark
Reliable distributed execution for agent workloads — without forking Pi.
A PostgreSQL control plane, pull-based workers, and a thin
Pi Extension
for durable remote runs, capability routing, leases, fencing, retries, and cancellation.
Quick start ·
Architecture ·
Embed a worker ·
Pi Extension ·
中文使用指南
> [!IMPORTANT]
> pi-spark is an **experimental distributed run engine**. APIs may change before the first stable release. The control plane enforces scoped client credentials and per-Worker identities, but it does not terminate TLS or provide rate limiting. Put every cross-machine deployment behind a trusted TLS/mTLS gateway or service mesh.
## Why pi-spark
| Thin integration | Durable coordination | Safe reassignment |
| --- | --- | --- |
| Integrate through REST, the worker library, or Pi's public Extension API. | PostgreSQL is the source of truth for runs, idempotency, events, placement, and retries. | Workers pull outbound, heartbeat a lease, and settle only with the current token and fencing epoch. |
The control plane is independent from any agent UI. Applications can submit work over REST, embed workers in their own process, place workers on separate machines, and route each run by executor ID and labels.
### Available today
- Durable submit, inspect, retry, cancel, and event APIs, with idempotency-key conflict detection.
- Pull-based worker registration and routing by executor ID plus arbitrary string labels.
- Database-backed idempotent claim replay, lease heartbeats, fencing epochs, bounded retry attempts, jittered reconnect backoff, cooperative cancellation, and graceful shutdown.
- Built-in deterministic `mock` and optional Pi SDK executor drivers.
- A driver SDK for registering application-specific executors without changing the control plane.
- Scoped client credentials, owner-isolated runs, and per-Worker credentials bound to capability ceilings.
- A Pi Extension that restores unfinished polling and injects terminal results into the originating session.
- Docker Compose verification for routing, idempotency, claim-response loss and replay, retry, fencing, concurrency, and cancellation.
## Quick start
Requirements: Node.js 24+, npm, Docker, and Docker Compose.
Install the published client, Worker runtime, or embeddable Control Plane:
```bash
npm install --global @anrans001/pi-spark-cli
npm install @anrans001/pi-spark-worker @anrans001/pi-spark-driver-sdk
npm install @anrans001/pi-spark-control-plane
```
The public packages are ESM-only. Run `pi-spark --help`,
`pi-spark-worker --help`, or `pi-spark-control-plane --help` after installation.
To develop from source:
```bash
npm ci
npm run build
npm test
```
Run the complete Docker E2E stack:
```bash
npm run test:docker
npm run docker:down
```
`test:docker` first recreates the dedicated Compose database volume, creates random local-only credentials under `.cache/docker-auth` when they are missing, reuses a complete valid bundle when present, then verifies both a fresh authenticated stack and a real upgrade from the legacy v1 schema to the current schema. The verifier drops an already-committed claim response at the TCP boundary, replays the same request concurrently, and checks conflict and stale fencing behavior. `docker:down` removes the checkout-scoped PostgreSQL and credential volumes, run history, and generated local credentials.
Keep the stack running for manual exploration:
```bash
npm run docker:up
export PI_SPARK_CLIENT_TOKEN_FILE="$PWD/.cache/docker-auth/client-primary.token"
node packages/cli/dist/index.js workers
node packages/cli/dist/index.js submit \
"hello from pi-spark" \
--kind mock \
--label slot=a \
--idempotency-key demo:hello:001 \
--json
node packages/cli/dist/index.js status
```
All CLI commands require exactly one of `PI_SPARK_CLIENT_TOKEN` or `PI_SPARK_CLIENT_TOKEN_FILE`. They accept `--url`; otherwise the URL comes from `PI_SPARK_URL`, then `CONTROL_PLANE_URL`, and finally `http://127.0.0.1:3000`. Tokens are intentionally not accepted as command-line arguments.
`docker:up` builds before invoking the idempotent `docker:auth` generator or reading credentials. It stops the existing application containers and creates service-scoped credential volumes only for the requested services and their dependencies, then streams the required bytes over stdin to a short-lived, network-disabled materializer. The materializer has only `CAP_CHOWN`; it seals each volume directory as UID/GID 1000 mode `0500` with regular mode-`0400` files. Application containers mount only their own volume read-only and continue to run as the non-root `node` user with a read-only root filesystem and no capabilities.
Canonical host files remain mode `0600`. Plaintext values are not placed in Compose YAML, container environments, image layers, command arguments, or logs; during a running local stack, a second plaintext copy exists in checkout-scoped Docker volumes and is removed by `npm run docker:down`. On one host, the lifecycle commands serialize concurrent operations, validate volume ownership labels, and pin a checked local Unix/npipe Docker endpoint. Remote Docker daemons are rejected: run lifecycle commands on each Docker host and connect remote Workers through the control-plane protocol. Custom credential paths must resolve outside the Docker build context. To rotate deliberately, run `npm run docker:stop`, then `npm run docker:auth -- --force`, followed by `npm run docker:up`. This local bridge caps the registry at 64 KiB. Production Swarm, Kubernetes, or another scheduler should use its native secret facility instead.
For complete CLI, REST, worker, Pi SDK, remote-machine, and environment-variable instructions, see the [中文使用指南](./docs/usage.zh-CN.md).
## Architecture
```mermaid
flowchart LR
APP["Your application"] -->|REST / CLI| CP["Control plane"]
PI["Pi session"] --> EXT["pi-spark Extension"]
EXT --> CP
CP <--> DB[("PostgreSQL")]
WA["Worker A"] -->|register · claim · heartbeat · settle| CP
WB["Worker B"] -->|register · claim · heartbeat · settle| CP
WA --> EA["Registered executor drivers"]
WB --> EB["Registered executor drivers"]
```
The worker connection is outbound: a remote worker only needs network access to the control plane. A claimed run carries a lease token and a monotonically increasing epoch. After lease expiry, another worker may reclaim the run, while the stale worker is fenced from committing.
The driver boundary deliberately excludes lease credentials and settlement APIs. A driver receives validated task input, worker context, and an `AbortSignal`; the worker runtime retains ownership of heartbeats, fencing, and result settlement.
See [Architecture and failure model](./docs/architecture.md) for the current invariants.
## Runnable examples
The [two-worker routing example](./examples/two-worker-routing) calls the REST API directly, waits for both workers, submits two tasks concurrently, and verifies label placement: `slot=a` runs on `worker-a`, while `slot=b` runs on `worker-b`.
```bash
npm run docker:up
PI_SPARK_CLIENT_TOKEN_FILE="$PWD/.cache/docker-auth/client-primary.token" \
npm run example:two-worker-routing
npm run docker:down
```
The example exercises the same pull, placement, lease, and settlement protocol used by workers on separate hosts.
The [Pi native Extension example](./examples/pi-native-extension) starts Pi
with the exact published npm package, disables every unrelated local resource,
and exposes only `spark_delegate`, `spark_status`, and `spark_cancel`. Against
the default Docker stack, ask it to submit `kind=mock` with `slot=a` and wait
for the automatic follow-up:
```bash
npm install --global --ignore-scripts \
@earendil-works/pi-coding-agent@0.80.10
npm run docker:up
export PI_SPARK_CLIENT_TOKEN_FILE="$PWD/.cache/docker-auth/client-primary.token"
npm run example:pi-native-extension
```
## Embed through REST
Applications do not need to load the Pi Extension. Submit a durable run directly:
```bash
(
set -eu
umask 077
curl_header="$(mktemp)"
trap 'rm -f "$curl_header"' EXIT
{
printf '%s' 'authorization: Bearer '
cat "$PWD/.cache/docker-auth/client-primary.token"
printf '\n'
} >"$curl_header"
curl -sS -X POST http://127.0.0.1:3000/v1/runs \
--header "@$curl_header" \
-H 'content-type: application/json' \
-d '{
"idempotencyKey": "demo:api:001",
"task": {
"kind": "mock",
"payload": { "prompt": "hello", "delayMs": 100 }
},
"placement": { "labels": { "slot": "a" } },
"maxAttempts": 3
}'
)
```
The subshell keeps the bearer out of exported environment variables and `curl` arguments; its `EXIT` trap removes the mode-`0600` temporary header file.
Use `GET /v1/runs/:id` for state, `GET /v1/runs/:id/events` for durable events, and `POST /v1/runs/:id/cancel` for cooperative cancellation.
### Authentication registry
Tokens use `ps1..`. Generate one without printing the plaintext secret to standard output:
```bash
npm run auth:create -- app-a-2026-07 /absolute/path/outside/repository/app-a.token
```
The command creates the token file exclusively with mode `0600` and prints its `id` plus `secretSha256` registry fragment. Prefer a token path outside the repository and every Docker build context; `.dockerignore` is defense in depth, not a secret store. `PI_SPARK_AUTH_CONFIG_FILE` must point to an absolute, versioned JSON registry. Client principals carry a subject, scopes, and a mandatory submit policy when `runs:submit` is present. Worker principals bind one Worker ID to allowed executors, exact labels, and maximum concurrency. See the [Chinese guide](./docs/usage.zh-CN.md#61-身份scope-与-owner) for a complete registry example.
Owner-scoped permissions are `runs:read`, `runs:cancel`, and `runs:retry`; the corresponding `:any` scopes are operator privileges. `workers:read` reveals Worker topology. Different credentials may share a principal during key rotation, while their key IDs and secret digests remain unique.
Upgrading a v1 database to the current schema is an offline Control Plane operation. Stop every Control Plane instance before starting exactly one current instance: the owner migration locks and rewrites `runs` while replacing global idempotency uniqueness. After it completes, start the remaining current instances. Historical v1 runs receive an internal empty owner sentinel that cannot be configured as an authenticated subject. They are accessible only to operator credentials with the relevant `runs:read:any`, `runs:cancel:any`, or `runs:retry:any` scope; an ordinary client cannot impersonate that sentinel.
For an existing v2 deployment, roll out v3 in Control-Plane-first order. Upgrade and verify every Control Plane replica while existing Workers continue to use the legacy `POST /v1/workers/claim`; only then roll the Workers to the idempotent `PUT /v1/workers/claim-requests/:claimRequestId` protocol. A new Worker that reaches an old Control Plane fails closed on `404` and never falls back to the non-idempotent route. Roll back in the reverse order: Workers first, then Control Plane replicas.
## Embed a worker
An executor ID is an open, operator-controlled routing key such as `mock`, `pi`, `codex`, or `acme/code-review.v1`. Runs can only be claimed by workers that both register a driver for the ID and advertise it in `WORKER_EXECUTORS`.
Install the Worker runtime and driver contract:
```bash
npm install @anrans001/pi-spark-worker @anrans001/pi-spark-driver-sdk
```
See the runnable [custom executor worker](./examples/custom-executor) for a complete source-checkout example with shutdown handling.
Implement `ExecutorDriver` from `@anrans001/pi-spark-driver-sdk`, then inject the driver into `SparkWorker`:
```ts
import { ExecutorError, type ExecutorDriver } from "@anrans001/pi-spark-driver-sdk";
import { loadConfig, SparkWorker } from "@anrans001/pi-spark-worker";
type EchoPayload = { prompt: string };
const echoDriver: ExecutorDriver = {
manifest: {
id: "echo",
apiVersion: 1,
displayName: "Echo",
},
parsePayload(payload) {
if (
payload === null ||
typeof payload !== "object" ||
!("prompt" in payload) ||
typeof payload.prompt !== "string"
) {
throw new ExecutorError({
code: "INVALID_PAYLOAD",
message: "payload.prompt must be a string",
retryable: false,
});
}
return { prompt: payload.prompt };
},
async execute(input, context, signal) {
if (signal.aborted) throw signal.reason;
return {
executor: "echo",
workerId: context.workerId,
prompt: input.payload.prompt,
};
},
};
const config = loadConfig({
...process.env,
WORKER_EXECUTORS: "echo",
});
const shutdown = new AbortController();
await new SparkWorker(config, {
drivers: [echoDriver],
includeBuiltIns: false,
}).run(shutdown.signal);
```
`loadConfig` requires exactly one of `PI_SPARK_WORKER_TOKEN` or `PI_SPARK_WORKER_TOKEN_FILE`. Each credential is bound server-side to one Worker ID, an executor allowlist, exact routing labels, and a maximum concurrency. A Worker cannot self-promote those capabilities in its registration payload.
Driver IDs contain at most 128 characters, use lowercase letters, digits, `.`, `_`, `/`, and `-`, and each slash-delimited segment must start and end with a letter or digit. Duplicate IDs and advertised-but-unregistered drivers fail during worker startup.
Driver results must be plain JSON values with at most 1,000,000 UTF-8 bytes, depth 64, and 100,000 visited values. Circular references, custom prototypes, hidden properties, symbols, `BigInt`, and non-finite numbers are rejected before settlement. Driver lifecycle hooks receive an `AbortSignal` and are bounded by `DRIVER_LIFECYCLE_TIMEOUT_MS`.
`ExecutorError.details` is persisted only when it is JSON-safe and within 64,000 bytes, depth 32, and 10,000 visited values; otherwise the worker replaces it with an omission marker.
Keep driver selection in trusted deployment configuration. A submitted task chooses among already registered IDs; it cannot cause a worker to import arbitrary code.
## Use from Pi
Install the published native Pi package, start a compatible worker, and then
launch Pi with the Control Plane client credential:
```bash
pi install npm:@anrans001/pi-spark-pi-extension@0.1.0
```
```bash
PI_SPARK_URL=http://127.0.0.1:3000 \
PI_SPARK_CLIENT_TOKEN_FILE=/absolute/path/client.token \
pi
```
Use `pi install npm:@anrans001/pi-spark-pi-extension@0.1.0 -l --approve` for a
project-local package, or try it for one process without changing settings:
```bash
PI_SPARK_URL=http://127.0.0.1:3000 \
PI_SPARK_CLIENT_TOKEN_FILE=/absolute/path/client.token \
pi -e npm:@anrans001/pi-spark-pi-extension@0.1.0
```
For an isolated, runnable local-stack walkthrough, see the
[Pi native Extension example](./examples/pi-native-extension). The package's
`pi.extensions` manifest points Pi at the compiled entry automatically; the
manual `npm install` plus filesystem `-e` form is only needed for source-level
development.
The Extension registers three model tools:
```text
spark_delegate({ task, kind?, payload?, labels?, maxAttempts? })
spark_status({ runId })
spark_cancel({ runId })
```
`spark_delegate` returns a durable run ID with `terminate: true`. Pi can become idle while the worker runs; the Extension polls in the background and injects the terminal result as a follow-up turn. It derives a stable idempotency key from Pi's session ID and tool-call ID, persists tracked runs, and restores unfinished polling on `session_start`.
The Extension snapshots its client credential during a session and rebuilds the client on every `session_start`. After rotating a client token file or environment value, start or restart the Pi session so unfinished polling resumes with the new credential.
> [!NOTE]
> The CLI defaults to `kind=mock`, while `spark_delegate` defaults to `kind=pi`. The Compose workers advertise only `mock`, so use `kind: "mock"` when testing the Extension against the default stack.
For real Pi execution, install the optional SDK peer in the dedicated worker image or deployment directory:
```bash
npm install --no-save --package-lock=false \
@earendil-works/pi-coding-agent@0.80.10
```
Then configure `WORKER_EXECUTORS=pi`, the required isolated `PI_WORKSPACE_ROOT`, an explicit `PI_ALLOWED_TOOLS` allowlist, and the provider credentials used by the Pi SDK. A task can only choose a subset of the worker allowlist; the default allowlist is empty. Do not put model credentials in the control-plane image.
## Packages
| Package | Responsibility |
| --- | --- |
| `@anrans001/pi-spark-protocol` | Schemas, API records, lease identities, and shared client errors |
| `@anrans001/pi-spark-driver-sdk` | Executor-driver contracts and JSON-safe result validation |
| `@anrans001/pi-spark-control-plane` | Fastify API, PostgreSQL state machine, placement, and lease reaper |
| `@anrans001/pi-spark-worker` | Pull loop, driver registry, concurrency, heartbeats, cancellation, and settlement |
| `@anrans001/pi-spark-pi-extension` | Public Pi API bridge, durable run tracking, and result injection |
| `@anrans001/pi-spark-cli` | Operator commands: `submit`, `status`, `cancel`, and `workers` |
| `@anrans001/pi-spark-e2e` (private) | Repository-only Docker Compose verifier for routing, retry, fencing, and cancellation |
## Security
- The Control Plane fails closed unless `PI_SPARK_AUTH_CONFIG_FILE` points to a valid static registry. It stores SHA-256 credential digests, applies client scopes, isolates runs by subject, and binds each Worker identity to a capability ceiling.
- Bearer credentials do not encrypt traffic. Never expose the HTTP port directly to the internet; use TLS/mTLS at a trusted gateway or service mesh and redact `Authorization` there as well.
- Static registry changes take effect after a Control Plane restart. Rotate credentials by overlapping two key IDs for the same principal, move clients or Workers, then remove the old key.
- Authentication is not rate limiting. Enforce request and connection limits at the trusted ingress.
- An agent executor can invoke tools with the worker's filesystem, network access, and credentials. `PI_WORKSPACE_ROOT` bounds the Pi executor's initial `cwd`; it is not a complete sandbox.
- Delivery is at-least-once with stable IDs. External effects such as deploys, messages, and writes still require their own idempotency key or reconciliation.
- Cancellation is cooperative. Fencing prevents a stale worker from committing, but cannot undo an external side effect that already occurred.
Please report vulnerabilities privately as described in [SECURITY.md](./SECURITY.md).
## Documentation
- [中文使用指南](./docs/usage.zh-CN.md)
- [Architecture and failure model](./docs/architecture.md)
- [Two-worker routing example](./examples/two-worker-routing)
- [Contributing](./CONTRIBUTING.md)
- [Security policy](./SECURITY.md)
Contributions are welcome. Start with [CONTRIBUTING.md](./CONTRIBUTING.md) and keep pull requests focused, tested, and explicit about compatibility changes.