# runway [![npm version](https://img.shields.io/npm/v/@valescoagency/runway?logo=npm)](https://www.npmjs.com/package/@valescoagency/runway) [![License](https://img.shields.io/npm/l/@valescoagency/runway)](https://github.com/ValescoAgency/runway/blob/master/LICENSE) [![CI status](https://img.shields.io/github/actions/workflow/status/ValescoAgency/runway/release.yml?label=release)](https://github.com/ValescoAgency/runway/actions/workflows/release.yml) [![Provenance](https://img.shields.io/badge/provenance-signed-blue?logo=sigstore)](https://www.npmjs.com/package/@valescoagency/runway) A small CLI for two jobs: **scaffold** a target repo for autonomous coding-agent runs, then **drain** a Linear queue against it. Wraps [Sandcastle](https://github.com/mattpocock/sandcastle) (Claude Code inside Docker), [varlock](https://varlock.dev) + 1Password for zero-secrets-at-rest, and the `gh` CLI for PR creation. ## Eight commands | | | |---|---| | `runway dash` | Bring up the operations dashboard (`up` / `logs` / `stop`). Wraps the published `ghcr.io/valescoagency/runway-dashboard` image so any runway-using project can run the dashboard without cloning runway. Ports bind to `127.0.0.1` only. | | `runway doctor` | Read-only preflight diagnostic: host tooling, env vars, repo state, and the agent docker image. Use when something stopped working and you want a sanity report. `--json` for CI / scripted health checks. | | `runway init` | Scaffold the cwd repo for runway: write `.sandcastle/Dockerfile` + (tier 2) `.env.schema` with op:// references. Run **once per target repo**. | | `runway planner` | Run the planner daemon — a long-running host process that will own cross-PR shepherding and queue selection per [ADR 0004](docs/adr/0004-planner-daemon-architecture.md). VA-472 epic in flight; VA-473 ships the foreground scaffold (`runway planner run` boots a tick loop, no capabilities wired yet). | | `runway review` | Run an IRA retrospective pass. `runway review run --drain --issue ` grades one issue-process post-drain — loads the captured agent + reviewer reports from the dashboard, fetches hindsight (PR merge state, human review-thread comments, Linear follow-ups in the past 48h), pulls rolling norms for the issue's category (last 30 drains, via the VA-399 read-model), asks an Anthropic model for a structured `Run Review` with absolute + relative grading axes, writes it to a `runway-meta` Linear project + the dashboard's `meta_reviews` table. Scheduler-agnostic; usually invoked from cron or GitHub Actions ~18h after a drain. The drain-age delay is enforced by `RUNWAY_REVIEW_DELAY_HOURS` (default 18, constrained to the 12–24h band) — pass `--force` to override for one-off operator bypass. **`runway review drain --id `** (VA-406) grades a whole drain: reads every per-issue Run Review for the drain, asks the model for a structured `Drain Review` covering composition / sequencing / cross-issue patterns, files it in `runway-meta`. When the model marks a finding `severity: critical` (drain-unsafe or captured-data-lost), the IRA also escalates a `Bug` + `runway-meta-promoted` issue into the runway-repo project (set `RUNWAY_REPO_PROJECT_NAME` to scope by project; otherwise team-level). The Drain Review fires automatically in-process the moment every Run Review for a drain has landed — manual invocation is for scheduled / catch-up runs. | | `runway run` | Drain a Linear queue. For each issue carrying the `ready-for-agent` label: branch, agent works, sub-agent reviews, PR opens (or `ready-for-human` label). Run **whenever you want a batch of work done**. | | `runway upgrade` | Update the runway CLI itself: `git pull` the local clone, `pnpm install`, typecheck. `--check` for a dry-run, `--force` to override dirty/branch refusals. | | `runway upgrade-repo` | Re-render the cwd repo's runway scaffold against the current vendored templates. Use after a runway version bump that changed the Dockerfile or template shape — `init` writes them, `upgrade-repo` keeps them current without re-prompting for op:// values. | `runway --help` for the full usage; `runway --help` per command. `runway init` and `runway upgrade-repo` are siblings: `init` is the first-time scaffold (you provide the op:// vault/items), `upgrade-repo` is the re-render (extracts those values from the existing `.env.schema` so you never re-type them). Override the extracted values via `--op-vault=…`, `--anthropic-item=…`, `--gh-token-item=…` if needed, or use `--check` for a CI dry-run that exits 1 on drift. ## What runway is and isn't | It is | It isn't | |---|---| | A small Node CLI that orchestrates Sandcastle runs from a Linear queue | A sandbox or agent runtime — Sandcastle does that | | The replacement for the AFK label-handler / preflight / contracts pipeline | A governance layer; trust comes from your review of the PR, not gates | | Repo-agnostic — runs in whatever cwd you launch it from | Multi-repo (yet) — one repo per `runway run` invocation | | Sequential | Parallel (yet) | ## Architecture ``` Linear (label=ready-for-agent, team=VA) ↓ poll runway (this CLI, on your Mac, run from inside the target repo) ↓ for each issue │ removeLabel(ready-for-agent) # claim signal — VA-423; skip │ # if already absent (lost race) │ sandcastle.run({ agent: claudeCode, sandbox: docker, cwd: process.cwd(), ... }) │ iter 1 → IMPL: DONE | IMPL: BLOCKED — | IMPL: CONTINUE │ iter 2 → same, with previous iteration's summary injected │ … │ → branch agent/, commits, tests │ │ if BLOCKED → HITL (skip review) │ else: │ sandcastle.run({ ..., prompt: review template }) │ → REVIEW: APPROVED | REVIEW: REJECTED — │ ├── approved → git push → gh pr create (Linear's GitHub integration │ auto-transitions: PR open │ → In Progress, merge → Done) └── rejected → Linear comment with reason, then `ready-for-human` label ↓ next issue [runway] per-issue outcomes: VA-312 APPROVED → PR opened https://github.com/.../pull/42 VA-313 HITL Sub-agent review rejected: TOCTOU race in … ``` ## Prerequisites - macOS or Linux - Docker Desktop (or Podman) - Node 22+ - `gh` CLI authenticated against the org that hosts your target repo - Linear API key with read+write on the team you're targeting - A Claude Code credential — **either** an Anthropic API key (`sk-ant-api03-…`, pay-per-token) **or** a Pro/Max OAuth token (`sk-ant-oat01-…`, generated via `claude setup-token`). The two are not interchangeable — see "Claude Code auth modes" below. Stored in the **target repo's** `.sandcastle/.env` (tier 1) or 1Password (tier 2); never in runway's own env. ## One-time setup per target repo ```bash cd /path/to/your/repo runway init \ --op-vault=runway \ --anthropic-item=anthropic-api-key \ --gh-token-item=gh-token \ --auth-mode=api-key # or --auth-mode=oauth for Pro/Max tokens ``` (No `--op-account` — runway uses 1Password service-account auth (`OP_SERVICE_ACCOUNT_TOKEN`) exclusively, and the token already encodes the tenant. `op://` URIs runway writes are `op:////credential`, not `op:////`. The `/credential` field selector is required for `API_CREDENTIAL` items, which is the canonical 1Password category for API keys.) This runs `npx sandcastle init`, patches the generated `.sandcastle/Dockerfile` to bake in `varlock` + the 1Password CLI + a `claude` shim, scaffolds `.env.schema` at the repo root with op:// references to your 1Password vault, and deletes `.sandcastle/.env` so no secrets sit at rest. Review the diff, commit on a feature branch, open a PR. Pass `--tier=1` if you want the plain Sandcastle setup with `.sandcastle/.env` and no varlock (faster but secrets land on disk). Architecture walkthrough: [`docs/secrets-with-varlock.md`](docs/secrets-with-varlock.md). ## Claude Code auth modes Claude Code accepts two distinct credentials, and they are **not interchangeable** — passing one as the other yields a generic `Invalid API key` inside the container with no useful diagnostic. | Mode | Env var | Token shape | Source | |---|---|---|---| | `api-key` (default) | `ANTHROPIC_API_KEY` | `sk-ant-api03-…` | [Anthropic console](https://console.anthropic.com), pay-per-token | | `oauth` | `CLAUDE_CODE_OAUTH_TOKEN` | `sk-ant-oat01-…` | `claude setup-token` on your Pro/Max account | Pick whichever matches what's stored in your 1Password item: ```bash runway init --tier=2 --op-vault=runway \ --anthropic-item=claude-pro-oauth-token \ --gh-token-item=gh-token \ --auth-mode=oauth ``` The `--anthropic-item` flag is the 1Password item name regardless of mode; only the env var written into `.env.schema` changes. `runway doctor` surfaces the resolved mode under Environment (`claude auth mode: oauth (…)`), and fails fast if `.env.schema` ends up with both env vars at once. If you switch modes later, run `runway upgrade-repo` — it extracts the existing op:// references, re-renders the template with the new mode (detected automatically from the schema), and writes back. You do not need to re-pass the op:// flags. ## Secrets — recommended: varlock + 1Password If you don't want any secret sitting at rest in any `.env` file, runway integrates [varlock](https://varlock.dev). Two layers (host + in-container), both opt-in. Full walk-through: [`docs/secrets-with-varlock.md`](docs/secrets-with-varlock.md). TL;DR for the host layer: ```bash varlock run --schema /path/to/runway/.env.schema -- runway --max 3 ``` Without varlock, runway falls back to plain `process.env` and sandcastle reads `.sandcastle/.env` per its docs. ## Signed agent commits (optional) Runway can sign every commit the agent produces so the PR lands on GitHub with a **Verified** badge. Off by default; opt in per target repo by populating four extra refs in `.env.schema`. Setup (one-time per bot): 1. Generate an Ed25519 SSH signing keypair for the runway agent: ```bash ssh-keygen -t ed25519 -C "runway-agent" -N "" -f ~/.ssh/runway_bot ``` 2. Store the keypair plus the bot's git identity in 1Password under `op:///runway-signing-ssh`: | field | value | |-----------|--------------------------------------------| | `private` | contents of `~/.ssh/runway_bot` | | `public` | contents of `~/.ssh/runway_bot.pub` | | `name` | the bot's git `user.name` (e.g. `Runway Agent`) | | `email` | the bot's git `user.email` (must match the GitHub agent account) | 3. Register the public key as a **Signing Key** on the GitHub account that owns the agent's `GH_TOKEN` (this is what makes GitHub render Verified — both Authentication and Signing Keys are managed under Settings → SSH and GPG keys). 4. Uncomment the four `RUNWAY_SIGNING_*` lines at the bottom of `.env.schema` (the file `runway init --tier=2` wrote at your repo root). The lines come pre-filled with `op://` references matching the layout above. Validate with `runway doctor`: the **Environment / agent commit signing** check goes from `warn (off)` to `ok` once all four refs are declared. Tear down by re-commenting the four lines — runway falls back to plain unsigned commits. Design rationale + future migration path (GitHub App identity) is in [`docs/adr/0003-agent-commit-signing.md`](docs/adr/0003-agent-commit-signing.md). ## Install ```bash pnpm add -g @valescoagency/runway # or npm i -g, yarn global add ``` Export runway's own env (in your shell rc, or wherever you keep API keys): ```bash export LINEAR_API_KEY=lin_api_... # Optional overrides: # export RUNWAY_LINEAR_TEAM=VA # export RUNWAY_LINEAR_PROJECT= # optional, scopes queue to one project # export RUNWAY_BASE_BRANCH=master # optional, overrides auto-detected default branch # export RUNWAY_READY_LABEL="ready-for-agent" # export RUNWAY_HITL_LABEL="ready-for-human" # export RUNWAY_MAX_ITERATIONS=5 # VA-482: the four RUNWAY_SHEPHERD_* env vars have been retired. # Mergeability / CI / reviewer-feedback work now flows through the # `runway planner` daemon (capabilities VA-477..479). Boot the # planner separately (`runway planner run`) and configure it with # the RUNWAY_PLANNER_* env vars; the planner reads from the # dashboard's SQLite file and emits OTLP back to the dashboard. # export RUNWAY_PR_REVIEWER_BOT_LOGIN="runway-reviewer-bot" # VA-463 / VA-479: GitHub login of the adversarial PR-reviewer # agent. Consumed by the planner's reviewer dispatcher to # distinguish bot CHANGES-REQUESTED comments from human feedback # (the latter routes through the LLM interpretation path). # Unset → every comment lands on the human-feedback path. # export RUNWAY_COMMENT_AUTHOR_ALLOWLIST="Reviewer Bot,Jane Reviewer" # optional, comma-separated Linear user names whose comments on a # re-queued issue surface as "Review feedback from prior attempts" # in the implement prompt. Defaults to the Linear user the API key # authenticates as (which is both the reviewer-agent's identity # and, in the solo-dev case, the repo owner). Set this only when # the reviewer agent and the repo owner are split across distinct # Linear identities. ``` Runway uses two labels from the [Flightplan](https://github.com/valescoagency/flightplan) v1.1.0 state-label contract: - `RUNWAY_READY_LABEL` (default `ready-for-agent`) marks an issue as "ready for the agent to pick up." Runway's drain queue filters by this label, **not** by workflow status — Linear's GitHub integration auto-mutates status when a PR cross-references an issue, which would drain a status-gated queue silently every time someone mentioned the issue from a PR. Labels are immune to that integration. Runway removes the label on pickup as the claim signal: the gateway returns whether the label was actually present at write time, and a runner that loses the claim race (label was already absent) skips the issue cleanly without posting a pickup comment or pushing an agent branch. Linear's API has no label-level compare-and-swap, so two runners that both read the labels before either has written can still both proceed — the design assumes the predominant operator pattern of a single drain instance running sequentially. - `RUNWAY_HITL_LABEL` (default `ready-for-human`) is applied when the agent or reviewer can't finish, AND when a run fails outright. Runway never re-applies the ready label on failure — terminal failures shouldn't retry indefinitely. The operator triages and re-applies `ready-for-agent` manually if the failure was transient. `runway doctor` validates that the configured team and both labels exist before any agent run — misconfiguration surfaces immediately instead of mid-drain. ### From source (development) ```bash git clone git@github.com:ValescoAgency/runway.git cd runway pnpm install pnpm build # compiles src/ → dist/ pnpm link --global # so `runway` is on your $PATH ``` `pnpm dev -- ` runs the TypeScript source via `tsx` without building, useful while iterating on runway itself. #### Tests ```bash pnpm test # one-shot run, used by CI pnpm test:watch # watch mode for local iteration ``` Vitest is the harness; tests live colocated with the source as `*.test.ts` files (e.g. `src/git.test.ts` next to `src/git.ts`). CI runs `pnpm typecheck && pnpm test` on every PR via `.github/workflows/ci.yml`. When adding logic that has a sharp pass/fail signal, add a test next to it. The seed suite covers `parseRunArgs`, `detectBaseBranch`, the `parseOpRefs` regex extraction, and the `drainQueue` error-handler branches — copy any of those as a shape for new tests. #### Git hooks (lefthook + commitlint) Hooks install automatically on `pnpm install` via the `prepare` script. What runs and when: | Hook | Runs | Why | |---|---|---| | `pre-commit` | `pnpm typecheck` | Catch TS errors before they land on a branch. | | `commit-msg` | `pnpm exec commitlint --edit` | Reject non-conventional commit messages (CLAUDE.md convention). | | `pre-push` | `pnpm test` | Block pushing red. | Skip a single hook invocation with `LEFTHOOK=0 git commit …` (or `… git push …`). To re-install after editing `lefthook.yml`, run `pnpm exec lefthook install -f`. ## Usage ```bash cd /path/to/the/repo/you/want/agents/working/on runway run # drain the entire ready queue runway run --max 3 # attempt at most 3 issues then exit runway --help ``` `runway` (no subcommand) is an alias for `runway run` for back-compat. `--max N` bounds **attempts**, not successes. Every issue picked up counts as one attempt, whether it ends in a PR, a `ready-for-human` label, or an infrastructure-error flag. An issue picked up in this invocation will not be re-picked in the same invocation — the pickup-time `removeLabel(ready-for-agent)` plus a same-invocation seen-set guard keep the drain from looping on it. The CLI exits with 0 even if some issues hit HITL or errored — those are normal outcomes. Every run prints a per-issue verdict trail on exit (`APPROVED → PR opened ` / `HITL ` / `INFRA_ERROR `) so you can scan results without opening Linear; the same content also lives on the issue as a Linear comment. ## Linear conventions Runway picks up issues that are: - in team `RUNWAY_LINEAR_TEAM` (default `VA`) - (optionally) in project `RUNWAY_LINEAR_PROJECT` (override per-run with `runway run --project=`; unset = team-wide) - carrying label `RUNWAY_READY_LABEL` (default `ready-for-agent`, the flightplan v1.1.0 contract — see VA-423) - not carrying `RUNWAY_HITL_LABEL` (default `ready-for-human`) - not blocked by a non-terminal `blocks` relation - **without any child sub-issues** — parent PRDs and umbrella tickets are skipped. The right unit of agent work is the leaf ticket, not the parent that spans it. If your ticket's scope is bigger than one PR, file children for the individual deliverables and let runway pick those up instead. Workflow status (`Triage` / `Todo` / `In Progress` / etc.) is **not** part of the queue contract: Linear's GitHub integration auto-mutates status whenever a PR cross-references the issue, which would drain a status-gated queue silently every time someone mentioned the issue from a PR. What runway does on each pickup: - removes the `ready-for-agent` label (claim signal — takes the issue out of the next drain's queue; a runner that finds the label already gone skips the issue without doing visible work) - comments and works the issue on `agent/` - on approve, pushes the branch, opens the PR; Linear's GitHub integration then auto-transitions the issue (`In Progress` on PR open, `Done` on merge via the `Closes ` line in the PR body) - on reject / HITL / startup failure / mid-run crash, applies `ready-for-human`. Runway never re-applies `ready-for-agent` on failure — terminal failures shouldn't retry indefinitely; the operator triages and re-applies the label manually if the cause was transient. ## Write-path policy Runway tells the impl agent which paths it must **not** write to. By default the denylist is: ``` .github/workflows/** .env* *.pem *.key pnpm-lock.yaml .sandcastle/** ``` When an issue's acceptance criteria require modifying a forbidden path, the agent is instructed to emit `IMPL: BLOCKED — issue requires modifying , which working-style policy forbids` rather than silently skipping the work. Runway routes those to HITL with the reason attached. Two layers of override: **Per repo** — drop a `.runway/policy.yml` in the target repo root: ```yaml # Grants write access to specific paths from the default denylist. allowedPaths: - .github/workflows/** # Or replace the denylist entirely (use with care). # forbiddenPaths: # - .env* # - "*.pem" ``` **Per invocation** — comma-separated globs, removed from the effective denylist for one `runway run`: ```bash runway run --allow-paths='.github/workflows/**' runway run --allow-paths='.github/workflows/**,scripts/ci/*.sh' ``` `runway doctor` surfaces the active policy under Environment so you can see what an agent run can and can't touch (e.g. `impl policy: .runway/policy.yml + --allow-paths (5 forbidden paths)`). ## Base branch Runway auto-detects the repo's default branch at the start of every `runway run` by reading `origin/HEAD` (with `git remote show origin` as a fallback for fresh clones). That branch is used for diffing the agent's work and as the `--base` for the PR. Set `RUNWAY_BASE_BRANCH=` to override detection — useful when you want runway to target a release branch instead of the default, or when `origin/HEAD` isn't set and you don't want to run `git remote set-head origin --auto`. `runway doctor` surfaces the resolved base branch (detected or overridden) in its Environment section. ## Implementation pass The impl agent runs in a Sandcastle container with [`prompts/implement.md`](prompts/implement.md). It iterates up to `RUNWAY_MAX_ITERATIONS` times (default 5) and must end every iteration with one of: ``` IMPL: DONE IMPL: BLOCKED — IMPL: CONTINUE ``` - `DONE` → runway stops the loop and runs the sub-agent reviewer. - `BLOCKED — ` → runway routes the issue to HITL with the reason attached; the reviewer pass does **not** run. - `CONTINUE` → runway runs another iteration (up to `maxIterations`). Between iterations, runway prepends a `## Previous iterations` block to the next prompt — running commit log + tail of the last iteration's final message — so the agent doesn't re-explore the repository from scratch every time. Converged issues typically exit after 1–2 iterations. ## Sub-agent review Every implementation run is followed by a fresh Sandcastle run with [`prompts/review.md`](prompts/review.md). The reviewer reads the diff and the issue body and outputs one of: ``` REVIEW: APPROVED REVIEW: REJECTED — ``` If approved, runway pushes the branch and opens the PR. If rejected, the issue gets the HITL label and a comment with the reviewer's reason. The reviewer is intentionally adversarial — its job is to find reasons NOT to ship, not to rubber-stamp. ## Dashboard `runway run` emits OpenTelemetry traces + logs for every drain. The operations dashboard projects those into a local SQLite db and serves a single-page web UI for browsing recent runs, drilling into per-issue timelines, and filtering by outcome / drain / date range. Binds to `127.0.0.1` only — no auth, no LAN exposure by default. ### v2: `runway dash` (recommended) ```bash # Bring up the dashboard from any directory — no runway clone required. runway dash up # In another shell, point runway run at it: export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 runway run # Open http://localhost:3001/ in a browser. # Tail the container logs. runway dash logs --follow # Tear down (volume + history preserved). runway dash stop # Tear down AND drop history. runway dash stop --purge ``` `runway dash up` pulls `ghcr.io/valescoagency/runway-dashboard:latest`, creates a named volume (`runway-dashboard-data`) for the SQLite db, and runs a detached container. Repeated `runway dash up` calls are idempotent: an already-running container stays as-is, a stopped one is started, otherwise a fresh container is created. Override the image with `--image=…` or `RUNWAY_DASHBOARD_IMAGE`; the ports with `--otlp-port=…` / `--dashboard-port=…`. Forward Linear sync by exporting `LINEAR_API_KEY` (and optionally `LINEAR_POLL_INTERVAL_SECONDS`, `RUNWAY_LINEAR_TEAM`, `RUNWAY_READY_LABEL`) before `runway dash up` — the CLI passes them through to the container without echoing the value through argv. ### v1: docker-compose (for hacking on the dashboard itself) The runway repo also ships a `docker-compose.yml` that builds the dashboard locally from `Dockerfile.dashboard`. Use this when you're developing the dashboard code and want to iterate without publishing an image: ```bash git clone https://github.com/ValescoAgency/runway && cd runway docker compose up # builds + runs the local image export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 ``` ### Migrating from compose to `runway dash` 1. `docker compose down` (in the runway repo). 2. `runway dash up` (anywhere). The default named volume is different (`runway-dashboard-data` vs compose's `runway-data`), so history doesn't carry over automatically. To preserve it, copy the volume contents before switching: ```bash docker run --rm -v runway-data:/from -v runway-dashboard-data:/to \ alpine sh -c 'cp -a /from/. /to/' ``` ## What's deliberately missing in v1 - Parallel runs (one issue at a time) - Multi-repo dispatch from a single command (cd into each repo) - Vercel Sandbox provider (Docker only — swap to `vercel()` later) - DAG / dependency awareness across issues - Auto-rebase / merge of approved PRs (you do that) These are tractable, just not v1. ## Status 0.15.0 — production-shaped and dogfooded against live Linear queues. The end-to-end pipeline (init → run → review → PR) is stable; surface may still shift as the orchestrator's policy and iteration mechanics mature. See [CHANGELOG.md](./CHANGELOG.md) for per-release detail.