# Self-Hosting and Bootstrap Process

**How SpecVerse builds itself, and the rules for extending it safely.**

---

## The Fundamental Problem

SpecVerse is a self-hosting system: it generates its own CLI from its own specification. This creates a chicken-and-egg dependency:

```
spec → [CLI runs infer+realize] → generates CLI → spec → ...
```

You need the CLI to build the CLI. This is the same bootstrapping problem that compilers face (you need a C compiler to compile the C compiler).

## The Solution: Bootstrap CLI

```
specverse-self/
├── bootstrap/
│   ├── cli/                   ← Frozen last-known-good generated CLI
│   └── engine-versions.json   ← Engine versions this bootstrap was built against
├── generated/
│   └── code/                  ← Fresh output — wiped by clean, regenerated every build
├── specs/
│   └── main.specly            ← The self-specification (source of truth)
└── bin/
    └── specverse.mjs          ← Shim that runs bootstrap/cli/index.ts via tsx
```

**Bootstrap** is the previous clean working version of the generated CLI. It is **frozen** — never edited directly. It's committed to git. It survives `npm run clean`. Its only job is to run `validate`, `infer`, and `realize` to produce the **next** version.

**Generated** is the fresh output from the current spec. Everything realize produces goes here: CLI commands, controllers, services, routes, views, frontend, tools. This is the **real** output that gets tested.

**engine-versions.json** records the exact engine package versions the bootstrap CLI was built and tested against. This is the emergency recovery key (see below).

## Build Lifecycle

```
npm run clean     →  rm -rf generated/code (bootstrap untouched)
npm run build     →  validate → infer → realize → build:vscode → build:mcp
                     (bootstrap CLI runs the pipeline)
npm run test      →  vitest (tests run against GENERATED output, not bootstrap)
npm run promote   →  cp generated CLI → bootstrap + update engine-versions.json
                     (only runs after tests pass)
```

### What Gets Tested

Tests validate the **generated** output — the new version, not the old bootstrap. The test suite checks:
- Generated CLI responds to `--help` for all commands
- Inferred spec validates
- Generated code structure is correct (controllers, views, tools)
- VSCode extension builds and packages
- MCP server responds to protocol

### What Gets Promoted

After tests pass, `npm run promote` replaces bootstrap with the generated CLI. This means the **next** clean build will use the CLI you just tested. The `engine-versions.json` file is updated to record which engine versions the new bootstrap was validated against.

## Normal Path: Engine Semver Compatibility

Bootstrap imports from `@specverse/engines` (and its subpaths) at runtime. When engines are upgraded, bootstrap must still work with the newer versions. This is the normal expectation — engines follow semver:

- **Patch** (4.0.4 → 4.0.5): Bug fixes. Bootstrap always works.
- **Minor** (4.0.x → 4.1.0): New features, backward compatible. Bootstrap always works.
- **Major** (4.x → 5.0.0): Breaking changes. Bootstrap may break.

**The rule**: Engine major version bumps must not break the bootstrap CLI's ability to run `validate`, `infer`, and `realize`. If a breaking change is necessary, the migration path is:

1. Publish the new engine version
2. Rebuild and test against the old bootstrap
3. If bootstrap breaks, use emergency recovery (below)
4. Promote the new generated CLI
5. Now bootstrap is compatible with the new engine version

## Emergency Recovery: Engine Breaking Change

If an engine upgrade breaks bootstrap (e.g. a method was renamed that bootstrap calls), `engine-versions.json` provides the recovery path:

```json
{
  "built": "2026-05-28",
  "note": "Engine versions this bootstrap was tested against",
  "engines": {
    "@specverse/assets": "^1.22.0",
    "@specverse/engines": "^6.90.0",
    "@specverse/entities": "^5.7.0",
    "@specverse/types": "^5.4.0"
  }
}
```

(`@specverse/runtime` is intentionally absent — it's consumed by *generated frontends*, not by the bootstrap CLI, so it isn't part of the bootstrap-compatibility set.)

Pipeline-stage engines (parser, inference, realize, generators, ai, registry) are subpaths of `@specverse/engines` — they don't need separate entries.

**Recovery steps**:

1. Read `engine-versions.json` for the working versions
2. Install those specific versions temporarily:
   ```bash
   npm install @specverse/types@5.4.0 @specverse/entities@5.7.0 @specverse/engines@6.90.0 @specverse/assets@1.22.0
   ```
3. Run `npm run build` — bootstrap works with the old engines
4. Run `npm run test` — generated CLI uses the old engines too (but that's fine for recovery)
5. Now install the new engine versions
6. Run tests against the generated CLI with new engines
7. If tests pass, promote — new bootstrap is compatible with new engines
8. Update `engine-versions.json`

This is a temporary fallback, not the normal workflow. The goal is to never need it by maintaining engine backward compatibility.

---

## Scenarios

### 1. Adding a New CLI Command (e.g. `lib`)

1. Add command to `specs/main.specly`
2. `npm run build` — realize generates the new CLI command file in `generated/code/`
3. `npm run test` — tests run against **generated** CLI (which has `lib`)
4. `npm run promote` — generated CLI (with `lib`) replaces bootstrap
5. Commit spec change + promoted bootstrap

Bootstrap never had `lib`. It didn't need to — it only needed to run infer+realize. The generated output has `lib` because the spec declares it and the CLI factory generates it.

**Prerequisite**: The CommanderJS CLI factory must be wired up in engine-realize for this flow to work. See "Current State" below.

### 2. Modifying an Existing CLI Command

1. Edit the command in `specs/main.specly` (add flag, change description, etc.)
2. `npm run build` — realize regenerates the command
3. `npm run test` — tests validate the updated command
4. `npm run promote` — new version replaces bootstrap
5. Commit

### 3. Fixing an Engine Bug

1. Edit engine source in `specverse-engines/engines/src/<subpath>/` (or the appropriate workspace package for types / entities / runtime)
2. Build: `npm run build` in the engine package
3. Copy `dist/` to specverse-self's `node_modules/` (dev) or publish to npm (release)
4. In specverse-self: `npm run clean && npm run build && npm run test`
5. If tests pass: commit engine fix, publish, bump specverse-self versions, promote

### 4. Clean Build (CI or Fresh Clone)

1. `git clone` — bootstrap is committed to git
2. `npm install` — engine packages from npm
3. `npm run clean && npm run build` — bootstrap drives the pipeline
4. `npm run test` — validates generated output
5. `npm run promote` — optional, updates bootstrap if generated output improved

This always works because bootstrap is in git and engines are on npm.

### 5. Engine Major Version Upgrade

1. Publish new engine version (e.g. `@specverse/engines@7.0.0` for a major bump)
2. Update `package.json` in specverse-self, `npm install`
3. `npm run clean && npm run build` — does bootstrap still work?
   - **Yes**: Continue normally. Test, promote.
   - **No**: Use emergency recovery with `engine-versions.json` to rebuild once with old engines, then promote, then rebuild with new engines.

### 6. Multiple Developers / CI

Bootstrap is committed to git, so all developers and CI share the same seed. The promote step should only be run deliberately (not automatically in CI builds), because it changes the seed for everyone.

**CI pipeline**: `clean → build → test` (no promote)
**Release pipeline**: `clean → build → test → promote → commit → tag`

---

## Golden Rules

### R30: Spec First, Always
Every change starts in `specs/main.specly`. Bootstrap is a frozen artifact generated from a previous version of the spec. The generated output reflects the current spec. Never modify bootstrap directly.

### R31: Bootstrap Is Frozen
`bootstrap/cli/` is the **previous clean working version** of the generated CLI. It is never edited. It exists only to bootstrap the next build. New features, new commands, new behavior — all go into the spec and flow through realize to the generated output.

### R31a: Bootstrap Recovery — Direct Edit Allowed Only When the Loop Is Locked
R31's "never edit bootstrap directly" has one narrow exception: when a bad promote leaves the regenerated bootstrap unable to run realize (syntax error / runtime crash). The standard cycle (edit template → realize → promote) cannot break itself out — the next realize-attempt fails because bootstrap is broken. Recovery: minimal direct patch to `bootstrap/cli/commands/*.ts`, rebuild `dist/cli.mjs`, confirm smoke green, then apply the same fix in the source template + run realize + promote (which overwrites the manual patch cleanly). See [GOLDEN-RULES.md](../GOLDEN-RULES.md#r31a) for the full procedure.

### R32: Promote Only After Green Tests
`npm run promote` replaces bootstrap with the tested generated CLI. It runs only after all tests pass. The promote script also updates `engine-versions.json`. Never promote untested output.

### R33: Clean Must Always Work
`npm run clean && npm run build && npm run test` must always pass from a fresh state. If it doesn't, either bootstrap is incompatible with current engines (use emergency recovery) or there's a real bug to fix.

### R34: Engines Follow Semver
Engine packages maintain backward compatibility within major versions. Bootstrap depends on this — it imports from `@specverse/engine-*` at runtime. Breaking changes require a major version bump and the engine upgrade migration path.

### R35: engine-versions.json Is the Emergency Key
`bootstrap/engine-versions.json` records exactly which engine versions the bootstrap CLI was tested against. If bootstrap breaks after an engine upgrade, install those specific versions to recover. This is insurance, not the normal path.

---

## Packaging and Distribution (R36)

SpecVerse self is published to npm as `@specverse/self`. The CLI must work when installed globally (`npm install -g @specverse/self`) on any machine — not just the development environment.

### The Problem: TypeScript Source in a Published Package

Bootstrap contains raw TypeScript (`.ts`) files. During development, these are executed via `tsx` (a TypeScript runner) in a subprocess. This creates three problems for published packages:

1. **tsx path resolution**: `bin/specverse.mjs` spawned tsx from the package's own `node_modules/.bin/tsx`, but npm hoists dependencies — tsx ends up in the parent `node_modules/`, not the package's own.
2. **Engine discovery**: `EngineRegistry.discover()` (in `@specverse/entities`) does dynamic `import('@specverse/engines/parser')`. Node resolves this from entities' file location, not from self's — so it can't find engines in self's `node_modules/`.
3. **Phantom dependencies**: Packages that work on the dev machine (via hoisted or globally-installed transitive deps like `ajv`) fail on clean installs where those deps don't exist.

### The Solution: Compiled CLI + Explicit Imports

**Pre-compile at publish time.** The `prepublishOnly` script bundles `bootstrap/cli/*.ts` into a single `dist/cli.mjs` using esbuild:

```
esbuild bootstrap/cli/index.ts --bundle --platform=node --format=esm --outfile=dist/cli.mjs --packages=external
```

`--packages=external` keeps all npm package imports (`commander`, `@specverse/engines`, etc.) as normal `import` statements — only the local bootstrap source files are compiled and bundled together.

**Explicit engine registration.** Instead of relying on `EngineRegistry.discover()` (which resolves from entities' location), the engine loader explicitly imports and registers each engine:

```typescript
import { engine as parserEngine } from '@specverse/engines/parser';
import { engine as inferenceEngine } from '@specverse/engines/inference';
// ... etc
const registry = new EngineRegistry({ disableAutoDiscovery: true });
registry.register(parserEngine);
```

These imports are in **self's** code, so Node resolves them from **self's** `node_modules/` — which always works.

**Dual-mode entry point.** `bin/specverse.mjs` checks for the compiled CLI first:

```javascript
if (existsSync(distCli)) {
  await import(distCli);   // Published: run compiled JS directly
} else {
  // Development: run TS source via tsx subprocess (as before)
}
```

### Package Hygiene

The `files` field in `package.json` controls what gets published:

```json
"files": ["bin", "dist", "bootstrap/cli", "bootstrap/engine-versions.json", "templates", "README.md"]
```

This reduced the published tarball from **936 files (8.7MB)** to **118 files** — excluding `generated/`, `documentation/`, `examples/`, `specs/`, and other development-only content.

### Key Decisions

| Decision | Rationale |
|----------|-----------|
| esbuild, not tsc | Single command, no config file, survives promote (re-bundles whatever is in bootstrap/) |
| `--packages=external` | Keeps @specverse/* as normal imports so Node resolution works; avoids bundling engine internals |
| Explicit engine imports | Moves resolution to self's code where we control the file location; eliminates cross-package dynamic import failures |
| `files` field | Prevents dev artifacts from bloating the published package |
| tsx in devDependencies only | Not needed at runtime; compiled JS runs with plain `node` |
| Dual-mode bin entry | Published packages use compiled JS; dev environment uses tsx as before |

### R36: Published Packages Must Work on Clean Installs

Every dependency must be declared. Every import must resolve from the package's own `node_modules/`. Never rely on hoisted transitive dependencies, global installs, or `~/node_modules/` phantom packages. Test with `npm pack` + fresh install before publishing.

### R36a: CI Catches What Local Doesn't
Local `npm test` and `smoke-all` are necessary but not sufficient. CI runs in a materially different environment (pinned Node 20, cold npm cache, fresh install from registry rather than workspace symlinks) and catches a distinct class of bug. Watch CI after every publish; treat green CI as the final certification, not local smoke.

### R36b: Engines Changes Require an End-to-End Smoke Before Publish
Vitest in `specverse-engines` exercises engines in isolation; it cannot see the cross-workspace round-trip from inference YAML emission → realize-time string expander → generated controller validator → live HTTP request. `npm run realize` does NOT exercise local engines edits — it runs the global `specverse` binary which uses its own bundled engines. Required before publishing any engines change that touches inference YAML emission, realize templates, convention/attribute processing, instance-factory templates, or schema rules: `node bin/specverse.mjs realize all ... && bash scripts/smoke.sh` (or `npm run verify:tarball`). 60s of local check beats post-publish patch-bump churn.

---

## Current State (June 2026)

### What works now
- Bootstrap CLI runs validate, infer, realize successfully
- Inference engine preserves all 5 self-spec components, multi-component fan-out generates code for all of them (engines 6.3.0+)
- Engines test suite: 3,200+ tests, zero skipped
- Handlebars rule-template engine rewrite complete — every inference rule renders via `Handlebars.compile() + yaml.load`, no more `generate*Spec` TypeScript shortcuts in `rule-engine.ts`
- Walker is the single view-render path across the ecosystem (runtime mode, starter-kit mode, app-demo)
- Clean build from bootstrap works end-to-end
- Global install (`npm install -g @specverse/self`) works on clean machines
- AI provider replatform shipped (engines 6.0+): four-mode `claude-cli` / `anthropic` / `openai-compatible` / `stub` via Vercel AI SDK
- Structural prepass shipped: three pluggable backends (grep-only / CodeGraph / GitNexus) extract deterministic facts before the LLM call
- Published versions live: `@specverse/types` 5.4.2, `@specverse/entities` 5.7.2, `@specverse/engines` 6.97.12, `@specverse/runtime` 5.12.25, `@specverse/assets` 1.25.0, `@specverse/self` 5.21.6

### What's next
- See `specverse-self/docs/TODO.md` for the current project backlog
- Coverage roadmap (analytics / integrations / pipelines / workflow orchestration) tracked in [SPECVERSE-INTRO.md](SPECVERSE-INTRO.md) Appendix A