# SJ2 Architecture

SJ2 is designed as an **engineering platform**, not a chatbot: the user states
intent, and an orchestration pipeline of specialized agents plans, implements,
reviews, and (eventually) ships the change — with a human approval gate before
anything is written or committed.

This document describes the target architecture from the full product spec,
and marks clearly what exists in this scaffold today vs. what's a defined next
milestone. Nothing below is aspirational marketing copy — it's the actual
contract between packages.

## 1. Monorepo layout

```
sj2/
  package.json            npm workspaces root
  tsconfig.base.json       shared compiler settings
  packages/
    shared/     [BUILT]  logger, typed errors, cross-package types
    core/       [BUILT]  .sj2/ project state, repo scanner, memory store
    ai/         [BUILT]  provider abstraction (9 providers) + router + model benchmarking
    agents/     [BUILT]  BaseAgent, 13 agents, agent registry, Orchestrator (fixed pipeline)
    workflows/  [BUILT]  user-defined pipelines, topological/parallel execution engine
    mcp/        [BUILT]  minimal MCP stdio client (JSON-RPC: initialize/tools-list/tools-call)
    auth/       [BUILT]  RBAC, audit logging, generic OIDC SSO client
    billing/    [BUILT]  pricing tiers, entitlements, NayaPay manual-confirmation payment flow
    plugins/    [BUILT]  plugin manifest schema + loader
    git/        [PLANNED] commit/branch/PR automation, rollback
    filesystem/ [PLANNED] safe write/patch engine with backups
    security/   [PLANNED] risk scoring beyond the orchestrator's current heuristic
    testing/    [PLANNED] test *execution* harness (QA agent designs tests today; nothing runs them yet)
    docs/       [PLANNED] README/CHANGELOG/migration-guide generation
    telemetry/  [PLANNED] structured metrics/tracing sink
    api/        [PLANNED] REST/GraphQL/WebSocket surface over core+agents+workflows
    context/    [PLANNED] relevance-ranked file retrieval (see §4)
    indexer/    [PLANNED] embeddings + dependency graph builder
    knowledge/  [PLANNED] org-wide knowledge graph (enterprise)
    vector/     [PLANNED] pgvector/LanceDB/ChromaDB adapters behind one interface
    config/     [PLANNED] layered config resolution (defaults -> project -> env -> flags)
    logging/    [MERGED into shared/ for now]
  apps/
    cli/        [BUILT]  sj2 init / learn / plan / explain / implement / workflow / doctor
    cloud/      [BUILT, minimal]  Fastify API: billing (NayaPay) + RBAC-gated audit log
    extension/  [PLANNED] VS Code extension (sidebar, inline chat, review window)
    desktop/    [PLANNED] Electron/Tauri shell
    dashboard/  [PLANNED] Next.js web UI over the API package
  examples/
    plugins/docker/  [BUILT]  working example plugin (manifest + activate())
```

See `ROADMAP.md` for how the remaining features from the Plus/Enterprise
vision (background agents, voice mode, visual dependency explorer, team
analytics, marketplace) map to concrete future phases and their real
infrastructure blockers.

Every package is an independent npm workspace with its own `package.json`,
`tsconfig.json`, and build/typecheck scripts — this is what "everything must
be independent, everything must be testable" means concretely: you can build,
test, and version `@stvdio/ai` without touching `@stvdio/agents`.

## 2. AI provider abstraction

**Contract:** everything upstream depends on `AIProvider` (one method:
`complete()`), never on a concrete SDK. `packages/ai/src/router.ts` holds the
priority list (`Ollama > Groq > Gemini > OpenRouter > OpenAI > Anthropic >
Azure > Mistral > Cohere` by default, matching the spec's local-first
ordering) and does retry-then-fallback: a provider gets `maxRetriesPerProvider`
attempts (for transient/429 errors) before the router moves to the next
configured provider.

Adding a 10th provider means writing one class that implements `AIProvider`
and registering it in `createRouterFromEnv()` — no changes to agents, CLI, or
orchestrator. Five of the nine providers (OpenAI, Groq, OpenRouter, Azure,
Mistral) share one `OpenAICompatibleProvider` base class since they speak the
same wire format; Anthropic, Gemini, Cohere, and Ollama each have their own
native HTTP client because their APIs genuinely differ.

## 3. Agents & orchestration

`BaseAgent` is the contract every specialized agent implements: it owns a
role, goals, constraints, a prompt-building method, and a confidence
estimate. `Orchestrator` currently runs a **fixed sequential pipeline**:

```
Architect (assess) -> Planner (break into steps) -> Engineer (implement)
  -> Reviewer (critique) -> requiresApproval flag
```

This matches the spec's pipeline shape (Intent -> Planning -> Agent Selection
-> Execution -> Validation -> Risk Analysis -> Approval) but **not yet** the
dynamic agent-selection and parallel-execution graph the full spec describes.
That's the defined next milestone: swap `Orchestrator.runWorkflow()`'s fixed
sequence for a task-graph scheduler that (a) lets the Planner's output
determine which agents run, and (b) executes independent steps concurrently.
The four other agent roles (Product Manager, Database Engineer, DevOps
Engineer, etc.) plug into the same `BaseAgent` contract — they're not built
yet because they need real tool access (git, DB introspection, cloud APIs)
to be more than a prompt with a different name.

**Nothing in this pipeline writes to disk or git.** `sj2 implement` prints the
reviewed diff and an approval verdict; wiring the Engineer's output through an
actual patch-apply step (with backups and a git checkpoint) is gated behind
building `packages/filesystem` and `packages/git` — this is a deliberate
sequencing decision, not an oversight: the spec's Security section requires
preview-diff + approval + rollback *before* any file gets touched, so those
packages have to exist before `implement` is allowed to write anything.

## 4. Context engine (planned, not built)

The spec is explicit: "never send entire repositories." The scanner in
`packages/core/src/scanner.ts` today only gathers repo-level metadata
(languages, frameworks, file counts) for `sj2 learn` — it does not yet do
per-task relevant-file retrieval. `packages/context` is where that goes:
embed files via `packages/indexer`, store vectors via `packages/vector`
(pgvector locally, LanceDB/Chroma as alternates), and at task time retrieve
top-k relevant files + past decisions + related tests, then hand a bounded
context string to the orchestrator. Every agent's `AgentTask.context` field
already assumes this shape — building the context engine is additive, not a
breaking change to `agents`.

## 5. Memory

`packages/core/src/memory.ts` defines `MemoryStore` (add/query/forget) and one
implementation, `JsonMemoryStore`, used for `.sj2/history.json`,
`knowledge.json`, `vectors.json`, `tasks.json`. This is intentionally a
stand-in for the spec's `history.db`/`knowledge.db`/`vectors.db` (SQLite +
pgvector) — the interface is what matters and is already what the CLI depends
on, so swapping in a `SqliteMemoryStore` later touches one file, not every
call site.

## 6. Security model (partially implemented)

Implemented today: `Orchestrator` computes a `requiresApproval` boolean from
reviewer verdict, risk keywords, and per-stage confidence, and the CLI prints
it prominently rather than silently proceeding.

Not yet implemented: preview diffs against the actual working tree, automatic
backups, git checkpoints before write, and rollback — these all depend on
`packages/filesystem` and `packages/git` existing first (see §3).

## 7. What "done" looks like for the next milestone

1. `packages/context` + `packages/indexer` + `packages/vector`: real
   relevant-file retrieval instead of full architecture-summary dumping.
2. `packages/filesystem` + `packages/git`: safe patch application, backups,
   git checkpoints, rollback — unblocks `sj2 implement` actually writing code.
3. Dynamic agent selection in `Orchestrator` instead of the fixed 4-stage
   pipeline.
4. `packages/api`: REST/MCP surface so the VS Code extension and dashboard
   aren't CLI-only consumers.

## 8. Workflow builder (new)

`packages/workflows` turns pipelines into data (`WorkflowDefinition`): each
step names an agent (resolved via `packages/agents`' `AGENT_REGISTRY`) and its
dependencies. `WorkflowEngine.run()` executes steps in topologically-sorted
batches — independent steps in a batch run in parallel via `Promise.all`,
each batch waits for the previous one. This is genuine parallel execution,
which the fixed `Orchestrator` pipeline (§3) doesn't have. Three templates
ship today (`standard`, `architecture-review`, `full-review`); `sj2 workflow
run <template> <goal>` executes them.

## 9. Model benchmarking (new)

`packages/ai/src/benchmark.ts` holds a hand-maintained table of
(provider, model) profiles scored on quality/speed/cost/privacy.
`ProviderRouter.completeOptimized(req, criterion)` picks the best-scoring
profile among currently-configured providers and routes to it, falling back
through the normal priority order if that specific provider fails at call
time. This is deliberately a static, editable table rather than a live eval
harness — see the comment in `benchmark.ts` for why, and what replacing it
with real evals would require.

## 10. MCP support (new, minimal)

`packages/mcp` implements the stdio-transport subset of MCP needed to call
external tool servers: spawn the server process, `initialize`, `tools/list`,
`tools/call`, all over newline-delimited JSON-RPC 2.0. It's a small
hand-written client rather than a full SDK wrapper — swapping in the
official `@modelcontextprotocol/sdk` later is a drop-in replacement since
nothing outside this package depends on the wire-format details.

## 11. Enterprise: RBAC, audit, SSO, billing (new)

- `packages/auth`: `Role`/`Permission` tables (`rbac.ts`), an
  `ApprovalChainPolicy` model for multi-approver gating, a JSON-backed audit
  logger (explicitly **not** tamper-evident — see the warning in `audit.ts`
  about what a compliance-grade version needs), and a generic OIDC client
  (`sso.ts`) implementing the real auth-code flow against any standards-compliant
  identity provider.
- `packages/billing`: the four pricing tiers as data (`plans.ts`), a single
  `checkEntitlement`/`checkUsageLimit` function every gated code path should
  call, and a NayaPay integration (`nayapay-service.ts`). **This is
  deliberately not an automated checkout/webhook flow** — NayaPay has no
  public, documented API for that (their merchant integration, "Arc", is
  dashboard-issued and PCI-DSS-gated; see the comment at the top of
  `nayapay-service.ts` for what was found and why an auto-checkout URL and a
  webhook "signature" that was really just a plaintext string compare were
  both removed rather than shipped). Instead: a customer requests to pay,
  gets your real payment details plus a unique reference code, pays you
  directly, and a human admin explicitly confirms the transfer before the
  plan activates. `verifyHmacSignature()` is provided, unused, for the day
  real NayaPay webhook docs exist — do not wire it up without confirming the
  exact signing scheme first.
- `apps/cloud`: a minimal Fastify service exposing billing routes
  (`/billing/plans`, `/billing/payment-request`, `/billing/confirm-payment`,
  `/billing/entitlement/:orgId`) and an RBAC-gated audit-log route. Its
  subscription and pending-payment stores are in-memory (reset on restart) —
  see the comments in `subscription-store.ts`/`pending-payment-store.ts` for
  what a production DB-backed version needs.
  **The `x-sj2-role`/`x-sj2-user` headers used for the confirm-payment and
  audit-log routes are a demo stand-in for a verified session, not real
  authentication** — `/billing/confirm-payment` is the route that actually
  activates a paid plan, so this needs a real session/JWT check before it
  touches real money, even more urgently than the audit log does.