# sagent Enhancement Requirements

**Status:** Approved for phased implementation
**Source:** [`enhancements.md`](./enhancements.md)
**Applies to:** Existing approved Phase 1–5 foundation plus A–D remediation

## 1. Goals

Enhance `sagent` with:

1. Recursive, bounded map-reduce delegation so any eligible persona can split genuinely complex work across multiple instances of the same persona.
2. Intelligent real-time status showing every run, agent, child shard, phase, progress, activity, model, usage, and failure state.
3. Reproducible CI, packaging, and public npm publication as `@sakiv/sagent`, installable with `pi install npm:@sakiv/sagent`.
4. A complete GitHub-facing `README.md` covering installation, concepts, operation, security boundaries, commands, examples, development, and release procedures.

## 2. Non-Goals and Preserved Gates

- Map-reduce does not enable mutating personas. Existing mutation/worktree gates remain unchanged.
- Recursive delegation cannot approve personas, alter policy, enable code upgrades, or bypass model/spend confirmation.
- Autonomous code upgrades remain unconditionally disabled.
- Status telemetry must never expose prompts, secrets, credentials, raw environment variables, or unrestricted model output.
- CI publication must never embed npm credentials in repository files or build artifacts.

## 3. Terminology

- **Parent task:** Task that requests decomposition.
- **Map plan:** Validated list of independent shards.
- **Shard:** One bounded child task handled by an instance of the same persona.
- **Reducer:** Same-persona synthesis step combining shard results with provenance.
- **Delegation depth:** Number of recursive map-reduce boundaries from the original task.
- **Run tree:** Durable parent/child relationship between tasks and shards.

## 4. Functional Requirements — Recursive Map-Reduce

### MR-1 Eligibility

- Every registered read-only persona, including approved dynamic personas, may request map-reduce.
- Delegation is optional and used only when independent partitioning is beneficial.
- Simple tasks must execute directly without decomposition.
- Mutating personas remain ineligible until the separately reviewed mutating pipeline exists.

### MR-2 Complexity Decision

Before delegation, the persona must produce a structured decision:

```typescript
interface ComplexityDecision {
  shouldSplit: boolean;
  rationale: string;
  estimatedItems: number;
  partitionKey?: string;
  requestedShards?: number;
}
```

Split when independent work units are numerous or context would exceed one agent’s safe processing envelope. Do not split tightly coupled tasks, tiny tasks, or tasks whose reducer would lose essential ordering/state.

### MR-3 Validated Map Plan

```typescript
interface MapReducePlan {
  parentTaskId: string;
  persona: string;
  depth: number;
  shards: Array<{
    shardId: string;
    task: string;
    inputRefs: string[];
    estimatedTokens: number;
  }>;
  reducerInstructions: string;
}
```

- Shard IDs are unique and stable.
- Shards must be independent and collectively cover the requested scope.
- Each child uses the same persona as its parent.
- Inputs use references or bounded excerpts, not unrestricted duplicated context.
- Plans with duplicate shards, empty tasks, excessive counts, or unsafe tools are rejected.

### MR-4 Bounds and Recursion Safety

Defaults, configurable only within hard ceilings:

- Maximum delegation depth: `2`.
- Maximum shards per map: `8`.
- Maximum concurrent shards within one map: `4`.
- Hard run-wide subprocess ceiling: `maxParallelReadOnlyAgents` (default `4`). Map shards, direct sibling DAG tasks, and reducers share the same fair semaphore; nested maps cannot multiply concurrency beyond it.
- Maximum total descendants per root task: `16`.
- Maximum reducer input: `200 KB` before deterministic compression.
- Existing session spend cap applies to the entire run tree, not each child independently; sibling `orchestrate` calls in the same session continue sharing the existing `accumulatedSpendUsd` ledger.
- Metered model approval must include projected map and reducer costs before dispatch.
- A task may not delegate to itself with an identical normalized task fingerprint.
- Repeated delegation fingerprints are rejected to prevent recursion loops.

### MR-5 Dispatch and Model Routing

- Each shard receives an independent Pi subprocess/context.
- Shards inherit the persona tool ceiling, resolved profile, knowledge scopes, and cancellation signal.
- Every shard is routed independently through the existing live model/auth router.
- Subscription models remain preferred.
- Metered approval is aggregated before map dispatch, including fallbacks and reducer.
- Failover remains limited to read-only/pre-mutation work.
- Enforce shared cross-process capacity limits before spawn: default maximum `2` concurrent requests per exact `provider/model` and `3` per provider, both configurable only within hard ceilings.
- If the preferred model or provider is at capacity, try an eligible fallback with a different provider and model before waiting; fallback diversity is preferred over launching another identical instance.
- HTTP 429 or provider rate-limit responses place that exact model and provider into a bounded cooldown/circuit state so queued shards route elsewhere when possible.
- Capacity and cooldown state is shared by direct DAG tasks, map shards, reducers, and sibling orchestration in the same session/root run.
- Fallbacks still obey capabilities, scoped models, billing classification, aggregate spend approval, and the read-only failover boundary.

### MR-6 Durable Run Tree

Extend task manifests with:

- `parentTaskId?: string`
- `rootTaskId: string`
- `delegationDepth: number`
- `taskKind: "direct" | "map" | "reduce"`
- `shardIndex?: number`
- `shardCount?: number`
- `progress: { completed: number; total: number; percent: number }`
- `activity: { phase: string; summary: string; updatedAt: number }`

All children use the existing seven-state lifecycle, atomic persistence, heartbeats, leases, cancellation, and recovery semantics. New run-tree fields are optional/defaulted during schema migration so existing Phase 3 manifests still open and reclaim safely.

### MR-7 Reduction

- Reduction runs only after all required shards reach terminal states.
- Reducer receives ordered results with shard IDs, status, provenance, usage, and truncation metadata.
- Reduction must identify partial failures and must not represent incomplete coverage as complete.
- Duplicate findings are consolidated deterministically.
- Conflicting findings remain visible with source attribution.
- Full shard results remain in artifacts; model-visible reducer input is bounded.

### MR-8 Partial Failure Policy

Configurable per request:

- `fail_fast`: cancel siblings after first terminal failure.
- `best_effort`: finish remaining shards and reduce successful results with explicit failure annotations.
- Default: `best_effort` for read-only analysis.

Cancellation propagates root → descendants and process trees.

### MR-9 Tool/API

Provide a dedicated tool available to eligible child personas, tentatively `delegate_map_reduce`, rather than unrestricted recursive access to the complete `orchestrate` API.

Required parameters:

- rationale
- shard tasks
- reducer instructions
- failure policy

The tool derives persona, depth, profile, root task, policy, and budgets from a supervisor-issued delegation envelope; callers cannot forge them.

- `delegate_map_reduce` is injected at runtime by the dispatcher after validating the supervisor envelope; it is never declared in candidate or approved-persona frontmatter and does not expand their persisted tool allowlist.
- The dispatcher omits the tool when `remainingDepth === 0`.
- Every invocation is bound to the envelope’s same persona, root task, profile, read-only tool ceiling, descendant budget, spend ledger, and cancellation signal.
- Callers cannot select another persona, add tools, increase depth/concurrency/budget, invoke mutating operations, or mint child envelopes.
- The dispatcher, not the model, decrements remaining depth and derives each child envelope.

## 5. Functional Requirements — Real-Time Agent Status

### ST-1 Status Model

```typescript
interface AgentStatusSnapshot {
  runId: string;
  taskId: string;
  parentTaskId?: string;
  persona: string;
  state: TaskState;
  phase: string;
  summary: string;
  progress: { completed: number; total: number; percent: number };
  model?: string;
  billingClass?: "subscription" | "metered";
  elapsedMs: number;
  lastActivityAt: number;
  usage: { input: number; output: number; costUsd: number };
  children: string[];
  errorSummary?: string;
}
```

### ST-2 Event Sources

Status updates derive from real execution events:

- task state transitions
- subprocess spawn/exit
- Pi JSON stream messages
- tool execution start/end
- heartbeat and lease updates
- shard completion
- reducer progress
- retries/failover
- cancellation and recovery

Do not infer fake percentages from elapsed time. Unknown progress displays as indeterminate.

### ST-3 TUI Presentation

- Add a compact swarm widget via `ctx.ui.setWidget` in TUI mode.
- Show root run summary and nested agent/shard rows.
- Include state icon, persona, concise activity, completed/total, elapsed time, selected model, and cost.
- Highlight stalled, retrying, blocked, failed, canceled, and recovered states.
- Throttle rendering to at most 10 updates/second and coalesce bursts.
- Remove completed widgets after a configurable retention period while preserving manifests and transcript details.
- Child processes (`SAGENT_CHILD=1`) must not render parent widgets.

### ST-4 Commands and Non-TUI Modes

- `/sagent status` provides current summary.
- `/sagent status --live` opens/focuses detailed live status in TUI.
- `/sagent runs` lists recent durable runs.
- `/sagent inspect <run-id>` displays a run tree and artifact locations; `/sagent run [--profile <name>] <task>` remains exclusively the start-work command.
- In JSON/RPC/print modes, `/sagent status --live` returns one structured snapshot with `live: false` and no widget; clients obtain subsequent snapshots through normal command polling. Terminal-only components are never created.

### ST-5 Privacy and Bounds

- Activity summaries are secret-scanned and capped at 200 characters.
- Never display full prompts, environment variables, tokens, approval challenges, or raw error stacks.
- Retain full diagnostic errors only in protected local artifacts where already permitted.

## 6. Functional Requirements — npm Packaging and GitHub Workflow

### PKG-1 Package Identity

- Public package name: `@sakiv/sagent`; the current development name `@sakiv-io/sagent` is implementation debt and must be changed during E3 before any publish.
- Install command: `pi install npm:@sakiv/sagent`.
- Remove `"private": true` before publication.
- Add repository, homepage, bugs, author, engines, and `publishConfig.access = "public"` metadata.
- Preserve `pi.extensions = ["./extensions/sagent/index.ts"]`.

### PKG-2 Artifact Allowlist

Use a `files` allowlist so the tarball contains only runtime and public documentation:

- `extensions/`
- `README.md`
- `LICENSE`
- selected architecture/security documentation if linked publicly

Exclude tests, local caches, `.pi/runs`, candidate data, credentials, review scratch files, and development-only artifacts.

### PKG-3 Runtime Dependencies

- Runtime dependencies remain under `dependencies`; Pi peer packages remain `peerDependencies`.
- Verify native packages (`better-sqlite3`, `sqlite-vec`) on supported OS/architecture matrix.
- `npm pack --dry-run` and installation from the generated tarball must succeed.
- A clean temporary Pi environment must load `/sagent status` from the packed tarball.

### PKG-4 CI Workflow

Add `.github/workflows/ci.yml`:

- Trigger on pull requests and pushes to `main`.
- Install from lockfile.
- Run typecheck and all tests.
- Run `npm pack --dry-run`.
- Test supported Node/Bun versions and the native dependency matrix: macOS arm64 and x64, plus Linux arm64 and x64. If a hosted runner is unavailable, a documented cross-architecture container/prebuild verification job must cover that target before release.
- Use least-privilege default permissions.

### PKG-5 Publish Workflow

Add `.github/workflows/publish.yml`:

- Trigger on GitHub Release publication or a version tag matching `v*`.
- Require CI success.
- Verify tag version equals `package.json` version.
- Build/test/package before publish.
- Publish with `npm publish --access public --provenance`.
- Prefer npm Trusted Publishing/OIDC (`id-token: write`, `contents: read`); if unavailable, use a GitHub Environment secret named `NPM_TOKEN` and never print it.
- Refuse publishing if the version already exists.
- Upload the `.tgz`, checksums, and package manifest as release artifacts.

### PKG-6 Release Safety

- No automatic version bump on every main push.
- Publishing requires an explicit version change and release/tag.
- Document semantic versioning and rollback/deprecation procedures.
- Initial publication must be manually confirmed immediately before running the publish workflow or `npm publish`.

## 7. README Requirements

The final `README.md` must include:

1. Project purpose and architecture overview.
2. Feature matrix distinguishing implemented, gated, and future capabilities.
3. Prerequisites and supported platforms.
4. npm, local package, and `-e` installation instructions.
5. Quick start and a reproducible smoke test.
6. Agent/persona catalog and privilege ceilings.
7. Direct, DAG, parallel, and map-reduce examples.
8. Live status UI and command reference.
9. Model routing, subscription/metered policy, and spend approval.
10. Knowledge hierarchy and local vector cache behavior.
11. Self-improvement candidate workflow and disabled code-upgrade boundary.
12. Configuration reference.
13. Durable state, recovery/state-reclamation semantics, and cleanup.
14. Security/threat model.
15. Development, testing, package verification, and release process.
16. Troubleshooting.
17. Architecture/review document links and license.

Claims must match implemented behavior exactly. Gated features must not be presented as active.

## 8. Testing Requirements

### Map-Reduce

- plan validation, shard limits, depth limits, descendant cap
- fingerprint loop rejection
- same-persona enforcement and tool ceiling preservation
- parallel execution/concurrency bounds
- fail-fast and best-effort behavior
- cancellation propagation
- deterministic reducer ordering/deduplication
- aggregate spend approval
- durable parent/child manifest recovery

### Status

- event-to-status projection
- progress calculations and indeterminate states
- hierarchy rendering and state colors/icons
- throttling/coalescing
- secret redaction and summary bounds
- cleanup on session shutdown/reload
- non-TUI structured behavior

### Packaging

- typecheck/tests on clean install
- tarball allowlist assertions
- extension load from packed tarball
- package/version/tag validation
- workflow syntax validation
- no secret or local path leakage in tarball

## 9. Acceptance Criteria

The enhancements are complete when:

1. A complex read-only persona task can map to multiple same-persona shards, reduce results, and persist a bounded durable run tree.
2. Depth, shard, descendant, cost, tool, cancellation, and recursion-loop controls are enforced by tests.
3. Pi displays accurate real-time nested status without leaking sensitive content or degrading stream performance.
4. Existing direct and DAG orchestration remain backward compatible.
5. `bun run verify` and packaging smoke tests pass.
6. A tarball installs and exposes `/sagent` in a clean Pi environment.
7. GitHub CI and explicit release publishing workflows are present and least-privilege.
8. The package is ready to publish as `@sakiv/sagent`; actual first publication occurs only after explicit confirmation.
9. README fully and accurately documents the extension.
10. Each enhancement phase receives reviewer approval before commit and before the next phase starts.

## 10. Implementation Phases

### Enhancement Phase E1 — Durable Map-Reduce Core

- delegation envelope and recursion limits
- map plan validation
- parent/child manifest schema
- bounded map scheduler and reducer
- cancellation, failure policies, usage aggregation
- tests and backward compatibility

### Enhancement Phase E2 — Real-Time Status

- status event projection/store
- Pi TUI swarm widget
- status/runs commands
- streaming updates, throttling, privacy controls
- TUI and non-TUI tests

### Enhancement Phase E3 — Packaging and CI/CD

- package identity/metadata/files
- lockfile and tarball validation
- CI matrix
- explicit provenance publish workflow
- packed-extension smoke test

### Enhancement Phase E4 — Complete GitHub Documentation

- rewrite README to satisfy §7
- validate all commands/examples against current behavior
- final full implementation and packaging review