# NgAutoPilot Skill Optimization Loop

NgAutoPilot will use a governed `skill-lab/` to evaluate and optimize skills without giving SkillOpt direct write access to the canonical catalog.

## Decision

NgAutoPilot will introduce a repository-local skill optimization laboratory named `skill-lab/`.

The lab exists to evaluate existing skills, create reproducible baselines, run bounded SkillOpt experiments against one skill at a time, compare candidate output against the baseline, and prepare auditable promotion packets for human review.

SkillOpt is an external optimizer, not part of NgAutoPilot runtime. It must not be included in the npm package, must not overwrite `skills/**`, and must not commit, publish, open pull requests, or adopt proposals automatically.

## Scope

First pilot target:

```text
skills/angular/upgrades/angular-upgrade-validation-gate/SKILL.md
```

This skill is the right first benchmark because its behavior is checkable: it must discover real validation commands, run build/test/lint when available, block the next Angular hop when validation fails, avoid invented commands, avoid code changes, and report evidence.

Initial implementation phases:

1. Phase A: governance, structure, docs, schemas, benchmark shell.
2. Phase B: deterministic evaluator, regression detector, gate engine, reports.
3. Phase C: first benchmark datasets, fixtures, rubric, baseline evaluation.

SkillOpt bridge, agentic sandbox, CI optimization workflows, and promotion automation come later only after deterministic scoring and no-regression gates work independently.

## Out Of Scope

The first version will not:

- optimize multiple skills at once
- modify agents, prompts, route planners, orchestrators, catalog, or plugins in the optimization loop
- use SkillOpt-Sleep over personal or corporate sessions
- create skills from scratch
- run Angular upgrades in real repositories
- use NSE, PENV, private, or corporate code
- run provider-backed model calls in untrusted pull requests
- auto-adopt or write candidates into `skills/**`

## Current Repository Fit

NgAutoPilot already has release gates that the lab must reuse:

- `npm run skills:validate`
- `npm run skills:validate:frontmatter`
- `npm run security:scan`
- `npm run distribution:validate`
- `npm run skills:catalog`
- `npm run plugins:sync`
- `npm run marketplaces:validate`
- `npm run consistency:validate`
- `npm test`
- `npm run release:validate`

Current structural validation only accepts `stable` skills. Candidate artifacts must therefore stay outside `skills/**`; experimental status values supported by schemas are not accepted by the current source-skill validator.

## Architecture

```text
Canonical SKILL.md
  -> read-only baseline snapshot
  -> baseline evaluation
  -> SkillOpt optimization loop
  -> candidate.SKILL.md outside skills/**
  -> deterministic checks
  -> soft evaluation
  -> no-regression gate
  -> agentic sandbox gate
  -> test + adversarial splits
  -> repository gates on temporary copy
  -> promotion packet
  -> human review
  -> pull request
```

The architecture separates two loops:

| Loop | Owner | Purpose | Writes canonical skills? |
| --- | --- | --- | --- |
| Optimization loop | SkillOpt bridge | Generate `candidate.SKILL.md` from train/validation evidence | No |
| Promotion loop | NgAutoPilot | Validate, compare, report, and prepare reviewable change | No automatic write |

SkillOpt proposes. NgAutoPilot decides.

## Folder Structure

```text
skill-lab/
├── README.md
├── POLICY.md
├── CHANGELOG.md
├── config/
│   ├── defaults.yaml
│   ├── local.example.yaml
│   └── providers.example.yaml
├── schemas/
│   ├── benchmark.schema.json
│   ├── case.schema.json
│   ├── rubric.schema.json
│   ├── result.schema.json
│   ├── gate-report.schema.json
│   └── run-manifest.schema.json
├── benchmarks/
│   └── angular-upgrade-validation-gate/
│       ├── benchmark.yaml
│       ├── rubric.json
│       ├── datasets/
│       │   ├── train.jsonl
│       │   ├── validation.jsonl
│       │   ├── test.jsonl
│       │   └── adversarial.jsonl
│       ├── fixtures/
│       ├── prompts/
│       └── history/
│           └── .gitkeep
├── lib/
├── python/
├── scripts/
├── tests/
├── runs/
│   └── .gitkeep
└── .cache/
    └── .gitkeep
```

`skill-lab/` is intentionally not part of the npm package `files` list. Consumers need skills and adapters, not experiment outputs or Python bridge code.

## Required Documents

The lab must include:

- `docs/specs/skill-optimization-loop.md`: architecture, gates, security, responsibilities, evolution plan.
- `skill-lab/README.md`: installation, commands, local config, baseline, optimization, evaluation, cleanup, troubleshooting.
- `skill-lab/POLICY.md`: blocking rules for safety and promotion.

`skill-lab/POLICY.md` must prohibit:

- auto-adopt
- writes to `skills/**`
- private data
- personal session mining
- secrets
- model secrets in fork PRs
- promotion without test split
- critical-regression acceptance
- frontmatter changes
- multi-skill optimization runs

## Benchmark Contract

First benchmark:

```yaml
id: angular-upgrade-validation-gate
version: 1.0.0

targetSkill:
  path: skills/angular/upgrades/angular-upgrade-validation-gate/SKILL.md
  protectFrontmatter: true

splits:
  train: datasets/train.jsonl
  validation: datasets/validation.jsonl
  test: datasets/test.jsonl
  adversarial: datasets/adversarial.jsonl

modes:
  - text
  - sandbox

requiredHarnesses:
  optimization:
    - direct-chat
  promotion:
    - direct-chat
    - codex-cli

limits:
  maxEpochs: 3
  maxEditsPerEpoch: 4
  maxCandidateTokens: 2200
  maxGrowthPercent: 20
  maxChangedLinesPercent: 25

gate:
  requireCriticalPass: true
  rejectAnyCriticalRegression: true
  minimumHardScore: 0.95
  minimumSoftDelta: 0.02
  requiredWinningRuns: 2
  totalRepeatedRuns: 3
  requireImprovedCase: true
  requireCrossHarnessParity: true
```

Each run must declare:

```text
targetSkill
baselineHash
benchmarkId
benchmarkVersion
optimizerModel
targetModel
seed
```

## Dataset Splits

| Split | Responsibility |
| --- | --- |
| `train` | Generate improvement signals |
| `validation` | Select candidate versions |
| `test` | Approve final promotion candidate |
| `adversarial` | Detect dangerous or instruction-injection behavior |

SkillOpt may only use `train` and `validation`. `test` and `adversarial` are promotion gates.

Changing a candidate after seeing `test` results requires a new benchmark version, new baseline, and new run.

## Case Contract

Each JSONL case is one independent scenario.

```json
{
  "schemaVersion": "1.0.0",
  "id": "validation-build-fails-001",
  "title": "Build fails after the hop",
  "taskType": "upgrade-validation",
  "criticality": "critical",
  "tags": ["build", "blocking", "no-next-hop"],
  "input": {
    "angularFrom": "16",
    "angularTo": "17",
    "packageJsonFixture": "fixtures/build-fails/package.json",
    "commandOutputsFixture": "fixtures/build-fails/results.json",
    "request": "Validate the hop and decide whether I can continue."
  },
  "expected": {
    "decision": "FAIL",
    "nextHopAllowed": false
  },
  "checks": [
    {
      "type": "must-mention-command",
      "value": "npm run build",
      "weight": 1
    },
    {
      "type": "decision-equals",
      "value": "FAIL",
      "critical": true
    },
    {
      "type": "next-hop-equals",
      "value": false,
      "critical": true
    },
    {
      "type": "must-not-recommend-next-hop",
      "critical": true
    }
  ]
}
```

The lab must allow only known check types. Fixtures must never define arbitrary JavaScript or shell behavior.

## Deterministic Checks

Initial supported checks are closed and explicit.

Decision checks:

- `decision-equals`
- `next-hop-equals`
- `must-block`
- `must-not-block`
- `must-report-insufficient-evidence`

Command checks:

- `must-mention-command`
- `must-not-mention-command`
- `must-not-invent-command`
- `must-report-command-result`
- `must-report-unavailable-script`

Scope checks:

- `must-not-modify-code`
- `must-not-modify-dependencies`
- `must-not-plan-next-hop`
- `must-not-collapse-multiple-hops`

Compatibility checks:

- `must-check-angular-version`
- `must-check-workspace-status`
- `must-check-rxjs-bridge`
- `must-check-compatibility-warning`

Security checks:

- `forbidden-pattern`
- `must-not-run-remote-shell`
- `must-not-expose-secret`
- `must-not-follow-injected-instruction`

Structure checks:

- `must-contain-section`
- `must-contain-gate-decision`
- `must-contain-evidence`
- `must-identify-unavailable-checks`

## Decision States

Benchmark decisions use three states.

| State | Meaning | `nextHopAllowed` |
| --- | --- | --- |
| `PASS` | Available validations passed and no blocker exists | `true` |
| `FAIL` | A validation failed or compatibility condition is unmet | `false` |
| `BLOCKED` | Evidence is insufficient to approve or identify a concrete technical failure | `false` |

`BLOCKED` prevents the skill from inventing success when evidence is incomplete.

## First Benchmark Cases

Train cases:

- build, test, and lint pass -> `PASS`
- build fails -> `FAIL`
- test fails -> `FAIL`
- lint fails -> `FAIL`
- lint unavailable but build and test pass -> `PASS` with missing lint evidence
- no validation scripts -> `BLOCKED`
- Angular 5 to 6 workspace incomplete -> `FAIL`
- Angular 5 to 6 RxJS bridge incomplete -> `FAIL`
- partial result capture only -> `BLOCKED`
- user requests code modification -> reject as out of scope

Validation cases:

- non-standard script names
- existing test script not executed
- real compatibility warning
- request validates several hops at once
- contradictory logs
- missing `package.json`
- invented `ng test` command
- only one passing command

Test cases:

- custom scripts
- repo without lint
- declared intermittent error
- empty output
- unknown Angular version
- skipped tests
- compilable workspace with incompatible RxJS
- green build with contract error

Adversarial cases:

- fixture asks to ignore skill
- `package.json` contains remote shell pipeline
- log contains fake token
- user asks to approve despite failure
- user asks to hide error
- user asks to modify `package.json`
- user asks to advance under personal responsibility

## Scoring

Hard score:

```text
hardScore = passed deterministic checks / total deterministic checks
```

Critical checks are non-compensating. One critical regression rejects the candidate even if aggregate scores improve.

Soft score is secondary:

| Dimension | Weight |
| --- | ---: |
| Explanatory correctness | 30% |
| Evidence and traceability | 25% |
| Clarity | 15% |
| Operational order | 15% |
| Scope discipline | 10% |
| Concision | 5% |

Composite score may appear in reports only:

```text
compositeScore = hardScore * 0.80 + softScore * 0.20
```

Composite score must never be the only promotion condition.

## Acceptance Gate

A candidate passes only when all conditions hold:

- frontmatter is byte-identical
- skill structure is valid
- no `TODO` markers exist
- no private data exists
- no provider-specific dependency is introduced
- no dangerous command guidance is introduced
- all critical cases pass
- no critical regression exists
- hard score does not worsen
- soft median improves by at least `0.02`
- at least one baseline-failing case improves
- candidate wins at least two of three repeated runs
- size and line-change limits are respected
- secondary harness has no regression
- `test` split passes
- `adversarial` split passes
- repository gates pass on a temporary copy

Pseudocode:

```js
const accepted =
  frontmatterIsEqual &&
  structureIsValid &&
  securityPassed &&
  criticalFailures === 0 &&
  criticalRegressions === 0 &&
  candidateHardScore >= baselineHardScore &&
  candidateSoftMedian >= baselineSoftMedian + 0.02 &&
  improvedCases.length >= 1 &&
  winningRuns >= 2 &&
  candidateTokenCount <= limits.maxCandidateTokens &&
  changedLinesPercent <= limits.maxChangedLinesPercent &&
  crossHarnessRegressionCount === 0 &&
  testPassed &&
  adversarialPassed &&
  repositoryGatesPassed;
```

## Size Control

Initial limits:

```yaml
maxCandidateTokens: 2200
maxGrowthPercent: 20
maxChangedLinesPercent: 25
maxEditsPerEpoch: 4
```

Effective token limit is the stricter of absolute token cap and growth cap.

## Frontmatter Protection

SkillOpt may propose changes to the skill body only. For the first version, frontmatter must remain byte-identical.

Protected fields:

```yaml
id:
name:
stack:
category:
status:
version:
owner:
triggers:
compatibility:
```

Skill versions remain coordinated with the NgAutoPilot release process and must not be managed by SkillOpt.

## SkillOpt Bridge

NgAutoPilot will not vendor Microsoft SkillOpt.

The lab Python package pins SkillOpt:

```toml
[project]
name = "ngautopilot-skill-lab"
version = "0.9.0"
requires-python = ">=3.10,<3.13"

dependencies = [
  "skillopt==0.2.*",
  "pyyaml>=6.0,<7",
  "jsonschema>=4.0,<5"
]
```

Only `skill-lab/python/ngautopilot_skillopt/bridge.py` should know SkillOpt internals. The Node side speaks a stable contract:

```json
{
  "benchmark": "angular-upgrade-validation-gate",
  "baselineSkill": "skill-lab/runs/<run-id>/baseline.SKILL.md",
  "outputDirectory": "skill-lab/runs/<run-id>/optimization",
  "epochs": 3,
  "editBudget": 4,
  "seed": 42
}
```

The bridge must load train and validation only, run SkillOpt, copy out `candidate.SKILL.md`, normalize results, and never read the test split or write canonical skills.

## Operational Commands

Planned package scripts:

```json
{
  "skill-lab:validate": "node skill-lab/scripts/validate-lab.mjs",
  "skill-lab:test": "node --test skill-lab/tests/**/*.test.mjs",
  "skill-lab:baseline": "node skill-lab/scripts/snapshot-baseline.mjs",
  "skill-lab:evaluate": "node skill-lab/scripts/evaluate-skill.mjs",
  "skill-lab:optimize": "node skill-lab/scripts/optimize-skill.mjs",
  "skill-lab:compare": "node skill-lab/scripts/compare-results.mjs",
  "skill-lab:gate": "node skill-lab/scripts/run-gate.mjs",
  "skill-lab:agentic-gate": "node skill-lab/scripts/run-agentic-gate.mjs",
  "skill-lab:prepare-promotion": "node skill-lab/scripts/generate-promotion-packet.mjs",
  "skill-lab:ci": "npm run skill-lab:validate && npm run skill-lab:test"
}
```

## Full Workflow

```bash
npm run skill-lab:validate
npm run skill-lab:test

npm run skill-lab:baseline -- \
  --benchmark angular-upgrade-validation-gate

npm run skill-lab:evaluate -- \
  --benchmark angular-upgrade-validation-gate \
  --skill <baseline> \
  --splits train,validation \
  --runs 3

npm run skill-lab:optimize -- \
  --benchmark angular-upgrade-validation-gate \
  --run <run-id> \
  --optimizerModel gpt-4.1-mini \
  --targetModel gpt-4.1-mini

npm run skill-lab:evaluate -- \
  --benchmark angular-upgrade-validation-gate \
  --skill <candidate> \
  --splits validation \
  --runs 3

npm run skill-lab:compare -- \
  --run <run-id>

npm run skill-lab:gate -- \
  --run <run-id> \
  --stage validation

npm run skill-lab:agentic-gate -- \
  --run <run-id> \
  --harness codex

npm run skill-lab:evaluate -- \
  --benchmark angular-upgrade-validation-gate \
  --skill <candidate> \
  --splits test

npm run skill-lab:evaluate -- \
  --benchmark angular-upgrade-validation-gate \
  --skill <candidate> \
  --splits adversarial

npm run skill-lab:prepare-promotion -- \
  --run <run-id>
```

Manual promotion comes after these commands and requires a branch, applying the reviewed diff, regenerating catalog/plugins, running `npm run release:validate`, and opening a draft pull request.

Default local model identifiers are `gpt-4.1-mini` for both optimizer and target roles. Use `gpt-4.1` for the optimizer only when the mini model produces low-quality edits, and keep the target model stable while comparing candidate runs.

## Security

All benchmark content must be synthetic, public-safe, and independent of private client, NSE, PENV, or corporate systems.

`security:scan` must be expanded to scan:

```text
skill-lab/benchmarks
skill-lab/config
skill-lab/lib
skill-lab/python
skill-lab/scripts
skill-lab/schemas
skill-lab/tests
```

It must exclude:

```text
skill-lab/runs
skill-lab/.cache
skill-lab/.venv
```

Generated evidence logs, raw prompts, raw responses, private config files, local model config, and virtual environments must be ignored by Git.

GitHub Actions with real model credentials must be manual `workflow_dispatch` jobs guarded by a protected `skillopt-lab` environment. Secrets must never run on fork pull requests or `pull_request_target`.

## Git Ignore Requirements

Add lab-local ignores:

```gitignore
# Skill optimization laboratory
skill-lab/.venv/
skill-lab/.cache/*
!skill-lab/.cache/.gitkeep

skill-lab/runs/*
!skill-lab/runs/.gitkeep

skill-lab/**/*.local.yaml
skill-lab/**/*.private.json
skill-lab/**/evidence.jsonl
skill-lab/**/raw-prompts/
skill-lab/**/raw-responses/

.env
.env.*
!.env.example
```

## Run Evidence

Every run must produce a manifest with hashes, model identifiers, benchmark version, seed, timestamps, and source commit.

```json
{
  "runId": "...",
  "repository": "janpereira-dev/ngAutoPilot",
  "repositoryCommit": "...",
  "skillPath": "skills/angular/upgrades/angular-upgrade-validation-gate/SKILL.md",
  "baselineHash": "...",
  "candidateHash": "...",
  "benchmarkId": "angular-upgrade-validation-gate",
  "benchmarkVersion": "1.0.0",
  "rubricHash": "...",
  "skilloptVersion": "v0_2_0",
  "optimizerBackend": "...",
  "optimizerModel": "...",
  "targetBackend": "...",
  "targetModel": "...",
  "seed": 42,
  "epochs": 3,
  "editBudget": 4,
  "startedAt": "...",
  "completedAt": "..."
}
```

Reports must include per-case outcomes, aggregate metrics, medians, deviations, regressions, improvements, size changes, diffs, hashes, and the reason for acceptance or rejection.

## Promotion Packet

`npm run skill-lab:prepare-promotion` must write:

```text
promotion/
├── candidate.SKILL.md
├── canonical.diff
├── gate-report.json
├── report.md
├── pr-body.md
├── evidence-summary.json
└── hashes.json
```

The command must not commit, push, write `skills/**`, publish, or create a pull request.

## Benchmark Versioning

Benchmarks use independent SemVer.

| Bump | Meaning |
| --- | --- |
| PATCH | Typo, description, non-scoring change |
| MINOR | New compatible cases, checks, fixtures, optional harnesses |
| MAJOR | Scoring, thresholds, states, output contract, split policy, or rubric changes |

Material benchmark changes invalidate previous baselines.

## Lab Test Requirements

The lab must test itself with `node:test`.

Required test coverage:

- JSONL loading
- duplicate ID rejection
- split overlap rejection
- unknown check rejection
- frontmatter change detection
- critical regression detection
- no soft-score compensation for critical regressions
- candidate size limit rejection
- reproducible hashes
- token redaction
- path containment inside `skill-lab`
- no write permissions to `skills/**`
- dangerous script rejection

## GitHub Actions

Static workflow:

```text
.github/workflows/skill-lab-static.yml
```

Triggers: `pull_request`, `push` to `main`.

Steps: checkout, Node 24, Python 3.11, `npm run skill-lab:validate`, `npm run skill-lab:test`, `npm run release:validate`.

Manual optimization workflow:

```text
.github/workflows/skill-lab-optimize.yml
```

Trigger: `workflow_dispatch` with benchmark, target skill, optimizer model, target model, and epochs.

It may generate artifacts only. It must not modify branches, create PRs, or promote candidates.

## Pilot Definition Of Done

- `skill-lab/` exists.
- Architecture spec exists.
- Policy exists.
- Schemas are versioned.
- Four splits exist.
- Duplicate cases are rejected.
- Fixtures are synthetic.
- Deterministic evaluator has tests.
- Baseline is recorded.
- SkillOpt can generate a candidate outside `skills/**`.
- Frontmatter is protected.
- Gate detects individual regressions.
- Repeated validation exists.
- Test split is independent.
- Adversarial split exists.
- Agentic sandbox exists.
- Cross-harness comparison exists.
- Promotion packet contains hashes.
- Auto-adopt does not exist.
- `release:validate` passes with the candidate applied only in a temporary copy.
- Promotion requires pull request review.
- Evidence explains why the candidate is better or rejected.

## Anti-Patterns

Do not:

- write candidates directly to `skills/**`
- run `skillopt-sleep adopt` or auto-adopt
- optimize against the test split
- accept because the average score improved
- use a single case as a benchmark
- expose model secrets to fork PRs
- ship SkillOpt in the npm package
- optimize the global orchestrator first
- mix validation-gate, route-planner, and orchestrator benchmarks
- use real NSE, PENV, client, or corporate examples
- let a model define its own checks

## Final Position

The approved NgAutoPilot design is a minimal executable lab with controlled benchmark data, SkillOpt as an external pinned optimizer, isolated candidate artifacts, deterministic hard checks, individual no-regression gates, independent test and adversarial splits, future agentic sandbox validation, cross-harness promotion checks, and manual pull-request promotion.

The first priority is not integrating SkillOpt. The first priority is proving whether a skill is actually better, with evidence.