# π€― pi-reason-harness
**Recursive self-improving reasoning harness for [pi](https://github.com/earendil-works/pi-coding-agent)**
_20-layer meta-system that discovers, adapts, evolves, transfers, and validates strategies autonomously._
[](https://github.com/earendil-works/pi-coding-agent)
[](./LICENSE)
---
Builds task-specific reasoning strategies on top of any LLM by running iterative solve-verify-feedback loops with multi-expert ensembling, voting, and a **20-layer meta-system** that discovers, adapts, evolves, transfers, and validates strategies autonomously.
**JS-exclusive** β LLM calls go through pi's native LLM infrastructure (`@earendil-works/pi-ai`). Code sandbox uses Node's `vm` module. Zero Python dependency.
## How It Works
The core insight (from first-principles analysis of SOTA reasoning systems): **LLMs are knowledge stores that require intelligent probing strategies to extract reliable answers.** The harness layer (open-source) iteratively generates, verifies, and refines. The meta-system layer (proprietary, rebuilt here) discovers and evolves the strategies themselves.
### The 20-Layer Meta-System
| Layer | Name | What it does |
|-------|------|-------------|
| 0 | **Problem Critic** | Inspects problems, proposes *targeted deltas* to proven templates (not writing from scratch) |
| 1 | **Strategy Library** | Persistent store of proven strategies with ROI + quality metrics |
| 2 | **Meta-Rule Engine** | Extracts cross-strategy principles that compound over time |
| 3 | **Model Router** | Thompson sampling for intelligent model selection per category |
| 4 | **Budget Bandit** | Early stopping, budget reallocation, re-exploration when stuck |
| 5 | **Auto-Trigger** | Self-improvement runs automatically (on success rate drops, new categories, periodic) |
| 6 | **Recursive Harness Generation** | Generates entire solve approach configurations (the "solver of solvers") |
| 7 | **Ensemble Diversification** | Each expert uses a fundamentally different approach strategy |
| 8 | **Sub-problem Decomposition** | Break hard problems into sub-problems, solve independently, combine |
| 9 | **Budget Optimization** | Marginal ROI estimation, reallocate iterations to high-ROI experts |
| 10 | **Cross-Domain Transfer** | Transfer proven strategies across analogous categories automatically |
| 11 | **Confidence-Weighted Voting** | Weight votes by self-assessed quality, not just output match |
| 12 | **Progressive Difficulty** | Train on easiest examples first, build up to harder ones |
| 13 | **Auto-Transfer** | Automatically transfer strategies when new categories are encountered |
| 14 | **Per-Problem Prompt Synthesis** | Generate + validate specialized prompts for novel problem types |
| 15 | **Meta-Meta Level** | Harness-of-harnesses β generate new approach types from performance data |
| 16 | **Gradient-Based Budget Optimization** | Trajectory-based improvement estimation with finite-difference gradients |
| 17 | **Recursive Meta-Meta Nesting** | Meta-harnesses feed back into solve; recursive evolution of underperformers |
| 18 | **Multi-Model Decomposition** | Route sub-questions to different models in parallel based on strengths |
| 19 | **Per-Iteration Prompt Adaptation** | Evolve the solver prompt mid-solve based on failure patterns |
| 20 | **ARC-AGI Benchmark Integration** | Validate against real ARC-AGI-2 challenges with scoring |
### Layer 6: Recursive Harness Generation β the "solver of solvers"
The biggest gap with Poetiq: their open-source code shows ONE harness configuration with fixed prompts. Their blog results prove they generate MULTIPLE different configurations per problem type.
A **HarnessSpec** defines a complete solve approach:
- **Approach type**: code-sandbox, decomposition, chain-of-questions, analogy, counter-factual, exhaustive-search, code-direct
- **Solver/feedback prompts**: Full templates with `$$problem$$` placeholders
- **Config overrides**: Temperature, iterations, reasoning level
- **Decomposition config**: Max sub-problems, depth, combine strategy
- **Validation data**: Score on held-out data, production stats
The system generates multiple specs per problem, validates them, and evolves them over time.
### Layer 7: Ensemble Diversification
Instead of N experts with the same prompt (just different seeds/models), each expert uses a **fundamentally different approach**:
| Expert | Approach | When to use |
|--------|----------|-------------|
| 1 | code-sandbox | Grid/array problems β generate code, execute, verify |
| 2 | decomposition | Complex problems β break into sub-problems |
| 3 | analogy | Hard problems β solve simpler version first |
| 4 | chain-of-questions | Knowledge tasks β hierarchical probing |
| 5 | counter-factual | Stubborn problems β generate wrong solutions, invert |
| 6 | exhaustive-search | Small search spaces β enumerate, filter |
Each approach has its own specialized prompt template.
### Layer 8: Sub-problem Decomposition
For hard problems that resist direct solving, the decomposer breaks them into independent sub-problems:
1. LLM analyzes the problem and proposes 2-4 sub-problems
2. Each sub-problem is solved independently
3. Sub-solutions are combined (sequentially, in parallel, or hierarchically)
Example: "Rotate 90Β° clockwise" β Sub-problem 1: "Transpose the grid" β Sub-problem 2: "Reverse each row"
### Layer 9: Budget Optimization via Marginal ROI
Not just "stop when stuck" but **"spend where ROI is highest"**:
- Estimate marginal ROI per expert based on recent improvement rate
- Reallocate remaining iterations to experts with highest expected improvement
- Phase-based execution: run all experts for half-iterations, then reallocate
### Layer 10-13: Cross-Domain Transfer + Auto-Transfer
When a strategy works in one domain, the system **automatically transfers** it to analogous domains:
- Category similarity map: grid-transformation β pattern-completion β spatial-reasoning
- Transfer adapts domain-specific parts while keeping universal insights
- Auto-triggered when a new category is encountered with no existing strategies
- Creates both a strategy entry and a harness spec for the new category
### Layer 11: Confidence-Weighted Voting
Voting is weighted by **self-assessed quality**:
- Solutions that pass in fewer iterations count more (efficiency bonus)
- Solutions with high soft scores count more (partial accuracy bonus)
- Failed solutions grouped by output similarity, ranked by total confidence
### Layer 12: Progressive Difficulty
Training examples are ordered from **easiest to hardest**:
- Difficulty proxy: grid size + unique value count + input/output asymmetry
- The solver sees simpler patterns first, building up to complex ones
- Mirrors Poetiq's per-iteration shuffle but with intelligence
### Layer 14: Per-Problem Prompt Synthesis
For truly novel problems where no proven strategy exists, the system **synthesizes specialized prompts**:
- Computes a problem fingerprint based on structural features (grid size, unique values, operation type)
- If a validated synthesized prompt matches the fingerprint, uses it instead of the generic template
- If no match, generates a new specialized prompt via LLM and validates it on training data
- Prompts with validation score > 0.5 are persisted for future use
- Fingerprint-based matching allows cross-problem generalization
### Layer 15: Meta-Meta Level β Harness-of-Harnesses
The biggest architectural gap with Poetiq: their system doesn't just generate strategies, it generates **new types of harness approaches**. Our meta-meta level:
- Analyzes performance data across all harness specs and meta-harnesses
- Uses LLM to propose NEW approach types that combine strengths of successful ones
- Each meta-harness has a name, description, solver prompt, config overrides, and rationale
- Meta-harnesses can evolve (generation counter, parent lineage) like strategies
- Example: "Decomposed-Sandbox-Synthesis" β combines decomposition's cognitive offloading with code-sandbox's deterministic verification
### Layer 16: Gradient-Based Budget Optimization
Replaces simple proportional reallocation with **finite-difference gradient estimation**:
- Estimates dScore/dIteration (improvement rate) using a 5-point window
- Estimates dΒ²Score/dIterationΒ² (acceleration/deceleration)
- Predicts expected next score: current + gradient + 0.5 Γ acceleration
- Allocates iterations proportional to (expected improvement Γ confidence)
- Detects when an expert should switch approaches: stuck (gradient β 0) + decelerating (acceleration < 0)
### Layer 17: Recursive Meta-Meta Nesting
Meta-harnesses don't just get generated β they **feed back into solve**:
- `selectMetaHarnessExpertConfig()` assigns the best meta-harness to one expert in the ensemble
- Meta-harness performance is tracked (useCount, avgScore, successCount)
- Underperforming meta-harnesses (useCount β₯ 2, avgScore < 0.5, generation < 3) are **recursively evolved** via `recursiveMetaEvolve()`
- This creates a true recursive loop: solve β generate meta-harness β use in solve β evolve if underperforming β repeat
### Layer 18: Multi-Model Decomposition
When multiple models are available, the system **routes sub-questions to the best-suited model**:
- `decomposeAndRoute()`: LLM analyzes the problem and assigns sub-problems to models based on heuristic strengths
- Model strength heuristics: Anthropic (complex reasoning, code), OpenAI (math, creative), Google (multimodal), Groq (fast), Wafer (reasoning), DeepSeek (code, math)
- Dependency tracking: sub-problems can depend on previous results
- `solveRoutedDecomposition()`: solves each sub-problem with its assigned model, combines results
- Triggered automatically in solve when `models.length > 1` and `useMeta=true`
### Layer 19: Per-Iteration Prompt Adaptation
The solver prompt **adapts mid-solve** based on failure patterns:
- After 3 consecutive failed iterations (score < 0.5), `adaptPromptMidSolve()` is called
- LLM analyzes the failure trajectory and suggests a prompt modification
- Three adaptation types: `pre-insert` (add before problem), `anti-pattern` (warn after problem), `section-replace` (replace a named section)
- `applyIterationAdaptation()` modifies the prompt for subsequent iterations
- The adaptation persists within the expert's solve loop
### Layer 20: ARC-AGI Benchmark Integration
The system can **validate against real ARC-AGI-2 challenges**:
- `loadArcChallenges()`: loads challenges from ARC-AGI JSON files
- `runArcBenchmark()`: runs the harness on a batch of challenges with budget limits
- Re-verifies test outputs against ground truth (when available)
- Computes: solved, partial solved, avg best score, total cost, total time
- CLI: `pi-reason-harness arc-benchmark --data-path ... --max-challenges 5`
- **Benchmark results with wafer/GLM-5.1**: 5/7 unique challenges solved (71%), 1 near-miss (0.97), cost ~$0.04/challenge
### Layers 0-5: Core Meta-System
These were implemented in the previous iteration and remain the foundation:
- **Layer 0: Critique, Don't Create** β The critic receives proven templates and proposes targeted deltas (insertions, anti-patterns, examples). This is code review, not writing from zero.
- **Layer 2: Meta-Rules Compound** β When a child strategy outperforms its parent, generalizable principles are extracted and applied to other categories.
- **Layer 3: Thompson Sampling** β Beta(Ξ±,Ξ²) sampling with Laplace smoothing picks the best model per category.
- **Layer 4: Budget Bandit** β Early stopping, re-exploration when all experts fail.
- **Layer 5: Auto-Trigger** β Runs automatically on success rate drops, new categories, and every 5th problem.
### The Harness Layer
Below the meta-system, the harness implements the iterative solve loop with Poetiq-parity features:
1. **Iterative solve-verify-feedback loops** β Generate code, sandbox-execute, build detailed feedback
2. **Multi-expert ensembling** β Parallel experts with diverse approaches
3. **Confidence-weighted voting** β Group by output, rank by confidence
4. **Poetiq-parity feedback** β Element-by-element diff grids, shape mismatch detection
5. **Poetiq-parity formatting** β `` text with Fisher-Yates shuffle
6. **Self-audit verification** β LLM checks its own answers
7. **Budget tracking** β Per-problem cost/time limits
### Task Types
| Type | Strategy | Verification |
|------|----------|--------------|
| `code-reasoning` | Generate JavaScript code β sandbox execute β verify against examples β feedback loop | Sandbox (default) or external |
| `knowledge-extraction` | Chain-of-questions probing β self-audit β confidence bucketing | Self-audit (recommended) |
| `hybrid` | Decide per-problem: code or direct answer β verify β feedback | Any method |
### Approach Types (for ensemble diversification)
| Approach | Description | Best for |
|----------|-------------|----------|
| `code-sandbox` | Generate JS code, execute in sandbox, verify output | Grid/array transformations |
| `code-direct` | Generate code, extract answer without execution | Computation-heavy |
| `decomposition` | Break into sub-problems, solve each, combine | Multi-step problems |
| `chain-of-questions` | Hierarchical probing from broad to specific | Knowledge questions |
| `analogy` | Solve simpler version first, then scale up | Hard spatial problems |
| `counter-factual` | Generate wrong solutions, analyze failures, invert | Stubborn problems |
| `exhaustive-search` | Enumerate possibilities, filter by constraints | Small search spaces |
### Persistent Data
The meta-system persists across server restarts at `~/.pi-reason-harness/`:
| File | Contents |
|------|----------|
| `strategies.json` | Strategy library with ROI, quality metrics, lineage |
| `meta-rules.json` | Cross-strategy principles with validation stats |
| `model-routes.json` | Per modelΓcategory routing stats |
| `harness-specs.json` | Complete harness specifications per categoryΓapproach |
| `synthesized-prompts.json` | Per-problem-type specialized prompts with validation |
| `meta-harnesses.json` | Generated approach types with evolution lineage |
## Quick Start
```bash
# Initialize a reasoning session
pi-reason-harness init --name "ARC solver" --type code-reasoning \
--models '["anthropic/claude-sonnet-4-5","openai/gpt-4o"]' --num-experts 3
# Solve with the full 13-layer meta-system pipeline
pi-reason-harness solve --meta --problem "Transform the grid..." \
--train-inputs '[[1,2],[3,4]]' \
--train-outputs '[[4,3],[2,1]]' \
--test-inputs '[[5,6]]'
# Analyze a problem without solving
pi-reason-harness meta-analyze --problem "Rotate a 2x2 grid 90 degrees clockwise"
# Decompose a hard problem into sub-problems
pi-reason-harness decompose --problem "Rotate a 3x3 grid 90 degrees clockwise. Input: [[1,2,3],[4,5,6],[7,8,9]]"
# Check harness specs
pi-reason-harness harness-specs
# Evolve the worst-performing spec
pi-reason-harness evolve-harness
# Check the strategy library
pi-reason-harness strategies
# Transfer a strategy from grid-transformation to pattern-completion
pi-reason-harness transfer --source-category grid-transformation --target-category pattern-completion
# Check meta-rules
pi-reason-harness meta-rules
# Check model routing stats
pi-reason-harness model-routes
```
## Architecture
```
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β META-SYSTEM V3 (16 layers β the proprietary layer) β
β β
β Layer 0: Problem Critic (critique-don't-create) β
β Layer 1: Strategy Library (ROI + quality metrics) β
β Layer 2: Meta-Rule Engine (cross-strategy principles) β
β Layer 3: Model Router (Thompson sampling) β
β Layer 4: Budget Bandit (early stopping + re-explore) β
β Layer 5: Auto-Trigger (self-improving loop) β
β Layer 6: Recursive Harness Generation (solver-of-solvers)β
β Layer 7: Ensemble Diversification (different approaches) β
β Layer 8: Sub-problem Decomposition (break & combine) β
β Layer 9: Budget Optimization (marginal ROI realloc) β
β Layer 10: Cross-Domain Transfer (analogous categories) β
β Layer 11: Confidence-Weighted Voting (quality-ranked) β
β Layer 12: Progressive Difficulty (easiest-first) β
β Layer 13: Auto-Transfer (new category handling) β
β Layer 14: Per-Problem Prompt Synthesis (novel types) β
β Layer 15: Meta-Meta Level (harness-of-harnesses) β
β Layer 16: Gradient-Based Budget Optimization β
β Layer 17: Recursive Meta-Meta Nesting (harnessβsolve) β
β Layer 18: Multi-Model Decomposition (model routing) β
β Layer 19: Per-Iteration Prompt Adaptation (mid-solve) β
β Layer 20: ARC-AGI Benchmark Integration (validation) β
βββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββ
β generates (with deltas + rules + specs)
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β HARNESS (iterative solve-verify-feedback) β
β β
β βββββββββββββββ ββββββββββββββββ βββββββββββββββ β
β β Expert 1 β β Expert 2 β β Expert N β β
β β code-sandboxβ β decompositionβ β analogy β β
β β (pi-ai β β (pi-ai β β (pi-ai β β
β β LLM call β β LLM call β β LLM call β β
β β + sandbox β β + sub- β β + analogy β β
β β + verify β β solve β β + verify β β
β β + feedback β β + combine β β + feedback β β
β ββββββ¬βββββββββ βββ¬βββββββββββββ ββββ¬βββββββββββ β
β βββββββββββββββΌβββββββββββββββββββ β
β βΌ β
β CONFIDENCE-WEIGHTED VOTING β
β (group by output, rank by confidence) β
β β β
β βΌ β
β LEARN + ADAPT + EVOLVE + TRANSFER β
β (update strategies, extract rules, evolve specs, β
β transfer to new categories, auto-improve) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
## CLI Reference
| Command | Description |
|---------|-------------|
| `init` | Initialize session with task config, models, verification |
| `solve` | Run iterative solve-verify-feedback loop |
| `status` | Show session state, budget, learned adaptations |
| `results` | Show iteration results |
| `learn` | Inspect strategy adaptations |
| `reset-learn` | Clear learned strategies |
| `clear` | Clear session |
| `meta-analyze` | Analyze a problem with the critic (no solving) |
| `meta-improve` | Manually trigger strategy evolution + rule extraction |
| `strategies` | List strategy library with ROI + quality metrics |
| `meta-rules` | List meta-rules with validation stats |
| `model-routes` | List model routing stats per modelΓcategory |
| `harness-specs` | List harness specifications with validation + production stats |
| `evolve-harness` | Evolve the worst-performing harness spec |
| `transfer` | Transfer strategy from one category to another |
| `decompose` | Decompose a problem into sub-problems |
| `synth-prompts` | List synthesized prompts with validation stats |
| `meta-harnesses` | List meta-harnesses (generated approach types) |
| `generate-meta-harness` | Generate a new approach type from performance data |
| `arc-benchmark` | Run ARC-AGI benchmark validation against real challenges |
| `route-decompose` | Decompose a problem across multiple models |
### init flags
`--name`, `--type`, `--models`, `--num-experts`, `--verification`, `--verify-command`, `--max-cost`, `--max-time`
### solve flags
`--problem`, `--train-inputs`, `--train-outputs`, `--test-inputs`, `--meta` / `-m`
### transfer flags
`--source-category`, `--target-category`
### decompose flags
`--problem`
## LLM Integration
The harness uses `@earendil-works/pi-ai` for all LLM calls. Models are specified in `provider/model` format (e.g., `anthropic/claude-sonnet-4-5`, `openai/gpt-4o`). API keys are resolved from the same environment variables pi uses:
- `ANTHROPIC_API_KEY` β Anthropic models
- `OPENAI_API_KEY` β OpenAI models
- `GEMINI_API_KEY` β Google models
- `GROQ_API_KEY` β Groq models
- `WAFER_API_KEY` β Wafer Pass models (GLM-5.1, Qwen3.5-397B-A17B)
- etc.
### Custom Providers
The harness also supports custom providers (like Wafer Pass) that aren't in pi-ai's built-in model registry. Custom providers use direct OpenAI-compatible API calls. Currently supported:
| Provider | Base URL | Models | Notes |
|----------|----------|--------|-------|
| `wafer` | `https://pass.wafer.ai/v1` | `GLM-5.1`, `Qwen3.5-397B-A17B` | Reasoning models with `reasoning_content` field |
To add a new custom provider, add it to the `CUSTOM_PROVIDERS` map in `server.ts`.
No additional setup required β if pi can call the model, so can the harness.
## Tests
```bash
npm test
```
104 tests covering: vm sandbox, formatProblem, arrayDiff, buildDetailedFeedback, PromptDelta application, budget bandit, Thompson sampling, meta-rule engine, prompt quality metrics, harness specs, ensemble diversification, budget optimization, cross-domain transfer, confidence-weighted voting, progressive difficulty, decomposition, problem fingerprinting, synthesized prompts, meta-harnesses, gradient estimation, approach switching, recursive meta-meta nesting, multi-model decomposition routing, per-iteration prompt adaptation, ARC-AGI benchmark.
## License
MIT