![DARKSOL](./assets/darksol-banner.png)
Built by DARKSOL 🌑

# modelab 🌑 — Repeatable Multi-Model Research

[![npm version](https://img.shields.io/npm/v/modelab)](https://www.npmjs.com/package/modelab)
[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
[![platform](https://img.shields.io/badge/platform-Windows%20%7C%20Linux%20%7C%20macOS-blue)](./)
[![node >=18](https://img.shields.io/badge/node-%3E%3D18-339933.svg)](https://nodejs.org)

> Run the same research question across multiple models, score the results, remember what worked, and get a concrete recommendation for what to test next.

modelab is used internally at DARKSOL for systematic research — comparing model strategies, pressure-testing prompts, auditing model behavior, and generating reproducible reports.

It is intentionally **model-agnostic underneath** and **workflow-opinionated on top**.
The point is not just to run prompts. The point is to make research runs **repeatable, comparable, and easier to improve over time**.

---

## What it does best

modelab is strongest when you want to:

- compare multiple model approaches to the same task
- keep a memory of what worked on similar past questions
- inspect routing choices instead of blindly trusting a default model
- get a suggested next experiment instead of just a pile of logs

If you only want a single prompt runner, this is overkill.
If you want a lightweight research loop with memory and planning, this is the lane.

---

## Product Thesis

modelab is built around one loop:

1. **Run** parallel experiments
2. **Score** outputs against a rubric
3. **Remember** lessons and summaries
4. **Recall** similar prior work
5. **Plan** the next best experiment

If it only ran experiments, it would be another eval harness.
The goal is to become an engine that learns from prior runs and gets sharper every cycle.

---

## Install

```bash
npm install modelab
modelab config --wizard   # guided setup for preset + export defaults
```

---

## Quick Start

```bash
# Run a research question across 3 model strategies in parallel
modelab run --goal "What causes migraines and what do the best treatments look like?"

# Check setup health before a real run
modelab doctor

# Ask modelab what it should test next
modelab plan --goal "Compare Postgres vs DynamoDB for a startup"

# Let modelab propose the next run and execute it immediately
modelab next --goal "Compare Postgres vs DynamoDB for a startup" --run

# Explain why routing picked a model
modelab route --task "Compare Postgres vs DynamoDB for a startup"

# Compare specific models head-to-head
modelab run --goal "Explain ZK rollup economics" --arms balanced,reasoning

# Use a built-in template
modelab run --goal "Review my REST API design" --template code-review

# Watch streaming tokens arrive in real-time
modelab run --goal "Write a生日贺卡" --template creative --stream

# Export a finished run to a shareable HTML report
modelab export <run-id> --format html --share
```

---

## Why it exists

Most AI tooling stops at execution.
modelab is for when you want a system that can:

- compare multiple approaches instead of trusting one answer
- preserve lessons instead of starting fresh every run
- route work using learned performance, not just keywords
- surface similar past work when a new question looks familiar
- recommend the next experiment instead of leaving you with raw logs
- turn that recommendation into an actual run without manual copy/paste

That makes it useful for:

- prompt and workflow R&D
- model comparison and provider audits
- research-heavy product decisions
- repeatable internal evaluation loops

---

## First-run checklist

```bash
npm install modelab
modelab config --wizard
modelab doctor
modelab plan --goal "Review my API design"
modelab run --goal "Review my API design"
```

If `modelab doctor` shows missing auth, add the right provider key and run it again before blaming the package.

If you want a different starting lane without prompts, use a preset directly:

```bash
modelab config --presets
modelab config --init --preset cheap
# or
modelab config --wizard --yes --preset research-heavy --default-format html --output-dir shared-reports
```

---

## What it does not do yet

- It does **not** automatically guarantee better research outcomes without a decent scoring setup.
- It does **not** replace a human reviewer for high-stakes decisions.
- It does **not** magically infer perfect provider credentials or budgets.
- Its learned routing is only as good as the run history you have built.

---

## What’s new in v0.6

### Research Planning
`modelab plan` turns routing, run history, lessons, model insight profiles, and semantic recall into a recommended next experiment. Instead of only telling you what happened, modelab now suggests what to run next: template, iterations, threshold, arms, a copy-paste command, and where to inspect/share the run after it finishes.

### One-Step Next Run
`modelab next` closes the gap between planning and execution. It uses the same routing, lessons, history, and semantic recall as `plan`, but can immediately launch the recommended experiment with `--run`. That makes the core workflow feel more like a real research loop instead of a planning screen plus manual follow-up.

### Guided Setup + Shareable Reports
modelab now has a guided first-run path via `modelab config --wizard` and a cleaner report handoff via `modelab export <run-id> --format html --share`. That means you can go from setup → plan → run → share without stitching together a bunch of one-off commands.

### Actionable Self-Iteration (lesson_engine)
Previous versions stored "lessons" as advisory notes nobody read. v0.4 closes the loop: after each run, the lesson_engine parses the scorer output, writes actual router adjustments to the DB, and the routing layer applies them before the next run. Score below 4? Model gets a -1 penalty. Score above 8? It gets boosted for that task type. Lessons become config changes — automatically.

### Model Profiles
Every model maintains a performance profile updated after each run: avg_score, avg_latency_ms, avg_cost_usd, strengths[], weaknesses[]. The router reads this, not just keyword matching.

### Semantic Memory (embedding_store)
Run summaries and lessons are embedded using TF-IDF vectors (Ollama nomic-embed-text when available) and stored in SQLite. Query past experiments semantically: `modelab recall "what did we learn about coding tasks?"`

### Learned Routing (routing_v2)
Replaces the keyword router with a performance-based router that considers: model strengths for the task type, historical scores, active adjustments, and similar past runs.

### Campaign Layer
Multi-run research campaigns: `modelab campaign new "My Hypothesis" --runs 10 --synthesize`. Coordinates a series of runs, synthesizes findings, and generates reports.

### Task Complexity Profiling
Goals are analyzed for complexity (question marks, structure, length) before routing. High-complexity tasks get more iterations; low-complexity get faster models.

### LCM Memory v2 — Cross-Run Persistence + Cross-Iteration Learning
Every run writes iteration summaries and full run summaries to SQLite. Before running, modelab loads prior context for the same goal ID — so lessons from last week actually influence today's experiment. The `iteration_context` template variable carries these lessons into new prompts automatically.

### Tiktoken Token Counting
Replaced rough `length/4` estimation with the GPT-2 vocab Tiktoken encoder. Token counts and cost estimates are now accurate to the actual tokenization scheme of the model being used.

### Proactive Rate-Limit Backoff
`RateLimitTracker` tracks 429 responses and their `Retry-After` headers. Before making a call, the system checks whether the endpoint is currently throttled and waits proactively — not just retrying after failure but preventing it.

### TTFT Latency Stats
Time-to-first-token is measured per arm per call and aggregated into p50/p95 stats. These are shown in comparison tables and stored in run logs. Arms can be configured with a `latencyTargetMs` to skip models that are historically slow.

### GLM-5.0 Routing
The keyword router now recognizes `glm`, `glm-5`, `glm5`, `智谱`, `zhipu` and routes to the GLM-5.1 model — enabling research on Chinese-language models and frontier Chinese providers.

### Cross-Run Learning System
- `outputPreview` (first 200 chars) and `outputTruncated` (boolean) stored in DB and cache
- `experiments` command: `modelab experiments --sort score|cost|date` for at-a-glance run history
- `review` command: `modelab review <run-id>` for detailed latency + lesson breakdown
- migration hardening for older local SQLite installs so new memory fields upgrade cleanly instead of breaking existing environments

### Structured Scorer with Optional Sub-Fields
The LLM judge returns `{ score, reasoning, clarity?, correctness?, completeness? }`. Sub-fields are optional — the scorer still works if the judge skips them.

---

## CLI Commands

```bash
modelab run --goal "..." [--iterations N] [--threshold N] [--arms m1,m2]
              [--template id] [--format json|md|html] [--output path]
              [--stream] [--no-cache]
              Run a research experiment

modelab experiments [--sort score|cost|date]    View all runs at a glance
modelab experiments --goal-id "..." --status completed
modelab review <run-id>                         Deep-dive: latency + lesson breakdown
modelab report <run-id>                         Winner summary + per-arm breakdown
modelab recall "what did we learn about..."    Semantic search across runs + lessons
modelab plan --goal "..."                      Recommend the next best experiment
modelab next --goal "..." [--run]             Generate the next best experiment and optionally run it now
modelab memory inspect --goal-id "..."         Inspect stored runs + lessons for a goal
modelab history                                 Show run history
modelab best [--goal-id]                        Show best result for a goal
modelab templates                               List built-in prompt templates
modelab export <run-id> [--format json|md|html] [--share] Re-export a past run with executive summary
modelab route --task "..."                      Show model routing decision
modelab cache --clear                           Clear the result cache
modelab config --init                           Create ~/.modelab/config.json
modelab config --wizard                         Guided setup for preset + export defaults
modelab config --presets                        Show starter config presets
modelab config --list                           Show current config
```

---

## Built-in Templates

| Template | Use case |
|----------|----------|
| `research` | Deep multi-perspective research |
| `code-review` | Bugs, security, performance review |
| `architecture` | System design and trade-offs |
| `bug-hunt` | Adversarial failure-mode analysis |
| `compare` | A/B decisions with scoring |
| `quick-answer` | Fast, concise responses |
| `creative` | Brainstorming and ideation |

---

## Model Routing

Routing is no longer just static keyword matching. modelab combines:

- task-shape heuristics
- learned model profiles
- active lesson adjustments
- prior run performance
- similar historical runs

The keyword layer still exists as a fallback/default shape:

| Task | Keywords | Routes to |
|------|----------|-----------|
| coding | code, refactor, bug, build, PR, function, class | `coding` |
| reasoning | proof, logic, analysis, theorem, 证明,推理 | `reasoning` |
| glm | glm, glm-5, glm5, 智谱, zhipu | `glm` |
| quick | quick, summary, what is, define, 什么是 | `fast` |
| default | everything else | `balanced` |

Override with `--arms fast,balanced,reasoning` or configure explicitly in config.

---

## Opinionated by workflow, not locked by provider

modelab should stay broad at the infrastructure layer:

- bring your own providers
- bring your own models
- bring your own scoring budgets

But at the product layer, it is intentionally opinionated:

- experiments should be comparable
- lessons should persist
- memory should influence future runs
- planning should be explicit
- research quality should compound over time

---

## Config

`~/.modelab/config.json`:

```json
{
  "models": {
    "fast":      { "provider": "openai",   "model": "gpt-4o-mini",             "costPerMillionInput": 0.15,  "costPerMillionOutput": 0.60 },
    "balanced":  { "provider": "anthropic", "model": "claude-sonnet-4-6",        "costPerMillionInput": 3,     "costPerMillionOutput": 15 },
    "reasoning": { "provider": "openai",   "model": "o1",                       "costPerMillionInput": 15,    "costPerMillionOutput": 60 },
    "coding":    { "provider": "ollama",   "model": "qwen3-coder",              "baseUrl": "http://localhost:11434" },
    "glm":       { "provider": "openai",   "model": "glm-z1-air",               "baseUrl": "https://open.bigmodel.cn/api/paas/v4" },
    "groq":      { "provider": "groq",     "model": "llama-3.3-70b-versatile",  "costPerMillionInput": 0.2,   "costPerMillionOutput": 0.8 }
  },
  "evalModel": "balanced",
  "budget": { "maxPerRun": 2.0, "maxPerExperiment": 0.5, "trackCosts": true },
  "parallelism": 3
}
```

---

## Use as a Library

```typescript
import { ResearchOrchestrator } from 'modelab';

const orch = new ResearchOrchestrator({
  models: {
    balanced: { provider: 'anthropic', model: 'claude-sonnet-4-6', costPerMillionInput: 3, costPerMillionOutput: 15 },
    reasoning: { provider: 'openai', model: 'o1', costPerMillionInput: 15, costPerMillionOutput: 60 },
  },
  budget: { maxPerRun: 2.0, maxPerExperiment: 0.5, trackCosts: true },
  evalModel: 'balanced',
  parallelism: 3,
  onProgress: msg => console.log(msg),
  onArmComplete: r => console.log(`Arm done: ${r.armId} → ${r.score}/10`),
});

const log = await orch.run({
  id: 'my-goal',
  question: 'What is the optimal block time for Ethereum L2s?',
  goal: 'Provide a technically rigorous analysis',
  qualityThreshold: 7.5,
  maxIterations: 3,
  arms: [
    { id: 'arm-1', name: 'balanced', model: 'balanced', promptTemplate: '...' },
    { id: 'arm-2', name: 'reasoning', model: 'reasoning', promptTemplate: '...' },
  ],
});

console.log(log.bestResult);
console.log(`Total cost: $${log.totalCostUsd}`);
```

You can also use the planning surface directly:

```typescript
import { buildResearchPlan } from 'modelab';
```

---

## Environment Variables

| Variable | Provider |
|---------|---------|
| `OPENAI_API_KEY` | OpenAI, Groq, OpenRouter, Perplexity |
| `ANTHROPIC_API_KEY` | Anthropic |
| `MINIMAX_API_KEY` | MiniMax |
| `GROQ_API_KEY` | Groq |
| `GEMINI_API_KEY` | Google Gemini |
| `PERPLEXITY_API_KEY` | Perplexity |

---

## Architecture

```
ResearchOrchestrator
  ├── router.ts         — keyword heuristic → best-fit model (GLM-5.0 aware)
  ├── routing_v2.ts    — learned routing: performance-based, reads from lesson_engine
  ├── evaluator.ts     — streaming calls across 10 providers, rate-limit tracking
  ├── scorer.ts        — LLM judge: structured rubric, Zod validation, retry
  ├── orchestrator.ts  — parallel arms, quality gate, budget guard, TTFT tracking
  ├── planner.ts       — recommends the next best experiment from routing + memory + recall
  ├── lesson_engine.ts — self-iteration: parses lessons, writes router adjustments
  ├── embedding_store.ts — TF-IDF / Ollama embeddings for semantic memory
  ├── complexity.ts    — task complexity profiling
  ├── memory.ts        — SQLite: ~/.modelab/memory.db (LCM Memory v2)
  ├── cache.ts         — SHA-256 hash cache: ~/.modelab/cache.json
  ├── templates.ts     — 7 built-in prompt templates
  └── export.ts        — json / markdown / html reports
```

---

## Links + License

- npm: <https://www.npmjs.com/package/modelab>
- Issues: <https://github.com/darks0l/modelab/issues>
- Changelog: [CHANGELOG.md](./CHANGELOG.md)

MIT License

Built with teeth. 🌑