# Benchmark & Sprint Burndown Open Orchestra measures the effectiveness of AI-assisted development across three modes and generates a sprint burndown from story point estimates. All cycle time and quality data is computed automatically from the event log — no manual input is required after the initial estimate declaration. ## Three Modes | Mode | How measured | |------|-------------| | **Solo (no LLM)** | Declared by PM or architect at story start — contrafactual estimate | | **AI-unguided** | Declared at story start — how long with a general LLM but no roles, gates, or skills | | **AI-guided** | Declared at story start — how long with AI plus Orchestra workflow, roles, gates, memory, and evidence | | **AI + Orchestra (actual)** | Measured automatically from `AUTONOMOUS_PHASE_DONE` timestamps in the event log | The first three are self-reported. Orchestra measures the actual governed run. The comparison is meaningful even with declared baselines because it creates a consistent, auditable record across many stories. ## Usage ### 1. Declare baselines at story start Record the three-mode estimate before work begins — ideally during the architect phase. Autonomous workflow runs also ensure an architect sizing decision exists before developer handoff; deterministic architect phases record `m [3 points]` when no architect sizing exists, while provider-backed phases record the normalized provider output. ```bash orchestra estimate \ --task FEAT-001 \ --sizing m \ --solo-days 5 \ --ai-unguided-days 3 \ --ai-guided-days 2 \ --confidence high \ --declared-by pm ``` Options: | Flag | Required | Default | Description | |------|----------|---------|-------------| | `--task` | Yes | — | Task ID | | `--sizing` | Yes | — | `xs`, `s`, `m`, `l`, `xl` | | `--solo-days` | Yes | — | Estimated days without any AI | | `--ai-unguided-days` | Yes | — | Estimated days with a general LLM, no Orchestra | | `--ai-guided-days` | Yes | — | Estimated days with AI guided by Orchestra workflow | | `--confidence` | No | `medium` | `low`, `medium`, `high` | | `--declared-by` | No | `pm` | Role recording the estimate | | `--json` | No | — | Structured output | ### 2. Run the story autonomously ```bash orchestra workflow run --task FEAT-001 --gates phase ``` The autonomous engine records all necessary data automatically as phases complete. ### 3. View the benchmark ```bash # Per-story orchestra benchmark --task FEAT-001 # All stories with estimates orchestra benchmark --summary orchestra benchmark --summary --json ``` Example per-story output: ``` Benchmark: FEAT-001 [complete] Sizing: m Solo: 5d (declared) AI-unguided: 3d (declared) AI-guided: 2d (declared) Actual: 1.4d vs Solo: -72% vs AI-U: -53% vs AI-G: -30% QA loops: 1 Reviews: 3 (0 blocking) Evidence: 5 artifacts Gate blocks: 0 Lessons: 2 Tokens: 17500in / 5000out Cost: $0.0257 ``` Example summary table: ``` Story Size Solo AI-U AI-G Actual vs Solo vs AI-U vs AI-G QA Rev Blk Ev Les ──────────────────────────────────────────────────────────────────────────────────────────────────── TASK-042 m 5d 3d 2d 1.4d -72% -53% -30% 1 3 0 5 2 TASK-089 l 8d 5d 3d 2.1d -74% -58% -30% 2 4 1 7 3 TASK-101 s 2d 1.5d 1d 0.6d -70% -60% -40% 0 2 0 3 1 Avg savings vs solo: -72% Avg savings vs AI-unguided: -57% Avg savings vs AI-guided: -33% Stories with actuals: 3/3 ``` ## Quality Signals All quality signals are computed automatically from the event log for the task. No additional commands are needed. | Signal | Source event | What it means | |--------|-------------|---------------| | `reviewCount` | `REVIEW_RECORDED` | Total reviews across all roles | | `blockingReviews` | `REVIEW_RECORDED` with `result=block` or `severity=high/critical` | Reviews that blocked progress | | `evidenceCount` | `EVIDENCE_ADDED` | Artifacts attached (commands, files, screenshots, traces) | | `gateBlockCount` | `GATE_BLOCKED` | Times a workflow gate evaluation failed | | `lessonCount` | `LESSON_RECORDED` | Lessons generated during the story | | `totalInputTokens` | `MODEL_PROVENANCE_RECORDED` | Input tokens across all LLM calls | | `totalOutputTokens` | `MODEL_PROVENANCE_RECORDED` | Output tokens across all LLM calls | | `estimatedCostUsd` | `MODEL_PROVENANCE_RECORDED` | Estimated cost of all LLM calls | Token and cost signals appear automatically once the autonomous engine makes real LLM calls. No changes to the benchmark layer are needed. ## Sprint Burndown The burndown uses story points from `DECISION_RECORDED` events. Developer points take priority over architect sizing when both exist. ```bash # ASCII chart with ideal and actual lines orchestra burndown --sprint FEAT-001,FEAT-002,FEAT-003 # JSON series for dashboards or external tools orchestra burndown --sprint FEAT-001,FEAT-002,FEAT-003 --json ``` Example output: ``` Sprint burndown total=18 pts tasks=3 18 │* 16 │ 14 │ · 12 │ * 10 │ 8 │ · 6 │ * 4 │ 2 │ 0 │ · └─────────────────────────────────────────────────── Day 0 Day 3 · ideal * actual Task breakdown: FEAT-001 arch=5 dev=8 resolved=8 done 2026-05-06 FEAT-002 arch=5 dev=— resolved=5 done 2026-05-06 FEAT-003 arch=5 dev=— resolved=5 pending ``` ### Developer Story Point Estimation Architect sizing (`xs/s/m/l/xl` + optional points) reflects technical scope. Developer points reflect implementation effort from the developer's perspective. Autonomous developer phases record implementation points when missing; provider-backed phases can return `developerPoints`, while deterministic phases use the architect point estimate or `3 points` fallback. When both exist, the burndown uses developer points — the divergence between the two is a calibration signal worth tracking. Developer records their estimate with: ```bash orchestra decision add \ --task FEAT-001 \ --owner developer \ --title "Dev estimate" \ --decision "M / 8 points" \ --context "Familiar codebase area, clean interfaces" \ --consequences "Fits in one sprint slot" \ --status accepted ``` The `metadata.points` value is extracted automatically from `DECISION_RECORDED` events where `actor=developer`. ## Sprint Velocity And Calibration Use explicit sprint boundaries when you want longitudinal velocity by sprint instead of a weekly benchmark rollup. ```bash orchestra sprint start --id SPRINT-1 --tasks FEAT-001,FEAT-002 orchestra sprint close --id SPRINT-1 orchestra velocity --sprints 3 orchestra velocity --sprints 3 --json ``` Calibration compares each estimator role's declared days against completed story cycle time. When the same role is more than 30% high or low across at least three completed stories, Orchestra records a `LESSON_RECORDED` event so future estimates can account for the bias. ```bash orchestra calibration orchestra calibration --role architect --json ``` ## JSON Output All benchmark and burndown commands support `--json` for structured output. `orchestra benchmark --task --json`: ```json { "taskId": "FEAT-001", "sizingLabel": "m", "soloEstimateDays": 5, "aiUnguidedEstimateDays": 3, "aiGuidedEstimateDays": 2, "actualDays": 1.4, "vsSoloPct": -72, "vsAiUnguidedPct": -53, "vsAiGuidedPct": -30, "qaIterations": 1, "quality": { "reviewCount": 3, "blockingReviews": 0, "evidenceCount": 5, "gateBlockCount": 0, "lessonCount": 2, "totalInputTokens": 17500, "totalOutputTokens": 5000, "estimatedCostUsd": 0.0257 }, "status": "complete" } ``` `orchestra burndown --sprint ... --json`: ```json { "sprintTaskIds": ["FEAT-001", "FEAT-002"], "totalPoints": 13, "idealLine": [{ "day": 0, "remaining": 13 }, { "day": 1, "remaining": 7 }, { "day": 2, "remaining": 0 }], "actualLine": [{ "day": 0, "remaining": 13, "completedTaskIds": [] }, { "day": 1, "remaining": 5, "completedTaskIds": ["FEAT-001"] }], "taskBreakdown": [ { "taskId": "FEAT-001", "architectPoints": 5, "developerPoints": 8, "resolvedPoints": 8, "completedAt": "2026-05-06T..." }, { "taskId": "FEAT-002", "architectPoints": 5, "developerPoints": null, "resolvedPoints": 5, "completedAt": null } ], "warnings": [] } ``` ## Persistent Files ```text .agent-workflow/ estimates.jsonl ← one JSON line per estimate (append-only, latest-per-task wins) ``` Estimates are append-only. Recording a new estimate for the same task replaces the previous one in reads (latest-wins by taskId).