purpose: Complete KPI system for measuring agentic development effectiveness type: measurement-framework domain: performance-metrics metrics: [attempts, size, streak, presence] prerequisites: [adw-fundamentals, zte] last-updated: 2025-09-30 Effective agentic development requires precise measurement. This document defines the Four Core Metrics that quantify agent effectiveness, plus supporting metrics for comprehensive system health monitoring. These metrics enable data-driven optimization and ROI calculation. # AGENTIC KPI SYSTEM ## THE FOUR CORE METRICS **Philosophy**: What gets measured gets improved. These four metrics capture the essence of agentic effectiveness. ``` METRIC 1: ATTEMPTS ↓ (Lower is Better) How many tries to get it right? METRIC 2: SIZE ↑ (Higher is Better) How much can be accomplished in one shot? METRIC 3: STREAK ↑ (Higher is Better) How many successes in a row? METRIC 4: PRESENCE ↓ (Lower is Better) How much human involvement required? ``` ## METRIC 1: ATTEMPTS (Lower is Better) ### Definition **Number of iterations required to complete a task successfully.** ``` ATTEMPTS = Total Execution Attempts / Total Completed Tasks Where: - Execution Attempt = One complete workflow run - Completed Task = Task marked as "complete" in task system - Count includes self-heal attempts ``` ### Measurement Method ```xml Task workflow execution Workflow started Self-heal attempt Human-requested retry Task marked complete Task abandoned (count as failure) ``` ### Example Calculation ``` TASK: Fix authentication bug Attempt 1: Implementation fails tests (self-heal triggered) Attempt 2: Tests pass but security scan fails (self-heal triggered) Attempt 3: All validations pass, marked complete ATTEMPTS = 3 ``` ### Target Values ``` BASELINE (Starting Point): - New system: 3-5 attempts - Typical developer: 2-3 attempts GOOD (Production Ready): - Attempts: 1.5-2.0 - One-shot success rate: 40-60% ELITE (World-Class): - Attempts: 1.0-1.3 - One-shot success rate: 70-90% THEORETICAL LIMIT: - Attempts: 1.0 (perfect one-shot) - One-shot success rate: 100% ``` ### Breakdown by Task Type ``` TASK TYPE | BASELINE | GOOD | ELITE -----------------------|----------|-------|------- Simple bug fix | 2.0 | 1.3 | 1.1 Integration bug | 4.0 | 2.5 | 1.5 New API endpoint | 3.5 | 2.0 | 1.3 UI component | 4.5 | 2.8 | 1.6 Refactoring | 3.0 | 1.8 | 1.2 Test generation | 2.5 | 1.5 | 1.1 Documentation | 2.0 | 1.2 | 1.0 ``` ### Improvement Strategies ``` HIGH ATTEMPTS (>3.0): DIAGNOSE: - Are prompts clear enough? - Are validation checks too strict? - Are self-heal strategies effective? - Is the task too complex for current capability? IMPROVE: 1. Refine prompts with specific examples 2. Add pre-flight validation (fail early) 3. Enhance self-heal logic 4. Break complex tasks into smaller workflows 5. Add more context to agent instructions ``` ## METRIC 2: SIZE (Higher is Better) ### Definition **Measure of work completed in a single successful workflow execution.** ``` SIZE = Complexity Points Per Successful Task Where Complexity Points are calculated: SIZE = (Files Changed × 2) + (Functions Modified × 3) + (Tests Added × 1) + (Dependencies Changed × 5) + (API Changes × 10) ``` ### Measurement Method ```xml Task completion Attempts (to get per-attempt size) ``` ### Example Calculation ``` TASK: Add user authentication endpoint Files Changed: 4 (route, handler, validation, test) Functions Modified: 2 (createUser, validateToken) Tests Added: 8 (unit + integration) Dependencies Changed: 1 (added bcrypt) API Changes: 1 (new /auth/login endpoint) SIZE = (4 × 2) + (2 × 3) + (8 × 1) + (1 × 5) + (1 × 10) SIZE = 8 + 6 + 8 + 5 + 10 SIZE = 37 complexity points ``` ### Target Values ``` BASELINE (Starting Point): - Size: 5-10 points per task - Scope: Single file, simple changes GOOD (Production Ready): - Size: 15-30 points per task - Scope: Multiple files, moderate complexity ELITE (World-Class): - Size: 40-80 points per task - Scope: Cross-cutting changes, high complexity THEORETICAL LIMIT: - Size: 100+ points per task - Scope: Full feature implementation ``` ### Breakdown by Task Type ``` TASK TYPE | BASELINE | GOOD | ELITE -----------------------|----------|-------|------- Simple bug fix | 5 | 10 | 15 Integration bug | 12 | 25 | 45 New API endpoint | 15 | 35 | 65 UI component | 18 | 40 | 75 Refactoring | 20 | 45 | 85 Full feature | 25 | 60 | 120 ``` ### Improvement Strategies ``` LOW SIZE (<10): DIAGNOSE: - Is agent being too conservative? - Are tasks broken down too much? - Is self-healing causing partial completions? - Are error thresholds too low? IMPROVE: 1. Combine related tasks 2. Increase agent confidence thresholds 3. Reduce unnecessary checkpoints 4. Allow more comprehensive changes 5. Improve parallel execution ``` ## METRIC 3: STREAK (Higher is Better) ### Definition **Number of consecutive successful completions without failure or human escalation.** ``` STREAK = Current Consecutive Successes Where: - Success = Task completed with all validations passed - Failure = Task failed, abandoned, or escalated to human - Streak resets to 0 on any failure ``` ### Measurement Method ```xml streak = 0 streak = streak + 1 streak = 0 ``` ### Example Tracking ``` TASK SEQUENCE: Task 1: Bug fix - SUCCESS (streak = 1) Task 2: API endpoint - SUCCESS (streak = 2) Task 3: UI component - SUCCESS (streak = 3) Task 4: Refactoring - FAILED (streak = 0) Task 5: Bug fix - SUCCESS (streak = 1) Task 6: Documentation - SUCCESS (streak = 2) Current Streak: 2 Max Streak: 3 Avg Streak: 2.0 ``` ### Target Values ``` BASELINE (Starting Point): - Avg Streak: 1-2 tasks - Max Streak: 3-5 tasks - Streak >5: Rare GOOD (Production Ready): - Avg Streak: 3-5 tasks - Max Streak: 8-12 tasks - Streak >5: 40-60% of time ELITE (World-Class): - Avg Streak: 7-12 tasks - Max Streak: 20-30 tasks - Streak >5: 80-90% of time THEORETICAL LIMIT: - Avg Streak: Unbounded (no failures) - Max Streak: Unbounded - Streak >5: 100% of time ``` ### Breakdown by Task Type ``` TASK TYPE | BASELINE | GOOD | ELITE -----------------------|----------|-------|------- Simple bugs only | 2-3 | 5-7 | 10-15 Mixed complexity | 1-2 | 3-5 | 7-12 High complexity only | 1 | 2-3 | 4-6 ``` ### Improvement Strategies ``` LOW STREAK (<3): DIAGNOSE: - What causes streak breaks? - Are certain task types more likely to fail? - Do failures cluster (cascading)? - Are validation thresholds causing false failures? IMPROVE: 1. Analyze failure patterns 2. Add pre-flight checks (fail before workflow starts) 3. Improve self-healing for common failures 4. Adjust validation thresholds 5. Better task sequencing (easier tasks first) ``` ## METRIC 4: PRESENCE (Lower is Better) ### Definition **Percentage of workflow time requiring human attention or intervention.** ``` PRESENCE = (Human Interaction Time / Total Workflow Time) × 100 Where: - Human Interaction Time = Sum of all human review/approval time - Total Workflow Time = Start to completion including agent + human time - Measured in minutes or percentage ``` ### Measurement Method ```xml Workflow begins Checkpoint requiring human input zero-presence Human attention required Human provides input/approval full-presence presence = (sum(human-phase durations) / total workflow time) × 100 ``` ### Example Calculation ``` TASK: API endpoint implementation Timeline: 00:00 - Workflow starts (agent) 00:08 - Discovery complete (agent) 00:20 - Implementation complete (agent) 00:21 - Validation complete, request approval (agent→human) 00:21 - Human notified 00:45 - Human reviews (24 min delay) 00:48 - Human approves (3 min review) 00:48 - Workflow completes Total Time: 48 minutes Agent Time: 21 minutes (autonomous) Waiting Time: 24 minutes (human not present) Human Review: 3 minutes (human present) PRESENCE = (3 / 48) × 100 = 6.25% ``` ### Target Values ``` BASELINE (Starting Point): - Presence: 40-60% (heavy human involvement) - Human time: Multiple reviews per task - Approval delays: Hours to days GOOD (Production Ready): - Presence: 10-20% (minimal involvement) - Human time: Single approval per task - Approval delays: Minutes to hours ELITE (World-Class): - Presence: 2-5% (strategic only) - Human time: Batch approvals - Approval delays: Seconds to minutes ZERO TOUCH (Theoretical): - Presence: 0% (full automation) - Human time: None - Approval delays: None ``` ### Breakdown by Workflow Type ``` WORKFLOW TYPE | BASELINE | GOOD | ELITE | ZTE -----------------------|----------|-------|-------|----- Simple bug fix | 30% | 8% | 2% | 0% Integration bug | 50% | 15% | 5% | 0% New API endpoint | 60% | 20% | 8% | 2% UI component | 70% | 25% | 10% | 5% Refactoring | 40% | 12% | 4% | 0% Full feature | 80% | 35% | 15% | 10% ``` ### Improvement Strategies ``` HIGH PRESENCE (>25%): DIAGNOSE: - Too many manual checkpoints? - Are escalations necessary? - Long approval delays? - Lack of batching? IMPROVE: 1. Reduce unnecessary checkpoints 2. Batch approvals (approve 10 tasks at once) 3. Improve agent confidence (reduce escalations) 4. Automate simple approvals 5. Use async approval mechanisms 6. Implement time-based auto-approval for low-risk ``` ## COMPOSITE METRICS ### EFFICIENCY SCORE **Overall measure of agentic effectiveness.** ``` EFFICIENCY = (SIZE × STREAK) / (ATTEMPTS × PRESENCE) Example: SIZE = 35 points STREAK = 5 tasks ATTEMPTS = 1.5 PRESENCE = 8% EFFICIENCY = (35 × 5) / (1.5 × 0.08) EFFICIENCY = 175 / 0.12 EFFICIENCY = 1458 Interpretation: < 100: Poor (manual development is faster) 100-500: Baseline (agentic provides some value) 500-1500: Good (clear productivity gains) 1500-5000: Elite (10-20× multiplier) > 5000: World-class (>20× multiplier) ``` ### VELOCITY **Work completed per time period.** ``` VELOCITY = Total Complexity Points Completed / Time Period Example (weekly): Week 1: 245 points across 12 tasks Week 2: 312 points across 14 tasks Week 3: 289 points across 13 tasks Avg Velocity: 282 points/week Trend: +18% week-over-week ``` ### RELIABILITY **Consistency of agent performance.** ``` RELIABILITY = (Successful Tasks / Total Attempted Tasks) × 100 Example: Total Attempted: 50 tasks Successful: 42 tasks Failed: 5 tasks Abandoned: 3 tasks RELIABILITY = (42 / 50) × 100 = 84% ``` ## DASHBOARD DESIGN ### Real-Time Dashboard ``` ┌─────────────────────────────────────────────────┐ │ AGENTIC KPI DASHBOARD Live: 2025-09-30 │ ├─────────────────────────────────────────────────┤ │ CORE METRICS │ │ │ │ ATTEMPTS ↓ 1.4 [━━━━━━━━░░] Target: <1.5 │ │ SIZE ↑ 42 [━━━━━━━━━░] Target: >35 │ │ STREAK ↑ 8 [━━━━━━━━━━] Target: >7 │ │ PRESENCE ↓ 6.2% [━━━━━━━━░░] Target: <8% │ │ │ │ COMPOSITE METRICS │ │ Efficiency: 2,154 (Elite) ▲ +12% vs last │ │ Velocity: 312 pts/week ▲ +8% vs last │ │ Reliability: 87% ▲ +3% vs last │ │ │ │ CURRENT ACTIVITY │ │ Tasks Running: 3 │ │ Current Streak: 8 🔥 │ │ Pending Approval: 2 │ │ │ │ RECENT TASKS (Last 10) │ │ ✓ Bug fix - auth.js 1 attempt 5 min │ │ ✓ API endpoint - /users 1 attempt 22 min │ │ ✓ UI component - Button 2 attempts 28 min │ │ ✓ Tests - auth tests 1 attempt 12 min │ │ ✓ Refactor - extractFn 1 attempt 8 min │ │ ✓ Docs - API reference 1 attempt 9 min │ │ ✓ Bug fix - validation 1 attempt 6 min │ │ ✓ Feature - user profile 2 attempts 35 min │ │ ● RUNNING - integration test Running... │ │ ⏸ WAITING - deployment Approval │ └─────────────────────────────────────────────────┘ ``` ### Trend Dashboard ``` ┌─────────────────────────────────────────────────┐ │ 30-DAY TRENDS │ ├─────────────────────────────────────────────────┤ │ ATTEMPTS (Lower is Better) │ │ 3.0 ┤ │ │ 2.5 ┤╮ │ │ 2.0 ┤╰╮ │ │ 1.5 ┤ ╰╮ │ │ 1.0 ┤ ╰─────────────────── │ │ └──────────────────────────── │ │ Day 1 Day 15 Day 30 │ │ │ │ SIZE (Higher is Better) │ │ 60 ┤ ╭───── │ │ 50 ┤ ╭─────╯ │ │ 40 ┤ ╭─────╯ │ │ 30 ┤ ╭─────╯ │ │ 20 ┤ ╭─────╯ │ │ └──────────────────────────── │ │ Day 1 Day 15 Day 30 │ │ │ │ KEY IMPROVEMENTS │ │ • Attempts: 3.2 → 1.4 (56% reduction) │ │ • Size: 18 → 42 (133% increase) │ │ • Streak: 2 → 8 (300% increase) │ │ • Presence: 22% → 6% (73% reduction) │ └─────────────────────────────────────────────────┘ ``` ### Task Type Breakdown ``` ┌─────────────────────────────────────────────────┐ │ PERFORMANCE BY TASK TYPE (Last 30 Days) │ ├─────────────────────────────────────────────────┤ │ TASK TYPE | TASKS | AVG ATT | AVG SIZE | % │ │───────────────|───────|─────────|──────────|────│ │ Bug Fix │ 45 │ 1.2 │ 12 │ 96%│ │ Feature │ 18 │ 1.8 │ 48 │ 83%│ │ Refactor │ 12 │ 1.4 │ 38 │ 92%│ │ Tests │ 32 │ 1.3 │ 15 │ 94%│ │ Docs │ 28 │ 1.1 │ 8 │ 98%│ │───────────────|───────|─────────|──────────|────│ │ TOTAL │ 135 │ 1.4 │ 24 │ 93%│ └─────────────────────────────────────────────────┘ ``` ## MEASUREMENT IMPLEMENTATION ### Data Collection ```javascript // Capture workflow metrics class WorkflowMetrics { constructor(taskId) { this.taskId = taskId; this.attempts = 0; this.startTime = Date.now(); this.humanInteractionTime = 0; this.complexityPoints = 0; } incrementAttempt() { this.attempts++; } startHumanInteraction() { this.humanStart = Date.now(); } endHumanInteraction() { this.humanInteractionTime += Date.now() - this.humanStart; } calculateSize(changes) { return ( changes.filesChanged * 2 + changes.functionsModified * 3 + changes.testsAdded * 1 + changes.dependenciesChanged * 5 + changes.apiChanges * 10 ); } complete(changes) { const totalTime = Date.now() - this.startTime; const presence = (this.humanInteractionTime / totalTime) * 100; const size = this.calculateSize(changes); return { taskId: this.taskId, attempts: this.attempts, size: size, presence: presence, totalTime: totalTime, timestamp: new Date().toISOString() }; } } ``` ### Aggregation ```javascript // Calculate metrics from stored data class MetricsAggregator { constructor(metricsDB) { this.db = metricsDB; } getAttempts(timeframe) { const tasks = this.db.getTasksInTimeframe(timeframe); const totalAttempts = tasks.reduce((sum, t) => sum + t.attempts, 0); return totalAttempts / tasks.length; } getAverageSize(timeframe) { const tasks = this.db.getTasksInTimeframe(timeframe); return tasks.reduce((sum, t) => sum + t.size, 0) / tasks.length; } getCurrentStreak() { const recentTasks = this.db.getTasksInOrder(); let streak = 0; for (const task of recentTasks.reverse()) { if (task.success) { streak++; } else { break; } } return streak; } getAveragePresence(timeframe) { const tasks = this.db.getTasksInTimeframe(timeframe); return tasks.reduce((sum, t) => sum + t.presence, 0) / tasks.length; } getCompositeMetrics(timeframe) { const attempts = this.getAttempts(timeframe); const size = this.getAverageSize(timeframe); const streak = this.getCurrentStreak(); const presence = this.getAveragePresence(timeframe); return { attempts, size, streak, presence, efficiency: (size * streak) / (attempts * (presence / 100)), velocity: this.getVelocity(timeframe), reliability: this.getReliability(timeframe) }; } } ``` ## ALERTING & THRESHOLDS ### Performance Alerts ```yaml alerts: attempts-high: condition: "attempts > 2.0" severity: warning action: "Review workflow prompts and validation" attempts-critical: condition: "attempts > 3.0" severity: critical action: "Stop workflow, require human review" streak-broken: condition: "streak reset to 0" severity: info action: "Log failure reason for analysis" streak-milestone: condition: "streak % 5 == 0" severity: info action: "Celebrate milestone" presence-high: condition: "presence > 15%" severity: warning action: "Review checkpoint necessity" efficiency-drop: condition: "efficiency < last_week * 0.8" severity: warning action: "Analyze degradation cause" ``` ## CONCLUSION **The Four Core Metrics provide complete visibility into agentic system performance.** Track them daily, analyze trends weekly, and optimize continuously. **Key Insights**: 1. **Attempts** measures first-time success 2. **Size** measures ambition and scope 3. **Streak** measures reliability 4. **Presence** measures automation level **Target State**: Elite performance across all four metrics results in 10-20× developer productivity gains. Implement measurement systems Build real-time dashboard Review one-shot-success.md for ultimate goal Study measuring-leverage.md for ROI calculation