purpose: Define one-shot success as the ultimate agentic development goal type: measurement-framework domain: performance-optimization metrics: [one-shot-rate, prerequisites, improvement-path] prerequisites: [agentic-kpis] last-updated: 2025-09-30 One-shot success represents the pinnacle of agentic effectiveness: completing tasks correctly on the first attempt, without retries, self-healing, or human intervention. This document defines one-shot success, prerequisites for achieving it, measurement methodology, and the path from 20% to 80%+ success rates. # ONE-SHOT SUCCESS ## DEFINITION **One-Shot Success**: A task is completed correctly on the first workflow execution, passing all validations without requiring: - Self-healing attempts - Human escalation - Retry execution - Manual correction ``` ONE-SHOT SUCCESS = { Attempts = 1 AND All Validations Pass AND No Escalation AND Human Approval (if required by policy, not due to failure) } ``` ## THE IMPORTANCE OF ONE-SHOT ### Why It Matters **1. Efficiency Multiplier** ``` COMPARISON: Average Attempts = 2.5 - First attempt: 15 minutes - Retry 1: 12 minutes - Retry 2: 8 minutes Total: 35 minutes One-Shot: - First attempt: 15 minutes Total: 15 minutes Time Saved: 20 minutes (57% reduction) ``` **2. System Trust** ``` PSYCHOLOGICAL IMPACT: Low one-shot rate (20-30%): - Developers don't trust agents - Manual verification becomes habit - "Let me check what it did..." - Presence remains high High one-shot rate (70-80%): - Developers trust agent output - Verification becomes exception - "I assume it worked correctly" - Presence drops dramatically ``` **3. Scaling Factor** ``` ONE-SHOT RATE vs TASK CAPACITY: At 20% one-shot: - 10 tasks attempted - 2 succeed immediately - 8 require retries (avg 2.5 attempts) - Total capacity: ~15 effective tasks/week At 80% one-shot: - 10 tasks attempted - 8 succeed immediately - 2 require retries (avg 2.0 attempts) - Total capacity: ~40 effective tasks/week 2.7× capacity increase from one-shot improvement alone ``` ## MEASURING ONE-SHOT RATE ### Primary Metric ``` ONE-SHOT RATE = (Tasks Completed in 1 Attempt / Total Tasks) × 100 Where: - Completed in 1 Attempt = First execution passes all validations - Total Tasks = All task attempts in measurement period - Excludes abandoned tasks (counted separately) ``` ### Detailed Tracking ```xml 2025-09-30T10:00:00Z success 180s success 420s false success 120s true true true true false approved 2025-09-30T10:12:00Z true failure Tests failed true false success false ``` ### Calculation Example ``` 30-DAY PERIOD: Total Tasks Completed: 120 - One-shot successes: 84 - Required 2 attempts: 28 - Required 3+ attempts: 8 ONE-SHOT RATE = (84 / 120) × 100 = 70% ``` ## INDUSTRY BENCHMARKS ### By Development Maturity ``` STAGE | ONE-SHOT RATE | ATTEMPTS AVG -------------------------|---------------|------------- Manual Development | N/A | 1.0* Early Agentic (Month 1) | 15-25% | 3.5-4.5 Growing (Month 2-3) | 30-45% | 2.5-3.0 Mature (Month 4-6) | 50-65% | 1.8-2.2 Elite (Month 7+) | 70-85% | 1.2-1.5 World-Class | 85-95% | 1.05-1.15 * Manual development is always "one-shot" but slower ``` ### By Task Complexity ``` TASK TYPE | BASELINE | MATURE | ELITE -------------------------|----------|--------|------- Trivial (docs update) | 60% | 85% | 95% Simple (single bug fix) | 40% | 70% | 88% Moderate (API endpoint) | 25% | 55% | 75% Complex (UI component) | 15% | 45% | 65% Very Complex (feature) | 8% | 30% | 50% ``` ### By Agent Specialization ``` AGENT TYPE | ONE-SHOT RATE ------------------------|--------------- Generalist (GPT-4) | 35-45% Specialist (bugsy) | 55-70% Custom-trained | 65-80% Human-in-loop hybrid | 75-90% ``` ## PREREQUISITES FOR HIGH ONE-SHOT RATES ### PREREQUISITE 1: Crystal Clear Requirements **Problem**: Vague requirements lead to wrong solutions, even if technically correct. ``` ❌ LOW ONE-SHOT: "Fix the login issue" - What login issue? - Which component? - What's the expected behavior? ✓ HIGH ONE-SHOT: "Fix bug in src/auth/login.ts where pressing Enter key doesn't submit the form. Expected: Form submits on Enter. Actual: Nothing happens. Root cause likely: Missing onKeyPress handler." ``` **Requirements Quality Checklist**: ``` □ Specific file/component mentioned (if known) □ Expected behavior described □ Actual behavior described □ Steps to reproduce (for bugs) □ Acceptance criteria clear □ Edge cases mentioned □ Non-goals specified (what NOT to change) ``` ### PREREQUISITE 2: Comprehensive Context **Problem**: Agents need full context to make correct decisions. ``` REQUIRED CONTEXT: 1. Existing Patterns - How does the codebase solve similar problems? - What libraries/utilities are already available? - What's the established code style? 2. Dependencies - What other components depend on this? - What does this component depend on? - What will break if this changes? 3. Constraints - Performance requirements - Security requirements - Backward compatibility needs - API contract stability 4. History - Why was it built this way? - What's been tried before? - What failed in the past? ``` **Context Provision Methods**: ```yaml methods: documentation: - Comprehensive README - Architecture decision records (ADRs) - Inline code comments - API documentation code-patterns: - Example implementations - Template files - Style guides - Best practice docs memory-systems: - Previous similar tasks - Known pitfalls - Success patterns - Failure patterns explicit-instructions: - "Follow the pattern in src/examples/auth.ts" - "Use the validateInput utility, don't write custom" - "Maintain backward compatibility with v1 API" ``` ### PREREQUISITE 3: Effective Validation **Problem**: Bad validation either blocks good solutions or passes bad ones. ``` VALIDATION BALANCE: TOO STRICT (False Negatives): - "Coverage must be 100%" - "Zero lint warnings allowed" - "Performance must improve by 20%" Result: Good solutions rejected, low one-shot rate TOO LOOSE (False Positives): - "Just check it runs" - "If tests pass, ship it" - "Security scan optional" Result: Bad solutions pass, breaks in production BALANCED: - "Coverage >= baseline, ideally +2%" - "No new lint errors, warnings acceptable" - "Performance not degraded >10%" Result: Good solutions pass, bad solutions caught ``` **Validation Layers**: ``` LAYER 1: Syntax & Structure - Code parses correctly - Imports resolve - Types valid (if TypeScript) LAYER 2: Unit Behavior - Unit tests pass - New tests added for changes - Coverage maintained LAYER 3: Integration - Integration tests pass - No breaking changes - API contracts maintained LAYER 4: Quality - Lint rules pass (errors only) - Security scan clean - Performance acceptable LAYER 5: Policy - License compliance - Documentation updated - Change log entry added ``` ### PREREQUISITE 4: Self-Healing Capability **Problem**: Minor issues should be auto-corrected, not escalated. ``` AUTO-CORRECTABLE ISSUES: ✓ Lint formatting errors ✓ Missing semicolons ✓ Import organization ✓ Simple test failures (typos) ✓ Missing docs for new functions ESCALATION REQUIRED: ✗ Logic errors ✗ Breaking API changes ✗ Security vulnerabilities ✗ Performance regressions ✗ Architectural mismatches ``` **Self-Heal Strategy**: ```yaml self-heal: attempt-1: strategy: "Analyze error, correct implementation" timeout: 5 minutes on-success: "Mark as one-shot success" on-failure: "Attempt 2" attempt-2: strategy: "Try alternative approach" timeout: 5 minutes on-success: "Not one-shot, but successful" on-failure: "Attempt 3" attempt-3: strategy: "Minimal fix for specific error" timeout: 3 minutes on-success: "Not one-shot, but successful" on-failure: "Escalate to human" escalation: provide: - Original task - All attempts made - Error history - Suggested next steps ``` ### PREREQUISITE 5: Prompt Engineering Excellence **Problem**: Poor prompts get poor results. ``` ❌ WEAK PROMPT: "Implement user authentication" ✓ STRONG PROMPT: "Implement user authentication following these specifications: REQUIREMENTS: 1. Add POST /api/auth/login endpoint 2. Accept {email, password} in request body 3. Validate input using validateUser utility (src/utils/validation.ts) 4. Check credentials against users table 5. Return JWT token on success 6. Return 401 on invalid credentials 7. Return 400 on validation errors PATTERNS TO FOLLOW: - Follow pattern in src/examples/auth-endpoint.ts - Use existing bcrypt comparison (security/crypto.ts) - Use JWT signing utility (security/jwt.ts) VALIDATION: - Add unit tests for handler function (>90% coverage) - Add integration test for full flow - Test error cases (invalid email, wrong password, missing fields) CONSTRAINTS: - No new dependencies - Maintain existing API response format - Complete in <25 minutes OUTPUT: - List all files changed - Provide test results - Explain any deviations from spec" ``` ## THE PATH FROM 20% TO 80% ### PHASE 1: Measurement (Weeks 1-2) **Goal**: Establish baseline and identify failure patterns. ``` ACTIONS: 1. Implement one-shot tracking 2. Collect data on 30-50 tasks 3. Categorize failure reasons 4. Identify patterns EXPECTED OUTCOME: - Baseline one-shot rate: 15-25% - Top 3-5 failure categories identified - Quick wins identified ``` **Common Failure Categories**: ``` CATEGORY | % OF FAILURES | IMPROVABLE? --------------------------|---------------|------------- Unclear requirements | 25-30% | ✓ Yes Missing context | 20-25% | ✓ Yes Validation too strict | 15-20% | ✓ Yes Agent capability limit | 15-20% | △ Partially External dependencies | 10-15% | △ Partially Edge cases not handled | 10-12% | ✓ Yes ``` ### PHASE 2: Quick Wins (Weeks 3-4) **Goal**: Address easiest failures first for rapid improvement. ``` TARGET: 20% → 35% one-shot rate ACTIONS: 1. Improve requirements templates - Add requirement quality checklist - Provide examples of good requirements - Reject vague requirements early 2. Enhance context provision - Add architecture docs - Create pattern library - Document common utilities 3. Optimize validation thresholds - Relax overly strict rules - Focus on critical validations - Allow warnings, block errors only EXPECTED OUTCOME: - 35-45% one-shot rate - ~15 percentage point improvement - Low-hanging fruit addressed ``` ### PHASE 3: Systematic Improvement (Weeks 5-8) **Goal**: Address structural issues systematically. ``` TARGET: 35% → 55% one-shot rate ACTIONS: 1. Build self-healing logic - Auto-fix lint issues - Retry simple test failures - Auto-update imports 2. Enhance prompts - Add specific examples - Clarify edge cases - Specify patterns to follow 3. Improve agent context - Implement memory systems - Add learning from failures - Provide success templates 4. Optimize workflows - Pre-flight validation - Early error detection - Incremental validation EXPECTED OUTCOME: - 55-65% one-shot rate - ~20 percentage point improvement - Solid foundation established ``` ### PHASE 4: Excellence (Weeks 9-16) **Goal**: Achieve elite-level performance. ``` TARGET: 55% → 75%+ one-shot rate ACTIONS: 1. Advanced self-healing - Multi-strategy attempts - Intelligent fallbacks - Context-aware corrections 2. Predictive failure prevention - Analyze task before starting - Identify likely issues - Pre-emptively handle edge cases 3. Continuous learning - Learn from every failure - Update prompts automatically - Evolve validation rules 4. Specialization - Task-specific agents - Custom-trained models - Domain-specific workflows EXPECTED OUTCOME: - 75-85% one-shot rate - ~20 percentage point improvement - Elite performance sustained ``` ## TRACKING IMPROVEMENT ### Weekly Review Template ```markdown # One-Shot Success Review - Week {{N}} ## Metrics - One-Shot Rate: {{current}}% (previous: {{previous}}%, change: {{delta}}%) - Total Tasks: {{total}} - One-Shot Successes: {{successes}} - Average Attempts: {{avg_attempts}} ## Successes This Week 1. [Task type] achieved {{rate}}% one-shot (up from {{previous}}%) 2. [Improvement] reduced failures by {{count}} ## Failures This Week 1. [Category]: {{count}} failures ({{percent}}% of total) - Root cause: [analysis] - Action: [what to do] ## Actions for Next Week - [ ] [Specific improvement action 1] - [ ] [Specific improvement action 2] - [ ] [Specific improvement action 3] ## Blockers - [Any systemic issues preventing improvement] ``` ### Monthly Deep Dive ```markdown # One-Shot Success Deep Dive - Month {{N}} ## Progress - Month Start: {{start}}% - Month End: {{end}}% - Improvement: {{delta}}% ({{trend}}) ## By Task Type | Task Type | One-Shot Rate | Change | Target | |-----------|---------------|--------|--------| | Bug Fix | {{rate}}% | {{delta}}% | 75% | | Feature | {{rate}}% | {{delta}}% | 60% | | Refactor | {{rate}}% | {{delta}}% | 70% | | Tests | {{rate}}% | {{delta}}% | 80% | | Docs | {{rate}}% | {{delta}}% | 85% | ## Failure Analysis Top Failure Reasons: 1. [Category]: {{count}} ({{percent}}%) - Actions taken: [what was done] - Result: [outcome] 2. [Category]: {{count}} ({{percent}}%) - Actions taken: [what was done] - Result: [outcome] ## Key Learnings - [Insight 1] - [Insight 2] - [Insight 3] ## Next Month Goals - Target one-shot rate: {{target}}% - Focus area: [specific category] - Key initiatives: [what to implement] ``` ## ADVANCED OPTIMIZATION ### Predictive One-Shot Probability **Concept**: Predict likelihood of one-shot success before starting task. ```javascript function predictOneShot(task) { const factors = { requirementClarity: scoreRequirements(task.description), taskComplexity: calculateComplexity(task.type, task.scope), contextAvailability: checkContext(task.files), historicalSuccess: getHistoricalRate(task.type), agentSpecialization: matchAgentToTask(task.type) }; const probability = factors.requirementClarity * 0.30 + factors.contextAvailability * 0.25 + factors.historicalSuccess * 0.25 + factors.agentSpecialization * 0.15 - factors.taskComplexity * 0.05; return { probability: probability, confidence: calculateConfidence(factors), recommendations: generateRecommendations(factors) }; } // Usage const prediction = predictOneShot(newTask); if (prediction.probability < 0.40) { console.log("Low one-shot probability. Suggestions:"); prediction.recommendations.forEach(r => console.log(`- ${r}`)); // Optionally: request human to clarify before starting } ``` ### Dynamic Prompt Adjustment **Concept**: Adjust prompts based on predicted difficulty. ```javascript function generatePrompt(task, prediction) { let prompt = basePrompt(task); if (prediction.probability < 0.50) { // Add extra guidance for difficult tasks prompt += `\n\nIMPORTANT: This task has challenges:\n`; if (prediction.factors.requirementClarity < 0.60) { prompt += `- Requirements may be unclear. Request clarification if ambiguous.\n`; } if (prediction.factors.contextAvailability < 0.60) { prompt += `- Limited context available. Search codebase thoroughly.\n`; } if (prediction.factors.taskComplexity > 0.70) { prompt += `- High complexity. Break into sub-tasks if needed.\n`; } } return prompt; } ``` ## ANTI-PATTERNS ### Anti-Pattern 1: Lowering Standards ❌ "To increase one-shot rate, we'll make validation less strict" ✓ "To increase one-shot rate, we'll improve context and prompts" ### Anti-Pattern 2: Cherry-Picking Tasks ❌ "Only give agents easy tasks to boost one-shot rate" ✓ "Track one-shot rate by task complexity, improve across all types" ### Anti-Pattern 3: Ignoring Failures ❌ "70% one-shot is good enough, ignore the 30% failures" ✓ "Analyze every failure to identify improvement opportunities" ### Anti-Pattern 4: Over-Automation ❌ "Remove all human checkpoints to maximize one-shot rate" ✓ "Keep strategic checkpoints, optimize agent capability" ## CONCLUSION **One-shot success is the ultimate measure of agentic effectiveness.** It represents not just technical capability, but the convergence of: - Clear communication - Comprehensive context - Intelligent validation - Effective self-healing - Continuous learning **The 20% → 80% journey takes 3-4 months** with systematic focus, but the productivity gains are exponential. **Key Insight**: Every percentage point improvement in one-shot rate compounds across all tasks, creating massive leverage. Implement one-shot tracking system Establish baseline with 30-50 tasks Begin Phase 1 measurement Review measuring-leverage.md for ROI calculation