# Production Patterns

<!-- TOC:START (generated by scripts/skill-toc.sh; do not edit) -->
## Contents

- [HN 2025 Production Patterns](#hn-2025-production-patterns)
- [Proactive Context Management (OpenCode Pattern)](#proactive-context-management-opencode-pattern)
- [Sub-Agents for Context Isolation](#sub-agents-for-context-isolation)
- [Git Worktree Isolation (Cursor Pattern)](#git-worktree-isolation-cursor-pattern)
- [Atomic Checkpoint/Rollback (Cursor Pattern)](#atomic-checkpointrollback-cursor-pattern)
- [CI/CD Automation (Zencoder Patterns)](#cicd-automation-zencoder-patterns)
  - [CI Failure Analysis and Auto-Resolution](#ci-failure-analysis-and-auto-resolution)
  - [Automated Review Comment Resolution](#automated-review-comment-resolution)
  - [Continuous Dependency Management](#continuous-dependency-management)
- [Batch Processing (Claude API)](#batch-processing-claude-api)
  - [When to Use Batch API](#when-to-use-batch-api)
  - [Batch API Limits](#batch-api-limits)
  - [Implementation Pattern](#implementation-pattern)
  - [Batch + Prompt Caching](#batch--prompt-caching)
- [Install Packages Locally](#install-packages-locally)
- [What Actually Works in Production](#what-actually-works-in-production)
  - [Human-in-the-Loop (HITL) is Non-Negotiable](#human-in-the-loop-hitl-is-non-negotiable)
  - [Narrow Scope Wins](#narrow-scope-wins)
  - [Confidence-Based Routing](#confidence-based-routing)
  - [Classification Before Automation](#classification-before-automation)
  - [Deterministic Outer Loops](#deterministic-outer-loops)
- [Context Engineering Patterns](#context-engineering-patterns)
  - [Context Curation Over Automatic Selection](#context-curation-over-automatic-selection)
  - [Information Abstraction](#information-abstraction)
  - [Separate Conversations Per Task](#separate-conversations-per-task)
- [Skills System Pattern](#skills-system-pattern)
  - [On-Demand Skill Loading](#on-demand-skill-loading)
  - [Sub-Agents for Context Isolation](#sub-agents-for-context-isolation-1)
- [Planning Before Execution](#planning-before-execution)
  - [Explicit Plan-Then-Code Workflow](#explicit-plan-then-code-workflow)
- [Multi-Agent Orchestration Patterns](#multi-agent-orchestration-patterns)
  - [Event-Driven Coordination](#event-driven-coordination)
  - [Policy-First Enforcement](#policy-first-enforcement)
  - [Simulation Layer](#simulation-layer)
- [Evaluation and Benchmarking](#evaluation-and-benchmarking)
  - [Problems with Current Benchmarks](#problems-with-current-benchmarks)
  - [Practical Evaluation Approach](#practical-evaluation-approach)
- [Cost and Token Economics](#cost-and-token-economics)
  - [Real-World Cost Patterns](#real-world-cost-patterns)
- [Cross-Process File Lock (v7.5.5)](#cross-process-file-lock-v755)
- [Sources](#sources)
<!-- TOC:END -->

## HN 2025 Production Patterns

Practitioner-tested patterns from Hacker News discussions and real-world deployments. These patterns represent what actually works in production, not theoretical frameworks. The deployment-side patterns come first; the practitioner patterns follow them.

This module consolidates battle-tested insights from:
- HN discussions on autonomous agents in production (2025)
- Coding with LLMs practitioner experiences
- Simon Willison's Superpowers coding agent patterns
- Multi-agent orchestration real-world deployments

**Per-surface constraints:** Claude Code has full network access and can install packages locally; the Claude API runs tools without network access and without runtime package installs; claude.ai installs packages from the standard registries only. Check the target surface before a pattern below assumes network or installs.

---

## Proactive Context Management (OpenCode Pattern)

**Prevent context overflow in long autonomous sessions:**

```yaml
compaction_strategy:
  trigger: "Every 25 iterations OR context feels heavy"

  preserve_always:
    - CONTINUITY.md content (current state)
    - Current task specification
    - Recent Mistakes & Learnings (last 5)
    - Active queue items

  consolidate:
    - Completed task summaries -> semantic memory
    - Resolved errors -> anti-patterns
    - Successful patterns -> procedural memory

  discard:
    - Verbose tool outputs
    - Intermediate reasoning
    - Superseded plans
```

---

## Sub-Agents for Context Isolation

**Run expensive explorations in isolated contexts:**

```python
# Heavy analysis that would bloat main context
Task(
    subagent_type="Explore",
    model="haiku",
    description="Find all auth-related files",
    prompt="Search codebase for authentication patterns. Return only file paths."
)
# Main context stays clean; only results return
```

---

## Git Worktree Isolation (Cursor Pattern)

**Use git worktrees for parallel implementation agents:**

```bash
# Create isolated worktree for feature
git worktree add ../project-feature-auth feature/auth

# Agent works in isolated worktree
cd ../project-feature-auth
# ... implement feature ...

# Merge back when complete
git checkout main
git merge feature/auth
git worktree remove ../project-feature-auth
```

**Benefits:**
- Multiple agents can work in parallel without conflicts
- Each agent has clean, isolated file state
- Merges happen explicitly, not through file racing

---

## Atomic Checkpoint/Rollback (Cursor Pattern)

```yaml
checkpoint_strategy:
  when:
    - Before spawning any subagent
    - Before any destructive operation
    - After completing a task successfully

  how:
    - git commit -m "CHECKPOINT: before {operation}"
    - Record commit hash in CONTINUITY.md

  rollback:
    - git reset --hard {checkpoint_hash}
    - Update CONTINUITY.md with rollback reason
    - Add to Mistakes & Learnings
```

---

## CI/CD Automation (Zencoder Patterns)

### CI Failure Analysis and Auto-Resolution

```yaml
ci_failure_workflow:
  1. Detect CI failure (webhook or poll)
  2. Parse error logs for root cause
  3. Classify failure type:
     - Test failure: Fix code, re-run tests
     - Lint failure: Auto-fix with --fix flag
     - Build failure: Check dependencies, configs
     - Flaky test: Mark and investigate separately
  4. Apply fix and push
  5. Monitor CI result
  6. If still failing after 3 attempts: escalate
```

### Automated Review Comment Resolution

```yaml
pr_comment_workflow:
  trigger: "New review comment on PR"

  workflow:
    1. Parse comment for actionable feedback
    2. Classify: bug, style, question, suggestion
    3. For bugs/style: implement fix
    4. For questions: add code comment or respond
    5. For suggestions: evaluate and implement if beneficial
    6. Push changes and mark comment resolved
```

### Continuous Dependency Management

```yaml
dependency_workflow:
  schedule: "Weekly or on security advisory"

  workflow:
    1. Run npm audit / pip-audit / cargo audit
    2. Classify vulnerabilities by severity
    3. For Critical/High: immediate update
    4. For Medium: schedule update
    5. Run full test suite after updates
    6. Create PR with changelog
```

---

## Batch Processing (Claude API)

**50% cost reduction for large-scale async operations.**

### When to Use Batch API

| Use Case | Batch? | Reasoning |
|----------|--------|-----------|
| Single code review | No | Immediate feedback needed |
| Review 100+ files | Yes | 50% savings, async OK |
| Generate tests for all modules | Yes | Bulk operation |
| Interactive development | No | Need real-time responses |
| Large-scale evaluations | Yes | Cost-critical at scale |
| QA phase bulk analysis | Yes | Can wait for results |

### Batch API Limits

```yaml
limits:
  max_requests: 100,000 per batch
  max_size: 256 MB per batch
  processing_time: Most < 1 hour (max 24h)
  results_available: 29 days

pricing:
  discount: 50% on all tokens
  stacks_with: Prompt caching (30-98% cache hits)
```

### Implementation Pattern

```python
import anthropic
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

client = anthropic.Anthropic()

# Batch all code review requests
def batch_code_review(files: list[str]) -> str:
    requests = [
        Request(
            custom_id=f"review-{i}-{file.replace('/', '-')}",
            params=MessageCreateParamsNonStreaming(
                model="claude-sonnet-4-6",
                max_tokens=2048,
                messages=[{
                    "role": "user",
                    "content": f"Review this code for bugs, security, performance:\n\n{open(file).read()}"
                }]
            )
        )
        for i, file in enumerate(files)
    ]

    batch = client.messages.batches.create(requests=requests)
    return batch.id  # Poll for results later

# Poll for completion
def wait_for_batch(batch_id: str):
    while True:
        batch = client.messages.batches.retrieve(batch_id)
        if batch.processing_status == "ended":
            return batch
        time.sleep(60)

# Stream results
def process_results(batch_id: str):
    for result in client.messages.batches.results(batch_id):
        if result.result.type == "succeeded":
            # Process successful review
            handle_review(result.custom_id, result.result.message)
        elif result.result.type == "errored":
            # Retry or log error
            handle_error(result.custom_id, result.result.error)
```

### Batch + Prompt Caching

**Stack discounts for maximum savings:**

```python
# All requests share cached system prompt
SHARED_SYSTEM = [
    {"type": "text", "text": "You are a code reviewer..."},
    {"type": "text", "text": CODING_STANDARDS,  # Large shared context
     "cache_control": {"type": "ephemeral"}}
]

requests = [
    Request(
        custom_id=f"review-{file}",
        params=MessageCreateParamsNonStreaming(
            model="claude-sonnet-4-6",
            max_tokens=2048,
            system=SHARED_SYSTEM,  # Identical across all requests
            messages=[{"role": "user", "content": f"Review: {code}"}]
        )
    )
    for file, code in files_with_code
]
```

**Cost math:**
- Base: $3/MTok input, $15/MTok output (Sonnet)
- Batch discount: 50% -> $1.50/$7.50
- Cache hit: 90% reduction on cached tokens
- Combined: Up to 95% savings on large batches


## Install Packages Locally

Install dependencies into the project (a local `node_modules`, virtualenv or vendor directory) rather than globally, so a run is reproducible, never alters the host, and is removed with the workspace.

---

## What Actually Works in Production

### Human-in-the-Loop (HITL) is Non-Negotiable

**Key Insight:** "Zero companies don't have a human in the loop" for customer-facing applications.

```yaml
hitl_patterns:
  always_human:
    - Customer-facing responses
    - Financial transactions
    - Security-critical operations
    - Legal/compliance decisions

  automation_candidates:
    - Internal tooling
    - Developer assistance
    - Data preprocessing
    - Code generation (with review)

  implementation:
    - Classification layer routes to human vs automated
    - Confidence thresholds trigger escalation
    - Audit trails for all automated decisions
```

### Narrow Scope Wins

**Key Insight:** Successful agents operate within tightly constrained domains.

```yaml
scope_constraints:
  max_steps_before_review: 3-5
  task_characteristics:
    - Specific, well-defined objectives
    - Pre-classified inputs
    - Deterministic success criteria
    - Verifiable outputs

  successful_domains:
    - Email scanning and classification
    - Invoice processing
    - Code refactoring (bounded)
    - Documentation generation
    - Test writing

  failure_prone_domains:
    - Open-ended feature implementation
    - Novel algorithm design
    - Security-critical code
    - Cross-system integrations
```

### Confidence-Based Routing

**Key Insight:** Treat agents as preprocessors, not decision-makers.

```python
def confidence_based_routing(agent_output):
    """
    Route based on confidence, not capability.
    Based on production practitioner patterns.
    """
    confidence = agent_output.confidence_score

    if confidence >= 0.95:
        # High confidence: auto-approve with logging
        return AutoApprove(audit_log=True)

    elif confidence >= 0.70:
        # Medium confidence: quick human review
        return HumanReview(priority="normal", timeout="1h")

    elif confidence >= 0.40:
        # Low confidence: detailed human review
        return HumanReview(priority="high", context="full")

    else:
        # Very low confidence: escalate immediately
        return Escalate(reason="low_confidence", require_senior=True)
```

### Classification Before Automation

**Key Insight:** Separate inputs before processing.

```yaml
classification_first:
  step_1_classify:
    workable:
      - Clear requirements
      - Existing patterns
      - Test coverage available
    non_workable:
      - Ambiguous requirements
      - Novel architecture
      - Missing dependencies
    escalate_immediately:
      - Security concerns
      - Compliance requirements
      - Customer-facing changes

  step_2_route:
    workable: "Automated pipeline"
    non_workable: "Human clarification"
    escalate: "Senior review"
```

### Deterministic Outer Loops

**Key Insight:** Wrap agent outputs with rule-based validation.

```python
def deterministic_validation_loop(task, max_attempts=3):
    """
    Use LLMs only where genuine ambiguity exists.
    Wrap with deterministic rules.
    """
    for attempt in range(max_attempts):
        # LLM handles the ambiguous part
        output = agent.execute(task)

        # Deterministic validation (NOT LLM)
        validation_errors = []

        # Rule: Must have tests
        if not output.has_tests:
            validation_errors.append("Missing tests")

        # Rule: Must pass linting
        lint_result = run_linter(output.code)
        if lint_result.errors:
            validation_errors.append(f"Lint errors: {lint_result.errors}")

        # Rule: Must compile
        compile_result = compile_code(output.code)
        if not compile_result.success:
            validation_errors.append(f"Compile error: {compile_result.error}")

        # Rule: Tests must pass
        if output.has_tests:
            test_result = run_tests(output.code)
            if not test_result.all_passed:
                validation_errors.append(f"Test failures: {test_result.failures}")

        if not validation_errors:
            return output

        # Feed errors back for retry
        task = task.with_feedback(validation_errors)

    return FailedResult(reason="Max attempts exceeded")
```

---

## Context Engineering Patterns

### Context Curation Over Automatic Selection

**Key Insight:** Manually choose which files and information to provide.

```yaml
context_curation:
  principles:
    - "Less is more" - focused context beats comprehensive context
    - Manual selection outperforms automatic RAG
    - Remove outdated information aggressively

  anti_patterns:
    - Dumping entire codebase into context
    - Relying on automatic context selection
    - Accumulating conversation history indefinitely

  implementation:
    per_task_context:
      - 2-5 most relevant files
      - Specific functions, not entire modules
      - Recent changes only (last 1-2 days)
      - Clear success criteria

    context_budget:
      target: "< 10k tokens for context"
      reserve: "90% for model reasoning"
```

### Information Abstraction

**Key Insight:** Summarize rather than feeding full data.

```python
def abstract_for_agent(raw_data, task_context):
    """
    Design abstractions that preserve decision-relevant information.
    Based on practitioner insights.
    """
    # BAD: Feed 10,000 database rows
    # raw_data = db.query("SELECT * FROM users")

    # GOOD: Summarize to decision-relevant info
    summary = {
        "query_status": "success",
        "total_results": len(raw_data),
        "sample": raw_data[:5],
        "schema": extract_schema(raw_data),
        "statistics": {
            "null_count": count_nulls(raw_data),
            "unique_values": count_uniques(raw_data),
            "date_range": get_date_range(raw_data)
        }
    }

    return summary
```

### Separate Conversations Per Task

**Key Insight:** Fresh contexts yield better results than accumulated sessions.

```yaml
conversation_management:
  new_conversation_triggers:
    - Different domain (backend -> frontend)
    - New feature vs bug fix
    - After completing major task
    - When errors accumulate (3+ in row)

  preserve_across_sessions:
    - CLAUDE.md / CONTINUITY.md
    - Architectural decisions
    - Key constraints

  discard_between_sessions:
    - Debugging attempts
    - Abandoned approaches
    - Intermediate drafts
```

---

## Skills System Pattern

### On-Demand Skill Loading

**Key Insight:** Skills remain dormant until the model actively seeks them out.

```yaml
skills_architecture:
  core_interaction: "< 2k tokens"
  skill_loading: "On-demand via search"

  implementation:
    skill_discovery:
      - Shell script searches skill files
      - Model requests specific skills by name
      - Skills loaded only when needed

    skill_structure:
      name: "unique-skill-name"
      trigger: "Pattern that activates skill"
      content: "Detailed instructions"
      dependencies: ["other-skills"]

  benefits:
    - Minimal base context
    - Extensible without bloat
    - Skills can be updated independently
```

### Sub-Agents for Context Isolation

**Key Insight:** Prevent massive token waste by isolating context-noisy subtasks.

```python
async def context_isolated_search(query, codebase_path):
    """
    Use sub-agent for grep/search to prevent context pollution.
    Based on Simon Willison's patterns.
    """
    # Main agent stays focused
    # Sub-agent handles noisy file searching

    search_agent = spawn_subagent(
        role="codebase-searcher",
        context_limit="10k tokens",
        permissions=["read-only"]
    )

    results = await search_agent.execute(
        task=f"Find files related to: {query}",
        codebase=codebase_path
    )

    # Return only relevant paths, not full content
    return FilteredResults(
        paths=results.relevant_files[:10],
        summaries=results.file_summaries,
        confidence=results.relevance_scores
    )
```

---

## Planning Before Execution

### Explicit Plan-Then-Code Workflow

**Key Insight:** Have models articulate detailed plans without immediately writing code.

```yaml
plan_then_code:
  phase_1_planning:
    outputs:
      - spec.md: "Detailed requirements"
      - todo.md: "Tagged tasks [BUG], [FEAT], [REFACTOR]"
      - approach.md: "Implementation strategy"
    constraints:
      - NO CODE in this phase
      - Human review before proceeding
      - Clear success criteria

  phase_2_review:
    checks:
      - Plan addresses all requirements
      - Approach is feasible
      - No missing dependencies
      - Tests are specified

  phase_3_implementation:
    constraints:
      - Follow plan exactly
      - One task at a time
      - Test after each change
      - Report deviations immediately
```

---

## Multi-Agent Orchestration Patterns

### Event-Driven Coordination

**Key Insight:** Move beyond synchronous prompt chaining to asynchronous, decoupled systems.

```yaml
event_driven_orchestration:
  problems_with_synchronous:
    - Doesn't scale
    - Mixes orchestration with prompt logic
    - Single failure breaks entire chain
    - No retry/recovery mechanism

  async_architecture:
    message_queue:
      - Agents communicate via events
      - Decoupled execution
      - Natural retry/dead-letter handling

    state_management:
      - Persistent task state
      - Checkpoint/resume capability
      - Clear ownership of data

    error_handling:
      - Per-agent retry policies
      - Circuit breakers
      - Graceful degradation
```

### Policy-First Enforcement

**Key Insight:** Govern agent behavior at runtime, not just training time.

```python
class PolicyEngine:
    """
    Runtime governance for agent behavior.
    Based on autonomous control plane patterns.
    """

    def __init__(self, policies):
        self.policies = policies

    async def enforce(self, agent_action, context):
        for policy in self.policies:
            result = await policy.evaluate(agent_action, context)

            if result.blocked:
                return BlockedAction(
                    reason=result.reason,
                    policy=policy.name,
                    remediation=result.suggested_action
                )

            if result.modified:
                agent_action = result.modified_action

        return AllowedAction(agent_action)

# Example policies
policies = [
    NoProductionDataDeletion(),
    NoSecretsInCode(),
    MaxTokenBudget(limit=100000),
    RequireTestsForCode(),
    BlockExternalNetworkCalls(in_sandbox=True)
]
```

### Simulation Layer

**Key Insight:** Evaluate changes before deploying to real environment.

```yaml
simulation_layer:
  purpose: "Test agent behavior in safe environment"

  implementation:
    sandbox_environment:
      - Isolated container
      - Mocked external services
      - Synthetic data
      - Full audit logging

    validation_checks:
      - Run tests in sandbox first
      - Compare outputs to expected
      - Check for policy violations
      - Measure resource consumption

    promotion_criteria:
      - All tests pass
      - No policy violations
      - Resource usage within limits
      - Human approval (for sensitive changes)
```

---

## Evaluation and Benchmarking

### Problems with Current Benchmarks

**Key Insight:** LLM-as-judge creates shared blind spots.

```yaml
benchmark_problems:
  llm_judge_issues:
    - Same architecture = same failure modes
    - Math errors accepted as correct
    - "Do-nothing" baseline passes 38% of time

  contamination:
    - Published benchmarks become training targets
    - Overfitting to specific datasets
    - Inflated scores don't reflect real performance

  solutions:
    held_back_sets: "90% public, 10% private"
    human_evaluation: "Final published results require humans"
    production_testing: "A/B tests measure actual value"
    objective_outcomes: "Simulated environments with verifiable results"
```

### Practical Evaluation Approach

```python
def evaluate_agent_change(before_agent, after_agent, task_set):
    """
    Production-oriented evaluation.
    Based on HN practitioner recommendations.
    """
    results = {
        "before": [],
        "after": [],
        "human_preference": []
    }

    for task in task_set:
        # Run both agents
        before_result = before_agent.execute(task)
        after_result = after_agent.execute(task)

        # Objective metrics (NOT LLM-judged)
        results["before"].append({
            "tests_pass": run_tests(before_result),
            "lint_clean": run_linter(before_result),
            "time_taken": before_result.duration,
            "tokens_used": before_result.tokens
        })

        results["after"].append({
            "tests_pass": run_tests(after_result),
            "lint_clean": run_linter(after_result),
            "time_taken": after_result.duration,
            "tokens_used": after_result.tokens
        })

        # Sample for human review
        if random.random() < 0.1:  # 10% sample
            results["human_preference"].append({
                "task": task,
                "before": before_result,
                "after": after_result,
                "pending_review": True
            })

    return EvaluationReport(results)
```

---

## Cost and Token Economics

### Real-World Cost Patterns

```yaml
cost_patterns:
  claude_code:
    heavy_use: "$25/1-2 hours on large codebases"
    api_range: "$1-5/hour depending on efficiency"
    max_tier: "$200/month often needs 2-3 subscriptions"

  token_economics:
    sub_agents_multiply_cost: "Each duplicates context"
    example: "5-task parallel job = 50,000+ tokens per subtask"

  optimization:
    context_isolation: "Use sub-agents for noisy tasks"
    information_abstraction: "Summarize, don't dump"
    fresh_conversations: "Reset after major tasks"
    skill_on_demand: "Load only when needed"
```

---

## Cross-Process File Lock (v7.5.5)

Concurrent loki invocations on the same `.loki/` workspace previously raced on `state/orchestrator.json`, `queue/pending.json`, and CONTINUITY.md. v7.5.5 introduces a `flock`-backed cross-process lock around all state writes. Concurrent runs serialize state mutations cleanly; readers continue without blocking.

---

## Sources

**Hacker News Discussions:**
- [What Actually Works in Production for Autonomous Agents](https://news.ycombinator.com/item?id=44623207)
- [Coding with LLMs in Summer 2025](https://news.ycombinator.com/item?id=44623953)
- [Superpowers: How I'm Using Coding Agents](https://news.ycombinator.com/item?id=45547344)
- [Claude Code Experience After Two Weeks](https://news.ycombinator.com/item?id=44596472)
- [AI Agent Benchmarks Are Broken](https://news.ycombinator.com/item?id=44531697)
- [How to Orchestrate Multi-Agent Workflows](https://news.ycombinator.com/item?id=45955997)
- [Context Engineering vs Prompt Engineering](https://news.ycombinator.com/item?id=44427757)

**Show HN Projects:**
- [Self-Evolving Agents Repository](https://news.ycombinator.com/item?id=45099226)
- [Package Manager for Agent Skills](https://news.ycombinator.com/item?id=46422264)
- [Wispbit - AI Code Review Agent](https://news.ycombinator.com/item?id=44722603)
- [Agtrace - Monitoring for AI Coding Agents](https://news.ycombinator.com/item?id=46425670)

---

References: `references/deployment.md` (infrastructure and deployment by platform), `references/task-queue.md` (task queue, dead letters, circuit breakers).