purpose: Complete guide to validation strategies and the 9-layer validation framework type: technical-reference course: Tactical Agentic Coding lesson: 05-supplemental difficulty: INTERMEDIATE dependencies: lesson-05-close-the-loops, closed-loop-architecture last-updated: 2025-09-30 Validation strategies define how agents verify their work meets quality standards. This document provides comprehensive coverage of the 9-layer validation framework, when to use each layer, command patterns for different languages and frameworks, and strategies for building effective validation suites. # VALIDATION STRATEGIES ## THE 9 VALIDATION LAYERS ### Overview ``` VALIDATION PYRAMID ┌────────────────┐ │ LLM-as-Judge │ Subjective quality └────────────────┘ ┌──────────────────────┐ │ Custom Evaluations │ Domain-specific └──────────────────────┘ ┌────────────────────────────┐ │ CI/CD Integration │ Production simulation └────────────────────────────┘ ┌────────────────────────────────────┐ │ Error Logs │ Runtime issues └────────────────────────────────────┘ ┌──────────────────────────────────────────┐ │ Build/Compile │ Deployment readiness └──────────────────────────────────────────┘ ┌───────────────────────────────────────────────────┐ │ UI Tests │ User experience └───────────────────────────────────────────────────┘ ┌──────────────────────────────────────────────────────────┐ │ Integration Tests │ Component interaction └──────────────────────────────────────────────────────────┘ ┌────────────────────────────────────────────────────────────────┐ │ Unit Tests │ Function logic └────────────────────────────────────────────────────────────────┘ ┌──────────────────────────────────────────────────────────────────┐ │ Linter │ Syntax & style └──────────────────────────────────────────────────────────────────┘ PRINCIPLE: Start at the bottom (fastest, cheapest) and work up ``` --- ## LAYER 1: LINTER **SPEED:** Instant (milliseconds) **COST:** Free (no API calls) **PURPOSE:** Catch syntax errors, style violations, basic code issues **WHEN:** Every code change, always first **FAILURE RATE:** High (catches many issues early) ### What Linters Validate - **Syntax errors:** Missing semicolons, brackets, quotes - **Style violations:** Indentation, spacing, naming conventions - **Common bugs:** Unused variables, unreachable code - **Best practices:** Deprecated APIs, security issues - **Type errors:** (in typed linters like TypeScript) ### Language-Specific Linter Commands ```bash # ESLint (most common) npx eslint . --ext .js,.jsx,.ts,.tsx # With auto-fix npx eslint . --ext .js,.jsx,.ts,.tsx --fix # Specific files npx eslint src/**/*.{js,ts} # Check exit code (agent validation) npx eslint . && echo "PASS" || echo "FAIL" ``` **INTEGRATION:** ```json // package.json { "scripts": { "lint": "eslint . --ext .js,.jsx,.ts,.tsx", "lint:fix": "eslint . --ext .js,.jsx,.ts,.tsx --fix" } } ``` ```bash # Pylint (comprehensive) pylint **/*.py # Flake8 (faster, less strict) flake8 . # Ruff (fastest, modern) ruff check . # With auto-fix ruff check . --fix # Black (formatter, but validates formatting) black --check . # Type checking (mypy) mypy src/ # All combined pylint src/ && flake8 src/ && mypy src/ ``` **INTEGRATION:** ```toml # pyproject.toml [tool.ruff] line-length = 88 target-version = "py311" [tool.black] line-length = 88 ``` ```bash # Go vet (official tool) go vet ./... # Golint (style checker) golint ./... # Staticcheck (comprehensive) staticcheck ./... # Combined go vet ./... && staticcheck ./... # With formatting check gofmt -l . && go vet ./... && staticcheck ./... ``` ```bash # Clippy (official linter) cargo clippy # Strict mode cargo clippy -- -D warnings # With formatting check cargo fmt --check && cargo clippy ``` ```bash # Checkstyle mvn checkstyle:check # SpotBugs mvn spotbugs:check # PMD mvn pmd:check # All combined mvn checkstyle:check && mvn spotbugs:check ``` ```bash # RuboCop rubocop # With auto-fix rubocop -a # Specific files rubocop app/ lib/ ``` ```bash # PHP_CodeSniffer phpcs --standard=PSR12 src/ # PHP Stan (static analysis) phpstan analyse src/ # Combined phpcs --standard=PSR12 src/ && phpstan analyse src/ ``` ### Linter Integration in Closed Loops ```python class LinterValidation: def __init__(self, language): self.language = language self.commands = { "javascript": ["npx", "eslint", "."], "python": ["ruff", "check", "."], "go": ["go", "vet", "./..."], "rust": ["cargo", "clippy"], } def validate(self, work_product): """Run linter and parse output.""" cmd = self.commands.get(self.language) if not cmd: raise ValueError(f"No linter for {self.language}") result = subprocess.run( cmd, capture_output=True, text=True ) return ValidationResult( passed=(result.returncode == 0), output=result.stdout, errors=result.stderr, exit_code=result.returncode ) ``` --- ## LAYER 2: UNIT TESTS **SPEED:** Seconds **COST:** Low (compute only) **PURPOSE:** Validate individual functions and components **WHEN:** After linter passes **FAILURE RATE:** Medium (catches logic errors) ### What Unit Tests Validate - **Function correctness:** Does function return expected output? - **Edge cases:** Handles empty input, null, extreme values - **Error handling:** Throws expected errors - **Pure logic:** Business logic without external dependencies - **Component isolation:** Individual units work correctly ### Framework-Specific Commands ```bash # Run all tests npm test # With coverage npm test -- --coverage # Watch mode (not for agents) npm test -- --watch # Specific test file npm test -- path/to/test.spec.ts # Silent mode (just pass/fail) npm test -- --silent # CI mode (no watch, deterministic) CI=true npm test ``` **INTEGRATION:** ```json // package.json { "scripts": { "test": "jest", "test:coverage": "jest --coverage", "test:ci": "CI=true jest --silent" } } ``` ```bash # Run all tests pytest # With coverage pytest --cov=src tests/ # Verbose output pytest -v # Stop on first failure pytest -x # Specific test file pytest tests/test_module.py # Specific test function pytest tests/test_module.py::test_function # Quiet mode pytest -q # Generate report pytest --html=report.html ``` **INTEGRATION:** ```toml # pyproject.toml [tool.pytest.ini_options] testpaths = ["tests"] python_files = ["test_*.py"] python_classes = ["Test*"] python_functions = ["test_*"] ``` ```bash # Run all tests go test ./... # With coverage go test -cover ./... # Verbose go test -v ./... # Short mode (skip slow tests) go test -short ./... # Specific package go test ./pkg/mypackage # With race detection go test -race ./... # Benchmark tests go test -bench=. ./... ``` ```bash # Run all tests cargo test # Verbose cargo test -- --nocapture # Specific test cargo test test_name # Doc tests cargo test --doc # Integration tests only cargo test --test integration_test # Release mode (optimized) cargo test --release ``` ```bash # Maven mvn test # Gradle ./gradlew test # Specific test class mvn test -Dtest=MyTestClass # With coverage (JaCoCo) mvn test jacoco:report ``` ### Unit Test Integration ```python class UnitTestValidation: def __init__(self, framework, test_path="tests/"): self.framework = framework self.test_path = test_path def validate(self, work_product): """Run unit tests and parse results.""" commands = { "jest": ["npm", "test", "--", "--silent"], "pytest": ["pytest", "-q", self.test_path], "go": ["go", "test", "./..."], "cargo": ["cargo", "test"], } cmd = commands.get(self.framework) result = subprocess.run(cmd, capture_output=True, text=True) # Parse output for failure details failures = self.parse_failures(result.stdout, result.stderr) return ValidationResult( passed=(result.returncode == 0), total_tests=self.count_tests(result.stdout), failures=failures, coverage=self.extract_coverage(result.stdout) ) def parse_failures(self, stdout, stderr): """Extract failure details for agent context.""" # Framework-specific parsing failures = [] # ... parsing logic ... return failures ``` --- ## LAYER 3: INTEGRATION TESTS **SPEED:** Seconds to minutes **COST:** Medium (may require services, databases) **PURPOSE:** Validate component interactions **WHEN:** After unit tests pass **FAILURE RATE:** Medium (catches integration issues) ### What Integration Tests Validate - **Database connections:** Can connect and query database - **API interactions:** Services communicate correctly - **File system operations:** Read/write operations work - **External services:** Third-party APIs respond correctly - **Message queues:** Pub/sub systems function - **Service meshes:** Microservices coordinate properly ### Integration Test Commands ```bash # Setup test database docker-compose up -d postgres # Run integration tests npm run test:integration pytest tests/integration/ go test -tags=integration ./... # Teardown docker-compose down ``` **EXAMPLE:** ```python # Integration test setup import pytest from testcontainers.postgres import PostgresContainer @pytest.fixture(scope="session") def postgres_container(): with PostgresContainer("postgres:15") as postgres: yield postgres def test_database_integration(postgres_container): """Test actual database operations.""" connection_url = postgres_container.get_connection_url() db = Database(connection_url) # Test CRUD operations user = db.create_user(name="test") assert db.get_user(user.id) == user ``` ```bash # Start test API server npm run api:test & API_PID=$! # Run API integration tests npm run test:api # Stop API server kill $API_PID ``` **EXAMPLE:** ```typescript // API integration test describe('API Integration', () => { let server: TestServer; beforeAll(async () => { server = await TestServer.start(); }); afterAll(async () => { await server.stop(); }); it('creates and retrieves user', async () => { const response = await fetch(`${server.url}/users`, { method: 'POST', body: JSON.stringify({ name: 'Test User' }) }); expect(response.status).toBe(201); const user = await response.json(); expect(user.name).toBe('Test User'); }); }); ``` ```bash # Docker Compose for full stack docker-compose -f docker-compose.test.yml up -d # Wait for services to be ready ./scripts/wait-for-services.sh # Run integration tests npm run test:integration:full # Cleanup docker-compose -f docker-compose.test.yml down ``` --- ## LAYER 4: UI TESTS **SPEED:** Minutes **COST:** Medium-High (browser automation) **PURPOSE:** Validate user interface behavior **WHEN:** After integration tests pass **FAILURE RATE:** Low-Medium (catches UI-specific issues) ### What UI Tests Validate - **User workflows:** Can complete key user journeys - **Element interactions:** Buttons, forms, navigation work - **Visual rendering:** Components display correctly - **Responsiveness:** Works on different screen sizes - **Accessibility:** ARIA labels, keyboard navigation - **Cross-browser:** Works in Chrome, Firefox, Safari ### UI Testing Framework Commands ```bash # Install browsers (one-time) npx playwright install # Run all tests npx playwright test # Headless mode (default) npx playwright test --headed # Specific browser npx playwright test --project=chromium npx playwright test --project=firefox npx playwright test --project=webkit # Debug mode npx playwright test --debug # UI mode (interactive) npx playwright test --ui # Generate report npx playwright show-report ``` **EXAMPLE TEST:** ```typescript import { test, expect } from '@playwright/test'; test('user can login', async ({ page }) => { await page.goto('http://localhost:3000/login'); await page.fill('input[name="email"]', 'test@example.com'); await page.fill('input[name="password"]', 'password123'); await page.click('button[type="submit"]'); await expect(page).toHaveURL('http://localhost:3000/dashboard'); await expect(page.locator('h1')).toHaveText('Dashboard'); }); ``` ```bash # Run all tests (headless) npx cypress run # Interactive mode npx cypress open # Specific browser npx cypress run --browser chrome # Specific test file npx cypress run --spec "cypress/e2e/login.cy.ts" # Record to Cypress Cloud npx cypress run --record --key ``` **EXAMPLE TEST:** ```typescript describe('Login Flow', () => { it('allows user to login', () => { cy.visit('/login'); cy.get('input[name="email"]').type('test@example.com'); cy.get('input[name="password"]').type('password123'); cy.get('button[type="submit"]').click(); cy.url().should('include', '/dashboard'); cy.contains('h1', 'Dashboard').should('be.visible'); }); }); ``` ```python # Python Selenium from selenium import webdriver from selenium.webdriver.common.by import By def test_login(): driver = webdriver.Chrome() driver.get("http://localhost:3000/login") driver.find_element(By.NAME, "email").send_keys("test@example.com") driver.find_element(By.NAME, "password").send_keys("password123") driver.find_element(By.CSS_SELECTOR, "button[type='submit']").click() assert "dashboard" in driver.current_url driver.quit() ``` **RUN:** ```bash pytest tests/ui/test_login.py ``` --- ## LAYER 5: BUILD/COMPILE **SPEED:** Seconds to minutes **COST:** Low (compute only) **PURPOSE:** Ensure code compiles and bundles successfully **WHEN:** After tests pass **FAILURE RATE:** Low (most errors caught earlier) ### What Build Validates - **Compilation:** Code compiles without errors - **Bundling:** Assets bundle correctly - **Dependencies:** All imports resolve - **Type checking:** (in compiled languages) - **Optimization:** Production build succeeds - **Asset generation:** Static files created ### Build Commands by Language/Framework ```bash # TypeScript compilation npx tsc --noEmit # Full build npm run build # Vite build npx vite build # Next.js build npm run build # Check build size npm run build && du -sh dist/ # Production build NODE_ENV=production npm run build ``` ```bash # Build binary go build -o app main.go # Build all packages go build ./... # Cross-compile GOOS=linux GOARCH=amd64 go build -o app-linux main.go # With optimizations go build -ldflags="-s -w" -o app main.go # Check for build errors without creating binary go build -o /dev/null ./... ``` ```bash # Debug build cargo build # Release build (optimized) cargo build --release # Check (fast compile check) cargo check # All targets cargo build --all-targets ``` ```bash # Maven mvn package # Skip tests during build mvn package -DskipTests # Gradle ./gradlew build # Just compile mvn compile ``` ```bash # Build package python -m build # Install locally pip install -e . # Create wheel python setup.py bdist_wheel # Check package twine check dist/* ``` --- ## LAYER 6: ERROR LOGS **SPEED:** Instant **COST:** Free **PURPOSE:** Check for runtime errors during execution **WHEN:** After execution completes **FAILURE RATE:** Variable (depends on execution) ### What Log Validation Checks - **Error messages:** No ERROR level logs - **Warnings:** Acceptable WARNING count - **Exceptions:** No uncaught exceptions - **Stack traces:** No stack traces in logs - **Fatal errors:** No FATAL/CRITICAL logs - **Specific patterns:** Known error signatures ### Log Validation Commands ```bash # Check for errors in logs grep -i "error" logs/app.log grep -i "exception" logs/app.log grep -i "fatal" logs/app.log # Count errors grep -c "ERROR" logs/app.log # Check exit status (agent validation) if grep -q "ERROR" logs/app.log; then echo "FAIL: Errors found in logs" exit 1 else echo "PASS: No errors in logs" exit 0 fi # Multiple patterns grep -E "(ERROR|FATAL|Exception)" logs/app.log # Exclude known acceptable errors grep "ERROR" logs/app.log | grep -v "ExpectedError" # Check last N lines tail -n 1000 logs/app.log | grep "ERROR" # Time-based (last 5 minutes) find logs/ -name "*.log" -mmin -5 -exec grep "ERROR" {} \; ``` ### Structured Log Parsing ```python import json def validate_logs(log_file): """Validate structured JSON logs.""" errors = [] warnings = [] with open(log_file) as f: for line in f: try: log_entry = json.loads(line) level = log_entry.get("level", "").upper() if level in ["ERROR", "FATAL", "CRITICAL"]: errors.append(log_entry) elif level == "WARNING": warnings.append(log_entry) except json.JSONDecodeError: # Non-JSON log line pass return ValidationResult( passed=(len(errors) == 0), errors=errors, warnings=warnings ) ``` --- ## LAYER 7: CI/CD INTEGRATION **SPEED:** Minutes **COST:** Medium (CI/CD compute) **PURPOSE:** Validate in production-like environment **WHEN:** Before merge/deployment **FAILURE RATE:** Low (most issues caught earlier) ### What CI/CD Validates - **Full pipeline:** All validation layers pass - **Environment simulation:** Production-like conditions - **Deployment readiness:** Can deploy successfully - **Cross-platform:** Works on different OS - **Dependency resolution:** Clean dependency install - **Security scans:** Vulnerability checks ### CI/CD Configuration Examples ```yaml name: Validation Pipeline on: [push, pull_request] jobs: validate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Setup Node.js uses: actions/setup-node@v3 with: node-version: '18' - name: Install dependencies run: npm ci - name: Lint run: npm run lint - name: Unit tests run: npm test - name: Build run: npm run build - name: Integration tests run: npm run test:integration - name: E2E tests run: npm run test:e2e ``` ```yaml stages: - lint - test - build - integration lint: stage: lint script: - npm run lint unit_tests: stage: test script: - npm test build: stage: build script: - npm run build artifacts: paths: - dist/ integration_tests: stage: integration script: - npm run test:integration dependencies: - build ``` --- ## LAYER 8: CUSTOM EVALUATIONS **SPEED:** Varies **COST:** Varies **PURPOSE:** Domain-specific validation logic **WHEN:** Task-specific requirements **FAILURE RATE:** Variable (depends on domain) ### Types of Custom Evaluations ```bash # Run benchmarks npm run bench # Compare against baseline node scripts/benchmark.js --compare baseline.json # Threshold validation node scripts/check-performance.js --max-time 100ms ``` **EXAMPLE:** ```javascript // benchmark.js const { performance } = require('perf_hooks'); function benchmark(fn, iterations = 1000) { const start = performance.now(); for (let i = 0; i < iterations; i++) { fn(); } const end = performance.now(); return (end - start) / iterations; } const avgTime = benchmark(myFunction); if (avgTime > 100) { console.error(`FAIL: Function too slow (${avgTime}ms > 100ms)`); process.exit(1); } ``` ```bash # NPM audit npm audit --audit-level=moderate # Snyk scan snyk test # OWASP dependency check dependency-check --project myapp --scan . # Trivy (container scanning) trivy image myapp:latest ``` ```bash # axe-core (automated a11y testing) npx @axe-core/cli http://localhost:3000 # Lighthouse accessibility lighthouse http://localhost:3000 --only-categories=accessibility # pa11y pa11y http://localhost:3000 ``` ```bash # Code complexity npx complexity-report src/ # Code coverage threshold jest --coverage --coverageThreshold='{"global":{"branches":80,"functions":80,"lines":80}}' # Duplicate code detection jscpd src/ ``` --- ## LAYER 9: LLM-AS-JUDGE **SPEED:** Seconds **COST:** API calls (variable) **PURPOSE:** Subjective quality evaluation **WHEN:** Subjective quality matters **FAILURE RATE:** Variable (depends on criteria) ### What LLM-as-Judge Validates - **Code review:** Quality, readability, best practices - **Documentation quality:** Clarity, completeness - **User experience:** Design, usability - **Style and tone:** Writing quality - **Business logic:** Correctness of domain logic - **Edge case coverage:** Comprehensive handling ### LLM Judge Implementation ```python import anthropic class LLMJudge: def __init__(self, model="claude-3-5-sonnet-20250219"): self.client = anthropic.Anthropic() self.model = model def evaluate_code_quality(self, code, criteria): """ Evaluate code quality using LLM. Args: code: Code to evaluate criteria: List of quality criteria Returns: ValidationResult with score and feedback """ prompt = f"""Evaluate this code against the following criteria: Criteria: {chr(10).join(f"- {c}" for c in criteria)} Code: ``` {code} ``` Provide: 1. PASS or FAIL 2. Score (0-100) 3. Specific issues found 4. Suggestions for improvement Format your response as JSON: {{ "result": "PASS" or "FAIL", "score": 85, "issues": ["issue 1", "issue 2"], "suggestions": ["suggestion 1"] }} """ response = self.client.messages.create( model=self.model, max_tokens=2000, messages=[{"role": "user", "content": prompt}] ) # Parse JSON response result = json.loads(response.content[0].text) return ValidationResult( passed=(result["result"] == "PASS"), score=result["score"], issues=result["issues"], suggestions=result["suggestions"] ) def evaluate_documentation(self, docs, requirements): """Evaluate documentation quality.""" prompt = f"""Evaluate this documentation: Requirements: - Clarity: Easy to understand - Completeness: All features documented - Examples: Code examples provided - Structure: Well-organized Documentation: {docs} Provide PASS/FAIL and specific feedback. """ # Similar implementation ``` ### Judge Patterns ```python criteria = [ "Code follows best practices", "Functions are small and focused", "Naming is clear and descriptive", "No obvious bugs or security issues", "Error handling is appropriate", "Code is maintainable" ] judge = LLMJudge() result = judge.evaluate_code_quality(code, criteria) if not result.passed: print(f"Code review failed: {result.issues}") # Agent can retry with feedback ``` ```python def validate_documentation(docs): judge = LLMJudge() requirements = [ "All public APIs are documented", "Examples are provided", "Clear and concise language", "No typos or grammatical errors" ] return judge.evaluate_documentation(docs, requirements) ``` --- ## WHEN TO USE EACH VALIDATION TYPE ### Decision Matrix | Situation | Recommended Layers | Priority | |-----------|-------------------|----------| | **Quick code changes** | Linter, Unit Tests | 1, 2 | | **New feature** | Linter, Unit, Integration, Build | 1, 2, 3, 5 | | **UI component** | Linter, Unit, UI Tests | 1, 2, 4 | | **API endpoint** | Linter, Unit, Integration, Logs | 1, 2, 3, 6 | | **Refactoring** | Linter, Unit, Integration | 1, 2, 3 | | **Bug fix** | Linter, Unit, Custom (regression test) | 1, 2, 8 | | **Performance critical** | Unit, Custom (benchmarks) | 2, 8 | | **Security sensitive** | Linter, Unit, Custom (security scan) | 1, 2, 8 | | **Documentation** | LLM-as-Judge | 9 | | **Pre-deployment** | All layers | 1-7 | ### Validation Selection Strategy ```python class ValidationSelector: def select_validations(self, task_type, risk_level): """Select appropriate validations based on task.""" base = ["linter", "unit_tests"] if task_type in ["feature", "bug_fix"]: base.append("integration_tests") if task_type == "ui_component": base.append("ui_tests") if risk_level == "high": base.extend(["build", "ci_cd", "custom_security"]) if task_type == "documentation": base.append("llm_judge") return base ``` --- ## VALIDATION COMMAND PATTERNS ### Universal Validation Script ```bash #!/bin/bash # validate.sh - Universal validation script set -e # Exit on first failure echo "🔍 Starting validation pipeline..." # 1. LINTER echo "📝 Running linter..." npm run lint || exit 1 # 2. UNIT TESTS echo "🧪 Running unit tests..." npm test || exit 1 # 3. BUILD echo "🏗️ Building..." npm run build || exit 1 # 4. INTEGRATION TESTS echo "🔗 Running integration tests..." npm run test:integration || exit 1 # 5. E2E TESTS if [ "$RUN_E2E" = "true" ]; then echo "🎭 Running E2E tests..." npm run test:e2e || exit 1 fi # 6. LOG CHECK echo "📋 Checking logs..." if grep -q "ERROR" logs/*.log 2>/dev/null; then echo "❌ Errors found in logs" exit 1 fi echo "✅ All validations passed!" ``` ### Framework-Agnostic Validation ```python class UniversalValidator: """Framework-agnostic validation orchestrator.""" def __init__(self, project_root): self.project_root = project_root self.detect_framework() def detect_framework(self): """Auto-detect project framework.""" if (self.project_root / "package.json").exists(): self.framework = "node" elif (self.project_root / "pyproject.toml").exists(): self.framework = "python" elif (self.project_root / "go.mod").exists(): self.framework = "go" elif (self.project_root / "Cargo.toml").exists(): self.framework = "rust" def validate_all(self): """Run all appropriate validations.""" validations = { "node": [ ("lint", ["npm", "run", "lint"]), ("test", ["npm", "test"]), ("build", ["npm", "run", "build"]), ], "python": [ ("lint", ["ruff", "check", "."]), ("test", ["pytest"]), ("typecheck", ["mypy", "src/"]), ], "go": [ ("lint", ["go", "vet", "./..."]), ("test", ["go", "test", "./..."]), ("build", ["go", "build", "./..."]), ], "rust": [ ("lint", ["cargo", "clippy"]), ("test", ["cargo", "test"]), ("build", ["cargo", "build"]), ], } for name, cmd in validations[self.framework]: print(f"Running {name}...") result = subprocess.run(cmd, capture_output=True) if result.returncode != 0: raise ValidationError(f"{name} failed", result.stderr) return ValidationSuccess() ``` --- ## BUILDING VALIDATION SUITES ### Progressive Validation ```python class ProgressiveValidation: """Run validations progressively from fastest to slowest.""" def __init__(self): self.layers = [ ("linter", self.run_linter, "instant"), ("unit", self.run_unit_tests, "seconds"), ("integration", self.run_integration_tests, "minutes"), ("ui", self.run_ui_tests, "minutes"), ("build", self.run_build, "minutes"), ] def validate(self, work_product): """Run validations in order, fail fast.""" results = [] for name, validator, expected_time in self.layers: print(f"Running {name} ({expected_time})...") start = time.time() result = validator(work_product) duration = time.time() - start results.append({ "name": name, "passed": result.passed, "duration": duration, "details": result.details }) if not result.passed: # Fail fast - don't run slower tests return ValidationSummary( passed=False, failed_at=name, results=results ) return ValidationSummary(passed=True, results=results) ``` ### Parallel Validation Suite ```python import asyncio class ParallelValidationSuite: """Run independent validations in parallel.""" async def validate_all(self, work_product): """Run all validations concurrently.""" tasks = [ self.run_linter_async(work_product), self.run_unit_tests_async(work_product), self.run_type_check_async(work_product), ] results = await asyncio.gather(*tasks, return_exceptions=True) # Check for failures failures = [ r for r in results if isinstance(r, Exception) or not r.passed ] return ValidationSummary( passed=(len(failures) == 0), total=len(tasks), failures=failures, results=results ) ``` --- ## SUMMARY

**9 VALIDATION LAYERS:** 1. **Linter** - Instant, syntax/style (always use) 2. **Unit Tests** - Seconds, function logic (almost always) 3. **Integration Tests** - Minutes, component interaction (frequently) 4. **UI Tests** - Minutes, user experience (UI changes) 5. **Build/Compile** - Minutes, deployment readiness (before deploy) 6. **Error Logs** - Instant, runtime issues (after execution) 7. **CI/CD** - Minutes, production simulation (before merge) 8. **Custom** - Varies, domain-specific (as needed) 9. **LLM-as-Judge** - Seconds, subjective quality (quality matters) **PRINCIPLES:** - Start with fastest validations (fail fast) - Use minimum necessary layers for task - Run independent validations in parallel - Progressive: fast → slow - Always provide specific commands - Parse output for agent context **COMMANDS MUST BE:** - Deterministic (same input = same output) - Automatable (no interactive prompts) - Machine-readable (parseable output) - Exit code based (0 = success, non-zero = failure)