---
description: Infrastructure testing—static analysis, Terratest, Helm/K8s tests, chaos, load, and DR verification. Keep tests automated and production-like.
alwaysApply: false
---

# Infrastructure Testing

Guidelines for testing infrastructure, platform components, and reliability.

## Core Principles

- **Test early, test often** — validate before apply, not after
- **Test like production** — realistic data, scale, and scenarios
- **Automate everything** — manual testing doesn't scale
- **Test the recovery** — verify backups, failover, and DR

## Testing Layers

- **Static analysis** — Terraform fmt/validate, TFLint, tfsec; K8s schema (kubeconform), Kyverno CLI
- **Unit (Terratest)** — module-level IaC tests; init, apply, assert outputs, destroy
- **Integration** — Helm chart tests (`helm.sh/hook: test`), multi-component in test namespace
- **Chaos** — pod/network failure via Chaos Mesh or Litmus; validate SLOs during and after
- **Load** — k6 or similar; define stages and thresholds (p99 latency, error rate)
- **DR** — backup restore to test DB; failover simulation and recovery verification

## Static Analysis

Run on every PR for changed paths (`terraform/**`, `kubernetes/**`):

- Terraform: `fmt -check`, `validate`, TFLint, tfsec/Checkov
- Kubernetes: YAML lint, kubeconform, Kyverno apply, Kubesec
- Pre-commit hooks: terraform_fmt, terraform_validate, yamllint, detect-private-key

## Unit Tests (Terratest)

```go
func TestVpcModule(t *testing.T) {
    opts := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
        TerraformDir: "../modules/networking/vpc",
        Vars: map[string]interface{}{"environment": "test", "vpc_cidr": "10.0.0.0/16"},
    })
    defer terraform.Destroy(t, opts)
    terraform.InitAndApply(t, opts)
    assert.NotEmpty(t, terraform.Output(t, opts, "vpc_id"))
}
```

## Integration, Chaos, Load, DR

- **Helm tests**: `helm.sh/hook: test` pods that curl healthz; `hook-delete-policy: hook-succeeded`
- **Chaos**: one experiment per file; run in staging on schedule; assert SLO after
- **Load**: ramp up → sustain → ramp down; thresholds (p99 < 500ms, errors < 1%); store results
- **DR**: periodic backup restore + sanity queries; staging failover + health verification

## Common Pitfalls

- **Happy path only** — test failure cases and validation errors too
- **Order-dependent tests** — isolate each test; use setup/teardown, no shared mutable state
- **No cleanup** — always `defer terraform.Destroy`; resources leak otherwise