{
  "version": 1,
  "id": "chaos-engineering",
  "title": "Chaos Engineering",
  "summary": "Steady-state hypotheses, blast-radius control and game days: breaking things on purpose, safely.",
  "estimatedMinutes": 20,
  "prerequisites": [],
  "source": "courses-notebook SE303_observability_sre 07_chaos_engineering",
  "quiz": [
    {
      "prompt": "The 'steady-state hypothesis' is associated with:",
      "options": [
        "Capacity planning",
        "Chaos engineering",
        "Pair programming",
        "Sharding"
      ],
      "answer": 1
    }
  ],
  "lesson": "Topics: chaos principles \u00b7 fault injection patterns \u00b7 steady-state hypothesis \u00b7 chaos runner\n\n## 1. What is Chaos Engineering?\n\n> *\"Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production.\"*  \n> \u2014 Principles of Chaos Engineering (principlesofchaos.org)\n\nKey idea: **proactively inject failures** in controlled conditions to find weaknesses before they become incidents.\n\nChaos is not random destruction \u2014 it is **scientific experimentation** with:\n1. A hypothesis about system behaviour\n2. A controlled blast radius\n3. Automated abort conditions\n4. Observable outcomes\n\n## 2. The Five Principles\n\n1. **Define steady state** \u2014 quantify \"normal\" system behaviour (SLIs)\n2. **Hypothesise** \u2014 predict the system will remain in steady state during the experiment\n3. **Introduce real-world events** \u2014 latency, failures, resource exhaustion\n4. **Disprove the hypothesis** \u2014 if steady state breaks, you found a weakness\n5. **Minimise blast radius** \u2014 start with small scope (canary, shadow traffic, test env)\n\nCommon fault types:\n- **Network**: latency injection, packet loss, DNS failures\n- **Resource**: CPU spike, memory pressure, disk full\n- **Dependency**: kill upstream service, inject 500 responses\n- **Process**: kill random instances, OOM kill\n\n```python\nimport time\nimport random\nfrom dataclasses import dataclass, field\nfrom typing import Callable, Dict, List, Optional, Tuple\nfrom enum import Enum\n\n\nclass FaultType(Enum):\n    LATENCY          = \"latency\"\n    ERROR_INJECTION  = \"error_injection\"\n    RESOURCE_LIMIT   = \"resource_limit\"\n    DEPENDENCY_KILL  = \"dependency_kill\"\n\n\n@dataclass\nclass FaultConfig:\n    fault_type:   FaultType\n    intensity:    float     # 0.0 \u2013 1.0 (e.g. 0.1 = inject into 10% of calls)\n    params:       Dict      = field(default_factory=dict)\n\n\nclass FaultInjector:\n    \"\"\"Wraps a callable and injects faults according to config.\"\"\"\n\n    def __init__(self, config: FaultConfig, seed: int = 0):\n        self.config = config\n        self._rng   = random.Random(seed)\n\n    def call(self, fn: Callable, *args, **kwargs):\n        if self._rng.random() > self.config.intensity:\n            return fn(*args, **kwargs)   # pass-through\n\n        ft = self.config.fault_type\n        if ft == FaultType.LATENCY:\n            delay = self.config.params.get(\"delay_ms\", 500) / 1000\n            time.sleep(delay)\n            return fn(*args, **kwargs)\n        elif ft == FaultType.ERROR_INJECTION:\n            raise RuntimeError(self.config.params.get(\"message\", \"injected error\"))\n        elif ft == FaultType.DEPENDENCY_KILL:\n            raise ConnectionError(\"dependency unavailable (injected)\")\n        else:\n            return fn(*args, **kwargs)\n\n\n# Demonstrate fault injection\ndef checkout_handler(order_id: str) -> Dict:\n    return {\"status\": \"ok\", \"order_id\": order_id}\n\n\ninjector = FaultInjector(\n    FaultConfig(FaultType.ERROR_INJECTION, intensity=0.3,\n                params={\"message\": \"payment gateway timeout\"}),\n    seed=42,\n)\n\nok = errors = 0\nfor i in range(20):\n    try:\n        result = injector.call(checkout_handler, f\"order-{i}\")\n        ok += 1\n    except RuntimeError as e:\n        errors += 1\n\nprint(f\"20 calls \u2014 ok={ok}, errors={errors} (injected ~30%)\")\n```\n\n## 3. Steady-State Hypothesis\n\nBefore running a chaos experiment, define measurable steady state:\n\n```\nHypothesis: \"During the experiment, error_rate < 1% AND p99_latency < 300ms\"\n```\n\nIf the hypothesis is violated, the experiment reveals a resilience gap.\n\n```python\n@dataclass\nclass SteadyStateProbe:\n    \"\"\"Checks system metrics against steady-state bounds.\"\"\"\n    name:      str\n    check:     Callable[[Dict], bool]  # returns True = healthy\n    metric_key: str\n\n\n@dataclass\nclass ChaosExperiment:\n    name:         str\n    hypothesis:   str\n    fault:        FaultConfig\n    probes:       List[SteadyStateProbe]\n\n    def run(self, service_fn: Callable, n_calls: int = 100) -> None:\n        print(f\"\\nChaos Experiment: {self.name}\")\n        print(f\"Hypothesis: {self.hypothesis}\")\n        print(\"-\" * 55)\n\n        injector = FaultInjector(self.fault, seed=77)\n        results = {\"ok\": 0, \"errors\": 0, \"latencies_ms\": []}\n\n        for i in range(n_calls):\n            t0 = time.monotonic()\n            try:\n                injector.call(service_fn, f\"req-{i}\")\n                results[\"ok\"] += 1\n            except Exception:\n                results[\"errors\"] += 1\n            results[\"latencies_ms\"].append((time.monotonic() - t0) * 1000)\n\n        lats = sorted(results[\"latencies_ms\"])\n        p99  = lats[int(len(lats) * 0.99)]\n        err_rate = results[\"errors\"] / n_calls\n\n        metrics = {\"error_rate\": err_rate, \"p99_latency_ms\": p99}\n        print(f\"  error_rate   : {err_rate*100:.1f}%\")\n        print(f\"  p99_latency  : {p99:.1f} ms\")\n\n        all_healthy = all(p.check(metrics) for p in self.probes)\n        verdict = \"HYPOTHESIS CONFIRMED\" if all_healthy else \"HYPOTHESIS DISPROVED \u2014 weakness found!\"\n        print(f\"  Result: {verdict}\")\n\n\nexperiment = ChaosExperiment(\n    name=\"Payment gateway latency injection\",\n    hypothesis=\"System maintains <1% errors and <300ms p99 under 20% latency injection\",\n    fault=FaultConfig(FaultType.LATENCY, intensity=0.2, params={\"delay_ms\": 400}),\n    probes=[\n        SteadyStateProbe(\"error_rate\", lambda m: m[\"error_rate\"] < 0.01, \"error_rate\"),\n        SteadyStateProbe(\"p99_latency\", lambda m: m[\"p99_latency_ms\"] < 300, \"p99_latency_ms\"),\n    ],\n)\n\nexperiment.run(checkout_handler, n_calls=50)\n```\n\n## 4. Blast Radius Control\n\nAlways limit blast radius:\n\n| Technique | How |\n|-----------|-----|\n| **Shadow traffic** | Replay production traffic to a chaos target (no user impact) |\n| **Canary targeting** | Apply fault only to 1-5% of instances |\n| **Time-boxed** | Run for 5 minutes, auto-stop |\n| **Abort conditions** | If error_rate > 5%, terminate experiment immediately |\n| **Non-peak hours** | Run experiments when traffic volume is lowest |\n\n```python\nclass SafeChaosRunner:\n    \"\"\"Runs chaos experiments with automatic abort on breach.\"\"\"\n\n    def __init__(self, max_error_rate: float = 0.05, max_duration_s: float = 30):\n        self.max_error_rate  = max_error_rate\n        self.max_duration_s  = max_duration_s\n\n    def run(self, experiment: ChaosExperiment, service_fn: Callable,\n            n_calls: int = 200) -> None:\n        print(f\"\\nSafe Chaos Run: {experiment.name}\")\n        injector   = FaultInjector(experiment.fault, seed=100)\n        errors = ok = 0\n        aborted = False\n        t_start = time.monotonic()\n\n        for i in range(n_calls):\n            if time.monotonic() - t_start > self.max_duration_s:\n                print(f\"  [ABORT] Time limit {self.max_duration_s}s reached after {i} calls\")\n                aborted = True\n                break\n\n            total = ok + errors\n            if total > 10 and errors / total > self.max_error_rate:\n                print(f\"  [ABORT] Error rate {errors/total*100:.1f}% exceeded {self.max_error_rate*100:.0f}% threshold\")\n                aborted = True\n                break\n\n            try:\n                injector.call(service_fn, f\"req-{i}\")\n                ok += 1\n            except Exception:\n                errors += 1\n\n        total = ok + errors\n        print(f\"  Completed: {total} calls, {errors} errors ({errors/total*100:.1f}%)\")\n        print(f\"  Aborted early: {aborted}\")\n\n\nsafe_runner = SafeChaosRunner(max_error_rate=0.25, max_duration_s=60)\n\nsafe_exp = ChaosExperiment(\n    name=\"Error injection at 40% \u2014 expect abort\",\n    hypothesis=\"System healthy under 40% error injection\",\n    fault=FaultConfig(FaultType.ERROR_INJECTION, intensity=0.40),\n    probes=[],\n)\nsafe_runner.run(safe_exp, checkout_handler, n_calls=100)\n```\n\n## Key Takeaways\n\n- Chaos engineering is **scientific** \u2014 hypothesis, experiment, measurement, conclusion\n- Always define **steady-state** before the experiment \u2014 you need a baseline to compare against\n- Control **blast radius**: canary, time-box, abort conditions\n- Inject real-world failure modes: latency, errors, dependency kills, resource pressure\n- Finding weaknesses in staging is far cheaper than discovering them during incidents\n\n## \ud83e\udde0 Quick Quiz\n\n**Q1.** Chaos engineering's core principle is:\n- A) Breaking things randomly in production\n- B) **Proactively injecting controlled failures to discover weaknesses before they cause incidents** \u2713\n- C) Disabling monitoring during tests\n- D) Only applicable to large organisations\n\n**Q2.** A \"Game Day\" in SRE is:\n- A) A team social event\n- B) **A planned exercise simulating a major failure to test response procedures** \u2713\n- C) A sprint retrospective\n- D) A scheduled maintenance window\n\n**Q3.** The blast radius in chaos experiments should be:\n- A) As large as possible for realism\n- B) **Minimised and controlled \u2014 start small, expand incrementally** \u2713\n- C) Unlimited in staging environments\n- D) Determined by random chance\n\n<details><summary>Answers</summary>1-B, 2-B, 3-B</details>\n"
}