{
  "version": 1,
  "id": "slo-alerting-case-study",
  "title": "Case Study: SLO Alerting with Error-Budget Burn Rate",
  "summary": "Design and test a multi-window burn-rate alerting system for an SLO from first principles.",
  "estimatedMinutes": 20,
  "prerequisites": [],
  "source": "courses-notebook SE303_observability_sre 08_case_study",
  "lesson": "## Implementing Service Level Objectives, Error Budgets, and Multi-Window Alerting\n\nThe Google SRE model centres on Service Level Objectives (SLOs): quantified reliability targets that define the acceptable failure rate for a service. When the error budget (the allowed failures within the SLO) burns faster than expected, alerts fire and reliability work takes priority over feature work. This case study implements a complete SLO alerting system: request-level SLI measurement, error budget calculation, and the multi-window burn rate alerting algorithm from the Google SRE Workbook.\n\n## Part 1: SLI Measurement and Error Budget Calculation\n\nWe generate synthetic request-level telemetry data, implement a rolling-window SLI aggregator, and calculate the error budget consumption rate. The error budget is not just a metric \u2014 it is the shared language between product and engineering that determines when reliability work takes precedence.\n\n```python\nimport math\nimport random\nimport statistics\nfrom dataclasses import dataclass\nfrom datetime import datetime, timedelta\n\nrandom.seed(42)\n\n# ============================================================\n# SLO Configuration\n# ============================================================\n\n@dataclass(frozen=True)\nclass SLO:\n    name: str\n    target: float        # e.g. 0.999 = 99.9%\n    window_days: int     # Rolling window (typically 30 days)\n    description: str\n\nAVAILABILITY_SLO = SLO(\n    name=\"api_availability\",\n    target=0.999,   # 99.9% \u2014 allows 43.2 minutes/month downtime\n    window_days=30,\n    description=\"Fraction of requests returning non-5xx status codes\",\n)\n\nLATENCY_SLO = SLO(\n    name=\"api_latency_p99\",\n    target=0.95,    # 95% of requests under 300ms\n    window_days=30,\n    description=\"Fraction of requests with response time < 300ms\",\n)\n\ndef error_budget_minutes(slo: SLO) -> float:\n    \"\"\"Total error budget in minutes for the SLO window.\"\"\"\n    return (1 - slo.target) * slo.window_days * 24 * 60\n\nprint(f\"SLO Configuration:\")\nprint(f\"  {AVAILABILITY_SLO.name}: {AVAILABILITY_SLO.target*100:.1f}% target\")\nprint(f\"    Error budget: {error_budget_minutes(AVAILABILITY_SLO):.1f} minutes / {AVAILABILITY_SLO.window_days} days\")\nprint(f\"  {LATENCY_SLO.name}: {LATENCY_SLO.target*100:.1f}% target\")\nprint(f\"    Error budget: {error_budget_minutes(LATENCY_SLO):.1f} minutes / {LATENCY_SLO.window_days} days\")\n\n# ============================================================\n# Synthetic request telemetry\n# ============================================================\n\n@dataclass\nclass Request:\n    timestamp: datetime\n    status_code: int\n    response_time_ms: float\n    endpoint: str\n\ndef generate_telemetry(hours: int = 72, rps: int = 100) -> list[Request]:\n    \"\"\"\n    Generate synthetic request log.\n    Includes a 4-hour incident window with elevated error rate.\n    \"\"\"\n    start = datetime(2024, 3, 1, 0, 0, 0)\n    requests = []\n    \n    INCIDENT_START = start + timedelta(hours=48)  # Incident at hour 48\n    INCIDENT_END   = INCIDENT_START + timedelta(hours=4)\n    \n    # Sample once per minute (scaled down for tractability)\n    for minute in range(hours * 60):\n        ts = start + timedelta(minutes=minute)\n        n_requests = max(1, int(rps/60 + random.gauss(0, 2)))\n        \n        in_incident = INCIDENT_START <= ts < INCIDENT_END\n        \n        for _ in range(n_requests):\n            error_prob = 0.05 if in_incident else 0.0005  # 5% vs 0.05% error rate\n            is_error = random.random() < error_prob\n            \n            latency = (\n                random.lognormvariate(5.0, 0.5) if in_incident  # Slower during incident\n                else random.lognormvariate(4.5, 0.4)\n            )\n            \n            requests.append(Request(\n                timestamp=ts,\n                status_code=500 if is_error else random.choice([200, 200, 200, 200, 201, 304]),\n                response_time_ms=latency,\n                endpoint=random.choice([\"/api/users\", \"/api/orders\", \"/api/products\", \"/health\"]),\n            ))\n    \n    return requests\n\ntelemetry = generate_telemetry(hours=72)\nn_errors = sum(1 for r in telemetry if r.status_code >= 500)\nn_slow   = sum(1 for r in telemetry if r.response_time_ms > 300)\nprint(f\"\\nGenerated {len(telemetry):,} requests over 72 hours\")\nprint(f\"  Error rate  : {n_errors/len(telemetry)*100:.3f}% ({n_errors:,} errors)\")\nprint(f\"  Slow rate   : {n_slow/len(telemetry)*100:.2f}% ({n_slow:,} > 300ms)\")\n\n# ============================================================\n# SLI measurement over rolling windows\n# ============================================================\n\ndef compute_sli_window(requests: list[Request], \n                        start: datetime, end: datetime,\n                        slo: SLO) -> dict:\n    \"\"\"Compute SLI for a time window.\"\"\"\n    window_requests = [r for r in requests if start <= r.timestamp < end]\n    \n    if not window_requests:\n        return {\"sli\": 1.0, \"total\": 0, \"bad\": 0, \"window_hours\": (end-start).seconds//3600}\n    \n    if slo.name == \"api_availability\":\n        bad = sum(1 for r in window_requests if r.status_code >= 500)\n    elif slo.name == \"api_latency_p99\":\n        bad = sum(1 for r in window_requests if r.response_time_ms > 300)\n    else:\n        bad = 0\n    \n    total = len(window_requests)\n    sli = 1.0 - (bad / total)\n    \n    return {\n        \"sli\": sli,\n        \"total\": total,\n        \"bad\": bad,\n        \"error_rate\": bad / total,\n        \"window_hours\": (end - start).total_seconds() / 3600,\n    }\n\n# Compute hourly SLI over the 72-hour window\nbase_time = datetime(2024, 3, 1, 0, 0, 0)\nhourly_slis = []\nfor hour in range(72):\n    window_start = base_time + timedelta(hours=hour)\n    window_end   = window_start + timedelta(hours=1)\n    sli_data = compute_sli_window(telemetry, window_start, window_end, AVAILABILITY_SLO)\n    hourly_slis.append({\"hour\": hour, **sli_data})\n\n# Error budget consumption over time\nTARGET = AVAILABILITY_SLO.target\ntotal_requests_72h = sum(h[\"total\"] for h in hourly_slis)\ntotal_bad_72h      = sum(h[\"bad\"] for h in hourly_slis)\noverall_sli = 1 - total_bad_72h / total_requests_72h\n\n# Scale error budget for 72-hour window (from 30-day SLO)\nbudget_requests_72h = total_requests_72h * (1 - TARGET)\nbudget_consumed = total_bad_72h / budget_requests_72h if budget_requests_72h > 0 else 0.0\n\nprint(f\"\\nSLI measurement (72-hour window):\")\nprint(f\"  Total requests: {total_requests_72h:,}\")\nprint(f\"  Total errors  : {total_bad_72h:,}\")\nprint(f\"  Overall SLI   : {overall_sli:.5f}\")\nprint(f\"  SLO target    : {TARGET:.3f}\")\nprint(f\"  SLO met       : {overall_sli >= TARGET}\")\nprint(f\"  Error budget consumed: {budget_consumed:.1%} (of 72-hour allotment)\")\n\n# Show hourly SLI around the incident\nprint(f\"\\nHourly SLI around incident (hours 46\u201352):\")\nprint(f\"{'Hour':>6s} {'SLI':>8s} {'Errors':>8s} {'Total':>8s} {'Budget hit':>12s}\")\nprint(\"-\" * 47)\nfor h in hourly_slis[46:53]:\n    budget_hit = h['sli'] < TARGET\n    print(f\"  {h['hour']:>4d} {h['sli']:>8.5f} {h['bad']:>8d} {h['total']:>8d} {'YES' if budget_hit else 'no':>12s}\")\n```\n\n### Key Takeaway \u2014 Part 1\n\nThe error budget translates the abstract SLO percentage into a concrete quantity that can be consumed and tracked. During normal operations, 72 hours should consume approximately (72 / (30*24)) = 10% of the monthly error budget. The incident at hours 48\u201352 concentrates error budget consumption dramatically \u2014 this asymmetric consumption pattern is exactly what the multi-window burn rate alert is designed to detect quickly.\n\n## Part 2: Multi-Window Burn Rate Alerting\n\nThe burn rate measures how quickly the error budget is being consumed relative to the expected rate. A burn rate of 1.0 means the budget is consumed at exactly the rate that would exhaust it at the end of the SLO window; a burn rate of 14.4x means the budget would be exhausted in ~2 hours. The Google SRE Workbook recommends a two-window alerting strategy: a short window catches fast burns, a long window prevents false positives from brief spikes.\n\n```python\n# ============================================================\n# Multi-window burn rate alerting\n# ============================================================\n\ndef compute_burn_rate(requests: list[Request], \n                       window_start: datetime, \n                       window_hours: float,\n                       slo: SLO) -> dict:\n    \"\"\"\n    Compute error budget burn rate for a time window.\n    Burn rate = (actual error rate) / (allowed error rate)\n    A burn rate > 1.0 means the budget is being exhausted faster than planned.\n    \"\"\"\n    window_end = window_start + timedelta(hours=window_hours)\n    sli_data = compute_sli_window(requests, window_start, window_end, slo)\n    \n    allowed_error_rate = 1 - slo.target\n    actual_error_rate = sli_data[\"error_rate\"]\n    \n    burn_rate = actual_error_rate / allowed_error_rate if allowed_error_rate > 0 else 0.0\n    \n    # Time-to-exhaustion at current burn rate\n    if burn_rate > 0:\n        time_to_exhaustion_days = slo.window_days / burn_rate\n    else:\n        time_to_exhaustion_days = float('inf')\n    \n    return {\n        \"window_hours\": window_hours,\n        \"burn_rate\": burn_rate,\n        \"sli\": sli_data[\"sli\"],\n        \"error_rate\": actual_error_rate,\n        \"time_to_exhaustion_days\": time_to_exhaustion_days,\n        \"total_requests\": sli_data[\"total\"],\n        \"bad_requests\": sli_data[\"bad\"],\n    }\n\n\n# Alert thresholds from Google SRE Workbook (Table 5-2)\n# Two-window strategy: short window catches fast burns, long window avoids false positives\nALERT_CONFIG = [\n    {\"name\": \"P1_CRITICAL\",  \"short_hours\": 1,   \"long_hours\": 5,   \"burn_rate_threshold\": 14.4},\n    {\"name\": \"P2_HIGH\",      \"short_hours\": 6,   \"long_hours\": 30,  \"burn_rate_threshold\": 6.0},\n    {\"name\": \"P3_MEDIUM\",    \"short_hours\": 24,  \"long_hours\": 72,  \"burn_rate_threshold\": 3.0},\n    {\"name\": \"P4_SLOW_BURN\", \"short_hours\": 72,  \"long_hours\": 360, \"burn_rate_threshold\": 1.0},\n]\n\ndef check_alerts(requests: list[Request], check_time: datetime, slo: SLO) -> list[dict]:\n    \"\"\"Evaluate all alert conditions at a given point in time.\"\"\"\n    fired_alerts = []\n    \n    for alert in ALERT_CONFIG:\n        short_start = check_time - timedelta(hours=alert[\"short_hours\"])\n        long_start  = check_time - timedelta(hours=alert[\"long_hours\"])\n        \n        short_br = compute_burn_rate(requests, short_start, alert[\"short_hours\"], slo)\n        long_br  = compute_burn_rate(requests, long_start,  alert[\"long_hours\"],  slo)\n        \n        # Alert fires only when BOTH windows show high burn rate\n        fires = (short_br[\"burn_rate\"] >= alert[\"burn_rate_threshold\"] and\n                 long_br[\"burn_rate\"]  >= alert[\"burn_rate_threshold\"])\n        \n        fired_alerts.append({\n            \"alert\": alert[\"name\"],\n            \"fires\": fires,\n            \"short_burn\": round(short_br[\"burn_rate\"], 2),\n            \"long_burn\":  round(long_br[\"burn_rate\"],  2),\n            \"threshold\":  alert[\"burn_rate_threshold\"],\n            \"time_to_exhaustion\": round(short_br[\"time_to_exhaustion_days\"] * 24, 1),\n        })\n    \n    return fired_alerts\n\n# Evaluate alerts at different points: before, during, and after incident\nbase_time = datetime(2024, 3, 1, 0, 0, 0)\nINCIDENT_START = base_time + timedelta(hours=48)\n\ncheck_points = [\n    (\"Pre-incident (hour 24)\",    base_time + timedelta(hours=24)),\n    (\"Incident +1h (hour 49)\",    base_time + timedelta(hours=49)),\n    (\"Incident +2h (hour 50)\",    base_time + timedelta(hours=50)),\n    (\"Incident +4h (hour 52)\",    base_time + timedelta(hours=52)),\n    (\"Post-incident (hour 56)\",   base_time + timedelta(hours=56)),\n]\n\nfor label, check_time in check_points:\n    alerts = check_alerts(telemetry, check_time, AVAILABILITY_SLO)\n    fired = [a for a in alerts if a[\"fires\"]]\n    print(f\"\\n{label}:\")\n    print(f\"  {'Alert':15s} {'Short burn':>12s} {'Long burn':>10s} {'Threshold':>10s} {'TTX (hrs)':>10s} {'Status':>8s}\")\n    print(f\"  {'-'*65}\")\n    for a in alerts:\n        status = \"\ud83d\udea8 FIRES\" if a[\"fires\"] else \"ok\"\n        ttx = f\"{a['time_to_exhaustion']:.1f}h\" if a['time_to_exhaustion'] < 1000 else \"\u221e\"\n        print(f\"  {a['alert']:15s} {a['short_burn']:>12.2f}x {a['long_burn']:>10.2f}x \"\n              f\"{a['threshold']:>10.1f}x {ttx:>10s} {status:>8s}\")\n\n# Error budget status report\nprint(f\"\\n\\nERROR BUDGET STATUS REPORT\")\nprint(\"=\" * 50)\nwindow_30d_start = base_time\nwindow_30d_end   = base_time + timedelta(hours=72)  # Only 72h of data\ntotal_req = len(telemetry)\ntotal_err = sum(1 for r in telemetry if r.status_code >= 500)\nsli_72h = 1 - total_err / total_req\nbudget_fraction_consumed = total_err / (total_req * (1 - AVAILABILITY_SLO.target)) if total_req > 0 else 0\n\nprint(f\"  SLO target            : {AVAILABILITY_SLO.target*100:.1f}%\")\nprint(f\"  Measured SLI (72h)    : {sli_72h*100:.3f}%\")\nprint(f\"  SLO status            : {'MET' if sli_72h >= AVAILABILITY_SLO.target else 'BREACHED'}\")\nprint(f\"  Error budget consumed : {budget_fraction_consumed:.1%} (72h window)\")\nprint(f\"  Remaining budget      : {max(0, 1-budget_fraction_consumed):.1%}\")\nprint()\nprint(\"Recommendation: \" + (\n    \"Budget healthy \u2014 feature work can proceed\" if budget_fraction_consumed < 0.5\n    else \"Budget depleted \u2014 reliability work should take priority\"\n))\n```\n\n### Key Takeaway \u2014 Part 2\n\nThe two-window alerting strategy eliminates two failure modes: a single-window alert would fire on brief spikes (false positives) or miss slow burns that don't exceed the threshold in any single window. The `short_burn AND long_burn` condition means: the service is currently burning fast (short window) AND has been burning fast for long enough that it's not just a transient spike (long window). The time-to-exhaustion metric converts an abstract burn rate into an actionable timeline: when TTX is 2 hours, the on-call engineer needs to act immediately; when TTX is 60 hours, a next-business-day response may be sufficient.\n\n## Reflection Questions\n\n1. The alerting thresholds (14.4x, 6x, 3x) in the Google SRE Workbook are derived from specific error budget consumption targets (5% in 1 hour, 10% in 6 hours, etc.). Derive the 14.4x threshold mathematically: if an incident burns 5% of a 30-day budget in 1 hour, what burn rate does that imply?\n2. The two-window alerting strategy has a specific recall/precision trade-off. A single short window has high recall (catches all fast burns) but low precision (many false positives). How does adding the long-window condition change this trade-off?\n3. This implementation computes burn rates on each check using the full request log. For a production system receiving millions of requests, how would you redesign the SLI computation to be efficient \u2014 specifically, what pre-aggregated time-series structure would enable O(1) window queries?\n4. The SLO in this case study is binary: 99.9% of requests are good. Some SLOs are multi-dimensional (e.g., 99.9% availability AND 95% of requests under 300ms). How would you define the error budget when multiple SLO dimensions must be satisfied simultaneously?\n\n## Going Deeper\n\n- **Google SRE Workbook Ch. 5**: \"Alerting on SLOs\" \u2014 the chapter this case study implements, including the mathematical derivation of all threshold values and the evaluation of single-window vs multi-window strategies.\n- **OpenTelemetry metrics documentation**: Study `Counter` and `Histogram` instrument types \u2014 the OpenTelemetry equivalents of the request-level telemetry collected here, with automatic aggregation by time window.\n- **Prometheus recording rules**: Implement the burn rate calculation as a Prometheus recording rule \u2014 the production approach that pre-computes burn rates as time-series for efficient alerting evaluation.\n- **Practical exercise**: Implement a \"budget burn analysis\" function that takes the request log and returns a breakdown of error budget consumption by endpoint, HTTP method, and hour of day. Identify which endpoint/time combination is responsible for the majority of budget consumption during the incident.\n",
  "quiz": []
}