{
  "version": 1,
  "id": "alerting-and-oncall",
  "title": "Alerting and On-Call",
  "summary": "Alert fatigue, symptom versus cause alerts, burn-rate alerting, runbooks and incident severity.",
  "estimatedMinutes": 20,
  "prerequisites": [],
  "source": "courses-notebook SE303_observability_sre 05_alerting_and_oncall",
  "quiz": [
    {
      "prompt": "You want to alert on 'we will burn the monthly error budget within an hour at the current rate'. Best alerting strategy:",
      "options": [
        "Threshold on raw 5xx count",
        "Multi-window burn-rate alert on SLO error budget",
        "CPU alert at 90%",
        "Manual log inspection"
      ],
      "answer": 1
    },
    {
      "prompt": "An on-call engineer is being paged dozens of times a day for non-actionable issues. The primary issue is:",
      "options": [
        "Too few alerts",
        "Alert fatigue caused by unactionable alerts",
        "Insufficient metrics",
        "Too many post-mortems"
      ],
      "answer": 1
    },
    {
      "prompt": "Why is alerting on 'CPU > 90%' often a bad SLO-based alert?",
      "options": [
        "It does not directly reflect user-visible reliability and can fire even when users are unaffected",
        "Because CPUs cannot reach 90%",
        "Because Prometheus cannot measure CPU",
        "Because it violates RFC 7807"
      ],
      "answer": 0
    }
  ],
  "lesson": "Topics: alert fatigue \u00b7 symptom vs cause alerts \u00b7 runbooks \u00b7 incident severity \u00b7 alert evaluator\n\n## 1. Alert Fatigue\n\n**Alert fatigue** occurs when on-call engineers receive so many alerts that they begin ignoring them.  \nIt is one of the most dangerous failure modes in an SRE organisation.\n\nCauses:\n- Alerts that fire but require no action (informational noise)\n- Duplicate alerts for the same root cause\n- Flapping alerts that oscillate around a threshold\n- Overly tight thresholds (too sensitive)\n\nThe fix: **every alert that pages someone must be actionable and urgent**.\n\n## 2. Symptom vs Cause Alerts\n\n**Cause-based** alerts fire on individual infrastructure signals:\n- CPU > 80%\n- Disk > 90%\n- Process restarted\n\n**Symptom-based** alerts fire on user-visible impact:\n- Error rate > 1% (SLO burn rate)\n- p99 latency > 500 ms\n- Checkout conversion dropped\n\n**Best practice**: page on symptoms, use cause-based alerts for tickets/dashboards only.  \nThe question to ask: *\"Is a user harmed right now?\"*\n\n```python\nfrom dataclasses import dataclass, field\nfrom typing import Callable, Dict, List, Optional, Any\nfrom enum import Enum\nimport datetime\n\n\nclass Severity(Enum):\n    PAGE   = \"PAGE\"    # wake someone up immediately\n    TICKET = \"TICKET\"  # create a ticket, handle next business day\n    INFO   = \"INFO\"    # log only, no action required\n\n\n@dataclass\nclass AlertRule:\n    name:        str\n    description: str\n    severity:    Severity\n    is_symptom:  bool\n    condition:   Callable[[Dict[str, float]], bool]\n    runbook_url: str = \"\"\n\n\n@dataclass\nclass FiredAlert:\n    rule:      AlertRule\n    fired_at:  str\n    context:   Dict[str, float]\n\n    def __str__(self) -> str:\n        return (\n            f\"[{self.rule.severity.value}] {self.rule.name} @ {self.fired_at}\\n\"\n            f\"  {self.rule.description}\\n\"\n            f\"  context: {self.context}\\n\"\n            f\"  runbook: {self.rule.runbook_url or 'N/A'}\"\n        )\n\n\nclass AlertEvaluator:\n    \"\"\"Evaluates a set of alert rules against current metric values.\"\"\"\n\n    def __init__(self):\n        self._rules: List[AlertRule] = []\n\n    def register(self, rule: AlertRule) -> None:\n        self._rules.append(rule)\n\n    def evaluate(self, metrics: Dict[str, float]) -> List[FiredAlert]:\n        fired = []\n        now = datetime.datetime.utcnow().isoformat() + \"Z\"\n        for rule in self._rules:\n            if rule.condition(metrics):\n                fired.append(FiredAlert(rule=rule, fired_at=now, context=metrics))\n        return fired\n\n\n# Define rules\nevaluator = AlertEvaluator()\n\nevaluator.register(AlertRule(\n    name=\"high_error_rate\",\n    description=\"Error rate exceeds SLO burn threshold\",\n    severity=Severity.PAGE,\n    is_symptom=True,\n    condition=lambda m: m.get(\"error_rate\", 0) > 0.01,\n    runbook_url=\"https://wiki/runbooks/high-error-rate\",\n))\n\nevaluator.register(AlertRule(\n    name=\"high_p99_latency\",\n    description=\"p99 latency > 500 ms\",\n    severity=Severity.PAGE,\n    is_symptom=True,\n    condition=lambda m: m.get(\"p99_latency_ms\", 0) > 500,\n    runbook_url=\"https://wiki/runbooks/high-latency\",\n))\n\nevaluator.register(AlertRule(\n    name=\"high_cpu\",\n    description=\"CPU utilisation > 80%\",\n    severity=Severity.TICKET,\n    is_symptom=False,\n    condition=lambda m: m.get(\"cpu_pct\", 0) > 80,\n))\n\n# Evaluate against current state\ncurrent_metrics = {\"error_rate\": 0.025, \"p99_latency_ms\": 320, \"cpu_pct\": 85}\nalerts = evaluator.evaluate(current_metrics)\n\nprint(f\"Fired {len(alerts)} alert(s):\")\nfor a in alerts:\n    print()\n    print(a)\n```\n\n## 3. Incident Severity Classification\n\n| Severity | Impact | Response time | Example |\n|----------|--------|---------------|---------|\n| **SEV-1** | Total outage, all users affected | Immediate | Payment system down |\n| **SEV-2** | Partial outage, major feature broken | < 15 min | Search unavailable |\n| **SEV-3** | Degraded performance, workaround exists | < 1 hour | Slow image uploads |\n| **SEV-4** | Minor issue, no user impact | Next business day | Flapping health check |\n\nSeverity drives **escalation policy** and **communication cadence**.\n\n```python\ndef classify_severity(\n    error_rate: float,\n    affected_users_pct: float,\n    feature_unavailable: bool,\n) -> int:\n    \"\"\"Classify incident severity 1-4.\"\"\"\n    if error_rate > 0.5 or affected_users_pct > 50:\n        return 1\n    if feature_unavailable or error_rate > 0.1 or affected_users_pct > 10:\n        return 2\n    if error_rate > 0.01 or affected_users_pct > 1:\n        return 3\n    return 4\n\n\nscenarios = [\n    dict(error_rate=0.60, affected_users_pct=70, feature_unavailable=True),\n    dict(error_rate=0.15, affected_users_pct=20, feature_unavailable=True),\n    dict(error_rate=0.02, affected_users_pct=3,  feature_unavailable=False),\n    dict(error_rate=0.001, affected_users_pct=0, feature_unavailable=False),\n]\n\nfor s in scenarios:\n    sev = classify_severity(**s)\n    print(f\"SEV-{sev}: error={s['error_rate']*100:.0f}%  users={s['affected_users_pct']}%  \"\n          f\"unavailable={s['feature_unavailable']}\")\n```\n\n## 4. Runbook Automation\n\nA **runbook** is a documented set of steps for responding to a specific alert.  \nAutomated runbooks can execute diagnostic commands and even remediation steps.\n\nStructure of a good runbook:\n1. **What fired** \u2014 alert name and threshold\n2. **Why it matters** \u2014 user impact\n3. **Immediate checks** \u2014 what to look at first\n4. **Remediation steps** \u2014 ordered, with expected outcomes\n5. **Escalation path** \u2014 who to page if steps fail\n\n```python\n@dataclass\nclass RunbookStep:\n    description: str\n    action:      Callable[[], str]   # returns outcome message\n\n\nclass Runbook:\n    def __init__(self, name: str):\n        self.name  = name\n        self.steps: List[RunbookStep] = []\n\n    def add_step(self, description: str, action: Callable[[], str]) -> None:\n        self.steps.append(RunbookStep(description=description, action=action))\n\n    def execute(self) -> None:\n        print(f\"\\nRunbook: {self.name}\")\n        print(\"=\" * 50)\n        for i, step in enumerate(self.steps, 1):\n            print(f\"Step {i}: {step.description}\")\n            result = step.action()\n            print(f\"  => {result}\")\n\n\n# Simulate a high-error-rate runbook\nimport random as _r\n_r.seed(5)\n\nrb = Runbook(\"high_error_rate_checkout\")\nrb.add_step(\"Check recent deployments\",\n            lambda: \"Last deploy: checkout-v2.3.1, 12 minutes ago\")\nrb.add_step(\"Check DB connection pool\",\n            lambda: f\"Pool utilisation: {_r.randint(60,95)}%\")\nrb.add_step(\"Check upstream payment gateway\",\n            lambda: \"Stripe status: degraded performance (official status page)\")\nrb.add_step(\"Enable fallback payment method\",\n            lambda: \"Fallback enabled: PayPal gateway active\")\n\nrb.execute()\n```\n\n## Key Takeaways\n\n- **Alert fatigue kills** \u2014 every page must be actionable, urgent, and non-duplicated\n- **Symptom-based alerts** (user harm) should page; cause-based alerts should create tickets\n- Severity classification drives response time and communication cadence\n- Runbooks should be **executable** \u2014 automated diagnostic steps save minutes during incidents\n- Review alert signal/noise ratio monthly; delete or demote any alert that fires without action\n\n## \ud83e\udde0 Quick Quiz\n\n**Q1.** Alert fatigue occurs when:\n- A) Alerts are too infrequent\n- B) **Too many low-quality alerts desensitise oncall responders to real incidents** \u2713\n- C) Alerting systems are unreliable\n- D) Teams are too small\n\n**Q2.** A good SLO (Service Level Objective) should be:\n- A) Set at 100% availability\n- B) **Achievable, meaningful to users, and based on error budget thinking** \u2713\n- C) Changed monthly\n- D) The same as SLA\n\n**Q3.** PagerDuty escalation policies ensure:\n- A) The fastest engineer is always notified first\n- B) **If the primary oncall doesn't respond, the alert escalates to the next person** \u2713\n- C) All team members receive all alerts\n- D) Alerts are automatically resolved\n\n<details><summary>Answers</summary>1-B, 2-B, 3-B</details>\n"
}