{
  "version": 1,
  "id": "incident-management",
  "title": "Incident Management and Blameless Postmortems",
  "summary": "Roles and timeline of an incident, and how a blameless postmortem turns it into durable fixes.",
  "estimatedMinutes": 20,
  "prerequisites": [],
  "source": "courses-notebook SE303_observability_sre 06_incident_management",
  "quiz": [
    {
      "prompt": "A 'blameless post-mortem' focuses on:",
      "options": [
        "Finding the responsible engineer",
        "Systemic causes and action items, not individual fault",
        "Avoiding documentation",
        "Cost recovery"
      ],
      "answer": 1
    }
  ],
  "lesson": "Topics: incident lifecycle \u00b7 blameless postmortems \u00b7 five whys \u00b7 timeline builder \u00b7 MTTR/MTBF\n\n## 1. Incident Lifecycle\n\n```\nDETECT \u2192 TRIAGE \u2192 MITIGATE \u2192 RESOLVE \u2192 POSTMORTEM\n```\n\n| Phase | Goal | Owner |\n|-------|------|-------|\n| **Detect** | Alert fires or user reports problem | Monitoring / on-call |\n| **Triage** | Confirm severity, assign IC (Incident Commander) | On-call engineer |\n| **Mitigate** | Reduce user impact (rollback, disable feature flag) | IC + responders |\n| **Resolve** | Root cause fixed, service fully restored | IC + engineering |\n| **Postmortem** | Learn and prevent recurrence | Team |\n\n```python\nimport datetime\nfrom dataclasses import dataclass, field\nfrom typing import Dict, List, Optional\nfrom enum import Enum\n\n\nclass IncidentPhase(Enum):\n    DETECTING   = \"detecting\"\n    TRIAGING    = \"triaging\"\n    MITIGATING  = \"mitigating\"\n    RESOLVED    = \"resolved\"\n    POSTMORTEM  = \"postmortem\"\n\n\n@dataclass\nclass TimelineEvent:\n    timestamp: datetime.datetime\n    actor:     str\n    message:   str\n    phase:     IncidentPhase\n\n\n@dataclass\nclass Incident:\n    id:          str\n    title:       str\n    severity:    int\n    opened_at:   datetime.datetime\n    resolved_at: Optional[datetime.datetime] = None\n    timeline:    List[TimelineEvent] = field(default_factory=list)\n\n    def add_event(self, actor: str, message: str, phase: IncidentPhase) -> None:\n        event = TimelineEvent(\n            timestamp=datetime.datetime.utcnow(),\n            actor=actor,\n            message=message,\n            phase=phase,\n        )\n        self.timeline.append(event)\n\n    def resolve(self) -> None:\n        self.resolved_at = datetime.datetime.utcnow()\n\n    @property\n    def ttd_minutes(self) -> Optional[float]:\n        \"\"\"Time to detect (first triage event).\"\"\"\n        for e in self.timeline:\n            if e.phase == IncidentPhase.TRIAGING:\n                return (e.timestamp - self.opened_at).total_seconds() / 60\n        return None\n\n    @property\n    def ttm_minutes(self) -> Optional[float]:\n        \"\"\"Time to mitigate.\"\"\"\n        for e in self.timeline:\n            if e.phase == IncidentPhase.MITIGATING:\n                return (e.timestamp - self.opened_at).total_seconds() / 60\n        return None\n\n\n# Create a sample incident\nimport time\n\ninc = Incident(\n    id=\"INC-2024-001\",\n    title=\"Checkout error rate spike \u2014 40% errors on payment endpoint\",\n    severity=1,\n    opened_at=datetime.datetime.utcnow(),\n)\n\ninc.add_event(\"monitoring\",    \"alert fired: error_rate > 1%\",              IncidentPhase.DETECTING)\ntime.sleep(0.01)\ninc.add_event(\"alice\",         \"acknowledged, starting triage\",             IncidentPhase.TRIAGING)\ntime.sleep(0.01)\ninc.add_event(\"alice\",         \"identified: v2.3.1 deploy 15 min ago\",      IncidentPhase.TRIAGING)\ntime.sleep(0.01)\ninc.add_event(\"alice\",         \"rolling back to v2.3.0\",                    IncidentPhase.MITIGATING)\ntime.sleep(0.01)\ninc.add_event(\"monitoring\",    \"error rate returned to baseline\",           IncidentPhase.RESOLVED)\ninc.resolve()\n\nprint(f\"Incident: {inc.id} \u2014 SEV-{inc.severity}\")\nprint(f\"Title   : {inc.title}\")\nprint()\nfor e in inc.timeline:\n    print(f\"  [{e.phase.value:10}] {e.actor:<12}: {e.message}\")\n```\n\n## 2. Blameless Postmortems\n\nA **postmortem** (or post-incident review) documents what happened, why, and how to prevent recurrence.\n\n**Blameless** means: assume people acted with the best intentions given the information available at the time. Focus on systemic fixes, not individual errors.\n\nAnatomy of a postmortem:\n1. **Summary** \u2014 2-3 sentence description of impact\n2. **Timeline** \u2014 chronological events\n3. **Root cause analysis** \u2014 what actually caused the failure\n4. **Contributing factors** \u2014 environmental conditions\n5. **Impact** \u2014 user/revenue/reliability numbers\n6. **Action items** \u2014 specific, owned, time-bounded\n\n```python\ndef five_whys(initial_symptom: str, whys: List[str]) -> str:\n    \"\"\"Format a five-whys root cause analysis.\"\"\"\n    lines = [f\"Symptom: {initial_symptom}\", \"\"]\n    for i, why in enumerate(whys, 1):\n        lines.append(f\"Why {i}: {why}\")\n    lines.append(\"\")\n    lines.append(f\"Root cause: {whys[-1]}\")\n    return \"\\n\".join(lines)\n\n\nanalysis = five_whys(\n    initial_symptom=\"40% of checkout requests returned 500 errors\",\n    whys=[\n        \"The payment service was returning database connection errors\",\n        \"The connection pool was exhausted (all 20 connections in use)\",\n        \"A slow DB query introduced in v2.3.1 held connections for 30s\",\n        \"The query was missing an index on orders.user_id\",\n        \"The migration adding that index was omitted from the deploy checklist\",\n    ],\n)\nprint(analysis)\n```\n\n## 3. MTTR & MTBF\n\n**MTTR** \u2014 Mean Time To Recover: average time from incident start to resolution  \n**MTBF** \u2014 Mean Time Between Failures: average time between incident resolutions\n\n```\nMTTR = total_downtime / incident_count\nMTBF = total_uptime / incident_count\nAvailability = MTBF / (MTBF + MTTR)\n```\n\nLow MTTR \u2192 fast recovery (good runbooks, good tooling)  \nHigh MTBF \u2192 infrequent failures (good testing, gradual rollouts)\n\n```python\nfrom typing import Tuple\n\n\n@dataclass\nclass IncidentRecord:\n    started_at:  datetime.datetime\n    resolved_at: datetime.datetime\n\n    @property\n    def duration_minutes(self) -> float:\n        return (self.resolved_at - self.started_at).total_seconds() / 60\n\n\ndef compute_reliability_metrics(\n    incidents: List[IncidentRecord],\n    observation_period_hours: float,\n) -> Dict[str, float]:\n    if not incidents:\n        return {\"mttr_min\": 0, \"mtbf_hours\": observation_period_hours, \"availability\": 1.0}\n\n    total_downtime_min = sum(i.duration_minutes for i in incidents)\n    mttr = total_downtime_min / len(incidents)\n\n    total_uptime_hours = observation_period_hours - (total_downtime_min / 60)\n    mtbf = total_uptime_hours / len(incidents) if len(incidents) > 0 else total_uptime_hours\n\n    mttr_hours = mttr / 60\n    availability = mtbf / (mtbf + mttr_hours) if (mtbf + mttr_hours) > 0 else 1.0\n\n    return {\n        \"incident_count\":   len(incidents),\n        \"total_downtime_min\": total_downtime_min,\n        \"mttr_min\":         mttr,\n        \"mtbf_hours\":       mtbf,\n        \"availability_pct\": availability * 100,\n    }\n\n\n# 30-day incident log\nbase = datetime.datetime(2024, 1, 1)\nrecords = [\n    IncidentRecord(base + datetime.timedelta(days=3,  hours=2),\n                   base + datetime.timedelta(days=3,  hours=2, minutes=45)),\n    IncidentRecord(base + datetime.timedelta(days=11, hours=14),\n                   base + datetime.timedelta(days=11, hours=14, minutes=12)),\n    IncidentRecord(base + datetime.timedelta(days=22, hours=9),\n                   base + datetime.timedelta(days=22, hours=9,  minutes=90)),\n]\n\nmetrics = compute_reliability_metrics(records, observation_period_hours=30*24)\nprint(\"30-day reliability metrics\")\nprint(\"-\" * 35)\nfor k, v in metrics.items():\n    print(f\"  {k:<25}: {v:.2f}\")\n```\n\n## Key Takeaways\n\n- The incident lifecycle: **Detect \u2192 Triage \u2192 Mitigate \u2192 Resolve \u2192 Postmortem**\n- Blameless postmortems improve system reliability; blame-driven ones destroy trust\n- **Five whys** reveals systemic root causes beneath surface symptoms\n- Track **MTTR** (speed of recovery) and **MTBF** (frequency of incidents) over time\n- Action items from postmortems must be **specific, owned, and time-bounded** \u2014 vague items rot\n\n## \ud83e\udde0 Quick Quiz\n\n**Q1.** A blameless post-mortem focuses on:\n- A) Identifying the responsible engineer\n- B) **Understanding systemic causes and improving processes to prevent recurrence** \u2713\n- C) Disciplinary actions\n- D) SLA penalties\n\n**Q2.** Mean Time To Detect (MTTD) measures:\n- A) How long it takes to fix an incident\n- B) **The average time from incident start to detection** \u2713\n- C) The frequency of incidents\n- D) The impact of an incident\n\n**Q3.** Runbooks in incident management provide:\n- A) Architecture diagrams\n- B) **Step-by-step procedures for responding to known failure scenarios** \u2713\n- C) Post-mortem templates\n- D) SLA definitions\n\n<details><summary>Answers</summary>1-B, 2-B, 3-B</details>\n"
}