---
description: Runbooks—alert, service, and procedure runbooks. Executable steps, current and tested; link from alerts; update after incidents.
alwaysApply: false
---

# Runbooks

Guidelines for operational runbooks.

## Core Principles

1. **Executable** - Step-by-step; anyone on-call can follow.
2. **Current** - Update after every incident that uses or improves them.
3. **Tested** - Periodically verify (e.g. game day or drill).
4. **Automated where possible** - Script steps; document the rest.

## Runbook Types

- **Alert runbook**: Tied to a specific alert. Overview, severity, verification steps, mitigation, escalation. Link via runbook_url in alert annotations.
- **Service runbook**: Per-service. Architecture, dependencies, common ops, troubleshooting, recovery.
- **Procedure runbook**: One-off or rare tasks—e.g. DB failover, cert rotation, scaling. Prerequisites, steps, verification, rollback.

## Structure (Alert Runbook)

- **Overview**: Alert name, severity, service, team.
- **Symptoms**: What users/dashboards show; how to verify.
- **Diagnosis**: 2–5 concrete steps (commands, links to dashboards).
- **Resolution**: Quick mitigation (scale, rollback, toggle) then root-cause fix. Numbered steps; include verification.
- **Escalation**: When and to whom (time-based or criteria).
- **History**: Changelog (date, author, change).

## Good Practices

- One runbook per alert or procedure; avoid mega-runbooks.
- Use code blocks for commands; specify env (e.g. namespace, region).
- Prerequisites: required access, permissions, tools.
- After incident: update runbook if steps were wrong or missing; add to history.

## Definition of Done (New Alert)

- [ ] Runbook exists and is linked from alert.
- [ ] Steps tested or reviewed; escalation path clear.
- [ ] Runbook in version control and discoverable.

## Common Pitfalls

- **Outdated** - Runbook says "run script X" but script was removed; update or remove.
- **Vague** - "Check the logs" instead of "kubectl logs -l app=api -n prod --tail=200".
- **No escalation** - Define when to escalate and to whom.