{
  "version": 1,
  "id": "k8s-split-brain-case-study",
  "title": "Case Study: Split-Brain Recovery in a Leader-Elected Controller",
  "summary": "Leader election, fencing tokens and idempotency: recovering from two controllers acting at once.",
  "estimatedMinutes": 25,
  "prerequisites": [],
  "source": "courses-notebook SYS401_kubernetes_devops 08_case_study",
  "lesson": "A custom controller `inventory-reconciler` running 3 replicas with Kubernetes lease-based leader election experienced a split-brain on 2026-04-02: a brief 70-second control-plane partition let two pods both believe they were the leader, and each one began applying a different reconciliation plan. The result was 142 inventory rows written twice and 18 written in conflicting orders.\n\nThis lesson reconstructs the failure mode, builds a corrected leader election with **fencing tokens**, simulates the recovery, and produces the runbook for the next time it happens (it will).\n\n## What \"split-brain\" actually means here\n\n```\n            kube-apiserver (lease holder of truth)\n                 |       |       |\n              partition  \u2713       \u2713\n                 |       |       |\n                 v       v       v\n              pod-A    pod-B    pod-C\n              (was leader, can't renew lease)   (sees lease expired, becomes leader)\n\n         partition heals\n                 |\n                 v\n   pod-A still thinks it's leader  +  pod-C is the real leader\n   => two writers, no fencing => corruption\n```\n\nThe mistake is assuming \"I hold the lease\" is *sufficient* to act. It isn't: by the time the action reaches the downstream system, you may no longer be the leader. The fix is **fencing tokens**: a monotonically increasing number issued with each lease grant, included in every downstream write, and validated by the downstream system.\n\n## Part 1 \u2014 Bad election: lease without fencing\n\n```python\nimport time, random\nfrom dataclasses import dataclass, field\n\n@dataclass\nclass Lease:\n    holder: str | None = None\n    expiry: float = 0.0\n    epoch: int = 0   # monotonically increasing per acquisition\n\nclass Apiserver:\n    def __init__(self, ttl=10.0):\n        self.lease = Lease()\n        self.ttl = ttl\n        self.now = 0.0\n    def tick(self, dt): self.now += dt\n    def try_acquire(self, holder):\n        if self.now >= self.lease.expiry or self.lease.holder == holder:\n            new_epoch = self.lease.epoch + (0 if self.lease.holder == holder else 1)\n            self.lease = Lease(holder, self.now + self.ttl, new_epoch)\n            return True, self.lease.epoch\n        return False, self.lease.epoch\n    def renew(self, holder):\n        if self.lease.holder == holder and self.now < self.lease.expiry:\n            self.lease.expiry = self.now + self.ttl\n            return True\n        return False\n\nclass UnsafePod:\n    def __init__(self, name, api): self.name = name; self.api = api; self.is_leader = False\n    def step(self):\n        if not self.is_leader:\n            ok, _ = self.api.try_acquire(self.name)\n            self.is_leader = ok\n        else:\n            ok = self.api.renew(self.name)\n            if not ok:  # we *think* we're still leader but lost the lease\n                pass     # bug: we don't notice until next renew, keep writing\n\napi = Apiserver(ttl=10.0)\npods = [UnsafePod(f\"pod-{x}\", api) for x in \"ABC\"]\n\n# Phase 1: normal -- pod-A becomes leader\napi.tick(0.1); pods[0].step()\nprint(f\"t={api.now:5.1f} leader={api.lease.holder} epoch={api.lease.epoch}\")\n# Phase 2: partition -- pod-A can't see apiserver for 12s, lease expires\napi.tick(12.0)\n# Phase 3: pod-C acquires\npods[2].step()\nprint(f\"t={api.now:5.1f} leader={api.lease.holder} epoch={api.lease.epoch}  pod-A still thinks={pods[0].is_leader}\")\n```\n\n**Where the bug is.** `pod-A` keeps `is_leader=True` across the partition. Any write it issues during the next few hundred ms after the partition heals (before its next failed renew) lands in the downstream system *without* being labelled as stale. The downstream has no way to reject it.\n\n## Part 2 \u2014 Fenced writes: the downstream is the arbiter\n\n```python\nclass InventoryDB:\n    def __init__(self): self.rows: dict[str, tuple] = {}; self.highest_epoch = 0\n    def write(self, key, value, epoch):\n        # Fencing rule: never accept a write with epoch < highest seen.\n        if epoch < self.highest_epoch:\n            return (\"REJECT_STALE_EPOCH\", self.highest_epoch)\n        self.highest_epoch = max(self.highest_epoch, epoch)\n        self.rows[key] = (value, epoch)\n        return (\"OK\", epoch)\n\nclass FencedPod:\n    def __init__(self, name, api, db):\n        self.name = name; self.api = api; self.db = db\n        self.is_leader = False; self.my_epoch = 0\n    def step(self):\n        if not self.is_leader:\n            ok, ep = self.api.try_acquire(self.name)\n            self.is_leader, self.my_epoch = ok, ep\n        else:\n            if not self.api.renew(self.name):\n                # demoted; remember our epoch but stop writing\n                self.is_leader = False\n    def reconcile(self, key, value):\n        if not self.is_leader: return (\"not-leader\", None)\n        return self.db.write(key, value, self.my_epoch)\n\napi = Apiserver(ttl=10.0); db = InventoryDB()\nA = FencedPod(\"pod-A\", api, db); C = FencedPod(\"pod-C\", api, db)\n\napi.tick(0.1); A.step()\nprint(\"A acquires:\", A.reconcile(\"widget-1\", \"qty=10\"))\n\n# Partition: A loses connectivity (cannot renew); time advances 12s\napi.tick(12.0)\n# pod-C acquires (new epoch)\nC.step()\nprint(\"C acquires:\", C.reconcile(\"widget-1\", \"qty=20\"))\n\n# Partition heals \u2014 A tries to write before it notices it's been demoted\nprint(\"A writes stale:\", A.reconcile(\"widget-1\", \"qty=99\"))\n```\n\n**What just happened.** `pod-A`'s reconcile carries `epoch=1`; the DB has already seen `epoch=2` from `pod-C` and rejects the write as `REJECT_STALE_EPOCH`. The split-brain still *occurred* \u2014 two pods believed they were leader \u2014 but the downstream made it *harmless*.\n\nThis is the same pattern as ZooKeeper's `zxid`, etcd's revision number, and (canonically) the fencing token in Kleppmann's *Designing Data-Intensive Applications* \u00a78.4.\n\n## Part 3 \u2014 Idempotency keys for the writes that do land\n\n```python\nclass IdempotentDB(InventoryDB):\n    def __init__(self):\n        super().__init__()\n        self.seen_keys: set[tuple] = set()    # (idempotency_key,)\n    def write(self, key, value, epoch, idem=None):\n        if idem is not None and idem in self.seen_keys:\n            return (\"OK_DUP\", epoch)\n        result = super().write(key, value, epoch)\n        if result[0] == \"OK\" and idem is not None:\n            self.seen_keys.add(idem)\n        return result\n\ndb2 = IdempotentDB()\nprint(db2.write(\"a\", 1, epoch=1, idem=\"r-001\"))\nprint(db2.write(\"a\", 1, epoch=1, idem=\"r-001\"))   # retry\nprint(db2.write(\"a\", 2, epoch=2, idem=\"r-002\"))\n```\n\n**Why both.** Fencing solves *split-brain* (two pods think they're leader). Idempotency keys solve *at-least-once retries* (the network ate the response and the controller retried). They are independent layers \u2014 production controllers need both.\n\n## Part 4 \u2014 Liveness vs safety: the CAP-flavoured trade-off\n\n| Knob | Smaller | Larger | Trade |\n|---|---|---|---|\n| Lease TTL | faster failover (good liveness) | fewer mistaken takeovers (good safety) | partition longer than TTL \u21d2 takeover; shorter \u21d2 stuck |\n| Renew interval | tolerate jitter | fewer apiserver writes | too large \u21d2 unnecessary failovers |\n| Lease leniency | strict (lose lease at TTL) | grace period | grace \u21d2 split-brain window |\n| Downstream check | fencing token | nothing | fencing makes safety apiserver-independent |\n\n**Rule:** *safety* belongs to the downstream system (fencing). *Liveness* belongs to the apiserver (lease TTL). Don't try to make one do the other's job.\n\n## Part 5 \u2014 Reconciler invariants worth testing\n\n```python\n# These are the invariants that should hold across any partition / failover.\n\nINVARIANTS = []\n\ndef inv(label):\n    def deco(fn):\n        INVARIANTS.append((label, fn)); return fn\n    return deco\n\n@inv(\"at most one epoch active at a time per row\")\ndef i1(db, history):\n    per_row = {}\n    for key, val, epoch, result in history:\n        if result[0] not in (\"OK\", \"OK_DUP\"): continue\n        if key in per_row and per_row[key] != epoch:\n            return False, f\"{key} had epoch {per_row[key]} then {epoch}\"\n        per_row[key] = epoch\n    return True, \"\"\n\n@inv(\"rejected writes never appear in db\")\ndef i2(db, history):\n    rejected = [(k,v,e) for k,v,e,r in history if r[0].startswith(\"REJECT\")]\n    for k, v, _ in rejected:\n        if k in db.rows and db.rows[k][0] == v:\n            return False, f\"rejected value for {k} ended up in db\"\n    return True, \"\"\n\n@inv(\"highest epoch is monotonic\")\ndef i3(db, history):\n    last = 0\n    for _,_,e,r in history:\n        if r[0] in (\"OK\",\"OK_DUP\"):\n            if e < last: return False, f\"epoch went {last} -> {e}\"\n            last = max(last, e)\n    return True, \"\"\n\n# Replay a richer scenario through the fenced design\napi2 = Apiserver(ttl=10.0); db3 = InventoryDB()\nA2 = FencedPod(\"pod-A\", api2, db3); C2 = FencedPod(\"pod-C\", api2, db3)\nhistory = []\ndef w(pod, k, v):\n    r = pod.reconcile(k, v)\n    history.append((k, v, pod.my_epoch, r))\napi2.tick(0.1); A2.step()\nw(A2, \"x\", 1); w(A2, \"y\", 2)\napi2.tick(12.0); C2.step()\nw(C2, \"x\", 10)\nw(A2, \"x\", 99)  # stale\nw(A2, \"y\", 3)   # also stale\n\nfor label, fn in INVARIANTS:\n    ok, msg = fn(db3, history)\n    print(f\"  [{'PASS' if ok else 'FAIL'}] {label}{'  -- '+msg if msg else ''}\")\nprint(\"\\nfinal db:\", db3.rows)\n```\n\n## Part 6 \u2014 Runbook\n\n| Alert | First 5 min | Next 30 min | Postmortem must answer |\n|---|---|---|---|\n| `leader_election_flap > 3/min` | Page on-call; confirm apiserver health, not controller bug | Diff lease TTL vs observed partition lengths | Is TTL too aggressive for this control plane? |\n| `REJECT_STALE_EPOCH from downstream` | Confirm no data corruption; identify stale leader | Audit logs for what stale leader tried to write | Did the stale leader self-terminate fast enough? |\n| `two distinct leader_id observed in same minute` | Treat as in-progress split-brain; verify fencing is on | Check downstream rejection rate | What was the partition root cause (apiserver/etcd)? |\n| `controller_restart_loop` | Roll back recent controller image | Bisect to faulty commit | Did the new code respect fencing? |\n\n**Rules of thumb**\n- Lease TTL should be **at least 3\u00d7 the worst observed network jitter** to the apiserver. Sub-second leases are how you create split-brain, not avoid it.\n- The downstream system, not the controller, decides what is safe. If the downstream can't fence, you do not have safety \u2014 period.\n- `kubectl get lease -A` is your first command; `kubectl logs --previous` is your second.\n\n## Reflection\n\n**Takeaways**\n- Lease-based leader election is a *liveness* mechanism. Safety needs a separate fencing layer.\n- Fencing tokens are cheap to add and convert split-brain from \"corruption\" to \"rejected writes\".\n- Idempotency keys and fencing tokens solve different problems; you need both for at-least-once delivery + safety.\n- Test the invariants (\"at most one active epoch per row\"), not just the happy path.\n- Lease TTLs are a network-jitter problem, not a UX problem \u2014 pick them from p99 latency, not from intuition.\n\n\n**Going further.** Replace the in-process apiserver with a real `k8s.io/client-go` lease, wire fencing into a PostgreSQL backend using an advisory lock keyed by epoch, and add a chaos-test that artificially extends the partition window and asserts zero downstream commits from the stale leader.\n",
  "quiz": []
}