# Bisecting, production and timing bugs

Companion to `../SKILL.md` for the `debugging-strategies` skill.

## 5. Bisect

When something used to work, bisection finds the change in log2(n) steps.

**Across commits:**

```bash
git bisect start
git bisect bad HEAD
git bisect good v4.11.0
git bisect run ./scripts/repro-total.sh   # exit 0 = good, 1 = bad, 125 = skip
git bisect reset
```

The script is your reproduction from step 2. Use exit code 125 for commits that
do not build, so bisect skips them.

**Across other axes:** dependency versions (pin half of the upgraded packages
back), configuration (revert half the changed keys), data (split the dataset),
time (replay half the event log). Anything ordered or divisible can be bisected.

## 8. Bugs that only show up in production or across services

- **Correlation ids.** Every inbound request gets an id that is propagated to
  downstream calls, queue messages and log lines. Without it you are joining
  logs by timestamp, which lies under load.
- **Distributed traces.** A trace shows which hop added the latency or returned
  the error. Look for the first span that deviates, not the one that finally
  threw.
- **Recent change inventory.** Deploys, config pushes, feature-flag flips,
  dependency upgrades, certificate rotations and traffic changes in the window
  before the first failure. Most incidents line up with one of them.
- **Reproduce with production data safely.** Capture the offending payload,
  scrub personal data, replay it in a lower environment.
- **Mitigate first.** In a live incident, roll back or flag off before root
  causing. The investigation continues on the captured evidence.

## 9. Heisenbugs: timing, concurrency, memory

Bugs that move when observed usually involve ordering.

- Suspect shared mutable state, missing awaits, callbacks on unexpected threads,
  caches read while being written, and assumptions about event order.
- Make timing deterministic: inject the clock, fix random seeds, run on a single
  thread or a controlled executor, add artificial delays at the suspected race
  window to widen it.
- Use the platform's race and memory detectors (thread sanitizer, address
  sanitizer, the runtime's race detector) on the looped reproduction.
- Logging changes timing. Prefer in-memory ring buffers dumped on failure over
  synchronous log writes inside the hot path.