---
name: debugging-strategies
description: "Language-neutral debugging method: reproduce, minimise, falsifiable hypotheses, git bisect, instrumentation, production traces, concurrency bugs, regression test. Use when a bug resists guesswork."
metadata:
  risk: safe
  source: multi-agent-pipeline
  date_added: "2026-09-25"
---

# Debugging Strategies

Debugging is the search for the smallest true statement that explains a
symptom. Every step below either makes the search space smaller or makes an
observation more trustworthy. If a step does neither, skip it.

Platform tools live elsewhere: `debugging-instruments` covers LLDB, the memory
graph and Instruments on Apple platforms; `agent-introspection-debugging` covers
an AI agent debugging its own failed runs. This skill is the method they plug
into.

## The loop

```
observe -> reproduce -> shrink -> hypothesise -> test one thing -> conclude
    ^                                                   |
    +-------------------- wrong? ----------------------+
```

Write each hypothesis and each result down as you go, in a scratch file or the
ticket. Most wasted debugging time is re-testing something already ruled out.

## 1. Pin down the symptom

Get the exact observable, not a paraphrase:

- The full error text and stack trace, the input that produced it, the version
  or commit, the environment (OS, runtime, config flags, locale, time zone).
- What was expected instead, stated so it can be checked.
- Whether it ever worked. If yes, when did it last work? That turns a search
  into a bisection.

"The order total is wrong" is a complaint. "An order with two discount codes
totals 18.00 instead of 16.20 on build 4.12.0, first seen after Tuesday's
release" is a symptom.

## 2. Reproduce on demand

A bug you cannot trigger is a bug you cannot confirm fixed. Push toward a
reproduction that runs in seconds:

1. Replay the exact input (request body, file, event payload) locally.
2. If it does not reproduce, list the differences between your run and the
   failing one: data, config, versions, concurrency, clock, network. Remove them
   one at a time until it does.
3. Turn the reproduction into an automated test as soon as it exists. It is the
   regression test later.

For an intermittent failure, loop it: run the test a few hundred times and count
failures. A rate is an observation you can compare before and after a change.

## 3. Shrink the failing case

A smaller case has fewer places for the bug to hide.

- **Input shrinking.** Halve the input and keep the half that still fails.
  Repeat. Two hundred order lines become the three that matter. Property-based
  testing tools do this automatically when they find a failure.
- **Code shrinking.** Stub out collaborators one by one. If the failure survives
  with the database replaced by an in-memory fake, the database is cleared.
- **Environment shrinking.** Disable caches, feature flags, background jobs.

Stop when removing anything else makes the failure disappear. What is left is
the cause or points straight at it.

## 4. Hypothesise, then try to kill each hypothesis

Write two or three candidate explanations, each specific enough to predict
something: "if the rounding happens per line instead of per order, then an order
with a single line will total correctly." Then run the experiment that could
prove it wrong.

Rules that keep this honest:

- Change one variable per experiment.
- Predict the outcome before running it. A result you did not predict means your
  model of the system is wrong, which is itself useful.
- Prefer the cheap experiment that splits the hypotheses over the expensive one
  that confirms your favourite.

## Details on demand

Load `references/hard-cases.md` for step 5 (bisecting commits, inputs and config), step 8 (production and cross-service bugs) and step 9 (timing, concurrency and memory bugs).

## 6. Instrument with intent

Add logging or breakpoints to answer a specific question, not to "see what
happens".

- Log the values that distinguish your hypotheses, at the boundary where they
  enter and leave the suspect code.
- Include an identifier (order id, request id) in every line so a single case
  can be followed.
- Conditional breakpoints and watchpoints beat stepping through a loop by hand.
- Remove temporary instrumentation before committing, or promote it to proper
  structured logging if it would have helped in production.

## 7. Differential debugging

When one run works and another fails, the cause is somewhere in the difference.
Capture both and diff them:

- Logs or traces of the same request on the good and bad build.
- Environment dumps (`env`, dependency lock files, config) from the working and
  failing machines.
- Serialised state before the failing step on both paths.

Normalise noise (timestamps, ids, addresses) first, then diff. The first
unexpected divergence is where to look.

## 10. Conclude properly

A bug is fixed when:

1. You can explain the mechanism in a sentence, including why it did not fail
   before.
2. The reproduction test fails without the fix and passes with it.
3. The test sits in the suite at the level closest to the defect (a unit test
   for a rounding rule, an integration test for a wrong query).
4. You checked for siblings: the same pattern elsewhere in the codebase.

If you cannot explain the mechanism, you have a change that makes the symptom go
away, not a fix. Say so in the pull request.

## When stuck

- Explain the problem aloud or in writing to someone, step by step. Gaps in the
  explanation are where the assumption is wrong.
- Re-read the actual error message and the actual code, not your memory of them.
- Question the "impossible": the file you edited is not the one being run, the
  cache is stale, the build did not pick up the change, the config you read is
  overridden later.
- Take a break with the experiment log written down. It survives the break;
  your working memory does not.