---
name: xiaoma-heap-dump-analysis
description: Analyze a JVM heap dump (.hprof) to find the TRUE root cause of a memory leak — not just "which objects are large" but "who retains them and why they are never released". Drives a class histogram, reverse-reference GC-root tracing, optional Eclipse MAT cross-validation, and precise collection/field measurement into a verified report. Use when the user provides a JVM .hprof / heap dump / OOM dump (or says 内存泄漏 / 堆转储 / 堆 dump（.hprof）分析) and asks to analyze it, find the leak, or 找出内存泄漏的真实原因. NOT for thread dumps (jstack), GC logs, or hs_err crash logs — this is heap (.hprof) memory-leak analysis only.
argument-hint: "[path to .hprof] [optional: suspected class name]"
---

# Heap Dump Leak Analysis

## Overview

This skill finds the **true root cause** of a JVM heap memory leak from a `.hprof` dump. A histogram alone only tells you *what* objects are numerous — it points at generic containers (`char[]`, `String`, `HashMap$Node`) and rarely names the bug. The real question is **who retains the leaked objects on a GC-root path, and why they are never released.**

The method is **multi-evidence and self-validating**: every conclusion is reached by at least two independent routes (a self-written reverse-reference tracer + Eclipse MAT's dominator tree, plus exact field/collection measurement). You act as a JVM memory forensics specialist.

The bundled `scripts/` are pure-stdlib streaming HPROF parsers (no third-party deps, memory usage independent of object count) and work on multi-GB dumps with tens of millions of objects. **MAT is optional** — the scripts alone can locate the holder chain; MAT is used as cross-validation when available.

**Out of scope (non-goals).** This skill analyzes JVM **heap dumps (`.hprof`)** for memory leaks only. It does **not** handle thread dumps (`jstack` / `*.tdump`), GC logs, or `hs_err_pid*` crash logs — those are different artifacts needing different tooling. The bundled parsers assume the HPROF binary format and now hard-fail with a clear `[err]` if the input doesn't start with the `JAVA PROFILE` magic. If the user hands you one of those other dump/log types, say so and stop rather than forcing it through this pipeline.

## Conventions

- Bare paths (e.g. `scripts/trace_referrers.py`, `resources/methodology.md`) resolve from this skill's root.
- All scripts run with the system `python3` (3.8+), no dependencies. Run any script with `--help` first.
- Class names accept dot or slash form: `com.foo.Bar` == `com/foo/Bar`.
- **Large dumps — run in background and poll.** Every script does a full streaming pass
  (~1–2 min per few GB; `trace_referrers` does one pass per hop). Don't block the foreground:

  ```bash
  nohup python3 scripts/trace_referrers.py <dump> <class> --hops 6 > /tmp/trace.out 2>&1 &
  until grep -q '\[done\]' /tmp/trace.out 2>/dev/null; do sleep 5; done
  cat /tmp/trace.out
  ```

  Scripts emit progress to stderr (`[passA]`, `heap segment #N`, per-hop headers) and finish
  with a `[done]` marker — poll for it.

## Prerequisites & Data Safety — DO THIS FIRST

Before any analysis, run the safety preflight. **These steps prevent the two failures that wasted the most time in practice.**

1. **Locate and verify the dump.** Confirm it is HPROF: the first 13 bytes are `JAVA PROFILE`. Note its size.

2. **⚠️ Special characters in the filename break tools.** Filenames containing `%`, spaces, or other shell/URI-sensitive characters (common when JVMs write `-XX:HeapDumpPath=app_%p.hprof`) cause Eclipse MAT and many launchers to fail silently. **Immediately create a hardlink with a clean name** and use it everywhere downstream:
   ```bash
   ln '/path/to/app_%p.hprof' '/path/to/app_clean.hprof'   # hardlink, instant, no extra disk
   ```

3. **⚠️ Protect the data — the dump may be the only copy.** A heap dump is irreplaceable, and external tooling (uploaders, splitters) may move or delete it mid-analysis. **Create a second hardlink as a backup** so the inode survives even if the original name is removed:
   ```bash
   ln '/path/to/app_clean.hprof' ~/app_dump_backup.hprof
   ```
   (A hardlink shares the inode — the data lives as long as ANY link exists, at zero extra disk cost.)

4. **Check tooling.** `python3 --version` (required). `java -version` (JDK 11+, only needed if using MAT). MAT presence is optional — see `resources/mat-headless-runbook.md` for install + headless invocation.

5. **If the user named a suspected class**, note it for Stage 2. Otherwise Stage 1's histogram will surface candidates.

## Stages

| # | Stage | Tool | Purpose |
|---|-------|------|---------|
| 1 | Histogram | `scripts/hprof_histogram.py` | Find which classes are abnormally numerous / large — especially business & third-party classes |
| 2 | Reverse trace | `scripts/trace_referrers.py` | Walk *up* from a suspect class to its GC-root holder chain (the "path to GC roots") |
| 3 | MAT cross-check *(optional)* | `resources/mat-headless-runbook.md` | Run MAT's Leak Suspects + dominator tree headless; corroborate Stage 2 |
| 4 | Precise measure | `scripts/inspect_objects.py` | Nail the exact entry count of the suspect collection / the exact value of config fields |
| 5 | Report | — | Synthesize: symptom → multi-evidence → leak chain → mechanism → fix |

### Stage 1: Histogram — what is abnormal

```bash
python3 scripts/hprof_histogram.py <dump.hprof> --top 50
```
Read the four leaderboards. The **business/third-party leaderboards (JDK excluded)** are the most diagnostic — they point at *your* objects, not generic containers. Look for a domain object whose instance count is wildly higher than the number of live "real" things it should represent (e.g. session objects ≫ live TCP connections). That mismatch is the leak signature; that class is the Stage 2 suspect.

Sanity-anchor the count against reality: compare the suspect against `io.netty.channel.socket.nio.NioSocketChannel`, `sun.nio.ch.SocketChannelImpl`, `java.io.FileDescriptor` (live connections), or a domain "online user" object. A large gap = retained zombies.

### Stage 2: Reverse trace — who retains them

```bash
python3 scripts/trace_referrers.py <dump.hprof> <suspect-class> --hops 6
```
Each hop reports who references the current set, **by referrer class and by exact field name**, and flags two GC-root signals:
- **★ static field holder** — a `static` field of some class directly references the objects (classic static-cache / registry leak).
- **★ referrer is itself a GC root** — thread, JNI global, sticky class, etc.

Follow the chain until it converges on a single container (e.g. one `ConcurrentHashMap$Node[]` table, one singleton holder) or hits a ★ anchor. That holder + field is the leak's retention point — the collection that should have been pruned but wasn't.

**Expect the object graph to contain cycles** (e.g. `ClientHead.clientsBox → ClientsBox → map → ClientHead`); the reverse-BFS `next` set will balloon in later hops. That is normal — focus on the *first* hops where a single container/field clearly dominates, and on the ★ anchors.

### Stage 3: MAT cross-validation *(OPTIONAL — skip if MAT isn't installed)*

**Quick gate:** if `/Applications/MemoryAnalyzer.app` (or a `ParseHeapDump.sh` on PATH) is
absent, **skip this stage entirely** — the pure-Python Stages 1/2/4 already locate and prove
the holder chain. Do this stage only when MAT is available and you want the authoritative
dominator-tree + retained-size corroboration.

If Eclipse MAT is available, generate the official Leak Suspects report headless and confirm it names the same holder. **Follow `resources/mat-headless-runbook.md` exactly** — the macOS `.app` launcher fails silently (`exit 14`) from a headless shell; you must invoke the Equinox launcher jar directly with a raised `-Xmx`, and the dump filename must be free of special characters (Stage Prereqs already handled this).

MAT's "Problem Suspect 1" (a dominator occupying the bulk of the heap) and its `Node[N]` accumulation point should match the container Stage 2 converged on. Two independent methods agreeing = high confidence.

### Stage 4: Precise measurement — nail it

Turn inference into hard numbers. First list the holder's fields, then measure the collection and/or read config fields:
```bash
# List the holder class's fields (decide what to read)
python3 scripts/inspect_objects.py <dump.hprof> --class <holder-class>
# Measure the suspect collection(s) — entry count via table capacity + non-empty buckets
python3 scripts/inspect_objects.py <dump.hprof> --class <holder-class> --map-fields <field1,field2>
# Read config / state fields (e.g. is a timeout misconfigured?)
python3 scripts/inspect_objects.py <dump.hprof> --class <config-class> --fields <field1,field2>
# Measure a STATIC cache/registry map (static fields are GC-root-level holders — a top leak source)
python3 scripts/inspect_objects.py <dump.hprof> --class <util-class> --static-fields <field1,field2>
```
This distinguishes the real leak collection from innocent ones (a sibling map with thousands of entries is not the 100k+ one), and reads the actual runtime config that governs cleanup (heartbeat/timeout values, flags). **Note:** `ConcurrentHashMap.baseCount`/`HashMap.size` can read low under concurrency; trust **non-empty bucket count** + **table capacity (a power of two)** for the real magnitude.

### Stage 5: Report

Synthesize a report in the user's language (Chinese if the user wrote in Chinese). Required structure:

1. **One-line conclusion** — the true root cause in a sentence.
2. **Heap overview** — file size, object count, app stack (framework versions if visible).
3. **Evidence (multi-route)** — a table: histogram counts, reverse-trace holder chain, MAT suspect %, exact measured entry counts. Show they agree.
4. **Leak chain** — `GC root → … → holder.field (collection) → leaked objects → their retained subtree`. Quantify the retained subtree (what fills the heap).
5. **Mechanism root cause** — *why* the collection is never pruned (missing remove on disconnect, listener never deregistered, unbounded cache, misconfigured timeout, framework bug, …). Cite the measured config/code evidence. Be explicit about what is *proven* vs *inferred*.
6. **Fix recommendations** — ordered: permanent fix (code/version upgrade), config correction, guardrails (limits/monitoring/alerts), temporary mitigation (heap bump + rolling restart / scheduled cleanup).
7. **Data-safety note** — if the original file was renamed/deleted, tell the user which hardlink now holds the data.

## Graceful degradation & scaling

- **No MAT / MAT won't run** → Stages 1, 2, 4 (pure-Python) are fully sufficient to locate and prove the holder chain. MAT is corroboration, not a dependency.
- **Very large dumps / multiple suspects** → run Stage 2 on each suspect; the scripts are streaming and re-runnable. You may fan out independent traces as subagents and synthesize.
- **Unfamiliar framework** → after Stage 2 names the holder class + field, look up that library's source for the add/remove lifecycle of that collection to explain the *mechanism* (Stage 5). Quote method names.

## Resources

- `resources/methodology.md` — the full five-stage leak-hunting methodology, decision heuristics, and a worked end-to-end example.
- `resources/mat-headless-runbook.md` — installing Eclipse MAT and running it **headless** (the exit-14 pitfall, Equinox launcher invocation, `-Xmx`, reading the report zips with `textutil`).
- `resources/hprof-internals.md` — HPROF binary format reference and how the bundled scripts parse it (so you can extend them for a new question).