# HPROF Binary Format — Reference & How the Scripts Parse It

The bundled scripts are self-contained streaming parsers of the HPROF binary format. This
note documents the format and the parsing strategy so you can **extend the scripts** for a
new question (e.g. "dump every field of object X", "list the keys of map Y", "histogram of
retained subtree of class Z").

## File layout

```
[format string, NUL-terminated]   e.g. "JAVA PROFILE 1.0.1" / "1.0.2"
[identifier size : u4]            usually 8 (64-bit, no compressed-oop in the dump format)
[timestamp : u8]
[record]*                         a flat sequence of top-level records
```

Each top-level record:

```
[tag : u1][time : u4][length : u4][body : <length> bytes]
```

### Top-level tags used

| tag | name | body |
|-----|------|------|
| 0x01 | STRING (UTF-8) | `id` + UTF-8 bytes → the string table (class names, field names) |
| 0x02 | LOAD_CLASS | class serial(u4) + **class object id** + stack serial(u4) + **name string id** |
| 0x0C | HEAP_DUMP | a container of heap sub-records |
| 0x1C | HEAP_DUMP_SEGMENT | same as 0x0C, just chunked (large dumps emit several) |
| 0x2C | HEAP_DUMP_END | marker |

Other tags (UTF8 stack frames/traces, alloc sites, etc.) are skipped via `length`.

## Heap-dump sub-records

Inside a 0x0C/0x1C body is a sequence of sub-records, each led by a 1-byte sub-tag. To stay
aligned you must consume **exactly** the right number of bytes for every sub-tag — including
GC-root records you don't care about.

### GC roots (fixed sizes — must skip precisely)

| sub-tag | root kind | after the leading `id`, extra bytes |
|---------|-----------|-------------------------------------|
| 0xFF | UNKNOWN | 0 |
| 0x01 | JNI GLOBAL | + 1 `id` (JNI ref) |
| 0x02 | JNI LOCAL | + u4 + u4 (= 8) |
| 0x03 | JAVA FRAME | + u4 + u4 (= 8) |
| 0x04 | NATIVE STACK | + u4 (= 4) |
| 0x05 | STICKY CLASS | 0 |
| 0x06 | THREAD BLOCK | + u4 (= 4) |
| 0x07 | MONITOR USED | 0 |
| 0x08 | THREAD OBJECT | + u4 + u4 (= 8) |

The leading `id` of each is the rooted object — collect these into a set to detect
"referrer is itself a GC root."

### 0x20 CLASS_DUMP (the tricky one)

```
class object id        : id
stack trace serial     : u4
super class object id  : id
class loader id        : id
signers id             : id
protection domain id   : id
reserved               : id
reserved               : id
instance size          : u4
constant pool count    : u2,  then each: cp-index(u2) + type(u1) + value(size-by-type)
static fields count    : u2,  then each: name string id(id) + type(u1) + value(size-by-type)
instance fields count  : u2,  then each: name string id(id) + type(u1)
```

From CLASS_DUMP the scripts cache: `super` (to walk the chain), the **instance field list**
`(type, name-id)` in declaration order, and **static field references** (object-typed
statics — these are GC-root-level holders and how static-cache leaks are detected).

### 0x21 INSTANCE_DUMP

```
object id           : id
stack trace serial  : u4
class object id     : id
num bytes that follow : u4
field values        : <num bytes>   (raw, laid out per the class's field layout)
```

**Field layout order:** the values are this class's instance fields in declaration order,
**then the super class's, then its super's**, … each field occupying `size-by-type` bytes.
To read a specific field you compute its byte offset by walking
`class → super → super…`, summing field sizes; reference fields (type 2) hold an `id`.

### 0x22 OBJECT_ARRAY_DUMP

```
array object id : id
stack serial    : u4
num elements    : u4
array class id  : id
elements        : num × id
```

### 0x23 PRIMITIVE_ARRAY_DUMP

```
array object id : id
stack serial    : u4
num elements    : u4
element type    : u1
elements        : num × size-by-type
```

## Basic type tags & sizes

| tag | type | size |
|-----|------|------|
| 2 | object (ref) | `id_size` (usually 8) |
| 4 | boolean | 1 |
| 5 | char | 2 |
| 6 | float | 4 |
| 7 | double | 8 |
| 8 | byte | 1 |
| 9 | short | 2 |
| 10 | int | 4 |
| 11 | long | 8 |

## Parsing strategy used by the scripts

- **Streaming, low memory.** Read each top-level record header (9 bytes), then read the
  body only for records of interest (STRING, LOAD_CLASS, HEAP_DUMP); `seek()` past the rest.
  Heap-dump bodies are walked sub-record by sub-record over a `memoryview` (no copies).
- **Class metadata is cached once**, object counts/fields are never all held in memory — so
  memory use is independent of object count (works on 40M-object dumps).
- **Reference offsets** (`get_refoffs`) precompute, per class, the byte offsets of all
  reference-typed fields (self + supers) for fast referrer scanning in `trace_referrers.py`.
- **Reverse tracing = repeated full streaming passes.** Each hop re-scans the file, testing
  every object's references against the current target id-set. Slower than building an
  in-memory graph but uses trivial memory and is robust on huge dumps.

## Extending the scripts

Common one-off questions and where to hook in:

- **Print every field of a specific object id** → in a heap walk, when
  `INSTANCE_DUMP.object id == target`, decode each field with the class's
  `layout` (see `inspect_objects.py`'s `read_val` + `layout`).
- **Enumerate a map's keys/values** → read the map's `table` (a `Node[]`), then for each
  non-null `Node` follow `key`/`val`/`next`; resolve value object classes via the
  class-id → name table.
- **Retained-style grouping** → for a target class, in one pass collect each instance's
  outgoing refs, then attribute their shallow sizes (approximate; real dominator/retained
  needs MAT).

Keep the GC-root and CLASS_DUMP skip logic byte-exact — a single miscounted field
desynchronizes the entire stream. The scripts raise on an unknown sub-tag, which is the
canary for a misalignment.