How to read an evidence bundle
Someone attached a run to a pull request and claimed it proves something. You have five minutes. Here is where to look, in what order, and what would make you distrust it.
Everything below is an actual run of the bundled wallet.smoke recipe on a real
Extension checkout. Paths, addresses, account identifiers, and internal package names are
redacted; structure, verdicts, and timings are untouched — including a wrinkle in the screenshot
step that is the most instructive part of the bundle.
Read it in this order
1 · report.md
The story in one page. Skim this first. If it does not match the PR description, stop here and ask.
2 · summary.json
The verdict and the counts. Is it green, how many nodes, and what content was in scope.
3 · trace.json
What actually happened per step. Every number you cite comes from here, never from prose.
The report
Written for humans. Six steps, each with its action, duration, and a one-line result.
Five steps take 220ms between them; the screenshot takes six seconds. The trace explains it, and the explanation is a reason to trust this bundle rather than doubt it.
summary.json — the verdict
The verdict for the whole run. pass only if every node passed; there is no partial credit here.
Node counts. Check these against the claim in the PR. "It proves six things" should mean total: 6, and if it does not, ask why.
The exact commit of the tooling that produced this. Two bundles that disagree are often two different refs — this is the first thing to compare.
Where the recipe came from and how much it is trusted. kind: bundled means it shipped with the harness. A recipe from an untrusted source is labelled as such rather than silently executed as equal.
Recipes that were overridden by a higher-priority library. If this is non-empty, the recipe that ran may not be the one you were reading. Check it.
App warnings and errors, kept out of the verdict (nonBlocking: true). Noise from an unrelated subsystem cannot fail someone's proof, and a clean verdict cannot hide it either. When this is not clean, open diagnostics.json.
trace.json — what actually happened
One entry per node, in execution order, each with the intent it declared, its timing, its verdict, and its full output. This is the authoritative record.
Written by the author, in the recipe, before the run. Read them top to bottom and you have the argument the bundle makes. If they do not add up to the claim, the bundle does not support it — however green.
The per-node verdict. To triage a failed run, find the first "ok": false; everything after it is consequence, not cause.
This step did not do anything; it confirmed a state that already held. A legitimate pass — but if a PR claims "proves unlocking works", this node did not prove it.
The file that executed — a real adapter on disk, not a stub. A missing adapter raises rather than returning a plausible object, which is why a green node means a real one ran.
The edge taken out of this node. Following next from the entry point reconstructs the path actually taken, which matters when a recipe branches.
The most instructive node in this bundle
Here is the six-second screenshot. It passed. Read what it recorded about how it passed:
The node passed, and still wrote down both failures. The preferred capture path failed, the fallback timed out after five seconds, and a DOM raster — weaker evidence — produced the PNG. All of it is in the metadata. Nothing forced it to tell you.
This is what to check when deciding whether to trust a bundle at all: a system that quietly upgraded "degraded capture" to "screenshot captured" would produce cleaner-looking and worse evidence. From this node specifically:
- The PNG is real but weaker than usual. A computed-style DOM raster is not a true screen capture. If the claim depends on pixel-accurate rendering, this image does not carry it.
- The six seconds were a timeout, not slow work. Timing anomalies in a bundle usually have an explanation in the metadata. Look before assuming.
- Always open the image. The trace tells you a PNG was registered and how; only your eyes tell you it shows what the author says it shows.
diagnostics.json — the quarantine
Application warnings and errors captured during the run, kept structurally separate from the verdict so that neither can contaminate the other.
The offsets are the honest part: the log is bracketed to the run window — 754 bytes — so "clean" is
a claim about this run, not that the log file is empty. truncated: false
confirms nothing was dropped from that window.
A reviewer's checklist
| Check | Where | What should worry you |
|---|---|---|
| Does the run match the claim? | node intents in trace.json |
Intents that do not add up to what the PR says was proven. |
| Are the counts real? | summary.json, not the description |
A number in prose that no field supports. |
| Did the recipe you read actually run? | recipe.json, shadowed |
A non-empty shadowed list. |
| Where did the recipe come from? | recipeLibraries.provenance |
A trust level the change does not warrant. |
| Did any step merely observe an existing state? | node outputs, e.g. alreadyUnlocked |
A confirmation being sold as a demonstration. |
| How was the visual evidence obtained? | artifact metadata.provider |
fallbackFrom present and the claim depends on rendering. |
| Was the app healthy while this ran? | diagnostics.json |
Errors in the window that the author never mentioned. |
| Does the screenshot show what they say? | screenshots/ |
Not opening it. Open it. |
Numbers come from trace.json — not the summary paragraph, not the PR description, not
what an agent wrote about its own work. A claim with no field behind it is not evidence yet, and
"which node shows that?" is a reasonable review comment.