# Tooling Reference — Host Capability · Tiers · The Bright Line

The reference for **Step 0** (`discover` harness, run once per corpus folder before Mode 1). Capture depth
is bounded by host tooling most machines lack. This file carries the **tier lists** (what to recommend),
the **ethics classification** (clean · gated · avoid), the **per-dimension gap map** (what a bare host
cannot do), and the **install caveats** that have actually bitten.

**Nothing here is ever installed automatically.** The harness detects what is present, recommends only what
is missing, and records the result in `.claude/discover-tooling.json` so it never re-prompts in the same
folder. The harness recommends; the user decides. A user who declines proceeds to Mode 1 on whatever is
present — a missing capability is re-surfaced at the Mode 2 capability-gap checkpoint (`ingestion.md` §8),
never silently degraded.

A host almost always arrives strong on languages and package managers (node, pnpm, python, uv, cargo, gh,
jq, ripgrep) and **bare on capture/analysis tooling**. That gap is the difference between a shallow capture
and a deep one.

---

## The three classes (this is an ethics classification, not a convenience one)

| Class | Meaning | Rule |
| --- | --- | --- |
| **clean** | passive / local-only; always in-bounds | recommend freely, offer the one-paste block |
| **gated** | in-bounds **only** first-party / user-owned / opt-in — the user installs the CA, routes *their own* client, owns the keys | surface the command, **never run it for the user** |
| **avoid** | crosses the harness bright line (active scanning, exploitation, third-party interception) | never recommend; listed so it is not added by reflex |

The harness is for **spec-writing / integration / competitive analysis** — passive or first-party only,
respecting TOS and robots (`ingestion.md` §7). No tool changes that; the gated ones simply need the user in
the loop, exactly as `session` Pass-2 and `wire-capture` rung 3 do.

---

## Why this file matters more since the §5 rewrite

`ingestion.md` §5 now requires **verbatim dumping** — the artifact lands on disk whole, and the digest is
written *beside* it, never *instead* of it. The obvious objection is "a 6 MB spec is unusable." **These
tools are the answer to that objection.** They are what make a verbatim corpus queryable, which is what
makes dumping-not-digesting practical:

| Instead of digesting… | Dump verbatim, then |
| --- | --- |
| a 6 MB OpenAPI spec → a hand-typed endpoint table | `curl -o raw/spec.json` → **spectral** to catalog, **oasdiff** to diff, **gron** to grep |
| a 9 MB minified bundle → prose about the routes | `curl -o raw/bundle.js` → **webcrack** / **source-map-explorer** to read, **ast-grep** to query |
| a HAR → a narrative of the calls | keep `raw/flows.har` → **gron** to flatten and grep |
| a docs site → distilled sections only | keep the page sources → **trafilatura** / **pandoc** to render |
| a package's exports → string-mined prose | `npm view --json` → **typescript + attw + publint**, **griffe** for Python |

**A digest is a lossy compression you cannot undo; a verbatim artifact plus a query tool is lossless.**
Recommend Tier 1 on that basis, not as a nice-to-have.

---

## Tier 1 — clean, install now (broad leverage)

Highest-consensus, always in-bounds. Offer as one paste:

```bash
# Homebrew (CLI analysis + data-wrangling + web→text)
brew install gron pandoc ast-grep scc universal-ctags oasdiff fd yq dasel monolith tokei sevenzip
# Rust
cargo install htmlq
# Python (via uv — pipx is typically absent; uv tool install is the path)
uv tool install trafilatura && uv tool install griffe
# Node global (JS/TS surface + bundle de-obfuscation + spec lint)
npm i -g webcrack source-map-explorer @arethetypeswrong/cli publint @electron/asar typescript @stoplight/spectral-cli
# Headless browser for JS-rendered docs/marketing (clean — the user's own headless instance)
npm i -g playwright && npx playwright install chromium
```

| Tool | Unlocks | Dimensions served |
| --- | --- | --- |
| **gron** | flatten any JSON → greppable/diffable `path = value` (specs, captures, manifests, HAR, configs) | **7** (codebase, api, session, bundle, infra, wire, cross-cutting) |
| **trafilatura** | boilerplate-stripped main content + dates from HTML → clean markdown; feed/sitemap discovery | **4** (docs, website, community, infra) |
| **pandoc** | deterministic HTML→GFM (headings, tables, code fences) at scale | 3 (docs, website, community) |
| **ast-grep** | tree-sitter structural search/count across many files + langs (call sites, routes, exports) — catches what regex misses | 2 (codebase, bundle) |
| **oasdiff** | structured breaking-vs-non-breaking OpenAPI diff — makes the published-vs-app-own diff machine-checked | 2 (api, cross-cutting) |
| **webcrack** | un-minify / un-webpack / de-obfuscate SPA & Electron JS into readable modules | 2 (bundle, distribution) |
| **scc** | instant LOC-by-language + complexity census of a fresh clone (JSON) — the codebase opening move | 1 (codebase) |
| **universal-ctags** | real multi-lang JSON symbol index (the host's BSD ctags is broken for this) | 1 (codebase) |
| **htmlq** | surgical CSS-selector extraction (pricing cards, feature tables) instead of regex | 1 (website) |
| **source-map-explorer** | reassemble a shipped `.map` → original `src/` tree (the `source-map-reassembly` method) | 1 (bundle) |
| **spectral** | validate/resolve an OpenAPI/AsyncAPI spec → structured endpoint/security/`$ref` catalog | 1 (api) |
| **griffe** | extract a PyPI package's full public API (the `.d.ts` equivalent for Python) | 1 (packages) |
| **@arethetypeswrong/cli + typescript + publint** | parse the real `.d.ts`/exports type graph + grade the publish surface (verbatim, not string-mined) | 1 (packages) |
| **@electron/asar** | extract Electron `app.asar` → JS module tree (the distribution step the playbook names but cannot otherwise run) | 1 (distribution) |
| **fd · yq · dasel · monolith · tokei · sevenzip** | gitignore-aware file discovery · YAML/XML/TOML query · RSS/XML→JSON · single-file page archive · LOC cross-check · multi-format extract | several |

---

## Tier 2 — gated, install when a run needs them (first-party / passive only)

The capture lanes. In-bounds **only** for the user's own session, the user's own first-party clients, or
passive sources. **Surface these; never run them for the user.**

```bash
brew install mitmproxy websocat grpcurl subfinder tlsx testssl jadx apktool
npm i -g chrome-remote-interface get-graphql-schema
# one-time, USER-RUN: expose a CDP port on the user's own Chrome (closes the WS-frame blind spot)
#   open -na "Google Chrome" --args --remote-debugging-port=9222
# mobile-only, the user's own device + explicit opt-in:  uv tool install frida-tools
```

| Tool | Unlocks | Ethics note |
| --- | --- | --- |
| **mitmproxy** | decode full HTTPS + WebSocket flows of any **first-party** client (CLI/desktop/mobile/SDK) — the `wire-capture` rung-3 keystone (`ingestion.md` §7.1) | user installs the CA and routes *their own* client only |
| **chrome-remote-interface** | raw CDP → `Network.webSocketFrameReceived`/`Created`, SSE, full bodies — **captures the on-load WS frames the browser MCP's HTTP-only `read_network_requests` drops** | the user's own logged-in session |
| **websocat** | curl-for-WebSockets: connect / frame / replay / hexdump a realtime endpoint outside the browser | first-party endpoint |
| **grpcurl** | list services + invoke RPCs via gRPC server reflection → readable op/message catalog | first-party endpoint |
| **get-graphql-schema** | introspect a first-party GraphQL endpoint → full SDL | first-party endpoint |
| **subfinder · tlsx · testssl.sh** | passive CT/DNS subdomain breadth + structured TLS posture (vs hand-rolled crt.sh + openssl text) | passive aggregation; respect rate limits |
| **jadx · apktool** | decompile Android `.apk`/`.dex` → Java + decode `AndroidManifest`/resources (capability map) | public binary; read the EULA first (`ingestion.md` §7.2) |
| **frida-tools** | cert-pinning bypass to capture **the user's own** mobile client's wire | the user's own device + explicit opt-in only |

> **The single biggest unlock for a realtime target** is `chrome-remote-interface` + a Chrome
> `--remote-debugging-port` + `websocat` + `mitmproxy` together: they give `session`/`wire-capture` a real
> WebSocket/SSE capture path instead of a client-side-remount workaround. The browser MCP's
> `read_network_requests` is **HTTP-only** — that is *why* a realtime tap comes back empty until the remount
> trick (`tradecraft.md` gotcha #13, `ingestion.md` §9.4). A CDP port removes the injection-timing problem
> entirely rather than working around it.
>
> (Origin: Emergent — with no CDP port open, the run fell back to in-page taps. That worked, but a
> post-boot `window.WebSocket` wrapper is structurally blind to a library socket opened at boot, and the
> prior run published a false negative because of it.)

---

## AVOID — never recommend (this is the bright line)

Out-of-bounds for a spec-writing / competitive-analysis harness. Listed so they are not added by reflex:

- **`nmap`, `masscan`, `naabu`** — active port sweeps / service-version probing of remote hosts.
- **`nuclei`** — template-based remote vulnerability scanning.
- **`sqlmap`** — injection / exploitation.
- **`amass` (active mode)** — active DNS brute-forcing. Its *passive* mode overlaps subfinder; prefer subfinder.
- **`mitmproxy` or `frida` against a third party**, **`socat` as a public relay** — intercepting traffic the
  user does not own is the line this harness will not cross.

These stay absent **by design**. The recon ethics depend on it, and a tool absent from the host is a
stronger guarantee than a rule telling the agent not to use it.

---

## Per-dimension gap map (what a bare host cannot do)

| Dimension | Biggest capability gap on a bare host | Top addition(s) |
| --- | --- | --- |
| **codebase** | no LOC census, no AST search; the host's BSD ctags is broken for this | scc · ast-grep · universal-ctags (+ fd) |
| **docs** | no HTML/MDX→md, no JS render → client-rendered docs come back empty | pandoc · trafilatura · playwright |
| **packages** | the real `.d.ts`/exports surface is string-mined, not parsed | typescript + attw + publint · griffe (Py) · yq |
| **api** | can fetch a spec but cannot lint / diff / introspect / exercise it | spectral · oasdiff · get-graphql-schema · grpcurl · websocat |
| **website** | no JS render, no clean table/text extraction → misses pricing and feature grids | trafilatura · htmlq · playwright |
| **community** | no clean changelog/blog→text, no RSS→JSON | trafilatura · pandoc · dasel |
| **session** | **structurally blind to WebSocket/SSE frames** (HTTP-only MCP, no CDP port) | chrome-remote-interface · mitmproxy · websocat |
| **deployed-client-bundle** | a minified/obfuscated blob stays opaque; no source-map reassembly | webcrack · source-map-explorer · ast-grep · gron |
| **infra-backend-fingerprint** | shallow CT coverage, TLS posture by hand, sub-processor pages scraped raw | subfinder · tlsx · trafilatura (passive only) |
| **wire-capture** | the keystone (mitmproxy) plus every non-browser replay/decode client is absent | mitmproxy · websocat · grpcurl |
| **distribution-artifacts** | only CRX/Mach-O readable; no Electron or Android decoders | @electron/asar · jadx · apktool · webcrack · sevenzip |
| **external-reputation / hiring-intel** | bot-walled review and ATS pages return 403 to plain fetch | playwright (own headless instance) · trafilatura · dasel |
| **cross-cutting (synthesis)** | no structured diff for the N-way seam reconciliation; no out-of-band WS capture | gron · oasdiff · chrome-remote-interface · yq |

---

## Known install caveats (each of these has actually bitten)

- **`webcrack` needs Node 22.** Its hard dependency `isolated-vm` is a native (node-gyp) addon that fails to
  build against Node 26, and `--omit=optional` does not help — the dep is non-optional. Run it under a Node
  22 LTS shim (`npx -y webcrack@latest …` in a Node-22 env), or lean on the already-clean
  `source-map-explorer` (the preferred `source-map-reassembly` path whenever the SPA ships `.map` files)
  plus `ast-grep`. De-obfuscating a **no-maps, minified** bundle is the only gap this leaves.
- **`chrome-remote-interface` is a library, not a CLI.** It installs as a Node module and is invoked from a
  script, not a shell binary. **Detect it with `npm ls -g chrome-remote-interface`, not `command -v`** — a
  `command -v` probe reports it missing when it is installed, and the run then wrongly records a capability
  gap.
- **`pipx` is typically absent; use `uv tool install`** for Python CLIs (uv is present). Simpler than adding pipx.
- **`playwright` is clean** — the user's own headless instance. It unblocks JS-rendered docs and marketing
  for `docs` + `website` **without contending for the gated `session` Chrome**, which matters because the
  browser is a singleton (`ingestion.md` §7 rule 9).

---

## How Step 0 uses this reference

Once per corpus folder, before Mode 1. If `.claude/discover-tooling.json` exists, tooling was already
reviewed here — **skip straight to Mode 1 and never re-prompt.** If it is absent:

1. Read this file — the tier lists, the classification, the AVOID list.
2. **Detect before recommending** — `command -v <tool>` per tool, and `npm ls -g <pkg>` for the Node
   libraries that ship no binary (see the caveat above). Detection only; nothing is installed at this step.
3. **Recommend only what is missing**, grouped **clean** (offer the Tier-1 one-paste block) vs **gated**
   (surface the command, never run it). Never suggest anything on the AVOID list. If nothing is missing,
   say so in one line.
4. Write the state file so the harness never asks again in this folder:
   ```json
   { "checked": "<ISO date>", "present": ["gron", "pandoc"], "missing": ["mitmproxy"], "recommended": true }
   ```

The state file is **host-local** (tools live on the machine, not the repo) — gitignore it in a shared
folder; delete it to force a fresh check after installing a batch. **This step never blocks a run.**

> **Maintenance loop.** When a run hits a capability wall — a format nothing on the host can read, a
> capture lane that needed a tool that was not listed, an install that failed in a new way — fold it back
> in here: the tool into the right tier with its ethics class, or the failure into the caveats. This file
> only stays useful if every new wall is recorded (the same loop `ingestion.md` §7.0 carries for redaction
> patterns).