# Step 1b  -  URL Enrichment (Phase 0)

> Loaded on demand from `phases/phase-0-init.md` Step 1b. It lives here rather
> than inline because it applies only to a task that carries URLs: a bare Jira
> ID, a GitHub issue number, or a free-text task never needs any of it, yet every
> run used to pay for all 9.6 kB of it. Deep-fetch sub-steps 1b.1-1b.3 are
> themselves conditional on `state.contextLinks[]` carrying that link type.


After Step 1 fetches the Jira description (or free-text input), every URL in the text gets classified and catalogued. Targeted deep fetches (Crashlytics, Fortify) run when the matching link type is present; the rest (Swagger, Confluence, Figma, generic doc) are catalogued so Phase 1 (Analysis) can consume them via the typed fetchers documented in `$HOME/.claude/multi-agent-refs/channels/*.md`.

##### Step 1b.0  -  Extract context links (always runs)

Pipe the description text through `~/.claude/lib/context-link-extractor.sh` and store the result on the state:

```bash
printf '%s' "$DESCRIPTION_TEXT" \
  | ~/.claude/lib/context-link-extractor.sh \
  > /tmp/context-links-${TASK_ID}.json

# Persist into agent-state.json under state.contextLinks
jq --slurpfile links /tmp/context-links-${TASK_ID}.json \
   '.contextLinks = $links[0]' \
   "$STATE_FILE" > "$STATE_FILE.tmp" && mv "$STATE_FILE.tmp" "$STATE_FILE"
```

Output schema (one array entry per URL  -  full contract in `~/.claude/lib/context-link-extractor.sh`):

```jsonc
"contextLinks": [
  { "type": "swagger",     "url": "<url>", "metadata": { "format": "json|yaml|unknown" } },
  { "type": "confluence",  "url": "<url>", "metadata": { "host": "<host>", "pageId": "<id|null>", "spaceKey": "<key|null>" } },
  { "type": "crashlytics", "url": "<url>", "metadata": { "projectId": "...", "platform": "ios|android", "bundle": "...", "issueId": "...", "sessionId": "<id|null>" } },
  { "type": "fortify",     "url": "<url|null>", "metadata": { "host": "<host|null>", "versionId": "<id|null>", "instanceId": "<id|null>", "source": "url|label" } },
  { "type": "document",    "url": "<url|null>", "metadata": { "path": "...", "format": "docx|pdf|md|txt", "source": "path|url" } },
  { "type": "graylog",     "url": null,    "metadata": { "idType": "trx|conversation", "id": "...", "label": "..." } },
  { "type": "figma",       "url": "<url>", "metadata": { "kind": "design|make|board|slides", "fileKey": "...", "nodeId": "<id|null>" } },
  { "type": "generic-doc", "url": "<url>", "metadata": { "host": "<host>", "kind": "notion|google-docs|gist|github-doc" } }
]
```

The catalogue is the single source of truth for downstream phases  -  they read `state.contextLinks[]` and dispatch per `type`. The extractor is deterministic: same input → same output, no network calls, no auth.

##### Step 1b.1  -  Firebase Crashlytics deep fetch (runs when `type == "crashlytics"` present in `state.contextLinks[]`)

URL pattern (caught by the extractor): `console.firebase.google.com/(u/[0-9]+/)?project/<projectId>/crashlytics/app/(ios|android)(:|%3A)<bundleOrPackage>/issues/<issueId>(/sessions/<sessionId>)?`

- Host is fixed (`console.firebase.google.com`); no `hosts.firebase` pref needed.
- Token resolution, in order: env `FIREBASE_TOKEN_KEY`; the `prefs.global.firebase.accounts[]` entry whose `projectId` equals the URL's project; `prefs.global.keychainMapping.firebase` (single-project setups). Then `~/.claude/lib/credential-store.sh get "<key>" | base64 -d` → Service Account JSON. `project_id` read from the decoded JSON  -  verify it matches the `<projectId>` from the URL; mismatch → the fetcher exits `5` naming the key it used and the `accounts[]` entry to add, and enrichment is skipped. Several Firebase projects per team is the normal case (legacy plus redesign, staging plus prod), which is what `accounts[]` exists for.
- Exchange the SA JSON for a short-lived GCP access token (scope `https://www.googleapis.com/auth/firebase https://www.googleapis.com/auth/cloud-platform`) using `google-auth` (Python) or an inline JWT exchange (OpenSSL sign → `https://oauth2.googleapis.com/token`). Cache in-memory for the pipeline run only  -  never persist.
- Resolve the opaque `appId` first: `GET https://firebase.googleapis.com/v1beta1/projects/<projectId>/(iosApps|androidApps)?pageSize=200` and match `bundleId` (iOS) / `packageName` (Android) against the bundle from the URL. The console URL only carries the bundle, and the Crashlytics API only accepts the appId (`1:1234567890:ios:abcdef`). No match in a multi-app project → exit `3`, reason `app-not-found`; never guess at another app.
- Fetch crash detail with two calls  -  v1alpha has no get-issue-by-id endpoint: `GET .../v1alpha/projects/<projectId>/apps/<appId>/reports/topIssues?pageSize=200` for the summary and metrics (filter to the issue id), and `GET .../apps/<appId>/events?filter.issue.id=<issueId>&pageSize=1` for the newest event, which carries the stack frames, device, OS and breadcrumbs.
- Store as `state.crashContext = { projectId, appId, issueId, title, subtitle, type, state, occurrences, impactedUsers, sessions, firstSeenVersion, lastSeenVersion, topStackFrame, stackTrace[], topDevices[], topOS[], affectedAppVersions[], sessionTimeline?, fetchedAt }`.
- Phase 1 Analysis prepends `state.crashContext` to the agent prompt under a **Known Crash Context** section (do not rediscover, use as ground truth).
- Soft-fail: any network/auth error → log warning, continue pipeline without enrichment (ticket still gets manually analysed).

##### Step 1b.2  -  Fortify SSC deep fetch (runs when `type == "fortify"` present in `state.contextLinks[]`)

Two entry shapes reach this step, and `metadata.source` says which:

- `source: "url"`  -  `https://<prefs.global.hosts.fortify>/ssc/html/(ssc/)?version/<versionId>(/.*)?(#/version/<versionId>)?(/audit|/issues|/artifacts)?(\?issue=<issueId>)?`, plus the short-link variant `/ssc/html/ssc/version/<versionId>/fix/<issueInstanceId>`. The version id is read from the path or the fragment, whichever the deployment uses.
- `source: "label"`  -  a bare `Fortify Instance ID: <id>` (also `issue instance id`, `fortify id`; the id must be at least 16 characters and carry a digit, so a word following the label is never mistaken for a finding) with no URL anywhere in the text, so the entry carries `"url": null`, exactly like a graylog entry. This is the common shape when a scanner-to-tracker bridge opens the ticket: it knows the instance id and the file:line, and never writes the SSC link. Call `~/.claude/lib/fetch-fortify.sh --instance-id <id>`; it resolves the project version by asking each id in `prefs.global.fortify.versionIds` in order and taking the first hit. No `versionIds` configured → the fetcher exits `6` with reason `version-not-configured`; log the setup hint and skip. Configured but no version owns the id → exit `3`, reason `instance-not-found`, also a skip.

- Host comes from `prefs.global.hosts.fortify` (collected during Token Save Flow Step 3.5). On the URL form the URL host must match, else warn and skip (protects against pasting another company's Fortify URL by accident); the label form has no host to disagree with.
- Token resolution: `prefs.global.keychainMapping.fortify` → `~/.claude/lib/credential-store.sh get "<key>"` → API token. Auth header: `Authorization: FortifyToken <base64(token)>` (legacy tokens) or `Bearer <token>` (scoped tokens)  -  try Bearer first, fall back to FortifyToken on 401.
- Fetch finding detail (VPN required; respect `vpnServices.fortify`):
  - Version: `GET /ssc/api/v1/projectVersions/<versionId>` → name, project name, current issue count.
  - Issue list / single issue: if `issueInstanceId` in URL → `GET /ssc/api/v1/projectVersions/<versionId>/issues?q=issueInstanceId:"<id>"&fields=id,issueName,friority,fullFileName,lineNumber,analyzer,kingdom`. Otherwise fetch top open issue of the version.
  - Issue details + recommendation: `GET /ssc/api/v1/issueDetails/<issueId>` → description, recommendation, code snippet (if the artifact is retained).
- Store as `state.fortifyFinding = { host, versionId, versionName, projectName, issueId?, issueName?, severity, friority, kingdom, analyzer, file, line, description, recommendation, codeSnippet?, fetchedAt }`.
- Phase 2 Planning injects `state.fortifyFinding` into the architectural-review prompt under a **Known Security Finding** section, and the generated task breakdown includes a task addressing the finding (naming convention: `sec(fortify-<issueId>): <issueName> at <file>:<line>`).
- Soft-fail: 401 → re-run Token Save Flow for `fortify`; 403 → warn and skip; network timeout → respect VPN fallback (skip enrichment, mark `state.fortifyFinding = { skipped: "vpn-unreachable" }`).

##### Step 1b.3  -  Graylog deep fetch (runs when `type == "graylog"` present in `state.contextLinks[]`)

The extractor emits `graylog` entries from free-text trx / conversation ids (labels like `trx`, `trxId`, `transaction id`, `conversationId`, `convId`, `X-conversationId`), not from a URL  -  so these entries carry `"url": null`. When at least one is present, pull the matching diagnostic logs.

- Hosts come from `prefs.global.hosts.graylog` (production) and the optional `prefs.global.hosts.graylogTest` (`GRAYLOG_HOST_OVERRIDE` forces a single host for tests/CI); no literal host lives in any source file. Missing host for the requested environment → the fetcher exits `6`; log a setup hint and skip.
- **Environment**: the fetcher defaults to `--env auto`, which searches production and falls back to test when production returns nothing or is unreachable. A trx id minted by a tester does not exist in production, so prod-only search reports "no logs" for a complaint that is fully logged one host over. The result carries `source.environment` and `source.searchedEnvironments[]`; quote the environment whenever the logs are cited, because evidence from test says something different about a production complaint than the same lines from prod. Pin one instance with `--env prod` / `--env test` when the ticket already says which.
- Token resolution: `prefs.global.keychainMapping.graylog` → `~/.claude/lib/credential-store.sh get "<key>"` → Graylog PAT (fallback key `${USER}_Graylog_Access_Token`). The test instance uses `keychainMapping.graylog_test`, falling back to the production key when unset. Auth is HTTP Basic with the token as username and the literal `token` as password; the token is fed to `curl` only via a `-K` config file (process substitution), never on argv.
- Fetch logs (VPN required; respect `vpnServices.graylog`): call `~/.claude/lib/fetch-graylog.sh --trx <id>` (or `--conv <id>` for conversation ids) once per extracted id. The fetcher emits a normalized JSON object of matching log messages.
- Store as `state.graylogContext = { idType, id, environment, searchedEnvironments[], messages[], fetchedAt }` (or an array keyed per id when several were extracted).
- Phase 1 Analysis prepends `state.graylogContext` to the agent prompt under the **Referenced External Sources** section as **diagnostic context, advisory only** (never contradicts code; no task is generated from it, no Phase 4 gate).
- Soft-fail (non-blocking, mirrors Crashlytics): unreachable host / VPN down / network timeout → the fetcher emits a normalized empty object and exits `0`; mark `state.graylogContext = { skipped: "vpn-unreachable" }` and continue. Only a genuine auth rejection (4xx on a reachable host, fetcher exit `3`) is logged as `skipped` with the auth reason. A log fetch never blocks the run.

##### Step 1b.3b  -  Document fetch (runs when `type == "document"` present)

A feature spec arrives as a Word file at least as often as a Confluence page, and until v15.20.0 nothing in the pipeline could read one. `~/.claude/lib/fetch-document.sh <path-or-url>` extracts the text and returns the fetcher-standard JSON.

- `.docx` needs no external tool: the file is a zip and `word/document.xml` is parsed with the python3 standard library, so the same code path runs on macOS, Linux and Windows.
- `.md` / `.txt` are read directly. A URL is downloaded first, then treated by extension.
- `.pdf` has no stdlib path: `pdftotext` is used when present, and its absence is exit `6` with `degraded: true`, `converter-not-available`. That is a soft skip, not an error - on Windows it is the normal case - and the document is cited in Section 21 as "referenced but not fetched".
- Store as `state.documentContext[]`; Phase 1 injects the text under **Referenced External Sources** and Section 21 cites it as `Doküman:<file>:<title>` (Locked 3).

##### Step 1b.4  -  Other link types (catalogued only at Phase 0; fetched at Phase 1)

For `swagger`, `confluence`, `figma`, `generic-doc` entries in `state.contextLinks[]`, no Phase 0 fetch runs. The entries pass through to Phase 1, which dispatches the matching fetcher (Block B work: `fetch-swagger.sh`, `fetch-confluence.sh`, etc.). Until those fetchers land, the entries are surfaced in Phase 1 context as "referenced but not fetched  -  agent should treat the URL as an authoritative external source and read it directly when relevant."

**Jira project key addendum**: Support multi-tenant Jira  -  `prefs.projects[{project}].jiraProjectKeys[]` is already a list. When Step 1 extracts a Jira key from input, `upsert` it at the head (dedup, cap 10). Sibling keys for the same project (e.g. one key per team or tracker board) coexist; `prefs.global.defaultJiraKey` is the fallback for ambiguous free-text input only.

**Log line shape** (progress contract):

```
→ context links: total=<N>, by-type={swagger:<n>, confluence:<n>, crashlytics:<n>, fortify:<n>, graylog:<n>, figma:<n>, document:<n>, generic-doc:<n>}
→ URL deep fetch: crashlytics=<hit|miss|skipped>, fortify=<hit|miss|skipped>, graylog=<hit|miss|skipped>, document=<hit|miss|skipped|degraded>
```

Two lines emit at the end of Step 1b. The first reports the catalogue from 1b.0 (always runs). The second reports the deep-fetch outcomes from 1b.1 / 1b.2 / 1b.3 / 1b.3b  -  `miss` = type not present in catalogue (normal), `skipped` = present but auth/VPN/mismatch skipped the fetch (for graylog this includes the non-blocking VPN-unreachable degrade), `hit` = context stored in state. `degraded` belongs to the document fetch alone: the file was found and named but its text could not be extracted (a PDF with no `pdftotext` on the machine), so Section 21 cites it as referenced-but-not-fetched instead of dropping it.