# Fetch Endpoint Parts Skill

## Scope of this skill

This skill is the **per-endpoint spec**. Given one entry from `api_coverage.json`, it defines exactly how to fetch its docs page, extract the 10 parts, build the entry, and write it to the per-resource YAML.

The **loop** — filtering `api_coverage.json` for a resource, iterating over its entries, batching, and verifying the resource's slice is complete — is owned by the **`resource-endpoint-fetcher`** sub-agent (the endpoint-extraction step of `/connector:research` spawns one per resource, in parallel batches — mandatory at ≥2 resources or >8 in-scope endpoints).

If you are reading this skill, you are either (a) inside a `resource-endpoint-fetcher` agent processing one entry at a time, or (b) directly invoking the skill for a single-endpoint debug session.

## The 10 endpoint parts

| # | Part | What it contains |
|---|------|-----------------|
| 1 | `base_url` | Provider base URL |
| 2 | `path` | Exact endpoint path (e.g. `/users/{id}`) |
| 3 | `method` | GET / POST / PUT / PATCH / DELETE |
| 4 | `path_params` | URL path parameters with types and possible values |
| 5 | `query_params` | Query string parameters; nested `properties[]` for objects, `items` for arrays |
| 6 | `body_params` | Request body fields; same nested rules as query_params |
| 7 | `headers` | ALL documented headers — but **classify** them: transport/auth-managed headers get `transport: true` (see below), the rest are ordinary parameters |
| 8 | `auth_methods` | Auth types this specific endpoint accepts |
| 9 | `error_codes` | HTTP status codes + provider error envelope + retry strategy |
| 10 | `pagination` | Pagination style for the endpoint (`null` if non-paginating) |

Every nested field of every object and array is included — no field is skipped.

## Supporting files (loaded on demand)

| File | Type | When to read |
|------|------|--------------|
| `examples/output-schema.md` | example | Before writing any endpoint entry — full per-endpoint JSON schema with every field and a worked example to follow |
| `references/fetch-strategies.md` | reference | When fetching the docs page in Step 1 — fallback chain (WebFetch → crwl → Playwright + dropdown extraction) AND the soft-404 gate that must pass before extraction begins |
| `references/extraction-rules.md` | reference | When extracting fields in Step 2 — per-field detection signals, recursive object expansion, LLM-optimized descriptions, the 100% coverage self-check, plus rules for `required_scopes`, `error_codes`, `pagination`, and conditional fields |

## Executable scripts

All scripts are invoked via `${CLAUDE_PLUGIN_ROOT}` so they resolve whether the plugin is installed locally or from the marketplace.

| Script | Purpose |
|--------|---------|
| `node ${CLAUDE_PLUGIN_ROOT}/scripts/research/load_scopes.js <provider>` | Load + dedupe scopes from all `auth/*.json` files. Called **once** by the executor before the loop, not per-endpoint. Output is the `scope_list` consumed by Step 2 here. |
| `node ${CLAUDE_PLUGIN_ROOT}/scripts/research/merge_resource_entry.js <provider> [entry.json]` | Step 3 — upsert this endpoint's entry into the per-resource YAML (stdin or file) |
| `node ${CLAUDE_PLUGIN_ROOT}/scripts/research/mark_investigated.js <provider> <action_id>` | Step 4 — mark this endpoint's `action_id` as `investigated: true` in `api_coverage.json` |
| `node ${CLAUDE_PLUGIN_ROOT}/scripts/research/verify_completion.js <provider>` | **Global** completion check across all resources. Called by the spawning command after all fetchers return, not by this skill. |

---

## Per-endpoint workflow

For a single endpoint entry `api` from `api_coverage.json`:

### Step 1: Fetch the docs page

If `api.resource_url` is `null`, **skip Step 1 + 2** and go straight to Step 3 with the stub shape.

Otherwise, fetch `api.resource_url` using the fallback chain in `references/fetch-strategies.md` (steps 1.1–1.3):
1. `WebFetch` — fast, always try first (1.1)
2. `crwl` bash — for JS-rendered pages (1.2)
3. Playwright — for interactive portals (1.3)

(The Playwright dropdown extraction for field-selector params — `opt_fields`, `fields`, `select`, `include`, `expand` — is fetch-strategies Step 2.4 and runs **during Step 2 extraction**, not here.)

### Step 1.5: Soft-404 verification (MANDATORY before extraction)

Providers frequently return HTTP 200 for stale or missing doc pages while the rendered body says "Page not found", "This page doesn't exist", or similar. Extracting from those pages produces empty or misleading entries whose real signal is "the `resource_url` is wrong."

After Step 1, verify **all** of the following before entering Step 2:
- HTTP status was 200 (or 2xx)
- The fetched body length is > 500 chars of actual doc content (nav-only bodies are also failures)
- The body does NOT contain any of these soft-404 markers as visible headings or main content:
  `"page not found"`, `"404"`, `"doesn't exist"`, `"no such page"`, `"we couldn't find"`, `"page you're looking for"`

If the page fails soft-404 verification:
1. Log the URL and detected marker(s) in the resource-fetcher summary under `soft_404_urls[]`.
2. Attempt one corrected URL guess (e.g. strip trailing slash, swap `docs.provider.com/api/` ↔ `developer.provider.com/reference/`, or use the OpenAPI spec `resource_url` if `api_coverage.json` recorded one).
3. If the second attempt also fails, write the entry as a **stub** (Step 3, stub shape) with `skip_reason: "soft_404"` instead of `"no_resource_url"`, and continue. Do **not** invent params for an unreachable page.

Full detection rules and marker list live in `references/fetch-strategies.md → Soft-404 detection`.

### Step 2: Extract the 10 parts

**STRICT SCHEMA RULE:** the output entry must follow `examples/output-schema.md` exactly — field names, structure, nesting. No additions, no omissions.

Each entry is assembled from two sources:

**Carried over from `api_coverage.json` (do NOT re-fetch):** `action_id`, `label`, `method`, `path`, `resource_url`, `category`, `tags`.

**`base_url` resolution.** If `coverage.base_urls.length > 1`, look up `api.base_url_service` in `coverage.base_urls[]` and use that registered URL — do NOT re-extract from the docs page. For single-service providers, use `coverage.base_url` directly. Full rules in `references/extraction-rules.md → base_url resolution`.

**Fetched from the docs page:** `description` (provider's action description, verbatim), `path_params`, `query_params`, `body_params`, `headers`, `auth_methods`, `required_scopes`, `error_codes`, `pagination`.

**Written by us (LLM-optimized layer):** `_action_summary` on the entry, and `usage_notes` on every param — purpose, when to use, constraints, gotchas. The provider's own text always stays untouched in `description`; the two-field split is defined in `references/extraction-rules.md → Description quality`.

Per-field detection signals and value constraints live in `references/extraction-rules.md`. It covers:
- `required_scopes` — scope matching against `scope_list` (loaded once by the executor)
- `error_codes` — HTTP status + error envelope + retry strategy
- `pagination` — style (cursor / offset / page_number / link_header) + cursor params
- Recursive object/array expansion — every `type: object` MUST enumerate `properties[]`, no matter how deep the nesting goes
- LLM-optimized descriptions — how to write action and field descriptions so downstream skills can reason about them
- Conditional fields (`example`, `pattern`, `min`, `max`) — always present as keys

### Step 2.5: 100% coverage self-check (MANDATORY)

After Step 2, before assembling the entry, run the parameter-parity audit described in `references/extraction-rules.md → Coverage self-check (100% field parity)`:

1. Re-scan the fetched page text for every parameter-name candidate — table headers, definition lists, code samples (JSON keys), OpenAPI `parameters` blocks, and inline `<code>` mentions.
2. Diff those candidates against the extracted `path_params + query_params + body_params + headers` (flattened, including recursed children).
3. For each candidate not in the extracted set: either (a) re-extract it if it was missed, or (b) record it in the entry's internal `_coverage_notes` array with a one-line reason (`"appears only in unrelated example"`, `"deprecated — provider says do not use"`, etc.).
4. Do **not** advance to Step 3 while any candidate is unresolved.

The goal is 100% field coverage. Silently dropping a documented param because "it looked optional" is the failure mode this step exists to catch.

### Step 3: Build the entry and merge

Two shapes — use the correct one based on whether `resource_url` is set.

**Normal entry** (resource_url is not null — fetched from docs):
```python
# base_url resolution: prefer registered base_urls[] over per-page extraction
if len(coverage["base_urls"]) > 1 and api.get("base_url_service"):
    base_url = next(b["url"] for b in coverage["base_urls"] if b["service"] == api["base_url_service"])
else:
    base_url = coverage["base_url"] or fetched["base_url"]

entry = {
    "action_id":       api["action_id"],
    "label":           api["label"],
    "method":          api["method"],
    "path":            api["path"],
    "resource_url":    api["resource_url"],
    "category":        api["category"],
    "tags":            api["tags"],                  # carried over — never empty (falls back to [category] in Pass 1)
    "base_url":        base_url,
    "description":     fetched["description"],       # provider's documented description, VERBATIM (None if not documented)
    "_action_summary": built["action_summary"],      # LLM-optimized layer: purpose / use when / do NOT use for / side effects
    "investigated_at": datetime.now(timezone.utc).isoformat(),
    "skipped":         False,
    "auth_methods":    fetched["auth_methods"],
    "required_scopes": fetched["required_scopes"],
    "headers":         fetched["headers"],
    "path_params":     fetched["path_params"],       # each param: description = provider verbatim, usage_notes = LLM layer
    "query_params":    fetched["query_params"],
    "body_params":     fetched["body_params"],
    "error_codes":     fetched["error_codes"],
    "pagination":      fetched["pagination"],
}
```

**Stub entry** (resource_url is null — cannot fetch docs):
```python
entry = {
    "action_id":       api["action_id"],
    "label":           api["label"],
    "method":          api["method"],
    "path":            api["path"],
    "resource_url":    None,
    "category":        api["category"],
    "tags":            api["tags"],
    "base_url":        None,
    "description":     None,
    "investigated_at": datetime.now(timezone.utc).isoformat(),
    "skipped":         True,
    "skip_reason":     "no_resource_url",  # or "soft_404" when Step 1.5 failed
    "auth_methods":    [],
    "required_scopes": [],
    "headers":         [],
    "path_params":     [],
    "query_params":    [],
    "body_params":     [],
    "error_codes":     [],
    "pagination":      None,
}
```

Pipe the entry to the merge script — it handles read → upsert → write back atomically, picks the right per-resource YAML file from `entry.category`, and stamps `last_updated`:

```bash
echo "$ENTRY_JSON" | node ${CLAUDE_PLUGIN_ROOT}/scripts/research/merge_resource_entry.js <provider>
```

The script writes `.connector-dev/<provider>/research/endpoints/resources/<resource>.yaml`, creating the file and directory if missing. If an entry with the same `action_id` already exists, it is replaced in place; otherwise appended.

### Step 4: Mark investigated

```bash
node ${CLAUDE_PLUGIN_ROOT}/scripts/research/mark_investigated.js <provider> <action_id>
```

Updates `all_apis[]` and the 4 tier arrays (`must_have`, `should_have`, `nice_to_have`, `would_not_have`) in `api_coverage.json` — sets `investigated: true` wherever the matching `action_id` appears.

---

## Anti-Patterns (per-endpoint)

- ❌ Extract from a soft-404 page — Step 1.5 must pass first; a fetched-but-broken page becomes a stub with `skip_reason: "soft_404"`, never a fake entry
- ❌ Stop object recursion at depth 1 — nested `type: object` inside another object MUST have its own `properties[]`, and so on until every leaf is a scalar / enum / array-of-scalar
- ❌ Skip Step 2.5 (coverage self-check) — a param that appears in the docs page but is missing from the extracted entry is a bug, not a judgement call
- ❌ Rewrite, summarize, or "improve" the provider's documented description — `description` (action-level and param-level) is the provider's text **verbatim**, `null` if none exists
- ❌ Put purpose / when-to-use / constraints / gotchas into `description` — that LLM-optimized layer lives in `usage_notes` (params) and `_action_summary` (action); see `references/extraction-rules.md → Description quality`
- ❌ Skip nested fields of objects — every `type: object` MUST have `properties[]`
- ❌ Leave `possible_values` as `null` — always use `[]` for freeform
- ❌ Use Playwright first — try WebFetch and crwl before it
- ❌ Return partial params — if the page has 12 body fields, all 12 must appear
- ❌ Guess param types — if the docs don't say, use `string` and note it in `description`
- ❌ Omit optional headers — include ALL headers the page documents, **classified**:
  mark transport/auth-managed headers `transport: true` so build never exposes them as
  inputs (B4 skips them). Transport headers are: `Authorization`, `Proxy-Authorization`,
  signature/date headers (`Date`, `x-ms-date`), size/framing headers (`Content-Length`,
  `Transfer-Encoding`, `Host`, `Expect`), and fixed API-version pins the caller never
  varies (`x-ms-version` and equivalents — build writes these as literal step args).
  Behavioral headers stay ordinary parameters under their **literal wire names**
  (`x-ms-lease-id`, `If-Match`, `Content-Type` when the caller chooses it). A measured
  azureblobstorage run recorded Azure's per-page auth boilerplate headers as ordinary
  parameters and paid repeated build repair rounds stripping them back out.
- ❌ Leave `possible_values: []` for field-selector params — always run the Playwright dropdown extraction in `references/fetch-strategies.md`
- ❌ Omit `example`, `pattern`, `min`, `max` keys — always present, `null` if not documented
- ❌ Infer values for `example`, `pattern`, `min`, `max` — only use what provider explicitly states
- ❌ Set `pattern` for soft format hints — put in `description`, set `pattern: null`
- ❌ Drop an entry with `null` resource_url — always write a stub via Step 3
- ❌ Deviate from the output schema in `examples/output-schema.md` — follow it exactly
- ❌ Use `operationId` — the correct field name is `action_id` (e.g. `list_messages`, not `gmail.users.messages.list`)
- ❌ Derive `action_id` from the provider's OpenAPI `operationId` — always copy it from `api_coverage.json` as-is
- ❌ Use `request_body` — the correct field name is `body_params`
- ❌ Include undeclared fields — if it is not in the schema, it does not belong in the output
- ❌ Set `required_scopes: null` or `error_codes: null` or `pagination: {}` for non-paginating — always `[]` / `null` per the schema
- ❌ Invent scope names for `required_scopes` — only include names that exist in the `scope_list` provided by the executor
- ❌ Skip scope checking because the endpoint page has no explicit "required scopes" section — also check header examples, inline descriptions, and OpenAPI security blocks