# Fetch Strategies — Tool Selection for Endpoint Docs

Detailed fallback chain for fetching a single endpoint's documentation page in **Step 1** of `SKILL.md`, the **Step 1.5** soft-404 gate, and the **Step 2.4** dropdown extraction that runs during extraction. Always try the chain in order. Stop when content is found.

| Step | Tool | When to use |
|------|------|-------------|
| 1.1 | `WebFetch` | Always try first — fast |
| 1.2 | `crwl` (bash) | If WebFetch returns empty/nav-only (JS-rendered page) |
| 1.3 | Playwright `browser_evaluate` | If crwl still misses content (interactive portals) |
| 1.5 | Soft-404 verification | MANDATORY gate before extraction — an HTTP 200 that renders a "page not found" body is a broken `resource_url`, not a real page (matches SKILL.md Step 1.5) |
| 2.4 | Playwright dropdown extraction | MANDATORY post-extraction check for field-selector params (runs during SKILL.md Step 2) |

**How to detect Step 1.1 failure:** fetched markdown has no parameter names, only nav links, or body is under 500 chars.

---

## Step 1.1: WebFetch

```
WebFetch(api.resource_url, "Extract the full endpoint documentation — every path/query/body parameter with type and required flag, all headers, auth methods, error codes, pagination, and examples")
```

If the page is long and WebFetch's response drops parameter detail, do not guess the missing fields — fall through to crwl (Step 1.2) for the raw page markdown.

---

## Step 1.2: crwl (bash fallback)

```bash
crwl crawl "<api.resource_url>" -c "wait_until=networkidle,delay_before_return_html=3.0" -o markdown
```

---

## Step 1.3: Playwright (last resort)

```javascript
mcp__plugin_stackone-agent-harness_playwright__browser_navigate("<api.resource_url>")
mcp__plugin_stackone-agent-harness_playwright__browser_evaluate(`({
  title: document.title,
  body: document.body.innerText.substring(0, 15000)
})`)
```

---

## Step 1.5: Soft-404 detection (MANDATORY gate before extraction)

Providers frequently serve HTTP 200 for URLs whose page body says "Page not found" or is essentially empty. Extracting from those pages produces empty/junk entries that hide the real problem: the `resource_url` in `api_coverage.json` is wrong.

Apply this gate to whichever step (1.1 / 1.2 / 1.3) succeeded in returning content:

### Verification checks

Pass **all three** before treating the fetch as real:

1. **HTTP status** — Must be 2xx. 3xx/4xx/5xx counts as fetch failure and falls through to the next step in the chain.
2. **Content length** — After stripping nav, headers, and footers, the visible body must be > 500 chars of doc content. Nav-only pages fail here.
3. **Soft-404 markers** — The visible body must NOT contain any of these strings as page-heading or main-content text (case-insensitive; ignore matches inside code samples):

   - `"page not found"`
   - `"404"` when it appears as a heading (`<h1>` / `<h2>`), not as a documented HTTP status
   - `"this page doesn't exist"` / `"page doesn't exist"`
   - `"no such page"`
   - `"we couldn't find the page"` / `"couldn't find that page"`
   - `"the page you're looking for"`
   - `"oops"` when followed by "gone" / "missing" / "not here"
   - Provider-specific variants observed in the wild: `"documentation not available"`, `"this endpoint was removed"`

### On failure

1. Record `{ action_id, url, marker_hit, fetch_step, resolved_by_retry }` in the fetcher's `soft_404_urls[]` summary (`fetch_step` = 1.1 / 1.2 / 1.3, whichever returned the broken page; shape matches the `resource-endpoint-fetcher` return summary).
2. Try **one** corrected-URL heuristic, in order until one produces a non-soft-404 page:
   - Strip trailing slash / trailing `#anchor` and refetch
   - Swap common host prefixes: `docs.provider.com/api/` ↔ `developer.provider.com/reference/` ↔ `docs.provider.com/reference/`
   - If `api_coverage.json` carries an `openapi_spec_url` and the provider renders docs from OpenAPI, derive the URL as `<docs_base_url>/<action_id_slug>` and try that
3. If the second attempt also fails, write the entry as a **stub** via SKILL.md Step 3 with `skip_reason: "soft_404"` (not `"no_resource_url"`). Do not fabricate params.

### Quick heuristic (fetch response)

```javascript
// Applied inside whichever fetch tool returned content
const body = fetched.text.toLowerCase();
const softMarkers = [
  'page not found', "this page doesn't exist", "page doesn't exist",
  'no such page', "we couldn't find", "couldn't find that page",
  "the page you're looking for", 'documentation not available',
  'this endpoint was removed'
];
const bare404 = /<h[12][^>]*>\s*404\s*<\/h[12]>/i.test(fetched.html || '');
const isSoft404 = bare404 || softMarkers.some(m => body.includes(m));
const isTooShort = body.replace(/\s+/g, ' ').trim().length < 500;
if (isSoft404 || isTooShort) {
  // treat as failure — see "On failure" above
}
```

---

## Step 2.4: Playwright — Interactive Dropdown Values (MANDATORY check during SKILL.md Step 2 extraction)

After extracting params from any source, check every param where `possible_values` is `[]` but the description mentions "comma-separated list of properties", "fields to include", "select fields", or similar field-selector language.

**Common names to look for:** `opt_fields`, `fields`, `select`, `include`, `expand`.

```javascript
// Step 1: try extracting dropdown options directly from the DOM
mcp__plugin_stackone-agent-harness_playwright__browser_navigate("<api.resource_url>")
mcp__plugin_stackone-agent-harness_playwright__browser_evaluate(`
  (() => {
    const selectors = [
      '[class*="dropdown"] li', '[class*="select"] option',
      '[class*="option"] [class*="item"]', '[class*="combobox"] li',
      '[role="option"]', '[role="listbox"] li',
      '[class*="autocomplete"] li', 'datalist option'
    ];
    const results = new Set();
    selectors.forEach(sel => {
      document.querySelectorAll(sel).forEach(el => {
        const t = el.textContent.trim();
        if (t.length > 0 && t.length < 100) results.add(t);
      });
    });
    return [...results].sort();
  })()
`)
```

If empty, click the param input first to trigger the dropdown, then re-run the evaluate.