---
name: resource-endpoint-fetcher
description: Autonomous agent that owns endpoint extraction for one resource — filters api_coverage.json for its slice, loops through each endpoint applying `research-provider/references/fetch-endpoint-parts.md`, verifies its slice is complete, and returns a JSON summary. Designed to be spawned in parallel — one agent per resource — so wall-clock time stays bounded by the slowest single resource.
model: sonnet
maxTurns: 40
tools:
  - Read
  - Grep
  - Glob
  - Bash
  - Write
  - WebSearch
  - WebFetch
  - mcp__plugin_stackone-agent-harness_playwright__browser_navigate
  - mcp__plugin_stackone-agent-harness_playwright__browser_snapshot
  - mcp__plugin_stackone-agent-harness_playwright__browser_click
  - mcp__plugin_stackone-agent-harness_playwright__browser_evaluate
---

# Resource Endpoint Fetcher Agent

## Purpose

Resource-level orchestrator. This agent owns:

- Filtering `api_coverage.json` for entries where `category == <its resource>`
- Looping through those entries
- Per-resource verification + summary

For each entry, it delegates the actual extraction work to the **`fetch-endpoint-parts.md`** — that skill defines the per-endpoint workflow (fetch → extract → build → merge → mark). This agent never duplicates the spec; it just drives the loop.

## Inputs

```json
{
  "provider": "gmail",
  "resource": "messages"
}
```

## Critical rules (resource-level only)

- ✅ **Process only your resource** — never touch entries where `category != <resource>`
- ✅ **Never modify any other resource's YAML** — only `endpoints/resources/<resource>.yaml`
- ✅ **Resume-aware** — skip entries where `investigated == true`
- ✅ **Return only the summary** — never bubble raw docs or extracted entries back to the parent

(Per-endpoint correctness rules — schema fidelity, scope validity, error_codes/pagination shape, etc. — live in `fetch-endpoint-parts/SKILL.md` Anti-Patterns. This agent inherits them by loading the skill.)

## Execution flow

### Step 1: Load the skill

Read these once at the start — they are the spec for what to do per endpoint:

```
${CLAUDE_PLUGIN_ROOT}/skills/research-provider/references/fetch-endpoint-parts.md
${CLAUDE_PLUGIN_ROOT}/skills/research-provider/references/fetch-endpoint-parts/output-schema.md
${CLAUDE_PLUGIN_ROOT}/skills/research-provider/references/fetch-endpoint-parts/fetch-strategies.md
${CLAUDE_PLUGIN_ROOT}/skills/research-provider/references/fetch-endpoint-parts/extraction-rules.md
```

### Step 2: Load scopes (once) + slice api_coverage

```bash
# Load deduplicated scope_list — passed to the per-endpoint workflow as required_scopes reference
node ${CLAUDE_PLUGIN_ROOT}/scripts/research/load_scopes.js ${provider}

# Filter api_coverage.json for your slice — entries where `selected: true`
# (written by select_apis_for_investigation.js in Phase 3.4 autonomous mode
# or Phase 3.5 after the single over-cap confirmation), narrowed to this
# agent's resource. Skips entries excluded by the 70-action cap, webhook
# exclusion, or the user's count/resource/tier reply.
jq --arg r "${resource}" '
  [.all_apis[] | select(.selected == true and .category == $r)]
' .connector-dev/${provider}/research/api_coverage.json
```

Bucket your slice:
- `already_done` — entries with `investigated == true` → skip
- `to_fetch` — entries with `investigated == false` AND `resource_url != null` → full per-endpoint workflow
- `no_resource_url` — entries with `resource_url == null` → write stub via the skill's Step 3

Announce:
```
🔍 [${resource}] N endpoints — to fetch: F, stubs: M, already done: A
```

### Step 3: Loop — apply skill's per-endpoint workflow to each entry

**Prefetch in batches, process one at a time.** The docs pages in your slice are
independent — issue their fetches ~5 per message (halve on 429s/truncation), then run the
per-endpoint workflow over each fetched page. A measured azure run issued 19 doc fetches
one per turn; batching the same fetches costs the same tokens and collapses most of that
round-trip time.

For each entry in `to_fetch ∪ no_resource_url`, execute the per-endpoint workflow defined in `fetch-endpoint-parts/SKILL.md` (its Step 1 → 4, including the two mandatory gates: Step 1.5 soft-404 verification and Step 2.5 coverage self-check — both still run per page, after its batched fetch). Use the scripts the skill references. Accumulate every Step 1.5 soft-404 hit and every Step 2.5 unresolved-candidate note — they go into `soft_404_urls[]` / `coverage_gaps[]` of your return summary.

Announce per-entry progress so the spawner can tail logs:
```
[<i>/<total>] <label> (<method> <path>) → merged
```

On any single fetch failure: log, write a stub via the skill's Step 3, continue. Never abort the loop.

### Step 4: Verify your slice

```bash
# Expected count for THIS resource
jq --arg r "${resource}" '
  [.all_apis[] | select(.selected == true and .category == $r)] | length
' .connector-dev/${provider}/research/api_coverage.json

# Actual count in the resource YAML
node -e "
  const yaml=require('${CLAUDE_PLUGIN_ROOT}/scripts/lib/yaml');
  const fs=require('fs');
  const d=yaml.load(fs.readFileSync('.connector-dev/${provider}/research/endpoints/resources/${resource}.yaml','utf8'));
  console.log(d.endpoints.length);
"
```

If counts don't match, identify the missing `action_id`s and re-run the per-endpoint workflow (or stub them) before returning. The global completion check is run by the spawning command — not by this agent.

## Return summary

Return **only** this JSON object to the parent:

```json
{
  "resource": "messages",
  "expected": 11,
  "actual": 11,
  "fetched": 10,
  "stubs": 1,
  "match": true,
  "missing_action_ids": [],
  "soft_404_urls": [
    { "action_id": "get_deprecated_thing", "url": "https://docs.provider.com/api/deprecated-thing", "marker_hit": "page not found", "fetch_step": "1.1", "resolved_by_retry": false }
  ],
  "coverage_gaps": [
    { "action_id": "list_messages", "coverage_notes": [ { "name": "internalDateSource", "reason": "example-only, not a real param" } ] }
  ],
  "completedAt": "<ISO timestamp>"
}
```

- `soft_404_urls[]` — every URL where SKILL.md Step 1.5 tripped a soft-404 marker, whether or not the retry heuristic recovered it. Empty array = none observed. Populated even on success so the spawner can flag stale `resource_url`s in `api_coverage.json`.
- `coverage_gaps[]` — one item per endpoint whose Step 2.5 self-check left `_coverage_notes[]` populated, echoing those notes as `coverage_notes: [{name, reason}]`. Endpoints with no notes are NOT listed; empty array = every documented param was extracted cleanly.
- If `match: false`, include `missing_action_ids[]` so the spawner can re-spawn just this agent (or escalate).

## Error recovery

- **Single fetch failure** — handled by the skill (stub + continue). Agent never aborts the loop.
- **Script error from `merge_resource_entry.js`** — entry shape is wrong (likely missing `category` or `action_id`). Fix and retry once; if it still fails, stub and continue.
- **Agent interruption** — on resume, `investigated == true` filters re-processed entries automatically.

## Why this agent exists

Per-endpoint docs pages are 5–20 KB of markdown that get parsed once and never re-read. Keeping that content in the parent's context window is wasteful — and processing 50–200 endpoints serially is slow. By spawning one agent per resource, the parent stays clean and extraction wall-clock time drops to roughly the slowest single resource (typically 1–3 minutes per resource group).

The spawner (the endpoint-extraction step of `/connector:research`) decides batching — usually 3–5 agents per message to stay within rate limits, **always foreground (`run_in_background: false`)** so the spawn message returns with the summaries in hand. A background spawn's summary was measured sitting uncollected while the parent re-did the slice inline. That step makes the fan-out **mandatory** at ≥2 in-scope resources or >8 in-scope endpoints: two otherwise-identical runs measured it, and the one that extracted inline died at the wall-clock cap while the fanned-out twin finished in half the time.