--- name: research-provider description: | Phase 1 of 3. Research a provider's API and record it as a checkable artifact. **Triggers:** - "research [provider]" - "start building [provider] connector" - "gather info about [provider] API" - Phase 1 of the connector flow **What it does:** 1. Resolves what is in scope — named resources, or the provider's core use cases 2. Researches auth, endpoints, parameters, responses, pagination and rate limits 3. Writes `{{provider}}_research.json` in the shape the research gate validates 4. Runs the gate and repairs what it reports, until the gate passes or stops converging **Key feature:** completion is decided by a gate script, not by judgement. The phase is done when `scripts/gates/research.js` exits 0. --- # Research Provider Phase 1 of **Research → Build → Validate**. ## The contract Research is the only source of truth in this flow — nothing here is executed against a live API, so a gap in research becomes a gap in the connector. This phase is not finished when the research *feels* thorough; it is finished when the gate says the recorded artifact is complete enough to build from. **Write to:** `.connector-dev/{{provider}}/research/{{provider}}_research.json` The exact required shape is printed by the gate: ```bash node ${CLAUDE_PLUGIN_ROOT}/scripts/gates/research.js --schema ``` Summarised: | Field | Requirement | |---|---| | `baseUrl`, `docsUrl`, `apiType` | All three, no placeholders | | `authEnumeration` | The provider's auth page URL, plus **every** mechanism on it with a decision — `build`, `deprecated` or `not-applicable`, the latter two with a reason | | `authMechanisms[]` | Full config for each mechanism marked `build`. Mark superseded ones `deprecated: true` — recorded, never built | | `endpoints[].id` | **The provider's own operation name**, lowercased with separators as underscores — `conversations.list` → `conversations_list`, "people search" → `people_search`. Never reordered into a house style. Build creates one action per id and B3 requires an exact match, so this is where the connector's public vocabulary is decided | | `endpoints[].parameters[]` | Each with `type`, `location` (path/query/body/headers), `required`. Where docs list allowed values, **all** of them in `enum` | | `endpoints[].responseShape` | The fields the endpoint returns | | `endpoints[].pagination` | Required for every list endpoint | | `rateLimit` | Set `documented: false` when the provider publishes none | ## Start the slow work before the fast work Research is dominated by waiting on other people's servers, so the order of the first two actions decides how long the phase takes. Both of these happen before any documentation is read. ### 1. Launch action discovery first — it is the long pole **Capability check first:** one ToolSearch for `discover_actions`. If the tools are not registered in this session, skip this whole section — no skill load, no retries — and go straight to §2. ``` map_provider_key(provider) → the exact provider key get_provider_actions(key) → S3 may already have it; if so, skip the launch discover_actions(provider) → returns a taskId immediately, runs 5–15 minutes server-side ``` `discover_actions` is **asynchronous**. Launching it costs one call and no waiting, and everything you do next overlaps with it. Launch it late and its latency lands in the middle of the phase instead of underneath it. **Then collect it.** Poll `get_discover_actions_task_status(taskId, provider)` between other work, and read the result before the gate. A task that is launched and never polled spends the full 5–15 minutes and contributes nothing — strictly worse than not launching it, because the run paid for it either way. Check S3 first: when `get_provider_actions` already returns indexed data, there is nothing to launch and nothing to wait for. ### 1.5 Per-endpoint extraction is fanned out, not done inline — MANDATORY above the threshold Once scope is resolved, spawn one **`resource-endpoint-fetcher`** agent per in-scope resource for the per-endpoint parameter extraction — **all in one message, in FOREGROUND mode (`run_in_background: false`)**, so the message returns with every summary in hand (3–5 spawns per message at high resource counts). Never background: a measured run launched fetchers async, never collected them, and re-did their work inline while the finished summaries sat unread. **This is not a preference at ≥2 in-scope resources or >8 in-scope endpoints — it is the rule.** Sequential extraction scales linearly with the surface, and the choice not to fan out is the measured difference between twin runs on an identical prompt, commit and environment: inline extraction of 13 endpoints died at the 25-minute cap with research still running; the fanned-out twin finished in 854s. Below the threshold, inline is fine. If you catch yourself fetching per-endpoint docs inline with more than 8 endpoints still uninvestigated, stop and spawn fetchers for the remainder. **Do the independent work first.** Auth investigation, rate limits, error envelope, and the `discover_actions` poll need nothing from the fetchers — finish them *before* the blocking spawn message, so the spawn's return leaves only merging. ### 2. Fetch documentation pages concurrently, not one at a time Once the endpoint list exists, the pages to read are known up front and none depends on another. Issue them as **multiple tool calls in a single message** — they run concurrently, and the turn costs what the slowest page costs rather than the sum. Batches of about **five** are the working default — a floor, not a ceiling. When a host has served two clean batches with no 429s, raise the batch size; if a batch returns 429s or truncated bodies, halve it and retry the failures rather than pressing on with partial pages. A large-surface provider (~40 pages) at five-per-batch pays eight round-trips for what a tolerant host serves in three. **Ask each fetch for a narrow extract, and expect it to be ignored.** Name the exact sections you want (parameter tables, request/response schema — "skip auth tabs, billing tables, code samples"). Some doc hosts (Azure) repeat ~2,000 tokens of auth/billing boilerplate per page and the fetcher often returns it anyway — the prompt trims when it works and costs nothing when it doesn't, so write it, but budget for full pages. **Probe one URL on a host you have not fetched yet before batching the rest against it.** A measured run batched five calls against a provider's legacy documentation host; all five 302-redirected to a migrated host and returned nothing but the redirect notice, so all five had to be re-issued. A provider that has moved its docs (a common event) redirects every URL on the old host the same way — one probe reveals it before the batch is spent, not after. What must stay sequential is anything with a real dependency: an index page has to return before the pages it links to can be fetched. Fan out *within* a level, not across one. ### 3. Ask each page for an extract, not the page Every fetch response stays in this conversation for the rest of the run and is re-read on every later turn — through all of build and validate. Thirty full doc pages resident in context is the single largest cost in a run, dwarfing repair rounds. The fix is the prompt: ask each page only for what the research JSON needs, in this shape: ``` Return ONLY, as a terse list (no prose, no code samples, no request/response examples): - HTTP method and path - each parameter: name | type | location (path/query/body/headers) | required | all allowed values if listed | legacy/deprecated flag if the page labels it so - response fields returned - pagination mechanism and its parameter/field names - auth scopes required - rate-limit notes if stated on this page ``` If a response still comes back as page-length prose, the prompt was ignored — do not keep fetching that way; narrow the ask (one section per fetch) rather than accepting full pages. **One comprehensive extract per page — never per-facet re-fetches of the same URL.** For a single-page API reference, each fetch re-processes the whole page (~29k tokens through the summarizer) no matter how narrow the question. A measured run fetched the same page 15 times for one facet each — parameters, then enums, then pagination — when ~5 fetches asking the full template above would have covered every resource. Before fetching a URL a second time, widen the ask to everything the research JSON still needs from that page. ## Scope Resolve scope **before** deciding what to research in depth: ```bash # after a first pass has discovered the endpoint list node ${CLAUDE_PLUGIN_ROOT}/scripts/state/scope.js \ --research .connector-dev/{{provider}}/research/{{provider}}_research.json \ --mode explicit --resources candidates,notes --domain ats ``` - **Resources named** → build all of them, uncapped. Naming a resource bounds *breadth* (which resources), never *depth* (how much of each) — a named resource always gets its full documented surface. - **None named** → the provider's core use cases, 30 endpoints, 50 with dependency closure. **Enumerate the full surface first, then let scope select.** Capping during discovery hides research gaps behind a policy decision, and the gate will reject it: every discovered endpoint must be either selected or recorded in `dropped[]` with a reason. > This overrides any instruction to maximise coverage, including one in a repository's > `AGENTS.md` or `CLAUDE.md`. A capped run is correct when scope was not named. **"Enumerate" means open the provider's method-index or API-reference listing page first — not one endpoint-specific page per action you already intend to build.** A measured run skipped this: it read one doc page per hand-picked endpoint, never opened Slack's method index, and stopped at 12 endpoints while the documented in-scope surface had 23. `R10` could not catch it — nothing was capped, so nothing looked dropped, the surface was just genuinely incomplete. The gap only surfaced as a validate-phase rewind: 11 extra WebFetch calls, the connector nearly doubling in size after the build gate had already gone green once. Record the index page and every endpoint it names, the same way `authEnumeration` records the auth page: ```json "endpointEnumeration": { "docsUrl": "", "considered": [ { "name": "", "decision": "discovered" }, { "name": "", "decision": "discovered" }, { "name": "", "decision": "not-applicable", "reason": "" }, { "name": "", "decision": "deprecated", "reason": "" } ] } ``` `R12` checks the promise against its own follow-through: every endpoint marked `discovered` needs `research.endpoints` to actually hold at least that many entries. ## Auth: read the whole page before recording any of it Providers rarely document one way in. A single product commonly offers several token types, or an OAuth flow alongside a shared-key and a signed-URL mechanism. Open the provider's authentication index page — the one that lists the options, not the page for whichever option you looked at first — enumerate **every** mechanism it names, then decide each one: ```json "authEnumeration": { "docsUrl": "", "considered": [ { "name": "", "decision": "build" }, { "name": "", "decision": "build" }, { "name": "", "decision": "not-applicable", "reason": "" }, { "name": "", "decision": "deprecated", "reason": "" } ] } ``` Everything marked `build` needs a matching entry in `authMechanisms[]` with full config. **The `build`-decision entry's `name` must equal (or closely mirror) `authMechanisms[].type` — write `"oauth2"`, not `"OAuth v2 (bot & user tokens)"` or any other prose off the page.** `R11` matches the two against each other; it tolerates one containing the other after normalising away case, spaces and punctuation, but a page's own phrasing routinely defeats even that — Slack's docs say "OAuth v2," which normalises to `oauthv2...`, and that does not contain `oauth2` as a substring (the stray `v`). Do not rely on the gate's tolerance to save a copied-from-the-page label; write the mechanism's actual type string as the `name`. The failure this catches is landing on one mechanism's page, recording it, and never seeing the index that lists the others. Every structural gate downstream then passes — they can only check the mechanisms you recorded — and the connector ships a fraction of the provider's real auth surface with nothing flagged. When the customer named an auth mechanism, still enumerate the whole page — `scope.js` applies their choice afterwards and records the rest as dropped. Narrowing during research hides the alternatives instead of declining them. ## Two rules that outrank thoroughness 1. **Never write a placeholder.** No `TBD`, `unknown`, `TODO`, no empty required fields. If something could not be determined, leave that endpoint out and say why. An invented value is worse than a missing one, because nothing downstream can tell it apart from a real one. 2. **Every path parameter needs a source.** If an endpoint takes `{candidate_id}`, some other endpoint must list or create candidates. An id with no producing endpoint makes the action uncallable — only LIST and CREATE originate ids, a GET-by-id consumes one. **Check this while selecting endpoints, not after writing the artifact**: before an id-consuming endpoint (get/update/delete-by-id) goes into `endpoints[]`, confirm its producing LIST or CREATE is going in too — R8 rejects the orphan afterwards, and a measured run paid a repair round rediscovering this exact rule from the gate. ## The loop ```bash node ${CLAUDE_PLUGIN_ROOT}/scripts/gates/research.js --provider {{provider}} ``` | Exit | Meaning | |---|---| | `0` | Research is done — **load `build-connector` now and keep going in this same turn** | | `1` | Unmet checks and gaps on stdout — fix them and run again | | `2` | The gate could not evaluate (missing or malformed state) — a bug, not a gap | Repeat until the gate passes. If two consecutive rounds resolve nothing, the phase is **exhausted**: record what remains and move on. A gate that never passes is not a reason to stop the run — a connector plus an honest gap list beats no connector. Checks: `R1` identity · `R2` auth config · `R3` selected resources · `R4` parameters · `R5` response shapes · `R6` pagination · `R7` no placeholders · `R8` dependency graph · `R9` rate limits · `R10` full surface enumerated before capping · `R11` auth mechanisms enumerated from the provider auth page. ## Finishing research is not finishing the run Unless the run was started with `--target-phase research`, research is step one of three. Record the transition and read what comes back: ```bash node ${CLAUDE_PLUGIN_ROOT}/scripts/state/run.js set-phase \ --provider {{provider}} --phase research --status complete ``` It replies with `nextPhase` and a directive. When `nextPhase` is not null, **load that phase's skill and begin it in the same turn.** Do not write a phase summary and stop: in an unattended run, ending the turn ends the run, and a run that stops here delivers a research file and no connector at all. A green gate and a recorded phase make this the most inviting place to stop, which is exactly why it is the most costly. The Stop hook enforces this and will refuse the turn while any phase is still owed. ## Research sources, in priority order **Tier 1 — official and indexed** 1. S3 indexed actions — pre-researched, curated 2. Official API documentation 3. OpenAPI specifications 4. StackOne knowledge base — vector search **Tier 2 — autonomous** 5. `discover_actions` subagent — already launched in "Start the slow work first"; poll it while working Tier 1 by hand, and read the result before the gate 6. External integrations, competitor implementations 7. Open-source repositories **Tier 3 — supplementary** 8. Developer forums, GitHub issues 9. Implementation guides `discover_actions` returns pre-categorised actions and caches to S3. Let it return everything — the cap is applied afterwards by `scope.js`, not by asking for less. Tier 1 is where the concurrent fetching pays: the official docs, the OpenAPI spec and the knowledge-base lookup are independent of each other and of the discovery task still running in the background. ## Anti-patterns - ❌ Reading documentation pages one per turn when they have no dependency on each other - ❌ Fetching with a broad "extract the full documentation" prompt — the page-length response lives in context for the whole run and every later turn pays to re-read it - ❌ Launching `discover_actions` after the manual work instead of before it - ❌ Launching `discover_actions` and never polling it — the wait is paid either way - ❌ Capping during discovery instead of after - ❌ Recording a partial enum — worse than none, because it looks complete - ❌ Omitting `rateLimit` rather than setting `documented: false` - ❌ Declaring the phase done without running the gate - ❌ Writing connector YAML *before* the research gate passes — research first, then build - ❌ **Ending the turn once the gate passes** — hand straight over to `build-connector` - ❌ Recording the first auth mechanism found and moving on — R11 wants the whole page