--- name: connector:research phase: 1 description: Phase 1 of 3 - Research a provider's API and record it as a checkable artifact, gated by scripts/gates/research.js triggers: - /connector:research - research api docs - investigate api - research connector --- # Phase 1: Research Provider Run phase 1 of **Research → Build → Validate** for `{{provider}}`. This command **never writes connector YAML or config code** — it produces one research artifact and drives it through the research gate. ## Trigger Conditions This command should auto-invoke when user requests match these patterns: - "Investigate [provider]" - "Research [provider] API for connector" - "Run full API investigation for [provider]" - "Research [provider] API docs" **Does NOT trigger for:** - General API research unrelated to connector development - Questions about a provider's API without connector context ## Output One artifact, in the shape the gate validates: ``` .connector-dev/{{provider}}/research/{{provider}}_research.json ``` The exact required shape is printed by the gate: ```bash node ${CLAUDE_PLUGIN_ROOT}/scripts/gates/research.js --schema ``` State lives in `.connector-dev/{{provider}}/run.json`, written only by `scripts/state/run.js` — never by hand. ## Execution Flow ### Step 0: Initialize the run ```bash node ${CLAUDE_PLUGIN_ROOT}/scripts/state/run.js init \ --provider {{provider}} --request "" \ --target-phase research ``` `--target-phase research` says this run stops after research and is finished there. It is what distinguishes "research was all that was asked for" from "a full build stopped after research" — without it the Stop hook treats the finished phase as an abandoned build and refuses to end the turn. Re-issuing the same request resumes from `run.json`; a completed research phase is skipped. ### Step 1: Launch action discovery before reading anything ``` map_provider_key({{provider}}) → get_provider_actions(key) → discover_actions({{provider}}) ``` `discover_actions` runs 5–15 minutes server-side and returns a taskId immediately, so launching it first puts its latency underneath the manual work instead of after it. Skip the launch when `get_provider_actions` already returns indexed data. Poll `get_discover_actions_task_status(taskId, {{provider}})` between other work and read the result **before** the gate. Launched-and-never-polled is the worst case: the wait is paid and nothing is collected. The same applies to documentation pages: once the endpoint list exists, the pages have no dependency on each other, so fetch them as several tool calls in one message — roughly five at a time — rather than one per turn. The phase skill carries the detail. ### Step 2: Load the phase skill and follow it **Load skill: skills/research-provider** — it owns this phase: what the artifact must contain, the two rules that outrank thoroughness (never write a placeholder; every path parameter needs a producing endpoint), the research source tiers, and the gate loop. For deeper provider investigation, the research reference docs carry the detailed technique — read the one that matches the work at hand rather than improvising: | Reference | Use for | |---|---| | `list-provider-apis.md` | Discovering the full endpoint surface (OpenAPI spec first, then deep crawl) | | `fetch-endpoint-parts.md` | Extracting one endpoint's parameters, headers, error codes and pagination from its docs page | | `investigate-auth-oauth2.md` | Every OAuth2 grant type the provider documents | | `investigate-auth-custom.md` | API key, PAT, basic, bearer and other non-OAuth2 mechanisms | | `use-case-catalog.md` | The provider's core use cases, when no resources were named | | `consolidate-research-output.md` | Merging findings into consolidated research output | Read them from `${CLAUDE_PLUGIN_ROOT}/skills/research-provider/references/`. They are documents, not skills — there is nothing to invoke and no slash command behind them. Whatever the sub-skills produce, the deliverable of this phase is the single artifact `{{provider}}_research.json` — findings that never reach it do not exist as far as the gates are concerned. ### Step 3: Resolve scope Once a first pass has discovered the endpoint list: ```bash node ${CLAUDE_PLUGIN_ROOT}/scripts/state/scope.js \ --research .connector-dev/{{provider}}/research/{{provider}}_research.json \ --mode [--resources a,b,c] [--operations search] \ [--auth "oauth,api key"] [--sandbox] [--domain ] \ | node ${CLAUDE_PLUGIN_ROOT}/scripts/state/run.js set-scope --provider {{provider}} ``` - **Resources named** → `mode: explicit`, build all of them, uncapped. - **None named** → `mode: full`, the provider's core use cases — 30 endpoints, 50 with dependency closure. - Pass `--operations`, `--auth` and `--sandbox` whenever the customer named them, using their words. - `--operations` only when a verb *restricts* the request ("actions for **searching** people"). "including X" is emphasis, not a restriction — leave it empty. **Enumerate the full surface first, then let scope select.** Every discovered endpoint must be either selected or recorded in `dropped[]` with a reason — the gate rejects capping during discovery. ### Step 4: Fan out endpoint extraction — MANDATORY above the threshold Once scope is resolved, per-endpoint parameter extraction is done by **`resource-endpoint-fetcher`** agents — spawn **one per in-scope resource**, each following `research-provider/references/fetch-endpoint-parts.md` for its slice and returning a JSON summary. **Spawn them all in ONE message, in FOREGROUND mode (`run_in_background: false`), and do the auth investigation BEFORE that message.** Foreground calls in one message still run concurrently — the message returns when the slowest fetcher finishes, with every summary in hand. Never launch them in background: a measured run spawned both fetchers async, got agentIds back in 10ms, never sent a single SendMessage, and re-did the whole extraction inline while the finished summaries sat uncollected for 17 minutes — the fan-out paid its full token cost and saved zero seconds. Order the phase: auth + provider-level facts first (they need no fetcher output), then the blocking spawn, then merge the summaries. At more than ~5 resources, split into consecutive blocking messages of 3–5 spawns each (doc-host rate limits) — each message still returns with its batch's summaries. **This fan-out is not optional when in-scope resources ≥ 2 OR in-scope endpoints > 8.** Sequential extraction scales with the surface and is the measured kill pattern: on the same prompt, commit and environment, the run that extracted 13 endpoints across 3 resources inline died at the 25-minute cap (research still running at 98% of its clock); its twin fanned out and finished in 854s. Below the threshold (a 1-resource, few-endpoint provider), inline extraction is fine. **Mid-course correction rule:** if you notice yourself fetching per-endpoint docs pages inline with more than 8 endpoints still uninvestigated, stop and spawn the fetchers for what remains — the pages you already read stay read. **Do the independent work BEFORE the spawn, not after.** Auth investigation (`investigate-auth-oauth2.md` / `investigate-auth-custom.md` + `merge_auth_config.js`), rate limits, error envelope, pagination style, and collecting `get_discover_actions_task_status` all need nothing from the fetchers — finish them first, then issue the blocking spawn message. The spawn returns with every summary; what remains is merging, not re-fetching. For full-mode runs that need the dynamic use-case catalogs, the **`use-case-discoverer`** agent owns Layers 3 and 4 the same way — spawn it after the API surface exists (Pass 1 of `list-provider-apis.md`), read its JSON summary, keep the docs-reading out of this session. ### Step 5: Run the gate loop ```bash node ${CLAUDE_PLUGIN_ROOT}/scripts/gates/research.js --provider {{provider}} ``` | Exit | Meaning | |---|---| | `0` | Research is done — this command ends here; `/connector:build` is the next step | | `1` | Unmet checks and gaps on stdout — fix them and run again | | `2` | The gate could not evaluate (missing or malformed state) — a bug, not a gap | Checks: `R1` identity · `R2` auth config · `R3` selected resources · `R4` parameters · `R5` response shapes · `R6` pagination · `R7` no placeholders · `R8` dependency graph · `R9` rate limits · `R10` full surface enumerated before capping · `R11` auth mechanisms enumerated from the provider auth page. `R11` wants the provider's authentication index page read in full and every mechanism on it recorded with a decision — `build`, `deprecated` or `not-applicable`, the latter two with a reason. Stopping at the first mechanism found is what it exists to catch. Repeat until the gate passes. If two consecutive rounds resolve nothing, the phase is **exhausted**: record what remains and move on — a connector plus an honest gap list beats no connector. ### Step 6: Record the transition ```bash node ${CLAUDE_PLUGIN_ROOT}/scripts/state/run.js set-phase \ --provider {{provider}} --phase research --status [--unmet R4,R6] ``` ## Output Report Two narrated lines, printed verbatim from the narrator — not composed by hand, so the counts match the artifact: ```bash node ${CLAUDE_PLUGIN_ROOT}/scripts/state/narrate.js --provider {{provider}} --event docs node ${CLAUDE_PLUGIN_ROOT}/scripts/state/narrate.js --provider {{provider}} --event researched ``` ``` ✓ Found Ashby's API docs — 3 resources, 41 endpoints ✓ Researched candidates, notes and applications ``` Then one line naming the next step, because this command was invoked on its own: ``` Next: /connector:build {{provider}} ``` No artifact paths, no gate rosters, no check ids — that detail is in `run.json` for anyone who wants it. If the phase was recorded `exhausted`, say in one sentence what remains. ## Behavioral Guidelines ### MUST DO: - Write everything into `{{provider}}_research.json` — the single artifact the gate reads - Record **every** documented auth mechanism, marking superseded ones `deprecated: true` - Give every list endpoint a `pagination` entry and every path parameter a producing endpoint - Set `rateLimit.documented: false` when the provider publishes none - Run the gate before declaring the phase done — completion is the gate's verdict, not judgement ### MUST NOT: - ❌ Write connector config or YAML — investigation only - ❌ Write placeholders — no `TBD`, `unknown`, `TODO`; leave the endpoint out and say why - ❌ Record a partial enum — worse than none, because it looks complete - ❌ Cap during discovery — enumerate first, let `scope.js` select - ❌ Edit `run.json` by hand — only `scripts/state/run.js` writes it - ❌ Declare the phase done without `research.js` exiting 0 (or recording `exhausted` with what remains)