# Extraction Rules — Per-Field Detail

Detailed extraction rules for the fields populated in `Step 3` of `SKILL.md`. Each field has its own detection signals and value constraints.

---

## `base_url` resolution (multi-service providers)

For most providers there is exactly one base URL (Gmail, Asana, Stripe …) and the docs page either restates it or it's stored at the top of `api_coverage.json`. Use it as-is.

For **multi-service providers** (e.g. Firebase has 14+ sub-services with different base URLs), `api_coverage.json` carries a `base_urls[]` array and each entry in `all_apis[]` is tagged with `base_url_service` naming which entry it belongs to. In that case:

1. If `api.base_url_service` is set, look it up in `coverage.base_urls[]` and use that as the authoritative `base_url` for this endpoint.
2. If `api.base_url_service` is null/missing but `coverage.base_urls.length > 1`, the extracted base URL from the docs page is your fallback — but log a warning so `list-provider-apis` can be improved to set the service tag.
3. If only one entry exists in `base_urls[]`, just use it (no lookup needed).

This avoids relying solely on per-endpoint extraction for providers where the docs page may not restate the base URL explicitly.

---

## `required_scopes` extraction

For each endpoint, check the documentation page for scope requirements. Cross-reference against the `scope_list` loaded once by the executor via `load_scopes.js` (see SKILL.md → Executable scripts).

**Detection signals (look for any of these on the endpoint's docs page):**
- Explicit scope list in a "Required scopes" / "Permissions" / "Auth" section
- Scope names embedded in the auth URL example (e.g. `scope=read:users`)
- Scope names mentioned inline next to the auth method description
- OpenAPI `security` block listing specific scopes for this operation

**Matching rules:**
- Only include scope names that are present in `scope_list` — never invent new scope names
- If the docs page explicitly names a scope that is in `scope_list`, add it to `required_scopes`
- If the docs say "requires read access" but don't name a specific scope, check `scope_list` for a matching read scope for this resource category and include it if confident
- If no scope info is found for this endpoint, set `required_scopes: []` — do NOT set `null`

---

## `error_codes` extraction

Capture every HTTP status code the endpoint can return, along with the provider's error envelope shape.

**For each error code, extract:**

| Field | Description |
|---|---|
| `status` | HTTP status (e.g. `400`, `401`, `403`, `404`, `409`, `429`, `5xx`) |
| `reason` | Provider's short error code/reason (e.g. `userRateLimitExceeded`, `invalid_grant`, `not_found`) — `null` if generic |
| `description` | What this error means in this provider's context |
| `error_envelope_path` | JSONPath into the response body where the error details live (e.g. `$.error.message`, `$.errors[0].detail`) |
| `retry_strategy` | `none` \| `retry_immediately` \| `exponential_backoff` \| `wait_for_retry_after_header` |

**Detection signals:**
- "Errors" / "Status codes" / "Response codes" section on docs page
- OpenAPI `responses` block with status codes other than `2xx`
- Inline mentions like "returns 429 when rate limited"

Only include error codes that are explicitly documented for the endpoint. If no error info is documented, set `error_codes: []` — do NOT set `null`.

---

## `pagination` extraction

If the endpoint returns a list, capture how it paginates. If the endpoint is not a list endpoint, set `pagination: null`.

| Field | Description |
|---|---|
| `style` | `cursor` \| `offset` \| `page_number` \| `link_header` \| `none` |
| `page_size_param` | Name of the param that controls page size (e.g. `limit`, `maxResults`, `pageSize`) — `null` if not configurable |
| `page_size_default` | Default page size if not specified |
| `page_size_max` | Maximum allowed page size |
| `cursor_request_param` | For cursor style: param name to send the cursor on next request (e.g. `pageToken`, `after`) |
| `cursor_response_path` | For cursor style: JSONPath to read the next cursor from response (e.g. `$.nextPageToken`, `$.meta.next_cursor`) |
| `total_count_path` | JSONPath to total count in response (e.g. `$.totalResults`) — `null` if not provided |
| `loop_until` | When to stop paginating (e.g. `"response omits nextPageToken"`, `"response has fewer items than page_size"`) |

---

## Critical extraction rules (apply to every param)

- `type: object` → MUST list ALL nested properties **recursively** in `properties[]`. This is not "list top-level keys". If a property is itself `type: object`, that property MUST have its own `properties[]`, and so on until every leaf is a scalar / enum / array-of-scalar. No opaque objects survive to Step 3.
- `type: array` → MUST include `items` describing element type; if elements are objects, list ALL their fields (again recursively).
- `possible_values` → list known enum values; use `[]` for freeform fields; NEVER use `null`
- `required` → true only if the API docs explicitly mark it as required
- Never skip a field because it seems obvious — include everything the page shows
- **Field-selector params** (`opt_fields`, `fields`, `select`, `include`, `expand`) → ALWAYS run Step 2.4 (see `references/fetch-strategies.md`) to extract dropdown values

### Recursive object expansion — worked example

Provider docs describe `body_params[0] = data` as:
```
data (object, required)
  ├─ type (string, required) — must be "user"
  ├─ attributes (object, required)
  │    ├─ name (string, required)
  │    ├─ profile (object)             ← NESTED OBJECT — recurse
  │    │    ├─ bio (string)
  │    │    └─ location (object)        ← STILL NESTED — recurse
  │    │         ├─ city (string)
  │    │         └─ country (string)
  │    └─ roles (array of string)
```

The **wrong** output stops recursion early:
```json
{ "name": "attributes", "type": "object", "properties": [
    { "name": "name", "type": "string" },
    { "name": "profile", "type": "object" }   ← MISSING properties[] — WRONG
]}
```

The **correct** output recurses all the way to leaves:
```json
{ "name": "attributes", "type": "object", "properties": [
    { "name": "name", "type": "string", ... },
    { "name": "profile", "type": "object", "properties": [
        { "name": "bio", "type": "string", ... },
        { "name": "location", "type": "object", "properties": [
            { "name": "city",    "type": "string", ... },
            { "name": "country", "type": "string", ... }
        ]}
    ]},
    { "name": "roles", "type": "array", "items": { "type": "string", "possible_values": [] } }
]}
```

If the docs only describe an object as "a `Foo` object" and hyperlink to a separate reference page, follow the link and extract the Foo schema too. An object without `properties[]` is a bug.

---

## Coverage self-check (100% field parity)

Run **before** Step 3 (SKILL.md Step 2.5). The goal: every param the docs page shows must end up in the extracted entry.

### Enumerate candidates on the page

Scan the fetched page for parameter-name candidates in all of these locations:
- Table rows in "Parameters", "Request Body", "Query Parameters", "Path Parameters", "Headers" sections
- Definition-list `<dt>` / bolded field-name entries
- Code samples — both request URLs (`?a=1&b=2`) and JSON bodies (top-level keys AND nested keys)
- OpenAPI `parameters:` and `requestBody.content.*.schema.properties` blocks if the page renders from a spec
- Inline `<code>foo_bar</code>` mentions in prose
- "Optional fields" / "Deprecated" side-notes — capture them so we consciously decide to include or record why not

### Diff against extraction

Build the extracted set as the union of:
- `path_params[].name`
- `query_params[].name`
- `body_params[].name` **plus every nested `properties[].name` recursively** and every `items.properties[].name` for arrays of objects
- `headers[].name`

Diff `candidates - extracted`. For each remaining name:

| Case | Action |
|---|---|
| Genuinely missed | Re-extract, add to the right list |
| Documented as deprecated / do-not-use | Add anyway, mark `usage_notes` with `"Deprecated: <reason>"` |
| Only appears in an unrelated example (e.g. auth code sample) | Record in the entry's `_coverage_notes[]` as `{ name, reason: "example-only, not a real param" }` |
| Belongs to a different endpoint that shares this page | Record in `_coverage_notes[]` as `{ name, reason: "belongs to <other_action_id>" }` |

**Blocker:** do not advance to Step 3 while any candidate is unresolved. If unsure whether a name is a param or example noise, include it — false positives are cheaper than silent drops.

---

## Description quality (two-field model)

Every description slot carries TWO complementary fields:

| Field | Content | Rule |
|---|---|---|
| `description` | The provider's **actual documented description, verbatim** | Faithful surface of the provider docs — trim whitespace/HTML only, never rewrite, summarize, or "improve" |
| `usage_notes` (param-level) / `_action_summary` (action-level) | The **LLM-optimized layer**: purpose, when to use, constraints, gotchas | Written by us for downstream skills (connector build, schema review, action testing) that reason about the API |

Why both: `description` must stay traceable to the provider (reviewers verify against docs, and the provider's wording is what users see in their portal). The LLM-optimized layer captures operational detail the provider prose usually lacks — but it must never overwrite the source of truth.

### `description` — provider verbatim

- Copy the provider's documented description for the action/field exactly (strip markup, collapse whitespace).
- If the provider documents no description → `description: null`. Never invent one.
- Do NOT merge soft format hints or deprecation warnings into it — those go in `usage_notes`.

### `_action_summary` — action-level LLM-optimized (entry-level key)

Whenever the docs page describes the action (or its purpose is evident from the endpoint), write:

```
<one-sentence purpose — imperative, ends with a period.>
Use when: <specific triggering scenarios, comma-separated>.
Do NOT use for: <adjacent but wrong scenarios, if any>.
Side effects: <state changes, notifications, downstream events — or "none">.
```

Skip any line whose value would be "unknown" — never fill with filler.

### `usage_notes` — param-level LLM-optimized

For every `path_params / query_params / body_params` entry (at every nesting depth), add `usage_notes` when there is operational detail beyond the provider's text; set `null` when there is nothing to add. For `headers`, the key is optional — include it only when there's a real gotcha. Template:

```
<how the API uses it — filter, identifier, mutation payload, etc.>. <constraints not captured elsewhere — format hints, valid ranges mentioned in prose, mutually-exclusive relationships with other fields>. <common mistake or gotcha if the docs call one out>.
```

Rules:
- Keep it 1–3 sentences. Longer notes get truncated by downstream tokenizers.
- Include semantic constraints the schema fields can't express: "must match the customer's email exactly", "cannot be set at the same time as `filter[status]`", "immutable after first save".
- Soft format hints with no regex ("use ISO 8601 format") go here, with `pattern: null`.
- For deprecated fields: `"Deprecated: <reason>. Use <replacement> instead."`.
- Do **not** repeat what other fields already say (`type`, `min`, `max`, `possible_values`) or what `description` already states.

### Example — the two fields together

```json
{
  "name": "filter",
  "type": "string",
  "required": false,
  "description": "Represents an optional string that can be used to filter results based on a variety of criteria including but not limited to the customer's status, region, and account tier.",
  "usage_notes": "Server-side filter applied before pagination. Comma-separated values combine as OR within a field, AND across fields. Use `filter[status]` for status alone — this generic filter is slower.",
  "possible_values": [],
  "example": null, "pattern": null, "min": null, "max": null
}
```

Action level:
```json
{
  "description": "This powerful endpoint lets you retrieve a user in a flexible way.",
  "_action_summary": "Fetch a single user by ID. Use when: displaying a user's profile, prefilling an edit form, checking existence before create. Do NOT use for: listing users (use list_users), searching by email (use search_users). Side effects: none — read-only."
}
```

---

## Conditional fields — always present as keys, value depends on provider docs

These 5 fields are ALWAYS present in every param object (path/query/body params at every nesting depth). Value is `null` when not documented / nothing to add — never omit the key.

| Field | Value when documented | Value when not documented |
|-------|-----------------------|--------------------------|
| `example` | Exact value shown in provider docs | `null` |
| `pattern` | Regex or strict format constraint | `null` |
| `min` | Min value / length / item count | `null` |
| `max` | Max value / length / item count | `null` |
| `usage_notes` | LLM-optimized layer (see Description quality above) | `null` |

**Soft format hints:** If provider says "use ISO 8601 format" (soft hint, no regex) — put it in `usage_notes`, set `pattern: null`. `description` stays provider-verbatim.