# Scripts Deterministic helpers for the three-phase connector flow. Everything here is plain Node with zero npm dependencies — the one library needed (`js-yaml`) is vendored at `lib/vendor/js-yaml.cjs` and loaded through `lib/yaml.js`, so the scripts run wherever the plugin directory is copied, with no npm install. Nothing requires `jq`. Skills invoke these via `${CLAUDE_PLUGIN_ROOT}/scripts/…`. ``` scripts/ ├── gates/ # phase gates — decide whether a phase may advance ├── state/ # run.json + scope contract — the durable run record ├── research/ # Phase 1 helpers — classification, merging, verification └── lib/ # shared helpers (safe-path, yaml loader + vendored js-yaml) ``` --- ## gates/ A gate answers exactly one question: **may this phase advance?** It is deterministic code, never a model call. The skill decides what to do about a gap; the gate decides whether the phase is done. ### Exit-code contract (all gates) ``` 0 pass — the phase may advance 1 fail — unmet checks and gap records printed as JSON on stdout 2 could not evaluate (missing or malformed state) — a bug, not a gap ``` ### Common CLI ```bash node gates/.js --provider [--scope ] [--work-dir ] [--only R1,R2] node gates/.js --schema # print what the gate reads and checks, as JSON ``` - `--schema` — self-documentation: the artifact shape the gate expects and the list of checks it runs. - `--only A1,A2` — re-run selected checks after a targeted fix. A scoped run reports `verdict: scoped-pass`, never `pass`: it answers "did my fix land?", not "may the phase advance?". Only a full run exiting 0 is a verdict. Scoped runs also skip ledger reconciliation, so they do not advance the round counter or the stall detector. - `--scope ` — re-check one resource instead of the whole surface. - `auth.js`, `build.js`, `conventions.js`, `validate.js` additionally take `--connector-path ` — the directory holding `*.connector.s1.yaml` and its partials. - `build.js` additionally takes `--cli-ok true|false [--cli-output "…"]` to carry the `stackone validate` verdict. Omitting it fails B1: an unrun check must never read as a passing one. ### Running a whole phase at once ```bash node gates/phase.js --phase build --provider --connector-path --cli-ok true ``` `phase.js` runs every gate a phase owns in one invocation — Build's three in order — and reports only the checks that failed. Each gate still runs as its own process, so ledger write-back and exit codes are unchanged; this removes the round-trips, not the semantics. Prefer it in the repair loop. Repair rounds are the normal case, and three Bash calls per round is real time — a run that spends its budget on round-trips can be cut off before it writes a connector. Reach for a single gate only to re-check one thing you just changed (`--only B4`). ### The five gates **`research.js` — R1–R11.** Research is the only source of truth in this flow, so this gate stops a thin investigation propagating into a thin connector. Reads `.connector-dev//research/_research.json`. Checks: provider identity (R1), captured auth mechanisms are configured (R2), every selected resource has endpoints (R3), parameters fully specified (R4), response shapes captured (R5), pagination on list endpoints (R6), no placeholder values (R7), dependency graph resolvable (R8), rate limits captured (R9), full surface enumerated before capping (R10), auth mechanisms enumerated from the provider's auth page (R11). R11 exists because R2 cannot ask the question that matters. R2 checks that what was recorded is well-formed, which a run that stopped at the first mechanism on the page passes with a single entry — indistinguishable from a provider that genuinely has one. Every downstream auth check inherits the same blind spot, since none of them can inspect a mechanism that was never written down. No deterministic check can prove completeness against a page it cannot read, so R11 does the next best thing: it requires `authEnumeration.docsUrl` plus every mechanism found there with a decision (`build`, `deprecated`, `not-applicable`) and a reason for each exclusion. An omission has to be written down as a decision instead of passing as silence. **`auth.js` — A1–A12.** A sub-gate of Build, not a fourth phase — run to green before any action is written, because a malformed auth block still parses and only reveals itself as an unexplainable 401 on every action at once. Checks: auth block present (A1), required block keys (A2), authorization shape (A3), credential fields well-formed (A4), referenced credentials declared (A5), OAuth redirect_uri platform-supplied (A6), refreshable mechanisms declare refresh (A7), testActions resolve and are read-only (A8), every in-scope mechanism configured (A9), provider auth documentation was read (A10 — requires `.connector-dev//build/auth-evidence.json`, the anti-tautology check), support text explains each field set (A11), every declared field used (A12). **`build.js` — B1–B16.** Measures the projection in the direction that matters: **built ÷ discovered-in-scope** — measuring tested ÷ built can read 100% while half the discovered API was never built. Checks: `stackone validate` exits 0 (B1), every in-scope resource has a partial (B2), every in-scope endpoint has an action (B3), every documented parameter exposed (B4), parameter fidelity — required/in/complete enums (B5), every in-scope auth mechanism configured (B6), substantive descriptions (B7), wiring — `$ref` present, every input reaches a step (B8), no duplicate actionIds (B9), corpus input form (B10), one expression format per field, JEXL quoted (B11), no unified actions (B12), actionType is custom or event (B13), no untyped catch-all body objects (B14), rateLimit carried across when research documented one (B15), result references a declared stepId (B16). B16 exists because it is the one silent failure nothing else caught: a `result.data` pointing at a stepId that does not exist parses cleanly, passes `stackone validate`, and returns an empty result at runtime. **`conventions.js` — N1–N20.** Every rule mined from the 484 shipped connectors (30,726 non-unified actions, 133,406 inputs) with its corpus adherence stated. N1–N8 and N13 are blocking; N9–N12 are real conventions with a legitimate tail, reported as `major` — both fail the gate. Checks: snake_case actionIds (N1), no bracket/type symbols in input names (N2), `required` always stated (N3), input type from the six-value set (N4), condition on every optional arg (N5), descriptions present and capitalised (N6), baseUrl never sandbox/demo/staging (N7), production-only environments unless sandbox in scope (N8), Title Case labels (N9), sandbox as one conditional baseUrl (N10), no condition on required-input args (N11), input descriptions ≥10 chars (N12), actionId semver version (N13), scope definitions resolve (N14), dynamic-baseUrl credentials declared and required (N15), every action declares valid effects (N16), authRequirements well-formed with no superseded requiredScopes (N17), environments name a deployment stage only (N18), every action cites a documentation URL (N19), exactly one step per action (N20). There is deliberately **no** actionId word-order rule. `actionId` mirrors the provider's own operation name — `people_search`, `conversations_list` — so a corpus-derived verb-first rule would fight the provider and flag connectors that are correct. N13–N15 were adopted from the sibling plugin's `review-connector` health check. Two of its rules were deliberately **not** taken: "version 1.0+ means releaseStage ga" is contradicted by our own corpus (9 of 11 shipped connectors sit at 1.0.0/preview), and an unused scope definition is reported rather than failed, since a scope can legitimately sit in an `includes` chain. A rule the corpus contradicts costs repair rounds for nothing. Connector **filenames** are not gated here. The rule — lowercase letters, digits and dots only, so `azureblobstorage` never `azure-blob-storage` — is a strict instruction in `connector-guide.md` and the build skill, deliberately not a check: a gate that fails on a filename is one more thing that can stall an unattended run. **`validate.js` — V1–V8.** Does not trust research — goes back to the provider's live documentation and asks whether the connector is the whole truth about that API. Reads per-resource comparison artifacts at `.connector-dev//validation/__doc_comparison.json`. Checks: structural review passes (V1), independent re-derivation complete (V2 — the evidence check: an empty `docsRead` list is rejected), no parameter discrepancies (V3), no endpoint omissions (V4), descriptions accurate (V5), dependency and pagination coherence (V6), scope reconciliation (V7), gap ledger clean (V8). ### Shared machinery - **`runner.js`** — the CLI wrapper and check executor shared by all gates: arg parsing, `--schema`/`--only` handling, gap materialisation and deduplication against the ledger, and the exit-code mapping. A check that throws becomes a failed check rather than aborting the run. - **`connector.js`** — connector YAML loader. Flattens the main file plus its `*.s1.partial.yaml` files into one action list, each action tagged with its source file and resource, and keeps the raw text per file for checks the parser would erase (unquoted `{{…}}`, duplicate keys). Nothing here throws — an unreadable connector is a gate failure, not a crash. --- ## state/ ### `run.js` — the run record All durable state for a run lives in `.connector-dev//run.json`, and this script is its **only** writer. Completion is computed from this file, never claimed from console output or memory. ```bash node state/run.js init --provider X --request "…" [--target-phase validate] [--work-dir D] node state/run.js read --provider X # full record node state/run.js summary --provider X # one-line progress node state/run.js set-scope --provider X # scope JSON on stdin (or --file) node state/run.js set-phase --provider X --phase build --status complete|exhausted [--unmet B3,B7] ``` **`targetPhase` — how far the run is meant to go.** `/connector:research` inits with `--target-phase research`; the full flow leaves it at `validate`. Without it a research-only run and a full build that stopped after research leave identical state, so nothing can tell a finished run from an abandoned one — and in a headless session an abandoned run ships no connector at all. Two things read it. `set-phase` replies with `nextPhase` and a directive rather than a bare `{"status":"complete"}` that reads like a finish line, and the Stop hook refuses a turn while any in-target phase is still owed. Also the library behind the gates' gap ledger: reconciliation (gaps no longer reported are resolved), oscillation detection (a gap that reopens `oscillationFlips` times is forced to `not_fixable`), the no-progress policy (`noProgressRounds: 2`), a rewind budget against phase ping-pong, and outcome derivation (`complete` / `partial`). Deliberately no time budget and no iteration cap — loops end on merit. **The no-progress detector.** Every full gate run increments `phases[].iterations` and fingerprints the phase's open, non-minor gaps. Two consecutive rounds with an identical fingerprint means the round resolved nothing, and at `policy.noProgressRounds` the result carries `stalled: true` with the `set-phase --status exhausted` command on stderr. An empty fingerprint is success, never a stall, and any resolved gap resets the counter — so a loop that is genuinely converging is never cut short. ### `narrate.js` — customer-facing progress Prints one line per milestone, computed from the artifacts: ```bash node state/narrate.js --provider ashby --event docs node state/narrate.js --provider ashby --event built --connector-path src/configs/ashby # docs | researched | built | validating | complete ``` ``` ✓ Found Ashby's API docs — 3 resources, 41 endpoints ✓ Researched candidates, notes and applications ✓ Built the connector — 41 actions, 318 parameters ⟳ Re-checking every action against Ashby's documentation… ✓ Complete — no gaps found ``` A script rather than a phrasing guideline for two reasons. The counts come off the research artifact and the connector YAML, so a narrated "41 actions" is 41 actions where a recalled one drifts. And the wording is fixed, where prose guidance produced a different multi-line status block every run — of the shape that immediately preceded three runs ending mid-flow. The completion line will not overstate: it says `2 gaps recorded` or `validate could not be driven to green` when that is the truth. Always exits 0 — narration must never be what fails a run. ### `scope.js` — the scope contract Resolves what the customer asked for into a durable contract across four axes — resources, operations, auth mechanisms, sandbox — each with `resolved` and `dropped` recorded. Named resources are built uncapped; an unscoped request is capped by endpoint count (base 30, up to 50 with the dependency-closure budget) ranked by domain. The cap bounds what is **built**, never what is **discovered**. ```bash node state/scope.js --research --mode explicit|full \ [--resources a,b,c] [--operations search,read] [--auth oauth2,api_key] [--sandbox] \ [--domain ats] [--base 30] [--closure 50] # prints the scope contract as JSON — pipe into: run.js set-scope ``` `--operations` bounds which operations to build on the named resources, for requests that restrict them — "actions for *searching* people and organizations". The words `search`, `read`, `get`, `create`, `update`, `delete`, `write` and `manage` map onto the research `operation` values; `search` resolves to `list`, because a provider's search endpoint is its collection reader whatever verb it is documented under. Excluded endpoints are recorded in `dropped[]` as `operation-out-of-scope`, never discarded, so a later "now add the writes" picks up where this left off. Omit it unless the request genuinely restricts: "including uploading a blob" is emphasis, not exclusion. An unrequested restriction loses real coverage, while a missing one only costs precision. ### `domain-catalog.json` Domain → core-entity catalog consumed by `scope.js` when ranking an unscoped provider's resources. --- ## research/ Deterministic helpers used by the Phase 1 skills. One line each: | Script | Does | |---|---| | `classify_priority.js` | Pass 2 of `list-provider-apis` — deterministic classification: logical paths, intrinsic role, use-case mapping, weighted priority scores, MoSCoW tiers | | `derive_summary_yamls.js` | Phase 5B — derives the 12 summary YAMLs (`api_type.yaml`, `actions.yaml`, `metadata/*` …) from raw research output | | `detect_collisions.js` | Finds resource-name collisions in `api_coverage.json` (same leaf `category`, different parent paths) before they overwrite each other's files | | `identify_category.js` | Phase 2A — keyword-based provider category match; returns `confidence: unmatched` so the SKILL can fall back to inference | | `load_scopes.js` | Collects and deduplicates every documented scope from `research/auth/*.json` | | `mark_investigated.js` | Sets `investigated: true` for one action_id in `api_coverage.json` (all tier arrays) | | `merge_auth_config.js` | Phase 5A — merges per-auth-type JSON files into the consolidated `auth_config.yaml` | | `merge_resource_entry.js` | Upserts a single endpoint entry (stdin or file) into its per-resource YAML | | `resolve_dependencies.js` | Makes the selected set self-consistent — auto-includes the lightest producer for path-parameter dependencies, or flags uncallable actions | | `select_apis_for_investigation.js` | Applies the user's scope-selection rules (count / resources / tier) to `api_coverage.json`, deterministically | | `validate_catalog.js` | Validates the static use-case catalog (universal + category files): required fields, regex compilation, unique ids | | `verify_completion.js` | Phase 4C — compares `selected: true` entries against entries written across per-resource YAMLs (stubs count as written) | | `verify_coverage.js` | Internal-consistency check of `api_coverage.json`, for both Pass-1 (discovery) and Pass-2 (classified) shapes | | `verify_dynamic_use_cases.js` | Validates the Layer 3/4 dynamic use-case files — schema, unique ids, no collisions with Layers 1/2 | | `write_use_cases.js` | Writes a generated Layer 3/4 use-case array (stdin JSON) as a properly-shaped catalog YAML | --- ## corpus/ ### `mine.js` — regenerate the corpus statistics Reads a local checkout of the shipped-connector corpus (never vendored — pass the path) and writes `skills/connector-patterns/references/corpus-stats.json`: leading-token census for actionIds (retained for reference; no gate consumes it), credential-key census (informs A15's synonym table), auth shapes, grant types, refresh coverage, step functions, input types and locations, field types. Every census counts distinct connectors, not raw occurrences — frequency-weighted consensus, because a single connector's habit is as likely a defect as a pattern. ```bash node scripts/corpus/mine.js --corpus "/connectors/src/configs" [--out ] ``` `gates/conventions.js` loads the stats file at startup (override with `CORPUS_STATS_PATH`; falls back to an inline verb list when absent). Re-run after meaningful corpus growth, and consult the census before extending `CREDENTIAL_KEY_SYNONYMS` in `gates/auth.js` — a minority spelling with real adoption (e.g. `apiToken`, 38 connectors) is vocabulary, not a defect. The curated per-case lookup lives in `skills/connector-patterns/references/exemplars.md`. --- ## Top-level helpers - **`lib/safe-path.js`** — `safeName()` / `safeJoin()` used by every script that joins a CLI-supplied name (`--provider`, resource, …) into a filesystem path; rejects traversal and separator characters at the entry boundary.