---
phase: 3
name: Validation & Benchmarking
description: Re-derive the connector from live provider documentation and reconcile every difference
skills:
  - provider-schema-comparison
agents:
  - provider-comparator
dependencies:
  - phase: 2
    requires: connector YAML built
  - phase: 1
    requires: research data exists
entryConditions:
  - Build phase recorded complete (or exhausted) in run.json
  - Connector YAML exists at connector_path
  - Research data exists in .connector-dev/{provider}/research/
exitConditions:
  - Provider comparison complete (all params verified)
  - No BLOCKING findings
  - validate.js (V1–V8) exits 0
  - Phase recorded via scripts/state/run.js set-phase
---

# Phase 3: Validation & Benchmarking

## Pre-Requisite: Research Data MUST Exist

**FAIL FAST if research data doesn't exist:**

```bash
ls .connector-dev/{provider}/research/endpoints/resources/
```

If missing, run `/connector:research {provider}` first.

**Why?** WebFetch cannot parse JS-rendered API docs (Redoc, ReadMe, Swagger UI). Research phase uses Playwright MCP to properly extract all parameters.

---

## Purpose

Run **provider-schema-comparison** to catch bugs before the connector ships. This is the **KEY BUG FINDER**.

**Key rule:** Process ACTION-BY-ACTION with strict verification to guarantee 100% coverage.

---

## Skill Available

| Skill | What It Catches |
|-------|-----------------|
| `provider-schema-comparison` | **Missing params, type mismatches, name mismatches, missing companions** |

### provider-schema-comparison (KEY BUG FINDER)

Compares connector schema against research data with **strict verification**:

**BLOCKING findings:**
- Missing companion actions (polling_result, async_status)
- Type mismatches (string vs number vs array)
- Name mismatches (from_date vs start_date)
- Missing pagination params

**HIGH findings:**
- Missing filter params
- Missing sort params
- Enum mismatches

**Strict Rules:**
1. NEVER trust existing schema — verify every param name and type
2. Type checking is STRICT — string ≠ number ≠ array
3. Missing companions are BLOCKING — async workflows break

---

## Agent

### provider-comparator (sonnet, 50 turns)

The main validation agent that:
1. **Verifies research data exists** (fails fast if missing)
2. **Processes ACTION-BY-ACTION** (guarantees 100% coverage)
3. **Loads research data** for each endpoint
4. **Strict comparison** — verifies name AND type for every param
5. **Checks companion actions** — polling, async status
6. **Writes shard immediately** after each action
7. **Uses Playwright MCP** for JS-rendered docs if needed

---

## Entry Checkpoint

Phase 3 requires:
- [ ] `run.json` records the build phase as `complete` (or `exhausted` with a gap list)
- [ ] Connector YAML exists at `connector_path` (from `build-state.yaml`)
- [ ] **Research data exists** in `.connector-dev/{provider}/research/`

---

## Execution Flow

### Step 1: Verify Research Data (FAIL FAST)

```bash
ls .connector-dev/{provider}/research/endpoints/resources/

# If not found, STOP:
echo "ERROR: Research data not found."
echo "Run: /connector:research {provider}"
```

### Step 2: Provider Schema Comparison

```
Load skill: provider-schema-comparison
Spawn agent: provider-comparator (sonnet, 50 turns)

PROCESS ACTION-BY-ACTION:

FOR each partial file:
  FOR each action:
    1. Read action from OUR schema
    2. Load research data for this endpoint
    3. STRICT comparison:
       - Verify every param NAME matches exactly
       - Verify every param TYPE matches
       - Check for missing params
    4. Check companion actions (polling, async)
    5. WRITE shard file immediately
  END FOR
END FOR

Aggregate all shards into final report
```

### Strict Verification Rules

**Rule 1: NEVER Trust Existing Schema**
```
❌ Wrong: "We have from_date, provider has date param, looks good"
✅ Right: "We have from_date, provider says start_date → NAME MISMATCH"
```

**Rule 2: Type Checking is STRICT**
```
string ≠ number ≠ integer ≠ array
team_id (string) vs teams (array) → TYPE MISMATCH + NAME MISMATCH
```

**Rule 3: Missing Companions are BLOCKING**
```
If we have: polling_onboard_employee
Must have:  polling_onboard_result → BLOCKING if missing
```

---

## Exit Checkpoint

Phase 3 is complete when:
- [ ] `provider-comparison.json` exists
- [ ] `shards/` directory with per-action evidence
- [ ] **No BLOCKING findings**
- [ ] All HIGH findings documented

---

## Artifacts Produced

```
.connector-dev/{provider}/validation/
├── provider-comparison.json       # Aggregated findings
├── provider-comparison.md         # Human-readable report
└── shards/                        # Per-action evidence
    ├── list_employees.json
    ├── get_employee.json
    └── ...
```

---

## Resume Support

If agent times out:

```bash
# Re-run - skips completed actions
/connector:validate {provider}

# Force fresh run
rm -rf .connector-dev/{provider}/validation/shards/
```

---

## Phase Transition

Validation is the **final phase** — there is nothing after it. Record the transition —
`run.json` is written only by `scripts/state/run.js`, never by hand:

```bash
node ${CLAUDE_PLUGIN_ROOT}/scripts/gates/validate.js \
  --provider {provider} --connector-path "$CONNECTOR_PATH"   # V1–V8, must exit 0

node ${CLAUDE_PLUGIN_ROOT}/scripts/state/run.js set-phase \
  --provider {provider} --phase validate --status complete
# or --status exhausted [--unmet V3,V5] when two consecutive rounds resolve nothing
```

`run.js` then derives the run's terminal outcome: **complete** (all three gates passed) or
**partial** (at least one phase exhausted — connector delivered with its gap list). The
connector is built and verified against documentation only; it is never deployed or
executed against a live account in this flow.