--- name: test-mcp-connector description: | ONLY trigger this skill when the user EXPLICITLY asks for MCP-based testing: **Required triggers (ALL must mention "MCP" explicitly):** - "test connector with mcp" - "test mcp connector" - "test [provider] with mcp" - "use mcp to test [provider]" - "run mcp connector test" - "mcp test for [provider]" **DO NOT trigger for:** - Generic "test the connector" requests (use stackone run / test_actions instead) - "test [provider]" without explicit MCP mention - Regular validation or testing requests - Any testing that doesn't explicitly mention MCP This skill builds a REAL agent with Claude Agent SDK that sends natural language prompts to evaluate if action descriptions are agent-friendly. It's more intensive than regular testing and should only be used when explicitly requested. --- # MCP Connector Testing ## Setup (REQUIRED - Collect ALL upfront) | Required | Description | |----------|-------------| | **Account ID** | StackOne account ID (linked provider account) | | **StackOne API Key** | StackOne API key (`credentials:read` scope) | | **StackOne Agent Token** | For the test agent (check `STACKONE_AGENT_TOKEN` env, ask if not set) | | **Profile** | CLI profile for `stackone push` | | **Provider** | Provider name (e.g., `datadog`, `intercom`) | ### ⛔ Account ID = Complete Authentication **If the user provides an Account ID, you have EVERYTHING needed for authentication.** The Account ID is a linked account where provider credentials (API keys, tokens, subdomains, secrets) are ALREADY stored in StackOne. **When Account ID is provided, DO NOT ask for:** - ❌ Provider API keys - ❌ Provider tokens or secrets - ❌ Any provider-specific credentials **Authentication = Account ID + StackOne API Key. Nothing else needed.** --- ## Phase 1: Quick Connectivity Check (Optional) ```bash # Test single action stackone run connectors//.connector.s1.yaml \ --account-id --action-id _ --profile= # Push after changes stackone push connectors//.connector.s1.yaml --profile= ``` --- ## Phase 2: Agent Simulation (PRIMARY) ### Core Principle Test via **agent conversations**, not direct tool calls. The goal: can an agent understand action descriptions and use them correctly from realistic user requests? ```typescript // ❌ WRONG - bypasses agent understanding await callMCPTool("provider_list_items", { path: {}, query: {} }); // ✅ CORRECT - agent decides what to call await agent.chat("Show me all items in the system"); ``` ### Test Agent Setup All test files: `test-agent//` (gitignored) ```bash mkdir -p test-agent/ && cd test-agent/ npm init -y && npm install @anthropic-ai/claude-agent-sdk tsx ``` **Agent code:** ```typescript import { query } from "@anthropic-ai/claude-agent-sdk"; const STACKONE_API_KEY = process.env.STACKONE_API_KEY; const ACCOUNT_ID = process.env.ACCOUNT_ID; async function testPrompt(prompt: string) { const toolCalls: any[] = []; let result: any; for await (const message of query({ prompt, options: { model: "claude-haiku-4-5", mcpServers: { "stackone": { type: "http", url: "https://api.stackone.com/mcp", headers: { "Authorization": `Basic ${Buffer.from(`${STACKONE_API_KEY}:`).toString("base64")}`, "x-account-id": ACCOUNT_ID } } }, allowedTools: ["mcp__stackone__*"] } })) { if (message.type === "assistant" && message.message?.content) { toolCalls.push(...message.message.content.filter((b: any) => b.type === "tool_use")); } if (message.type === "result" && message.subtype === "success") { result = message.result; } } return { prompt, toolCalls, result }; } ``` Run: `cd test-agent/ && npx tsx test.ts` --- ## Phase 3: Prompt Generation Strategy ### ⚠️ CRITICAL: Realistic Prompts, Not Literal Descriptions **DO NOT** generate prompts that mirror the action name or description. Users don't know internal action names. ``` ❌ BAD (too literal): - "List all labels" → list_labels - "Get message by ID" → get_message - "Create a new filter" → create_filter ✅ GOOD (realistic user intent): - "What folders do I have in my inbox?" → list_labels - "Show me that email from John about the contract" → get_message (after search) - "Can you automatically archive newsletters?" → create_filter ``` ### Prompt Variation Strategy For EACH action, generate **3 prompt variations**: 1. **Direct request** - Clear, straightforward ask - "Show me my recent emails" 2. **Contextual/conversational** - Natural speech with context - "I'm looking for that email my boss sent last week about the budget" 3. **Indirect/goal-oriented** - User states goal, not method - "I need to clean up my inbox - what labels are available?" ### Multi-Turn Workflow Tests **REQUIRED**: Test realistic multi-step workflows, not just single actions. **Example Gmail workflow:** ``` Turn 1: "Show me unread emails from today" → list_messages (with query filter) Turn 2: "Open the one from Sarah" → get_message (using ID from previous) Turn 3: "Star it and add it to my Projects folder" → modify_message (add STARRED + label) Turn 4: "Reply saying I'll review it tomorrow" → create_draft (as reply to thread) ``` **Example CRM workflow:** ``` Turn 1: "Find all deals closing this month" → list_deals (with date filter) Turn 2: "What's the total value?" → Agent calculates from previous results Turn 3: "Show me the contact for the biggest one" → get_contact (using ID from deal) Turn 4: "Schedule a follow-up call with them" → create_task or create_activity ``` ### Edge Case Prompts Also test prompts that might confuse the agent: - **Ambiguous requests**: "Check my settings" (which settings?) - **Similar actions**: "Get my email" vs "Get my profile" vs "Get my aliases" - **Compound requests**: "Star this email and move it to archive" - **Negations**: "Remove the label" vs "Add the label" - **Synonyms**: "folders" = "labels", "mark as read" = "modify message" --- ## Phase 4: Iterative Testing Process ### Round 1: Basic Coverage (Quick Pass) 1. Generate 1 simple prompt per action 2. Run all tests 3. Identify failure patterns 4. **Don't fix yet** - just collect data ### Round 2: Fix & Verify 1. Analyze failures by category: - Description unclear → improve `description` and `details` - Wrong tool called → check for naming conflicts - No tool called → action may be too complex for agent 2. Fix connector YAML 3. Push: `stackone push --profile=` 4. Re-test ONLY failed actions ### Round 3: Complex Scenarios 1. Add multi-turn workflows 2. Add edge case prompts 3. Add synonym variations 4. Test prompt variations (3 per action) ### Round 4: Stress Testing 1. Test prompts that SHOULD fail (wrong provider, impossible requests) 2. Test boundary conditions (empty results, large payloads) 3. Test concurrent/rapid requests --- ## Phase 5: Coverage Requirements ### Minimum Requirements - **100% action coverage** - Every active action tested at least once - **Multi-turn tests** - At least 3 workflow scenarios - **Prompt variations** - At least 2 variations for actions that failed initially - **Edge cases** - At least 5 ambiguous/edge case prompts ### Skip Criteria (Document Clearly) Actions can be skipped ONLY if: - **Destructive** - Would delete real data (delete_*, trash_*) - **Side effects** - Would send emails, notifications, etc. - **Requires setup** - Needs data that doesn't exist (untrash requires trashed item) - **Commented out** - Action exists in YAML but is disabled **Always document WHY each action was skipped.** --- ## Phase 6: Fix Loop When an action fails: 1. **Diagnose** the failure: - Wrong tool called → naming conflict or unclear description - No tool called → complex input requirements or missing context - Error returned → API issue or invalid parameters 2. **Fix** the connector YAML: - Improve `description` (short, shown in tool list) - Improve `details` (longer, used for tool selection) - Add keywords that match user vocabulary - Clarify what makes this action unique vs similar actions 3. **Push**: `stackone push --profile=` 4. **Retry** with multiple prompt variations 5. **Document** the fix and move on ### Description Writing Tips ```yaml # ❌ BAD - too generic description: Get a message details: Gets a message by ID # ✅ GOOD - specific and user-oriented description: Retrieve full email content including body, attachments, and headers details: "Fetches complete details of a specific email by its ID. Use this to read email content, view attachments, check headers, or get thread context. Returns subject, sender, recipients, body (text/html), attachment list, and labels." ``` --- ## Output Report ```markdown ## Test Results: connector ### Summary - Total actions: 50 | Active: 45 | Commented out: 5 - Tested: 40 | Skipped (safety): 5 - Initial pass rate: 75% (30/40) - Final pass rate: 95% (38/40) - Fixes applied: 12 ### Test Rounds | Round | Focus | Pass Rate | |-------|-------|-----------| | 1 | Basic coverage | 75% | | 2 | Fix & verify | 90% | | 3 | Multi-turn workflows | 88% | | 4 | Edge cases | 95% | ### Multi-Turn Workflows Tested 1. Email triage workflow (4 turns) ✅ 2. Contact lookup workflow (3 turns) ✅ 3. Filter creation workflow (2 turns) ✅ ### Changes Made | Action | Issue | Fix | |--------|-------|-----| | list_labels | Called for "folders" prompt | Added "folders" synonym to details | | get_message | Confused with get_thread | Clarified "single email" vs "conversation" | | modify_message | Agent tried create_label first | Added "use this to add/remove labels from existing emails" | ### Remaining Failures (Accepted) | Action | Reason | |--------|--------| | update_draft | Complex RFC 2822 input - agent cannot construct payload | | batch_modify | Requires array of IDs from previous context | ### Issue Patterns - Description clarity: 8 fixes - Naming conflicts: 2 fixes - Missing synonyms: 2 fixes ### Files Modified - connectors//.connector.s1.yaml - connectors//.messages.s1.partial.yaml - connectors//.labels.s1.partial.yaml ``` **Required in report:** - Total vs active vs tested (account for commented-out actions) - Pass rates per round - Multi-turn workflow results - Every fix with before/after - Accepted failures with reasoning