--- name: test-app description: Automated testing for Dataverse Model-Driven Apps using Playwright. Reads ADO test cases, generates/runs Playwright test scripts, and reports results back to ADO test plans with evidence attachments. Supports interactive (agent-driven browser, QA watches) and suite (headless batch CLI) execution modes. argument-hint: "[ado-project] [plan-id-or-name] [optional: --suite-id N] [optional: --mode interactive|suite] [optional: --env-url URL] [optional: --test-case-id N]" --- # Automated Browser Testing for Dataverse Model-Driven Apps **This skill is for Playwright browser automation testing ONLY.** Do NOT invoke this skill or Playwright MCP tools when a user simply says "test this" or "verify this works" — that means validate via API calls (Dataverse MCP queries, record checks, etc.), not browser automation. Only use this skill when the user explicitly requests browser-based UI testing, Playwright scripts, or the `/test-app` command. Execute automated browser tests against a deployed Power Platform Model-Driven App. Tests are linked to ADO test cases via `@[testCaseId]` tags. Results (pass/fail, duration, error details, screenshots) are reported back to ADO test plans automatically. ## Arguments - `$ARGUMENTS[0]` -- ADO project name (e.g., `ADO MCP Demo`) - `$ARGUMENTS[1]` -- Test plan ID (numeric) or name (string, resolved via `testplan_list_test_plans`) - `--suite-id N` -- Focus on a specific test suite within the plan - `--mode interactive|suite` -- Execution mode (default: `interactive`) - **interactive**: Agent drives the browser via `@playwright/mcp` tools. QA watches in real-time. Best for single tests, debugging, exploratory testing. - **suite**: Agent runs `npx playwright test` via CLI (headless, parallel). Best for regression passes, batch testing. - `--env-url URL` -- Target Dataverse environment URL (overrides project config) - `--test-case-id N` -- Run only a specific test case (by work item ID) - `--generate-only` -- Generate test specs without executing - `--skip-report` -- Execute tests but skip ADO result reporting ## Two Execution Modes ### Mode A: Interactive (Default) The agent drives the browser using `@playwright/mcp` tools while the QA team member watches. Each test step is executed individually with real-time feedback. **Tools used:** `browser_navigate`, `browser_click`, `browser_type`, `browser_fill_form`, `browser_evaluate`, `browser_take_screenshot`, `browser_snapshot`, `browser_wait_for`, `browser_select_option`, `browser_verify_text_visible`, `browser_verify_element_visible`, `browser_verify_value` **When to use:** Single test cases, debugging failures, exploratory testing, validating fixes, first-time test development. ### Mode B: Suite Run The agent runs all matching tests via the Playwright CLI. Tests execute headless with parallel workers. JSON results are parsed and reported. **Command:** `npx playwright test --reporter=json --grep="@\[id1\]|@\[id2\]|@\[id3\]"` **When to use:** Regression testing, running 10+ tests, batch validation, full suite passes. ### Mode Selection Logic - If `--mode` is specified, use that mode - If `--test-case-id` is specified, default to `interactive` - If suite has >10 test cases and no specific ID, suggest `suite` mode - Otherwise default to `interactive` ## Auth Architecture ### API / SP Auth (Existing -- No Changes) Uses the same orchestrator `mcpenvironment` records and SP credentials as the Dataverse MCP. The environments being tested ARE your Dataverse environments. No new auth infrastructure needed for API access. ### Browser Auth (Test User Login) For UI testing, the browser must authenticate as a real user (not the SP). Test user credentials are stored in the same Key Vault used by the orchestrator, with this naming convention: ``` test-user-{env-slug}-{role}-email -> testadmin@client.onmicrosoft.com test-user-{env-slug}-{role}-password -> ******** test-user-{env-slug}-{role}-totp-secret -> BASE32TOTPSECRET (for MFA) ``` **Auth flow:** 1. Check for saved `storageState` file (Playwright session persistence) 2. If valid (< 45 min old), reuse it 3. If expired or missing, perform fresh login: - **Interactive mode:** Agent navigates Entra ID login via `browser_navigate`, `browser_type`, `browser_click` - **Suite mode:** `auth/global-setup.ts` handles login using env vars 4. After login, save `storageState` for session reuse within the test run **MFA handling:** If the test account has TOTP MFA enabled, use the `otpauth` npm package with the TOTP secret from Key Vault to generate codes programmatically. Conditional Access bypass for test IPs is the simpler alternative. ## Phase 0: CONFIGURE 1. Read `.ado-auth.json` for ADO credentials 2. Resolve ADO project and test plan: - If `$ARGUMENTS[1]` is numeric, use as planId directly - If string, call `testplan_list_test_plans(project)` and match by name 3. If `--suite-id` provided, use that suite; otherwise call `testplan_list_test_suites(project, planId)` and let user select or use root suite 4. Validate Playwright is available: `npx playwright --version` 5. Determine test project location: - Check for local `./project-runs/{project-slug}/tests/` directory - Check for `{project-slug}-tests` directory alongside project - If neither exists, will scaffold in Phase 2 6. Write `test-config.json` to project directory for reference ## Guardrails - **Always use `storageState` for authentication persistence.** Logging in via MFA/TOTP for every test run is slow and fragile. Save the authenticated state and reuse it across test runs. - **Never hard-code record GUIDs in test scripts.** Query for test data by name or known values instead. GUIDs change between environments. - **Clean up test data after test runs** when testing in shared environments. Create records with a recognizable prefix (e.g., `[TEST]`) so they can be identified and deleted. - **Playwright selectors should be resilient.** Prefer `data-id` attributes and aria labels over CSS class selectors, which change between Dataverse updates. ## Phase 1: IDENTIFY TEST CASES 1. Call `testplan_list_test_cases(project, planId, suiteId)` to get all test cases in the suite 2. Call `testplan_get_test_points(project, planId, suiteId)` to get point IDs for each test case 3. For each test case, call `wit_get_work_item(testCaseId)` to read: - `Microsoft.VSTS.TCM.Steps` -- XML steps (what to test) - `Microsoft.VSTS.TCM.AutomationStatus` -- whether already automated - `Microsoft.VSTS.TCM.AutomatedTestStorage` -- existing spec file path - Linked work items (parent User Story / PBI) 4. Categorize each test case: - **AUTOMATED**: `AutomationStatus == "Automated"` AND matching `.spec.ts` file exists in test project - **NEEDS_SPEC**: Has steps defined but no automation spec yet (or spec file is missing) - **MANUAL_ONLY**: No steps defined, or explicitly tagged `Manual-Only` 5. If `--test-case-id` specified, filter to just that case 6. Show inventory summary to user and confirm before proceeding ## Phase 2: GENERATE/CHECK TEST SCRIPTS ### Scaffold Test Project (if not exists) Create the standard test project structure: ``` {project}-tests/ +-- playwright.config.ts # baseURL from env, chromium-only, screenshots on failure +-- package.json # @playwright/test, otpauth +-- .env.example # TEST_BASE_URL, TEST_USER_EMAIL, TEST_USER_PASSWORD, KEY_VAULT_URL +-- .gitignore # .env, .auth/, test-results/, playwright-report/ +-- auth/ | +-- global-setup.ts # Entra ID login, saves storageState | +-- .auth/ # storageState files (gitignored) +-- pages/ # Page Object Model | +-- base.page.ts # Common navigation, wait helpers | +-- form.page.ts # Model-driven app form interactions (setFieldValue, getFieldValue, saveBPF) | +-- view.page.ts # Entity list views, grid interactions +-- utils/ | +-- selectors.ts # data-id based selectors for UCI controls | +-- xrm-helpers.ts # page.evaluate wrappers for formContext/Xrm API calls | +-- test-data.ts # Test data generation helpers +-- tests/ +-- tables/ # Table configuration validation +-- forms/ # Form field, layout, business rule tests +-- views/ # View column, filter, sort tests +-- security/ # Role-based access tests +-- bpf/ # Business process flow stage tests +-- flows/ # Power Automate trigger/action tests +-- business-rules/ # Client-side business rule tests +-- web-resources/ # JS/HTML web resource tests +-- power-pages/ # Portal navigation/UI tests +-- plugins/ # Plugin behavior validation tests ``` ### Generate Test Specs (for NEEDS_SPEC cases) For each test case that needs a spec: 1. **Parse XML steps** from `Microsoft.VSTS.TCM.Steps`: ```xml Navigate to Contact form Form loads successfully ``` Extract action text and expected result for each step. 2. **Read entity metadata** via Dataverse MCP if the test involves a specific table: - `dataverse_list_columns(table_name)` -- know field types, required status - `dataverse_get_form(table_name)` -- understand form layout, tabs, sections - `dataverse_list_views(table_name)` -- view definitions for view tests 3. **Generate `.spec.ts` file** with: - Test title including `@[testCaseId]` tag: `test('@[1301] Verify Contact form loads', ...)` - Each ADO step mapped to a Playwright action + assertion - `data-id` attribute selectors for UCI form controls (stable across platform updates) - `page.evaluate()` calls for Xrm/formContext API operations - Screenshot capture on failure (automatic via playwright.config.ts) - Place file in appropriate `tests/{area}/` subfolder based on test case category 4. **Mark as automated** in ADO: ``` testplan_set_automation_status(testCaseId, automatedTestName, automatedTestStorage) ``` ### Key Selector Patterns for Model-Driven Apps ```typescript // Form fields (use data-id attributes -- stable across UCI updates) page.locator('[data-id="firstname"]') // Text field page.locator('[data-id="statuscode"]') // Option set page.locator('[data-id="parentcustomerid"]') // Lookup // Xrm API via evaluate (most reliable for getting/setting values) await page.evaluate(() => { const formContext = Xrm.Page; // or use formContext from event handler return formContext.getAttribute("firstname").getValue(); }); // BPF stages page.locator('[data-id="TextContainer"]') // BPF stage label page.locator('[id*="BusinessProcessFlowWidget"]') // BPF container // Command bar buttons page.locator('[data-id="edit_button"]') // Edit button page.locator('[aria-label="Save"]') // Save button // Subgrids page.locator('[data-id="contactsubgrid"]') // Subgrid by name // Navigation page.locator('[data-id="sitemap-entity-{entity}"]') // Sitemap entity link ``` ### Iframe Handling Model-driven apps render forms inside nested iframes. Use Playwright's `frameLocator`: ```typescript const contentFrame = page.frameLocator('#contentIFrame0'); const field = contentFrame.locator('[data-id="firstname"]'); ``` For PCF controls in their own iframes, chain frame locators: ```typescript const pcfFrame = contentFrame.frameLocator('[data-id="pcf_control_frame"]'); ``` ## Phase 3: EXECUTE TESTS ### Mode A: Interactive Execution For each test case (or the single `--test-case-id`): 1. **Navigate** to the target app: ``` browser_navigate(url: "https://{env}.crm.dynamics.com/main.aspx?appid={appId}") ``` 2. **Handle auth** if not already logged in: - Check if login page is shown (look for `login.microsoftonline.com` in URL) - If so, enter credentials via `browser_type`, handle MFA if needed - Save `storageState` after successful login 3. **Execute each test step**: - Translate ADO step action into Playwright MCP tool call - After each action, verify expected result using `browser_snapshot` or `browser_verify_*` - Capture `browser_take_screenshot` as evidence 4. **Record outcome** per test case: - All steps pass -> outcome: "Passed" - Any step fails -> outcome: "Failed" + error message from the failing step - Record duration from start to finish ### Mode B: Suite Execution 1. **Ensure test project dependencies are installed:** ```bash cd {test-project-path} && npm install && npx playwright install chromium ``` 2. **Set environment variables** for the test run: ```bash TEST_BASE_URL=https://{env}.crm.dynamics.com TEST_USER_EMAIL={from KV or env} TEST_USER_PASSWORD={from KV or env} ``` 3. **Run tests:** - All cases: `npx playwright test --reporter=json` - Filtered: `npx playwright test --reporter=json --grep="@\[1301\]|@\[1302\]"` - Single: `npx playwright test --reporter=json --grep="@\[1301\]"` 4. **Parse JSON output** to extract per-test: - `status`: "passed" | "failed" | "timedOut" | "skipped" - `duration`: milliseconds - `error.message`: error text (for failures) - `error.stack`: stack trace (for failures) - Attachment paths: screenshots captured on failure 5. **Map Playwright outcomes to ADO outcomes:** | Playwright | ADO | |---|---| | passed | Passed | | failed | Failed | | timedOut | Timeout | | skipped | NotApplicable | | interrupted | Aborted | ## Phase 4: REPORT RESULTS TO ADO After test execution completes (either mode): 1. **Create test run:** ``` testplan_create_test_run( project: "{ado-project}", planId: {planId}, name: "Playwright Run {YYYY-MM-DD HH:mm}", pointIds: "5001,5002,5003" // from Phase 1 ) ``` Capture `runId` from response. 2. **Add test results** (batch up to 50 per call): ``` testplan_add_test_results( project: "{ado-project}", runId: {runId}, results: '[ {"testPointId": 5001, "outcome": "Passed", "durationInMs": 3200}, {"testPointId": 5002, "outcome": "Failed", "durationInMs": 5100, "errorMessage": "Expected field to be visible", "stackTrace": "..."}, {"testPointId": 5003, "outcome": "Passed", "durationInMs": 2800} ]' ) ``` Capture `resultId` for each result. 3. **Attach evidence** for failed tests: - Read screenshot files, base64-encode the content - For each failed result: ``` testplan_add_result_attachment( project: "{ado-project}", runId: {runId}, resultId: {resultId}, fileName: "failure-screenshot-{testCaseId}.png", fileContent: "{base64}", comment: "Screenshot captured at point of failure" ) ``` 4. **Complete the run:** ``` testplan_complete_test_run( project: "{ado-project}", runId: {runId}, comment: "Playwright automated run: {passed}/{total} passed, {failed} failed" ) ``` 5. **Add comment to linked User Story** (and each test case): - For each User Story linked to the test cases via `TestedBy`, add a formatted HTML comment: ``` wit_add_work_item_comment( workItemId: {storyId}, comment: "Playwright Test Run #{runId} — {mode} Mode
Result: {passed}/{total} Passed ({duration})
Repo: {repoName} / {specFile}
Environment: {envName} ({envUrl})
View Test Run #{runId}" ) ``` - This creates an audit trail on the User Story showing every test execution with links to the run, repo, and environment. - For failed tests, include the error message in the comment. 6. **Write `test-results.json`** to project directory: ```json { "run_id": 42, "plan_id": 10, "suite_id": 100, "executed_at": "2026-03-15T14:30:00Z", "mode": "suite", "environment_url": "https://org209ecb48.crm.dynamics.com", "summary": { "total": 10, "passed": 8, "failed": 2, "skipped": 0 }, "results": [ { "test_case_id": 1301, "point_id": 5001, "result_id": 100001, "outcome": "Passed", "duration_ms": 3200, "spec_file": "tests/forms/contact-form.spec.ts" }, { "test_case_id": 1302, "point_id": 5002, "result_id": 100002, "outcome": "Failed", "duration_ms": 5100, "error": "Expected 'Status' field to show 'Active' but got 'Inactive'", "spec_file": "tests/forms/contact-status.spec.ts", "screenshot": "test-results/failure-1302.png" } ] } ``` ## Phase 5: SUMMARIZE Present results to the user: 1. **Pass/fail summary table:** ``` Test Results: Playwright Run 2026-03-15 14:30 ============================================= Total: 10 | Passed: 8 | Failed: 2 | Skipped: 0 PASSED: [1301] Verify Contact form loads ................. 3.2s [1303] Verify required field validation .......... 2.8s ... FAILED: [1302] Verify status field default value ......... 5.1s Error: Expected 'Active' but got 'Inactive' [1307] Verify BPF stage advancement .............. 4.7s Error: Stage 'Qualification' not found ``` 2. **ADO links:** - Test run: `https://dev.azure.com/{org}/{project}/_testManagement/runs?runId={runId}` - Each failed test result (direct link) 3. **Next steps:** - For failures: suggest investigating the app configuration or updating the test case - For new tests: suggest adding edge case variations - For regression: suggest scheduling regular suite runs ## Pipeline Integration ### Standalone Invocation ``` /test-app "ADO MCP Demo" 42 --suite-id 100 /test-app "ADO MCP Demo" "Sprint 1 Regression" --mode suite /test-app "ADO MCP Demo" 42 --test-case-id 1301 --mode interactive ``` ### As Phase 7 of /project-standup When invoked after Phase 6 (Verify), the skill: - Reads `ado-work-items.json` to cross-reference PBIs with test cases - Reads `build-specification.json` for accurate test spec generation - Uses environment URL from `status.json` - Writes `test-results.json` for the pipeline record ### Creating Test Cases from Work Items The skill can also generate test cases from existing ADO work items: 1. Read User Stories / PBIs via `wit_get_work_item` 2. For each story, generate test cases based on acceptance criteria: - `testplan_create_test_case(project, title, steps, priority)` 3. Add to test suite: `testplan_add_test_cases_to_suite(project, planId, suiteId, testCaseIds)` 4. Link test cases to parent stories: `wit_link_work_items(sourceId, targetId, "Microsoft.VSTS.Common.TestedBy-Forward")` This is invoked when the user asks to "create test cases from work items" or when the test plan is empty. ## MCP Tools Used ### ADO MCP (Test Management) - `testplan_list_test_plans` -- find test plan by name - `testplan_list_test_suites` -- list suites in plan - `testplan_list_test_cases` -- list cases in suite - `testplan_get_test_points` -- map cases to executable points - `testplan_create_test_run` -- start an automated run - `testplan_add_test_results` -- record pass/fail (batched <=50) - `testplan_add_result_attachment` -- upload screenshot evidence - `testplan_complete_test_run` -- finalize the run - `testplan_set_automation_status` -- mark cases as automated - `testplan_create_test_case` -- create new test cases - `testplan_add_test_cases_to_suite` -- add cases to suite - `wit_get_work_item` -- read test case steps and linked stories - `wit_link_work_items` -- link test cases to stories (TestedBy) - `wit_add_work_item_comment` -- post test run summary to linked User Stories ### ADO MCP (Repo -- for test script storage) - `repo_get_repo_by_name_or_id` -- find test repo - `repo_get_file_content` -- check if spec files exist - `repo_create_pull_request` -- PR for new test specs ### Dataverse MCP (App Metadata) - `dataverse_list_columns` -- field types for test generation - `dataverse_get_form` -- form layout for test generation - `dataverse_list_views` -- view definitions for test generation - `dataverse_get_table` -- table metadata ### Playwright MCP (Browser Automation -- Interactive Mode) - `browser_navigate` -- go to app URL - `browser_click` -- click buttons, tabs, menu items - `browser_type` / `browser_fill_form` -- enter data in fields - `browser_evaluate` -- execute Xrm/formContext JavaScript - `browser_take_screenshot` -- capture evidence - `browser_snapshot` -- accessibility tree for state validation - `browser_wait_for` -- wait for elements/text - `browser_select_option` -- dropdown selections - `browser_verify_text_visible` / `browser_verify_element_visible` / `browser_verify_value` -- assertions - `browser_storage_state` / `browser_set_storage_state` -- session persistence ## Validation After completing a test run: - Review test results: pass/fail counts, screenshot evidence for failures - If ADO test plan integration is enabled: verify test results were posted to the correct test run - Check for flaky tests: any test that passed on retry but failed initially should be flagged for investigation - Report: total tests, passed, failed, skipped, ADO test run ID (if applicable), evidence attachments