---
name: e2e-testing
description: |
  End-to-end testing of a running web or mobile app through the
  multi-agent-toolkit MCP server: element locators instead of coordinates,
  expect steps that return passed / failed / blocked verdicts with codes,
  secrets by secret_ref, failure traces, saved and replayed step sequences,
  parallel web sessions, and synthetic passport / residence permit / visa
  fixtures for MRZ and document-scan tests. Use when driving an app on a
  simulator, emulator or browser to check a flow, writing a repro for a UI
  bug, or testing an MRZ or document-scan feature.
metadata:
  source: multi-agent-pipeline
  version: 1.0.0
---

# E2E testing with the toolkit

Drive the app, assert exact outcomes deterministically, leave judgement only
to what cannot be compared, and keep a step that could not run apart from a
product that ran and was wrong.

Requires `@mmerterden/multi-agent-toolkit-mcp` ≥ v3.19.0 for everything below
except the base `ios_*` / `android_*` / `web_*` tools. On an older server, read
the UI tree, tap frame centres, and say in the result that the run used the
fallback. Tool inputs come from the server's own schema; the names here are
the contract.

## 1. Locate, then act

| Platform | Locate | Act |
|---|---|---|
| iOS | `ui_locate` | `ios_tap_element`, `ios_type_text` |
| Android | `ui_locate` | `android_tap_element`, `android_type_text` |
| Web | `web_snapshot` / `ui_locate` | `web_click`, `web_type`, `web_select_option` |

Locator order: accessibility identifier, then role plus label, then visible
text. Text is last because it changes with locale and copy edits.

- `LOCATOR_NOT_FOUND`: nothing matched. Read the tree; the element may be
  off-screen (scroll), not loaded yet (wait), or renamed.
- `LOCATOR_AMBIGUOUS`: several matched. Narrow by role or container; never pick
  the first one.

Both are automation failures, not product bugs. Fix the locator and retry once;
still failing, record the step as blocked (automation).

Coordinates (`ios_tap`, `android_tap`) are for what has no element: a canvas, a
map, a chart point.

## 2. Assert with expect steps

Inside `agent_run_steps`, an `expect_text`, `expect_element` or `expect_url`
step checks one exact claim and the run returns a verdict:

| Verdict | Code | Meaning |
|---|---|---|
| `passed` | - | every expect step held |
| `failed` | `ASSERTION_FAILED` | an expect step did not hold: a product failure |
| `blocked` | `CREDENTIAL_MISSING` | a secret_ref named an unknown credential |
| `blocked` | `ENVIRONMENT_UNAVAILABLE` | device, browser or backend not reachable |
| `blocked` | `SEED_DATA_MISSING` | the data the flow needs is not there |
| `blocked` | `BUDGET_EXCEEDED` | the step or time budget ran out |
| `blocked` | `REPLAY_DIVERGED` | a replayed sequence no longer matches the screen |

Report blocked apart from failed. "Checkout is broken" and "the run had no test
account" must never be the same line.

Use an expect step for anything exact: a total, a label, a route, an element
present or absent. A visual or semantic claim (clipping, contrast, "the error
explains what to do") is not an expect step: hand it to a fresh judge that sees
only the claim and the current screenshot or UI tree, never the run that led
there. Inconclusive is a failure, not a pass.

## 3. Secrets

Type credentials with `secret_ref: "<name>"`, never the value. The value never
reaches the transcript, a log, a screenshot caption, a finding or a test file.
A finding names the account (`secret_ref:checkout-user`), not its password.

## 4. Traces, save and replay

- A failed `agent_run_steps` writes a `trace.md`: the steps, the failing step,
  the screen at failure. Attach its path to the bug, not a paraphrase of it.
- `save_as: "<name>"` stores a sequence that reached a state; `replay` runs it
  again. A replay ending in an expect step is a repro: it fails with
  `ASSERTION_FAILED` while the bug exists and passes once it is fixed.
- `REPLAY_DIVERGED` means the app no longer follows the recorded path. Look at
  what changed before re-recording.

## 5. Parallel web sessions

Pass `session: "<name>"` to the web tools to keep sessions apart: one per
explorer, one per test account. Two explorers on one session or one account
report each other's edits as bugs.

## 6. A repro that proves a bug

1. Follow the bug's repro steps with locators.
2. End with the assertion that encodes the expected behaviour.
3. Run it alone. It proves the bug only when it fails with `ASSERTION_FAILED`
   on that assertion. Any other failure (a locator, a timeout, a build error, a
   different assertion) means the test is wrong. A pass means the bug did not
   reproduce.
4. Keep it out of the gating suite until the fix lands; then it is the
   regression test.

In a project test framework the same rule holds: a failing `XCTAssert`,
JUnit / Compose assertion or Playwright `expect` on the encoding line is
`ASSERTION_FAILED`.

## 7. MRZ and document-scan features

`document_mrz_build` and `document_generate` produce synthetic passport (TD3),
residence permit (TD1) and visa (MRV-A / MRV-B) fixtures labelled TEST
SPECIMEN, with every ICAO 9303 check digit computed. Recipes, edge cases and
how to get the image into a simulator: `references/document-fixtures.md`.

## Related

- `/multi-agent:test` and `/multi-agent:bug-bash` run this contract with
  budgets, stop rules and a findings schema.
- `accessibility-compliance-accessibility-audit` for the accessibility half of
  a screen check.