Run mobile tests on real devices in the cloud
Run mobile tests on real devices in the cloud
Website · Documentation · npm
---
` | Geographic IP location (ISO country code, e.g., "US", "DE") |
**Tunnel:**
| Option | Description |
| -------------------------- | ------------------------------------------------------------------------------- |
| `-t, --tunnel` | Start a TestingBot tunnel for this test run (cannot be combined with `--async`) |
| `--tunnel-identifier ` | Identifier for the tunnel, allowing multiple tunnels in parallel |
**Output Options:**
| Option | Description |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--async` | Start tests and exit without waiting for results |
| `-q, --quiet` | Suppress progress output |
| `--json` | Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail |
| `--json-file` | Write results as JSON to a file (default: `_testingbot.json` in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file |
| `--json-file-name ` | Custom path for the JSON results file (requires `--json-file`) |
| `--report ` | Download report after completion: `html`, `html-detailed`, `junit` or `allure` |
| `--report-output-dir ` | Directory to save reports (required with --report) |
| `--download-artifacts [mode]` | Download test artifacts (logs, screenshots, video). Mode: `all` (default) or `failed` |
| `--artifacts-output-dir ` | Directory to save artifacts zip (defaults to current directory) |
**Advanced Options:**
| Option | Description |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--shard-split ` | Split flows into N parallel sessions for faster execution |
| `--retry ` | Retry failed flows up to N times (0-2, default 0). Re-runs only the flows (or shards) that failed, the moment they fail, while the rest of the run continues. Cannot be combined with `--async`. |
| `--ignore-checksum-check` | Skip checksum verification and always upload the app |
> **Note on `--retry`:** a failed flow/shard is retried immediately — as soon as it fails — without waiting for the other flows in the run to finish. Retry attempts appear live in the flow table marked with a `↻` icon. Each flow is retried independently up to N times, stopping as soon as that flow passes. Pass/fail uses the result of the **last** attempt per flow (last-attempt-wins), consistently across the CLI exit code, the TestingBot dashboard, and reports downloaded via `--report`.
**CI/CD Integration:**
| Option | Description |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `--branch ` | Git branch this test run was built from |
| `--commit-sha ` | Git commit SHA associated with this test run |
| `--pull-request-id ` | Pull request ID this test run originated from |
| `--pr-url ` | Pull request URL this test run originated from |
| `--repo-name ` | Repository name (e.g., GitHub repo slug) |
| `--repo-owner ` | Repository owner (e.g., GitHub organization or username) |
| `-m, --metadata ` | Free-form metadata attached to the run and shown in the dashboard (repeatable, e.g. `-m team=mobile -m env=staging`) |
| `--check-name ` | Name the GitHub pull request check `TestingBot / ` so several runs on one commit (iOS, Android) post separate checks that branch protection can require individually |
**Allure reports:** `--report allure` converts each run's results into Allure result files under `/allure-results/`, one JSON per flow with its steps, status and failure details. Render them with `allure serve /allure-results` (requires the [Allure CLI](https://allurereport.org/docs/install/)). Results from several runs or shards accumulate in the same directory.
**Migrating from Maestro Cloud:** the `maestro cloud` spelling of common flags is accepted as hidden aliases, so an existing command line runs unchanged: `--app-file`, `--flows `, `--apiKey`, `--device-model iPhone-17-Pro`, `--device-os iOS-18-2` / `android-34`, `--format JUNIT|HTML`, `--output ` (its directory becomes `--report-output-dir`) and `--test-suite-name`. The canonical flag wins when both are given.
**Examples:**
```sh
# Basic usage
testingbot maestro app.apk ./flows
# Multiple flow directories
testingbot maestro app.apk ./flows/smoke ./flows/regression ./flows/e2e
# With device selection
testingbot maestro app.apk ./flows --device "Pixel 8" --deviceVersion "14"
# Android app on real device with tags
testingbot maestro app.apk ./flows --device "Samsung Galaxy S24" --real-device --include-tags "smoke,regression"
# Tag the test session with groups (visible in the dashboard)
testingbot maestro app.apk ./flows --groups "smoke,critical"
# With environment variables
testingbot maestro app.apk ./flows -e API_URL=https://staging.example.com -e API_KEY=secret
# With companion apps installed alongside the main app (up to 4)
# Each --other-app can be a local file (uploaded) or a tb:// / http(s):// URL (passed through)
testingbot maestro --app main.apk \
--other-app helper.apk \
--other-app tb://existing-appkey \
--other-app https://example.com/mock-server.apk \
./flows
# Download JUnit report
testingbot maestro app.apk ./flows --report junit --report-output-dir ./reports
# Download all artifacts (logs, screenshots, video)
testingbot maestro app.apk ./flows --download-artifacts --build "build-123"
# Download artifacts only for failed tests
testingbot maestro app.apk ./flows --download-artifacts failed --artifacts-output-dir ./artifacts
# Use a custom config file
testingbot maestro app.apk ./flows --config .maestro/ci-config.yaml
# Run in background (async)
testingbot maestro app.apk ./flows --async
# Split flows across 3 shards, grouping all flows over 3 parallel sessions
testingbot maestro app.apk ./flows --shard-split 3
# Retry failed flows up to 2 times (re-runs only the flows that failed)
testingbot maestro app.apk ./flows --retry 2
# CI/CD integration with Git metadata
testingbot maestro app.apk ./flows \
--commit-sha "abc123def" \
--pull-request-id "42" \
--repo-owner "myorg" \
--repo-name "myapp"
```
#### Device matrix
Run the same flows across several devices in a single command. Each cell names exactly one device; there is no cross-product, because not every device exists in every OS version. Every flow runs once per device, so the cost is devices × flows, and the CLI prints that summary before submitting.
```sh
testingbot maestro app.apk ./flows \
--device-matrix "Pixel 9:14" \
--device-matrix "Samsung Galaxy S24:14:real" \
--device-matrix "Pixel 8"
```
Each device becomes its own run with its own results, live table rows and dashboard link; `--json` lists the `device` per run, and `--retry` re-runs only the flow that failed on the device it failed on. If any cell is not a valid device/OS combination the whole request is rejected and nothing runs, so a matrix never partially submits.
#### Organizing flows and subflows
Every top-level flow you pass runs as its own test. A **subflow** (a reusable
flow another flow pulls in with `runFlow`) should **not** be passed as a
top-level flow — if it is, it runs twice: once standalone and once as part of
the flow that calls it.
> Maestro has no notion of a "subflow-only" file. A `.yaml` sitting alongside
> your real flows is a runnable flow, regardless of its name. Naming it
> `*.shared.yaml` does **not** make Maestro treat it as shared.
Recommended structure — keep subflows in their own directory:
```
flows/
login.yaml # top-level, runs
checkout.yaml # top-level, runs
subflows/
sign-in.yaml # only runs when a flow calls it via runFlow
```
```yaml
# flows/login.yaml
- runFlow:
file: subflows/sign-in.yaml
env:
APP_ID: com.example.app
```
Then pass only the directory of top-level flows:
```sh
# Runs login.yaml and checkout.yaml; sign-in.yaml is bundled automatically
# (as a runFlow dependency) but never runs on its own.
testingbot maestro app.apk ./flows
```
When you pass individual files, list only the flows you want to run — their
`runFlow` targets are discovered and uploaded for you:
```sh
# Correct: only the top-level flow. sign-in.yaml is bundled automatically.
testingbot maestro app.apk ./flows/login.yaml
# Wrong: sign-in.yaml would run twice.
testingbot maestro app.apk ./flows/login.yaml ./flows/subflows/sign-in.yaml
```
Other ways to keep a subflow out of a run:
- **`config.yaml` globs** — list only the folders that hold top-level flows
(e.g. `flows: ["*.yaml"]`), leaving subflow folders out of discovery.
- **Tags** — add `tags: [subflow]` to the subflow's header and pass
`--exclude-tags subflow`.
**Preview before you run.** `--dry-run` prints exactly which flows run
standalone and which are bundled as `runFlow` subflows, without spending any
device minutes:
```sh
testingbot maestro app.apk ./flows --dry-run
```
`testingbotctl` also prints a warning if a flow you passed will run more than
once because another top-level flow calls it via `runFlow`.
---
### Upload once, run many times
`testingbot upload` pushes an app once and prints a Project ID. Later runs pass that ID with `--app-binary-id` and skip the upload entirely; each run still gets its own project and results.
```sh
testingbot upload app.apk
# Uploaded app.apk. Project ID: 4321
# Run flows against it with: testingbot maestro --app-binary-id 4321 ./flows
APP_ID=$(testingbot upload app.apk --json | jq -r .appId)
testingbot maestro --app-binary-id "$APP_ID" ./flows/smoke
testingbot maestro --app-binary-id "$APP_ID" ./flows/regression --device "Pixel 9"
```
Every `maestro` run also prints its Project ID after the app upload, so any previous run's ID works with `--app-binary-id` too. Unchanged binaries are deduplicated by checksum on upload as well; pass `--ignore-checksum-check` to force a fresh upload.
**`upload `**
| Option | Description |
| ------------------------- | ---------------------------------------------------- |
| `--ignore-checksum-check` | Skip checksum verification and always upload the app |
| `-q, --quiet` | Suppress upload progress |
`--json` returns `{ provider, appId, file, url }`. Fails with exit code `1` if the upload was rejected.
### Expo / EAS Build
`eas build` produces a download URL rather than a local file, and for iOS simulator builds the artifact is a `.tar.gz` containing the `.app`. Both are handled directly:
```sh
# iOS simulator build from EAS (tar.gz with the .app inside)
URL=$(eas build --platform ios --profile preview --json --non-interactive | jq -r '.[0].artifacts.buildUrl')
testingbot maestro --app-url "$URL" ./flows --device "iPhone 16"
# Android build from EAS
URL=$(eas build --platform android --profile preview --json --non-interactive | jq -r '.[0].artifacts.buildUrl')
testingbot maestro --app-url "$URL" ./flows --device "Pixel 9"
# A local tar.gz works too
testingbot maestro build.tar.gz ./flows
```
Inside an EAS Build or EAS Workflows job, the run is tagged automatically with the EAS build id, profile and platform, and `EAS_BUILD_GIT_COMMIT_HASH` is used as the commit SHA unless `--commit-sha` is given. `testingbot upload ` accepts URLs as well.
### Status, artifacts and list
Commands for working with Maestro projects after they were started, typically together with `--async`. Every command accepts `--api-key` / `--api-secret`, `--debug`, and the `--json`, `--json-file`, `--json-file-name` output flags described under [JSON Output](#json-output).
```sh
# Start tests without waiting and capture the project id
testingbot maestro app.apk ./flows --async --json | jq -r .appId
# Check on it later; --wait blocks with live progress and exits 2 on failure
testingbot status --id 1234
testingbot status --id 1234 --wait
# Fetch reports and artifacts once it finished
testingbot artifacts --id 1234 --report junit --report-output-dir ./reports
testingbot artifacts --id 1234 --download-artifacts failed --artifacts-output-dir ./artifacts
# Browse recent projects
testingbot list
testingbot list --count 25 --offset 25 --json
```
**`status --id `**
| Option | Description |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `-w, --wait` | Block until every run has finished, showing the same live flow table as a foreground run. Ctrl-C detaches without cancelling the runs |
| `-q, --quiet` | Suppress progress output |
Exit code is `0` while the project is still running (JSON `outcome: "running"`), `0`/`2` once it completed, `1` on errors.
**`artifacts --id `**
| Option | Description |
| ------------------------------- | ----------------------------------------------------------------------- |
| `--report ` | Download report: `html`, `html-detailed`, `junit` or `allure` |
| `--report-output-dir ` | Directory to save reports (required with `--report`) |
| `--download-artifacts [mode]` | Download logs, screenshots and video. Mode: `all` (default) or `failed` |
| `--artifacts-output-dir ` | Directory to save the artifacts zip (defaults to current directory) |
Fails with exit code `1` if the project is still running; use `status --wait` first.
**`list`**
| Option | Description |
| ------------------- | ------------------------------------------------- |
| `--count ` | Maximum number of projects to return (default 10) |
| `--offset ` | Number of projects to skip, for pagination |
Projects are listed newest first with id, name, state, run and flow counts. `--json` returns `{ provider, meta: { offset, count, total }, projects: [...] }` with a dashboard `url` per project.
### Espresso
Run Android Espresso tests on real devices and emulators.
```sh
testingbot espresso [appFile] [testAppFile] [options]
```
**Arguments:**
- `appFile` - Path to application APK file
- `testAppFile` - Path to test APK file containing Espresso tests
**Device Options:**
| Option | Description |
| ------------------------------ | ---------------------------------------------------- |
| `--app ` | Path to application APK file |
| `--test-app ` | Path to test APK file |
| `--device ` | Device name (e.g., "Pixel 6", "Samsung.\*") |
| `--platform-version ` | Android OS version (e.g., "12", "13", "14") |
| `--real-device` | Use a real device instead of an emulator |
| `--tablet-only` | Only allocate tablet devices |
| `--phone-only` | Only allocate phone devices |
| `--locale ` | Device locale (e.g., "en_US", "de_DE") |
| `--timezone ` | Timezone (e.g., "America/New_York", "Europe/London") |
**Test Configuration:**
| Option | Description |
| ------------------------ | ----------------------------------------------------- |
| `--name ` | Test name for dashboard identification |
| `--build ` | Build identifier for grouping test runs |
| `--test-runner ` | Custom test instrumentation runner |
| `--language ` | App language (ISO 639-1 code, e.g., "en", "fr", "de") |
**Test Filtering:**
| Option | Description |
| -------------------------------- | --------------------------------------------------------------------- |
| `--class ` | Run tests in specific classes (comma-separated fully qualified names) |
| `--not-class ` | Exclude tests in specific classes |
| `--package ` | Run tests in specific packages (comma-separated) |
| `--not-package ` | Exclude tests in specific packages |
| `--annotation ` | Run tests with specific annotations (comma-separated) |
| `--not-annotation ` | Exclude tests with specific annotations |
| `--size ` | Run tests by size: small, medium, large (comma-separated) |
**Network & Location:**
| Option | Description |
| ---------------------------- | ----------------------------------------------------------- |
| `--throttle-network ` | Network throttling: 4G, 3G, Edge, or airplane |
| `--geo-location ` | Geographic IP location (ISO country code, e.g., "US", "DE") |
**Tunnel:**
| Option | Description |
| -------------------------- | ------------------------------------------------------------------------------- |
| `-t, --tunnel` | Start a TestingBot tunnel for this test run (cannot be combined with `--async`) |
| `--tunnel-identifier ` | Identifier for the tunnel, allowing multiple tunnels in parallel |
**Output Options:**
| Option | Description |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--async` | Start tests and exit without waiting for results |
| `-q, --quiet` | Suppress progress output |
| `--json` | Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail |
| `--json-file` | Write results as JSON to a file (default: `_testingbot.json` in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file |
| `--json-file-name ` | Custom path for the JSON results file (requires `--json-file`) |
| `--report ` | Download report after completion: html or junit |
| `--report-output-dir ` | Directory to save reports (required with --report) |
**Examples:**
```sh
# Basic usage with positional arguments
testingbot espresso app.apk app-test.apk --device "Pixel 8"
# Using named options
testingbot espresso --app app.apk --test-app app-test.apk --device "Pixel 8"
# Real device with specific Android version
testingbot espresso app.apk app-test.apk \
--device "Samsung Galaxy S24" \
--platform-version "14" \
--real-device
# Run specific test classes
testingbot espresso app.apk app-test.apk \
--device "Pixel 8" \
--class "com.example.LoginTest,com.example.HomeTest"
# Run tests with annotations
testingbot espresso app.apk app-test.apk \
--device "Pixel 8" \
--annotation "com.example.SmokeTest" \
--size "small,medium"
# With network throttling and geolocation
testingbot espresso app.apk app-test.apk \
--device "Pixel 8" \
--throttle-network "3G" \
--geo-location "DE" \
--language "de"
# Download JUnit report
testingbot espresso app.apk app-test.apk \
--device "Pixel 8" \
--report junit \
--report-output-dir ./reports
```
---
### XCUITest
Run iOS XCUITest tests on real devices and simulators.
```sh
testingbot xcuitest [appFile] [testAppFile] [options]
```
**Arguments:**
- `appFile` - Path to application IPA file
- `testAppFile` - Path to test ZIP file containing XCUITests
**Device Options:**
| Option | Description |
| ------------------------------ | ---------------------------------------------------- |
| `--app ` | Path to application IPA file |
| `--test-app ` | Path to test ZIP file |
| `--device ` | Device name (e.g., "iPhone 15", "iPad.\*") |
| `--platform-version ` | iOS version (e.g., "17.0", "18.2") |
| `--real-device` | Use a real device instead of a simulator |
| `--tablet-only` | Only allocate tablet devices |
| `--phone-only` | Only allocate phone devices |
| `--orientation ` | Screen orientation: PORTRAIT or LANDSCAPE |
| `--locale ` | Device locale (e.g., "DE", "US") |
| `--timezone ` | Timezone (e.g., "America/New_York", "Europe/London") |
**Test Configuration:**
| Option | Description |
| ------------------- | ----------------------------------------------------- |
| `--name ` | Test name for dashboard identification |
| `--build ` | Build identifier for grouping test runs |
| `--language ` | App language (ISO 639-1 code, e.g., "en", "fr", "de") |
**Network & Location:**
| Option | Description |
| ---------------------------- | ----------------------------------------------------------- |
| `--throttle-network ` | Network throttling: 4G, 3G, Edge, or airplane |
| `--geo-location ` | Geographic IP location (ISO country code, e.g., "US", "DE") |
**Tunnel:**
| Option | Description |
| -------------------------- | ------------------------------------------------------------------------------- |
| `-t, --tunnel` | Start a TestingBot tunnel for this test run (cannot be combined with `--async`) |
| `--tunnel-identifier ` | Identifier for the tunnel, allowing multiple tunnels in parallel |
**Output Options:**
| Option | Description |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--async` | Start tests and exit without waiting for results |
| `-q, --quiet` | Suppress progress output |
| `--json` | Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail |
| `--json-file` | Write results as JSON to a file (default: `_testingbot.json` in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file |
| `--json-file-name ` | Custom path for the JSON results file (requires `--json-file`) |
| `--report ` | Download report after completion: html or junit |
| `--report-output-dir ` | Directory to save reports (required with --report) |
**Examples:**
```sh
# Basic usage with positional arguments
testingbot xcuitest app.ipa app-test.zip --device "iPhone 16"
# Using named options
testingbot xcuitest --app app.ipa --test-app app-test.zip --device "iPhone 16"
# Real device with specific iOS version
testingbot xcuitest app.ipa app-test.zip \
--device "iPhone 15 Pro" \
--platform-version "17.2" \
--real-device
# iPad in landscape mode
testingbot xcuitest app.ipa app-test.zip \
--device "iPad Pro" \
--tablet-only \
--orientation LANDSCAPE
# With localization settings
testingbot xcuitest app.ipa app-test.zip \
--device "iPhone 16" \
--locale "DE" \
--language "de" \
--timezone "Europe/Berlin"
# With network throttling and geolocation
testingbot xcuitest app.ipa app-test.zip \
--device "iPhone 16" \
--throttle-network "3G" \
--geo-location "DE"
# Download HTML report
testingbot xcuitest app.ipa app-test.zip \
--device "iPhone 16" \
--report html \
--report-output-dir ./reports
# Run in background
testingbot xcuitest app.ipa app-test.zip \
--device "iPhone 16" \
--async
```
---
## Common Features
### Real-time Progress
By default, the CLI shows real-time progress updates including:
- Test status updates with actual device names (even when using wildcards)
- Device allocation status
- Live output from Maestro flows
Use `--quiet` to suppress progress output.
### Graceful Shutdown
Press `Ctrl+C` to gracefully stop running tests. The CLI will:
1. Stop all active test runs on TestingBot
2. Clean up resources
3. Exit with appropriate status code
Press `Ctrl+C` twice to force exit immediately.
### Report Downloads
All test frameworks support downloading reports after completion:
```sh
# JUnit XML format (for CI integration)
--report junit --report-output-dir ./reports
# HTML format (for human viewing)
--report html --report-output-dir ./reports
```
### Artifact Downloads (Maestro only)
Download all test artifacts including logs, screenshots, and video recordings:
```sh
testingbot maestro app.apk ./flows --download-artifacts --build "my-build"
```
Artifacts are saved as a zip file named after the `--build` value (or with a timestamp if not provided).
## Exit Codes
| Code | Meaning |
| ---- | -------------------------------------------------------------------------------------------- |
| `0` | All tests passed (also for `--async`, `--dry-run`, and failed tests with `--json-file`) |
| `1` | CLI or infrastructure error: invalid arguments, missing credentials, upload failure, timeout |
| `2` | One or more tests failed |
Distinguishing `1` from `2` lets CI decide whether to retry the job or fail the build.
## JSON Output
`--json` prints one JSON document on stdout and moves all log lines to stderr, so `testingbot maestro app.apk ./flows --json | jq` works. `--json-file` writes the same document to disk while keeping the normal console output. Both flags imply `--quiet`.
```json
{
"provider": "maestro",
"outcome": "failed",
"success": false,
"appId": 1234,
"url": "https://testingbot.com/members/maestro/1234",
"runs": [
{
"id": 5678,
"status": "DONE",
"passed": false,
"device": { "name": "Pixel 6", "platform": "Android", "version": "14" },
"url": "https://testingbot.com/members/maestro/1234/runs/5678",
"flows": [
{
"id": 1,
"runId": 5678,
"name": "login",
"status": "DONE",
"passed": true,
"attempt": 1,
"latest": true,
"startedAt": "2026-01-01T00:00:00Z",
"completedAt": "2026-01-01T00:00:30Z",
"durationSeconds": 30,
"errors": []
}
]
}
]
}
```
- `outcome` is one of `passed`, `failed`, `started` (`--async`), `dry-run`, or `error`. On `error` the document carries an `error` message and the exit code is `1`.
- `flows` (Maestro only) lists every attempt, including `--retry` re-runs. `attempt` counts from 1; `latest` marks the attempt whose verdict counts for the run.
- `runs` is empty for `--async`, `--dry-run`, and errors raised before tests were submitted.
## Documentation
For more information, visit [TestingBot Documentation](https://testingbot.com/support).
## License
MIT