# @validate.qa/runner

Local test runner for the **[validate.qa](https://validate.qa)** AI testing platform.

This package enables you to execute Playwright and Appium tests generated from your voice narrations directly on your local machine, securely connecting to your validate.qa cloud project.

**Features:**
- 🎭 **Web Testing** - Execute Playwright tests in headless Chromium
- 📱 **Mobile Testing** - Execute Appium tests on iOS Simulator, Android Emulator, AND real devices via USB
- 🔧 **Doctor Command** - Check system prerequisites for web + mobile testing
- 🔐 **Secure Authentication** - Project-scoped runner tokens
- 🌐 **Tunnel Support** - Connect HTTPS dashboards to local devices with token-based auth

---

## Quick Start

### 1. Install (Global or NPX)

You don't need to install it globally if you use `npx`, but it's completely supported:

```bash
npm install -g @validate.qa/runner
```

### 2. Login

Generate a runner token from your validate.qa **Project Settings → Local Runner**. Your credentials will be saved securely to `~/.validate.qa/credentials.json`.

```bash
npx @validate.qa/runner login --token urt_xxxxxxxxxxxx --server https://your-server.com
```

### 3. Start the Runner

Starts the runner daemon, polling for pending test runs from the cloud and executing them locally:

```bash
npx @validate.qa/runner start
```

### Auto-update (required)

Every published runner version can change the client↔server workflow. On
`login` / `start` / `status`, and again every ~15 minutes while running, the
CLI checks the npm `latest` dist-tag for `@validate.qa/runner`. If your
install is behind, the process **exits** with install instructions:

```bash
npm install -g @validate.qa/runner@latest
validate-runner start
```

The server also rejects heartbeats from outdated local runners (HTTP 426) so
they cannot claim new work. Cloud / shared runners are not gated by this.

| Env var | Effect |
|---------|--------|
| `VALIDATE_RUNNER_ALLOW_OUTDATED=1` | Emergency bypass (debug / air-gapped only) |
| `VALIDATE_RUNNER_VERSION_CHECK_MS` | Recheck interval while running (min 60s, default 15m) |
| `RUNNER_CLOUD=true` | Skips the check (shared-runner / cloud images) |
| `MIN_LOCAL_RUNNER_VERSION` *(server)* | Optional floor version if npm is unreachable |

---

## Commands

| Command | Description |
|---------|-------------|
| `login` | Saves credentials for future test executions |
| `start` | Starts the local runner (uses saved credentials or CLI flags) |
| `status` | Pings the server to verify your runner token and connection |
| `logout` | Clears all saved runner credentials from your machine |
| `doctor` | Checks system prerequisites for web and mobile testing |

### Command Options

#### `login`
```bash
npx @validate.qa/runner login \
  --token <token> \        # Runner token (urt_...) from dashboard
  --server <url> \         # validate.qa server URL
  --project <id>           # Optional: Project ID (embedded in urt_ tokens)
```

#### `start`
```bash
npx @validate.qa/runner start \
  --token <token> \        # Override saved token
  --server <url> \         # Override saved server URL
  --project <id> \         # Override saved project ID
  --mobile \               # Enable mobile testing (Appium + bridge)
  --tunnel \               # Enable ngrok tunnel for HTTPS dashboard access (with token auth)
  --mode <mode> \          # Execution mode: playwright-native (default)
  --poll-interval <ms>     # Poll interval: 2000 (default)
```

#### `status`
```bash
npx @validate.qa/runner status
# Shows: Server URL, Project ID, Token (masked), Connection status
```

#### `doctor`
```bash
npx @validate.qa/runner doctor
# Checks: Node.js, Playwright, Xcode, Android SDK, Appium
```

---

## 📱 Mobile Testing

### Overview

The runner supports native mobile app testing on:
- **iOS:** Simulators (macOS) + Real devices via USB
- **Android:** Emulators + Real devices via USB

Tests are executed using **Appium** with XCUITest (iOS) and UiAutomator2 (Android) drivers.

### How It Works

```
┌─────────────────────────────────────────────────────────────┐
│ 1. START RUNNER WITH MOBILE                                 │
│    npx @validate.qa/runner start --mobile                 │
│    → Starts Appium server (port 4723)                       │
│    → Starts mobile bridge (port 9515)                       │
│    → Discovers available devices                            │
└─────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────┐
│ 2. RECORD FROM DASHBOARD                                    │
│    - Open validate.qa dashboard                             │
│    - Start Recording → Select iOS/Android Native            │
│    - Interact with mirrored device screen                   │
│    - Actions captured with screenshots + UI trees           │
└─────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────┐
│ 3. GENERATE TESTS                                           │
│    - Stop recording                                         │
│    - Click "Generate from Recording"                        │
│    - Deterministic compiler creates MobileTestStep[] array  │
│    - Test saved with framework: 'APPIUM'                    │
└─────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────┐
│ 4. EXECUTE TESTS                                            │
│    - Click "Run" in dashboard (or queue from API)           │
│    - Runner claims mobile test run                          │
│    → Boots simulator/emulator                               │
│    → Launches app via Appium                                │
│    → Executes each step (tap, type, swipe, assert)          │
│    → Captures screenshots                                   │
│    → Reports results back to server                         │
└─────────────────────────────────────────────────────────────┘
```

### Prerequisites

#### macOS (Required for iOS)
- **Xcode** - Install from Mac App Store
- **Xcode Command Line Tools**: `xcode-select --install`
- **iOS Simulator** - Included with Xcode

#### Android (Optional)
- **Android Studio** - https://developer.android.com/studio
- **Android SDK** - Installed via Android Studio
- **Environment variables**: `ANDROID_HOME`, `ANDROID_SDK_ROOT`
- **Emulator** - Create via Android Studio → Device Manager

#### Node.js
- **Version**: Node 20 or higher
- **npm**: Latest stable version

### Setup Steps

#### 1. Check Prerequisites
```bash
npx @validate.qa/runner doctor
```

Expected output:
```
✓  Node.js v20.x.x
✓  Playwright installed
✓  Xcode 15.x (iOS SDK)
✓  Android SDK (API 34)
✓  Appium available
```

#### 2. Boot a Simulator/Emulator

**iOS:**
```bash
# List available simulators
xcrun simctl list devices

# Boot a specific simulator
xcrun simctl boot "iPhone 15"

# Or open Simulator.app and boot from menu
open -a Simulator
```

**Android:**
```bash
# List available emulators
emulator -list-avds

# Start an emulator
emulator -avd Pixel_7_API_34

# Or use Android Studio → Device Manager → Start
```

#### 3. Install Your App

**iOS:**
```bash
# Install .app or .ipa on simulator
xcrun simctl install booted /path/to/YourApp.app
```

**Android:**
```bash
# Install APK on emulator
adb install /path/to/YourApp.apk

# Or drag-and-drop APK onto emulator window
```

#### 4. Start Runner with Mobile

**For local dashboard (http://localhost):**
```bash
npx @validate.qa/runner start --mobile
```

**For production dashboard (HTTPS):**
```bash
npx @validate.qa/runner start --mobile --tunnel
```

Expected output (with tunnel):
```
validate.qa Local Runner
─────────────────────────────────────
ℹ  Server:   https://your-server.com
ℹ  Project:  your-project-id
ℹ  Mode:     🎭 Playwright Native + 📱 Mobile 🌐 Tunnel
ℹ  Interval: 2000ms
✓  Browser:  Chrome with user profile
ℹ  📱 Mobile: iOS ✓ | Android ✓ | 2 device(s)
✓  📱 Appium server started
🌐 Public tunnel started: https://abc123.ngrok.io
🔐 Tunnel token: a7f3b2c8d9e1f4a5b6c7d8e9f0a1b2c3d4e5f6a7
   Add this token to your dashboard Project Settings

✓  Runner started - waiting for test runs... (web + mobile)
```

> **Note:** The tunnel token is automatically sent to your dashboard via the server API. You don't need to manually configure it.

#### 5. Record Your First Mobile Test

1. **Open Dashboard** → Your Project
2. **Click "Start Recording"** → Select **"iOS Native"** or **"Android Native"**
3. **Enter App ID**:
   - iOS: Bundle ID (e.g., `com.example.myapp`)
   - Android: Package name (e.g., `com.example.myapp`)
4. **Select Device** from the dropdown (detected automatically)
5. **Click "Start"** - Device screen mirrors in your browser
6. **Interact**:
   - Click on screen → Taps recorded
   - Use swipe buttons → Swipes recorded
   - Click "Type Text" → Keyboard input
   - Back/Home buttons → Navigation
7. **Click "Stop Recording"** when done
8. **Generate Tests** → Click "Generate from Recording"

#### 6. Run Mobile Tests

The **Run Test** button queues Appium tests directly. A queued mobile run starts when:

1. You queue a run from the dashboard
2. The runner is online with `--mobile` flag
3. A matching device is available

The runner will:
- Claim the mobile test run from the queue
- Launch the app on the target device
- Execute each step via Appium
- Capture screenshots after each step
- Report pass/fail results

---

## Runner Modes

### Web Testing (Default)
```bash
npx @validate.qa/runner start
# or explicitly:
npx @validate.qa/runner start --mode playwright-native
```

Executes Playwright tests in headless Chromium with:
- Persistent browser profiles (cookies/logins persist)
- Test variable injection
- AI healing support (optional)

### Mobile Testing
```bash
npx @validate.qa/runner start --mobile
```

Enables:
- Appium server (port 4723)
- Mobile bridge server (port 9515)
- Device discovery and capabilities reporting
- iOS Simulator + Android Emulator support

### Execution Flow Comparison

| Feature | Web Tests | Mobile Tests |
|---------|-----------|--------------|
| Framework | Playwright | Appium |
| Browser/Device | Headless Chromium | iOS Simulator / Android Emulator |
| Locator Strategy | CSS selectors, XPath | Accessibility ID, XPath, Resource ID |
| Actions | click, fill, navigate | tap, type, swipe, back, home |
| Screenshots | On failure | After each step |
| AI Healing | ✅ Supported | ❌ Not supported |
| Cloud Runner | ✅ Supported | ❌ Local only |

---

## Configuration

### Saved Credentials

Credentials are stored in `~/.validate.qa/credentials.json`:

```json
{
  "token": "urt_xxxxxxxxxxxx",
  "serverUrl": "https://your-server.com",
  "projectId": "your-project-id"
}
```

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `SERVER_URL` | validate.qa server URL | - |
| `AUTH_TOKEN` | Runner token | - |
| `PROJECT_ID` | Project ID | - |
| `POLL_INTERVAL_MS` | Poll interval | `2000` |
| `RUNS_DIR` | Directory for test run artifacts | `~/.validate.qa/runs` |
| `CHROME_USER_DATA_DIR` | Chrome profile path | Auto-detected |

### Example: Custom Configuration

```bash
# Run with environment variables
SERVER_URL=https://your-server.com \
AUTH_TOKEN=urt_xxxx \
POLL_INTERVAL_MS=5000 \
npx @validate.qa/runner start --mobile
```

---

## Troubleshooting

### "No devices found"
```bash
# iOS: Boot a simulator
xcrun simctl boot "iPhone 15"

# Android: Start an emulator
emulator -avd Pixel_7_API_34

# Then restart runner with --mobile
```

### "App not installed"
```bash
# iOS
xcrun simctl install booted /path/to/App.app

# Android
adb install /path/to/App.apk
```

### "Bridge not responding"
```bash
# Check if bridge is running
curl http://127.0.0.1:9515/bridge/health

# Restart runner with --mobile flag
```

### "Appium failed to start"
```bash
# Check Appium installation
appium --version

# Reinstall Appium drivers
appium driver install xcuitest
appium driver install uiautomator2

# Run doctor to diagnose
npx @validate.qa/runner doctor
```

### "Tests stay PENDING"
```bash
# 1. Verify runner is online
npx @validate.qa/runner status

# 2. Check runner is started with --mobile
npx @validate.qa/runner start --mobile

# 3. Verify device is booted
xcrun simctl list devices  # iOS
adb devices                # Android
```

---

## Security

### Runner Token Security

`urt_` runner tokens are:
- **Project-scoped:** They cannot access sessions, user accounts, or cross-project data.
- **Revocable:** Instantly cancel tokens from the dashboard if compromised.
- **Client-only:** Your local browser/device instances run locally and post test results back.
- **No code upload:** Tests execute on your machine - source code never leaves your infrastructure.

### Mobile Tunnel Security

When using `--tunnel` for mobile testing from an HTTPS dashboard:

**Security Features:**
- **Token-based authentication:** Each tunnel session generates a random 64-character hex token
- **Per-session tokens:** Tokens are regenerated every time you restart the runner
- **Header validation:** All bridge requests require `X-Bridge-Token` header or `tunnelToken` query param
- **Health endpoint public:** Only `/bridge/health` is accessible without auth (for monitoring)
- **All other endpoints protected:** Device control, screenshots, and actions require valid token

**How It Works:**
```
1. Runner starts with --tunnel
   → Generates random token (e.g., a7f3b2c8d9e1f4a5...)
   → Starts ngrok tunnel to your local bridge
   → Displays token in console

2. Dashboard fetches tunnel URL + token from server API
   → Server receives tunnel info via runner heartbeat
   → Dashboard stores token in memory (never persisted)

3. All bridge requests include token:
   → HTTP header: X-Bridge-Token: a7f3b2c8d9e1f4a5...
   → Or query param: ?tunnelToken=a7f3b2c8d9e1f4a5...

4. Bridge validates token on every request:
   → Invalid/missing token → 401 Unauthorized
   → Valid token → Request processed
```

**Best Practices:**
- ✅ **Safe for production use:** Token is required for all device control operations
- ✅ **Token never exposed to client:** Dashboard fetches from server API, not console
- ✅ **Session-limited:** Token changes each time runner restarts
- ⚠️ **Don't share console output:** The full token is shown in runner console
- ⚠️ **Use HTTPS dashboard:** Prevents man-in-the-middle attacks on token transmission

**For Enterprise Deployments:**
Consider using a paid ngrok account for:
- Fixed domain URLs (no changing URLs)
- IP whitelisting
- Request inspection
- Higher bandwidth limits

---

## Architecture

```
┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   Dashboard     │────▶│   Server API     │◀────│   Local Runner  │
│  (validate.qa)  │     │  (your-infra)    │     │  (your-machine) │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                │                        │
                                │                        ├──→ Playwright (web)
                                │                        └──→ Appium (mobile)
                                │                               │
                                │                               ├──→ iOS Simulator
                                │                               └──→ Android Emulator
                                ▼
                        ┌──────────────────┐
                        │   PostgreSQL     │
                        │   (test metadata)│
                        └──────────────────┘
```

---

## Deploying

### GitHub Actions (Automated)

Publishing is automated via GitHub Actions. Push a tag to trigger the workflow:

```bash
# 1. Bump version in packages/runner/package.json
# 2. Create and push a tag (format: runner-v<version>)
git tag runner-v0.1.11
git push origin runner-v0.1.11
```

The workflow (`.github/workflows/publish-runner.yml`) will:
1. Install dependencies
2. Build shared package
3. Build runner package
4. Publish to npmjs as `@validate.qa/runner`

### Manual (Local)

If GitHub Actions is unavailable:

```bash
# 1. Login to npm
npm login

# 2. Navigate to runner package
cd packages/runner

# 3. Bump version (patch, minor, or major)
npm version patch

# 4. Build and publish
npm run build
npm publish --access public
```

### Version

**Current:** 1.0.36

**Changelog:** See [git history](https://github.com/Validate-QA/validate.qa/commits/main/packages/runner) for the full change log. Recent highlights:

- **1.0.36** - Mobile Appium pipeline hardening (cross-platform runs, launch/reset policy, per-test heal, credential redaction)
- **1.0.35** - Force-upgrade gate: outdated installs exit when a newer package is on npm (workflow changes every release)
- **1.0.0** - First stable release
- **0.1.10** - Rename package to `@validate.qa/runner`; API test capabilities; discovery pipeline refactor
- **0.1.9** - Mobile execution improvements; mobile bridge security hardening
- **0.1.8** - Mobile implementation (Appium 2, device discovery, bridge server)
- **0.1.7** - Security hardening; audio race condition fix; Voice Insights filter fix
- **0.1.6** - Pre-deploy bug fixes
- **0.1.5** - Redesigned local runner UI; publish workflow fixes
- **0.1.2** - Build workflow fix
- **0.1.0** - Initial standalone runner package with secure runner tokens

---

## Support

- **Documentation:** [validate.qa docs](https://github.com/Validate-QA/validate.qa/tree/main/docs)
- **npm Package:** https://www.npmjs.com/package/@validate.qa/runner
- **GitHub:** https://github.com/Validate-QA/validate.qa

---

*Generated by validate.qa*