---
name: quality-test-execution
description: "Use when a locked test plan must execute — triggered by phrases like 'run the full test plan', 'execute test-plan.md', 'run quality gate tests', 'run all the tests', 'execute the test matrix', 'run the integration + E2E pass'. Walks every test type the plan specifies (unit, integration, E2E, smoke, load, contract) and reports per-type pass/fail with traceability back to the plan. Skip when there is no test plan (run quality-test-plan first), or when the user wants ad-hoc testing of a single function/file — that's TDD territory, route to build-tdd."
---

# Test Execution

## Overview

Execute every test defined in the test plan. Map each scenario to pass/fail. Generate a results report with full traceability back to requirements. No skipping.

**Core principle:** The test plan says what to test. This skill executes it ALL. No exceptions, no shortcuts, no "we'll test that later."

**Announce at start:** "I'm using the quality-test-execution skill to execute the full test plan."

## When to Use

- After `quality-test-plan` has produced a test plan document
- During `/feature` and `/greenfield` commands at the test execution phase
- Before deployment (quality gate requirement)

**Not for:**
- Writing tests during implementation (that is build-tdd)
- Generating test plans (that is quality-test-plan)
- Ad-hoc testing (just run the tests directly)

## Prerequisites

Before executing, the following must exist:

1. **Test plan** -- `.forge/work/{type}/{name}/test-plan.md` (the plan to execute)
2. **Implementation code** -- All code from the build-tdd phase
3. **Test infrastructure** -- Test runners, databases, Playwright, etc. configured
4. **All test files written** -- Tests from the plan must exist as actual code

If any are missing, stop. Do not execute a partial test suite.

## The Execution Process

### Step 1: Read the Test Plan

Parse `.forge/work/{type}/{name}/test-plan.md` and extract:

- All test IDs (UT-001, IT-001, E2E-001, SMOKE-001, LOAD-001, CONTRACT-001)
- Their expected locations (file paths)
- Their traceability (which requirements they cover)
- Pass/fail thresholds (coverage %, response time, error rate)

**Verify all test files exist.** If a test from the plan has no corresponding test file, flag it immediately.

```
MISSING TEST FILES:
- IT-003: tests/integration/auth/login.test.ts (file not found)
- E2E-002: tests/e2e/auth/login.spec.ts (file not found)

Cannot proceed. Write missing tests first.
```

### Step 1.5: Codex Mode Check

Now that the test plan is parsed and files are verified, run the Codex consent flow from `protocols/codex.md`. The selected mode applies for the rest of this skill's invocation.

- **Takeover:** Dispatch Codex with the test plan to execute tests and produce the results report. Claude reviews test quality and coverage.
- **Verify** or **Skip / Codex unavailable:** Proceed with the steps below. The Codex Verify note in Step 9 will dispatch Codex to review test quality (Verify only).

### Step 1.6: Oracle Coverage Check

Before any test runs, verify that every oracle in `aiwiki/oracles/` for this work item is mapped to a test in the plan. Oracles are the Phase 5 (harden) contract — production code must reproduce the prototype's verified behavior. This is the load-bearing closure of the mock-pass / real-break wiring gap; running tests without checking oracle coverage re-creates the gap.

1. Enumerate `aiwiki/oracles/*.md`. Filter to oracles whose `prototype_path` is within this work item's scope (typically `pocs/{name}-prototype/`).
2. For each in-scope oracle, find the matching test ID(s) in the test plan's "Map Oracles to Tests" table (produced by `quality-test-plan` Step 2.3).
3. **If any in-scope oracle is unmapped, halt:**

```
ORACLE COVERAGE FAILURE:
Unmapped oracle: aiwiki/oracles/auth-login-success.md
  Type: interaction
  prototype_path: pocs/myapp-prototype/src/auth/Login.tsx
  No test in test-plan.md cites this oracle.

Cannot execute the test plan. Mock-heavy tests pass while real integration
breaks (the wiring gap) — the chain closes only when every oracle has a
test. Resolve by:
  (a) Returning to quality-test-plan and adding a test that satisfies the
      oracle, OR
  (b) Re-running harden if the oracle is stale or out of scope.
```

4. **Skip condition:** If no oracles exist for this work item (the test plan recorded "No oracles in scope" in Step 2.3, or `aiwiki/oracles/` is empty for the scope), record "No oracles in scope" in the results and proceed. The check is non-blocking when oracle capture wasn't part of the lifecycle.

### Step 2: External Service Availability Check

Before executing any tests, determine which external services are available for real testing.

1. **Extract** all external services from the architecture's API contract (`architecture/api-contract.md` — look for external service sections)
2. **Attempt automated connectivity** — hit health endpoints or make minimal requests to each service
3. **If check succeeds** — service confirmed available, record the endpoint
4. **If check fails or is ambiguous** — ask the user: "Is [service] running right now? If yes, confirm the endpoint."
5. **Record the availability matrix** — this feeds into all subsequent test steps

```markdown
## External Service Availability

| Service | Endpoint | Status | Verified |
|---|---|---|---|
| Payment API | api.stripe.com/v1/charges | Available ✓ | Real request returned 200 |
| Email Service | smtp.sendgrid.net | Not available | Connection refused |
```

**Downstream enforcement based on availability:**

| Service Status | Unit Tests | Integration Tests | E2E Tests |
|---|---|---|---|
| **Confirmed available** | Mocks OK | SHOULD use real, mocks require documented justification | **MUST use real service** |
| **Not available** | Mocks OK | Mocks OK, flagged | **BLOCKED** — not faked |

**Note:** This step is skipped during `/hotfix` execution, consistent with existing gate exemptions (smoke tests only).

### Step 3: Execute Unit Tests

```bash
# Run unit tests with coverage
npm test -- --coverage
# or: pytest --cov=src --cov-report=term-missing
# or: go test ./... -coverprofile=coverage.out
```

**Record:**
- Total tests: passed / failed / skipped
- Coverage percentage (overall and per-file)
- Any skipped tests (MUST be justified)
- Failed test details with error messages

**Coverage check:**
```
Coverage: 87% overall
  src/auth/service.ts: 95%
  src/auth/middleware.ts: 100%
  src/auth/validation.ts: 82%
  src/users/repository.ts: 73%  <-- BELOW THRESHOLD (80%)

FAIL: src/users/repository.ts coverage below 80% threshold
```

**Rules:**
- 80% minimum overall coverage
- 100% for critical paths (auth, payments, data validation)
- Zero skipped tests without documented justification
- Zero failed tests (all must pass)

### Step 4: Execute Integration Tests

```bash
# Run integration tests (typically against test database)
npm run test:integration
# or: pytest tests/integration/
# or: go test ./tests/integration/...
```

**Pre-execution setup:**
- Verify test database is running and accessible (use real database, not mocks)
- Run migrations on test database
- Seed test data if required by the plan
- For external services confirmed available in Step 2: use real service (mocks require documented justification)
- For external services not available: verify stubs/mocks are configured before running the suite (suite is flagged as incomplete, but must still execute)

**Record:**
- Total tests: passed / failed / skipped
- API contract compliance (response shapes match)
- Database constraint validation results
- Failed test details with full error output

**Rules:**
- Use real database, not mocks
- Each test cleans up its own data
- Validate against API contract shapes exactly
- All integration tests from the plan must execute

### Step 5: Execute E2E Tests (Playwright)

Dispatch the **e2e-runner** subagent for E2E test execution. The e2e-runner has Bash access, specializes in Playwright, and frees your context from long test output.

```bash
# Run Playwright E2E tests
npx playwright test
# or: npx playwright test tests/e2e/auth/
```

**Pre-execution setup:**
- Verify application is running (start if needed)
- Verify database is seeded with E2E test data
- Configure Playwright browsers (chromium minimum, cross-browser if specified)
- Set base URL and auth credentials for test environment

**Execute each E2E scenario from the plan:**

```typescript
// Example: Execute E2E-001 from test plan
test('E2E-001: User registration flow', async ({ page }) => {
  // Steps from test plan
  await page.goto('/register');
  await page.fill('[name="email"]', 'e2e-test@example.com');
  await page.fill('[name="password"]', 'ValidPass123');
  await page.fill('[name="name"]', 'E2E Test User');
  await page.click('button[type="submit"]');

  // Expected results from test plan
  await expect(page).toHaveURL('/dashboard');
  await expect(page.locator('.welcome-message')).toContainText('E2E Test User');
});
```

**Record:**
- Total scenarios: passed / failed / skipped
- Screenshots captured per step (for visual verification)
- Video recordings of failed scenarios
- Browser(s) tested against
- Timing information per scenario

**Rules:**
- Every E2E scenario from the plan must execute
- Capture screenshots at key steps
- Record video for failed tests
- Test both happy path AND error scenarios from the plan
- No "flaky test" excuses -- if a test flakes, fix it

### Step 6: Execute Smoke Tests

```bash
# Run smoke tests (typically lightweight HTTP checks)
npm run test:smoke
# or: curl-based health checks
```

**Execute each smoke check from the plan:**

```bash
# SMOKE-001: Application starts and responds
curl -f http://localhost:3000/health --max-time 5
# Expected: 200 OK

# SMOKE-002: Authentication works
curl -X POST http://localhost:3000/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"test@example.com","password":"testpass"}' \
  --max-time 10
# Expected: 200 + token in response

# SMOKE-003: Core API responds
curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/users \
  --max-time 10
# Expected: 200
```

**Record:**
- Each check: pass / fail
- Response time for each check
- Any timeout or connection failures

**Rules:**
- All smoke checks must pass
- Timeout thresholds from the plan are hard limits
- Smoke tests must complete within 60 seconds total

### Step 7: Execute Load/Stress Tests

**Only execute if specified in the test plan.** Check the project profile -- not all projects need load testing.

```bash
# Run load tests with k6
k6 run tests/load/registration-load.js

# Or with artillery
npx artillery run tests/load/registration.yml
```

**Execute each load scenario from the plan:**

```javascript
// k6 example for LOAD-001
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    normal_load: {
      executor: 'constant-vus',
      vus: 50,
      duration: '5m',
    },
    peak_load: {
      executor: 'constant-vus',
      vus: 200,
      duration: '2m',
      startTime: '5m',
    },
  },
  thresholds: {
    http_req_duration: ['p(95)<500', 'p(99)<2000'],
    http_req_failed: ['rate<0.01'],
  },
};
```

**Record:**
- Requests per second achieved
- Response time percentiles (p50, p95, p99)
- Error rate under each load level
- Breaking point (if stress test included)
- Threshold pass/fail status

**Rules:**
- Use thresholds from the test plan (not arbitrary values)
- Run against a clean environment (no other traffic)
- Report actual numbers, not just pass/fail
- If thresholds fail, this is a blocking issue

### Step 8: Execute API Contract Tests

```bash
# Run contract tests
npm run test:contract
# or: pytest tests/contract/
```

**Verify each contract from the plan:**

```typescript
// CONTRACT-001: POST /api/users request/response shape
test('POST /api/users matches API contract', async () => {
  const response = await request(app)
    .post('/api/users')
    .send(validUserPayload)
    .expect(201);

  // Validate response shape matches api-contract.md exactly
  expect(response.body).toMatchSchema(userCreatedSchema);
});
```

**Record:**
- Each contract: valid / violated
- Schema mismatches with details
- Missing fields, extra fields, wrong types

**Rules:**
- Validate exact shapes (not just "has some fields")
- Test all status codes defined in the contract
- Any contract violation is a blocking issue

### Step 9: Generate Results Report

Map every test result back to the plan and produce the report.

**Codex Verify:** Before presenting the report, check the mode recorded at Step 1.5. If **Verify** was selected, dispatch Codex to review for tests that pass but test the wrong thing, misleading coverage, and mock-only blind spots. If **Takeover** was selected, skip this step (Codex already ran). If **Skip**, do nothing. Do NOT re-run the consent flow. See **Codex Integration** section below for full details.

**Output to:** `.forge/work/{type}/{name}/test-results.md`

Required sections:
1. **Metadata** — feature, date, duration, link to test plan
2. **Summary table** — each test type: Total / Passed / Failed / Skipped / Status
3. **Coverage report** — per-file and overall, against thresholds
4. **Traceability matrix** — each test ID mapped to requirement and result
5. **Failed tests** — error, screenshot/video path, root requirement traced
6. **Load test results** — p50/p95/p99, error rate, threshold status (if applicable)
7. **Quality gate status** — each gate criteria, pass/fail
8. **External service verification** — from Step 2 availability matrix: service, verified status, real tests run

## Enforcement Rules

### No Skipping

Every test in the plan MUST execute. If a test cannot execute:

1. **Infrastructure missing** -- Set it up. Do not skip.
2. **Test is flaky** -- Fix it. Do not skip.
3. **Test takes too long** -- Optimize it. Do not skip.
4. **External service confirmed available** -- E2E tests MUST use real service. Integration tests SHOULD use real service. Do not mock what is available.
5. **External service not available** -- E2E tests that depend on the service are BLOCKED (not faked, not worked around with pre-existing data). Integration tests may use stubs. Flag the suite as INCOMPLETE.
6. **E2E tests MUST NOT fall back to cached, pre-seeded, or pre-existing data** as a substitute for real service responses. If a real service is slow or times out, the test FAILS — it does not degrade to fake data.

The ONLY acceptable skip is a test that the user explicitly requests to defer, with documented justification.

### No Partial Execution

Do not execute only unit tests and declare success. All test types from the plan must run:
- Unit tests
- Integration tests
- E2E tests
- Smoke tests
- Load tests (if in plan)
- Contract tests (if in plan)

### Failure Handling

| Failures | Action |
|----------|--------|
| 0 failures | PASS -- proceed to deployment gate |
| 1-3 failures | FAIL -- report details, return to build-tdd to fix |
| 4+ failures | FAIL -- report details, may need architecture review |
| Infrastructure failure | BLOCKED -- fix infrastructure, re-run entire suite |
| External service unavailable | INCOMPLETE -- can proceed to PR with flag visible, cannot deploy to production without resolution |

## Quality Enforcement

| Do NOT | DO |
|---|---|
| Run only unit tests | Execute ALL test types from the plan |
| Ignore flaky tests | Fix them — flaky = bug |
| Skip load tests | Run if in the plan |
| Proceed with failures | Zero failures = quality gate |
| Skip E2E screenshots/video | Capture at every key step |
| Test against production data | Use isolated test environment |
| Declare success without traceability | Map every result to the plan |

## I/O Contract

| Field | Value |
|---|---|
| **Requires** | Test plan (`.forge/work/{type}/{name}/test-plan.md`) + implementation code (from `build-tdd`) + architecture artifacts (`.forge/work/{type}/{name}/architecture/api-contract.md` for external service verification) + oracle pages (`aiwiki/oracles/*.md` from harden Step 2.5) when codify ran |
| **Produces** | `.forge/work/{type}/{name}/test-results.md` |
| **Feeds into** | `deliver-deploy` (via quality gate -- zero failures required) |
| **Updates manifest** | `artifacts.test-results: test-results.md`, `phases.quality.test-execution: { status: complete, gate-passed: true }` |

## Codex Integration
**Modes:** Verify or Takeover | **Protocol:** `protocols/codex.md`

- **Verify:** Claude executes tests and reports, Codex reviews test quality.
- **Takeover:** Codex executes tests and reports, Claude reviews results.

**When:** After test execution, when reviewing results.

**Context to pass:**
- Path to `test-results.md` or test output
- Path to `test-plan.md`
- Path to test source files

**What Codex reviews:**
- Tests that pass but test the wrong thing (assertions on mocks, not real behavior)
- Coverage numbers that are misleading (high line coverage, low branch coverage)
- Integration boundaries tested only via mocks

**Prompt focus:** "Review these test results and test source code. Identify tests that pass but don't verify real behavior — especially mock-based tests at integration boundaries. Flag misleading coverage metrics. Which passing tests would still pass if the feature were completely broken?"

**Presentation:** Codex findings presented as "Test Quality Review" alongside pass/fail results.

---

## Integration

**Called by:**
- `/feature` command (test execution phase)
- `/greenfield` command (test execution phase)

**Pairs with:**
- `quality-test-plan` (provides the plan to execute)
- `build-tdd` (provides the tests and implementation)
- `deliver-deploy` (test results are a quality gate for deployment)
- `quality-code-review` (test failures may trigger re-review)