# Test Report Protocol

**Purpose:** Shared protocol for all test report templates (`FeatureTestReport`, and future `APITestReport`, `E2ETestReport`). Defines execution status semantics, defect triage, merge gate algorithm, and metric formulas. Test report templates REFERENCE this file — they do not redefine these rules.

**Usage:** Any agent populating a test report template must Read this file first. Templates signal this via `<!-- PROTOCOL: .tas/rules/common/test-report-protocol.md -->`.

---

## 1. Execution Status Definitions

| Status | Meaning | Severity Required | Defect Link |
|--------|---------|-------------------|-------------|
| ✅ PASS | Assertion ran AND passed | — | None |
| ❌ FAIL | Assertion ran AND failed (code/logic bug) | Required | DEF-ID → RCA + Defect Registry |
| ⏭️ SKIP | Did NOT run — intentional (flag OFF / env unavailable / pre-condition unmet / deferred) | — | None |
| ⛔ BLOCKED | Did NOT run — prior dependency TC already FAIL/BLOCKED. Note which TC blocked it. | — | None |
| 🔴 HOLD | TC exists but fixture not implemented yet (Draft status) | — | None |

**Key distinctions:** FAIL = ran + wrong result. SKIP = never ran (own reason). BLOCKED = another TC's failure caused this. Infra issue: ran + wrong result → FAIL[INFRA]; couldn't run at all → BLOCKED[INFRA].

---

## 2. Metric Formulas (GLOSSARY)

- **execution_mode:** Manual = human tester runs TCs | Automation = CI job runs via framework
- **test_coverage_rate:** (PASS + FAIL) / Total TCs — % of TCs actually executed (SKIP/BLOCKED/HOLD = coverage gap)
- **pass_rate:** PASS / (PASS + FAIL) — quality of executed tests only (excludes SKIP/BLOCKED/HOLD)
- **p0_pass_rate:** (P0 PASS) / (P0 PASS + P0 FAIL) — P0 rows in results table only
- **p1_pass_rate:** (P1 PASS) / (P1 PASS + P1 FAIL) — P1 rows only
- **p2_pass_rate:** (P2 PASS) / (P2 PASS + P2 FAIL) — P2 rows only
- **automation_coverage:** (TCs with Autoable=YES in test spec) / (total TCs) — automation capability scope, NOT pass rate
- **critical_defects:** count of bugs with Severity = HIGH (per § 3 Defect Triage Matrix below; typically Business Logic defects per ISTQB)
- **medium_defects:** count of bugs with Severity = MEDIUM (per § 3 Defect Triage Matrix; typically Unhappy Path, Performance, or blocked functional defects)
- **flaky_tests:** TCs in Automation mode with ≥2 consecutive FAIL→PASS transitions within same CI pipeline execution (single trigger, not cumulative across multiple re-runs) — Manual mode: N/A
- **execution_time:** total elapsed time from CI job start → report generated (Automation only)
- **defect_id:** Manual = ADO#XXXX | Automation = DEF-001 (internal sequence)
- **failure_impact** (results table column): blast-radius of THIS TC failing — BLAST-CRITICAL / BLAST-MAJOR / BLAST-MINOR / BLAST-TRIVIAL. Lookup: P0→BLAST-CRITICAL | P1+has dependents→BLAST-MAJOR | P1 alone→BLAST-MINOR | P2+has dependents→BLAST-MINOR | P2 alone→BLAST-TRIVIAL. Use Func-Test § 4 “Depends” to check dependents. **KEY:** BLAST-* are TC-impact labels, NOT bug Severity. Bug Severity = § 3 Defect Triage Matrix (LOW / MEDIUM / HIGH). Merge gate uses bug_severity only.

**Status legend for metric tables:** rates — 🟢 ≥95% · 🟡 90–94% · 🔴 <90%. P0/P1 are binary: 🟢 100% / 🔴 <100% (per merge gate below). Defect counts — 🔴 ≥1 HIGH · 🟠 ≥1 MEDIUM. Flaky — ⚠️ ≥1. `—` = N/A.

---

## 3. Defect Triage Matrix (Severity SSoT)

> This table is the ONLY severity source for all test reports. Any Defect Registry (in any test report template) and any severity metrics read severity from this table ONLY — they do NOT infer from TC priority. Severity is assigned first by **Defect Type** (business impact), then validated against **Test Type** (category/technique that discovered it) for context.
> (Defect Registry and metrics are typically in template § 2.3 and § 1.1, but those section numbers vary by template.)

### Primary Defect Classification — by Business Impact

| Defect Type | Definition | ISTQB Severity | Block Release | Action | Defer Window |
|-------------|-----------|----------|---|---|---------|
| **Business Logic** | Violates AC, calc wrong, data lost, security breach, blocks user flow | 🔴 **HIGH** | ✅ YES, ALWAYS | Fix before release | None (must fix) |
| **Unhappy Path / Functional Impact** | NEG TC failed (invalid input handling), edge case with impact, failed workaround flow | 🟠 **MEDIUM** | ⚠️ ASSESS | Fix or track in backlog + monitor | Sprint N+1 if applicable |
| **Performance (NFR)** | Misses SLA baseline (see test-design.md Platform NFR Baselines): web LCP <2.5s / app cold start <2s, etc. | 🟠 **MEDIUM** | ⚠️ ASSESS | If critical path: fix. Else: monitor → backlog | Sprint N+1 |
| **UI/UX** | Layout, color, spacing, label, alignment, responsive broken, accessibility gap | 🟢 **LOW** | ❌ NO → defer OK | Design backlog | Sprint N+1 |
| **Edge Case** | <0.1% real path, handled gracefully by validation layer, no user impact | 🟢 **LOW** | ❌ NO → defer OK | Add to backlog | Sprint N+1 |
| **Cosmetic** | Typo, icon misalign, unused var, minor wording | 🟢 **LOW** | ❌ NO → defer OK | Maintenance backlog | Sprint N+2 |
| **Infrastructure** | Staging/CI down, ECONNREFUSED, flaky deploy, Ops issue (NOT feature code) | — (N/A) | ❌ NO → defer to Ops | → Infra backlog (Ops team) | Ops SLA |

### Secondary Classification — by Test Type (Technique Context)

> Reference [test-design.md § Test Techniques](test-design.md) to map TC Category to Primary Technique. Use this to validate Defect Type assignment:

| Test Type (Primary Technique) | When Bug Found | Recommended Severity | Rationale |
|---|---|---|---|
| **Business Logic** (Decision Table, State Transition) | Logic/state bug | → Assign Business Logic defect type | Critical path failure |
| **Validation** (Boundary, Equivalence) | NEG TC failed (invalid input not rejected) | → Assign Unhappy Path type | Input validation is part of BL |
| **Exception** (Negative Testing, Error Handling) | Error not shown / recovery failed | → Assign Unhappy Path type (if blocking) or UI/UX (if UX-only) | Determines user ability to continue |
| **Security** (Input Sanitization, Auth Bypass, PII) | XSS/injection/auth bypass/data leak | → Assign Business Logic type | Always HIGH (security breach) |
| **GUI & UX** (Visual Inspection, UI Manual) | Layout/responsive/accessibility broken | → Assign UI/UX or Accessibility type | User experience, not functionality |
| **Performance** (Lighthouse, Load Time) | Misses SLA baseline | → Assign Performance type; check test-design § Platform NFR Baselines | Measured against quantified NFR |
| **Integration** (API Mock, Webhook) | 3rd-party call fails / data mismatch | → Assign Business Logic (if blocks flow) or Unhappy Path (if gracefully handled) | Depends on recovery mechanism |

**Business Logic test (apply when unsure of Defect Type):** a defect is Business Logic IFF it does ≥1 of —
(1) user cannot complete the required flow, (2) data integrity broken (calc wrong / lost / corrupted), (3) security boundary breached (auth bypass / XSS / data exposure).
Otherwise: Unhappy Path (failed NEG TC, edge case with impact), UI/UX (wording, layout, alignment), Edge Case (<0.1% path, handled), Performance (miss SLA %), or Cosmetic (typo). AC ambiguous → escalate to PE, do not guess.

**Deferred bug** (for merge gate counting): a defect with Status = ⏳ Pending (awaiting fix). Fixed + Verified = not counted (remove from deferred list). **Defect Status = BLOCKED** (fix is awaiting external blocker, not in team's control) → escalate to PE (not counted as deferred; must be resolved before merge). **Note:** "BLOCKED" here is a defect-level status, distinct from "⛔ BLOCKED" TC execution status (§ 1 Execution Status) — same word, different meanings.

**Severity order (ascending):** LOW < MEDIUM < HIGH (use this to evaluate "Severity ≤ MEDIUM" in rules below). Note: ISTQB standard uses HIGH/MEDIUM/LOW; this protocol does NOT use "CRITICAL" or "TRIVIAL" severity levels — those terms appear only in Risk Scoring (test-design.md) and are not defect severity values.

**KEY RULE:**
- **Business Logic defects = NEVER defer** (always HIGH, always blocks release)
- **All other types = defer OK IFF Severity ≤ MEDIUM** (track in defect registry → next-sprint backlog; UI/UX / Edge Case / Cosmetic are always LOW; Unhappy Path / Performance may be MEDIUM or deferred)

---

## 4. Merge Gate Algorithm

### Step 1 — Escalation Check (apply FIRST)

**🔴 FAIL immediately if ANY exist** (skip Step 2):

| Condition | Source |
|-----------|--------|
| Any P0 or P1 TC has status ≠ PASS (FAIL **or** SKIP/BLOCKED/HOLD) | results table |
| More than 2 P2 TCs have status ≠ PASS (i.e., ≥3 P2 not-PASS) | results table |
| Any deferred bug has Severity = HIGH (Business Logic, always blocks) | § 3 above → defect registry |

> **Step 1 vs Step 2 — the key distinction:** Step 1 (Escalation Check) checks raw **execution status** of individual TCs (a SKIPped P0 fails here — it was never proven, even though pass_rate would be undefined). Step 2 (Release Decision) checks computed **pass rate** (SKIP/BLOCKED excluded from denominator). Example: P0 set of 4 PASS + 1 SKIP = 100% pass_rate but FAILS Step 1 escalation check because one P0 TC status ≠ PASS. Step 1 is the gatekeeper; Step 2 only runs if Step 1 finds no triggers.

**If none match → apply Step 2.**

### Step 2 — Release Decision Gate

> Applies only when Step 1 finds no escalation triggers.

| Merge Gate | P0 Pass Rate | P1 Pass Rate | P2 Pass Rate | Deferred Bugs | Decision |
|-----------|------|------|------|---|---|
| **✅ PASS** | 100% | 100% | ≥95% | 0 deferred bugs | Release immediately ✅ |
| **🟡 PASS WITH RISK** | 100% | 100% | 90–94% | ≤2 **deferred bugs** total, ALL Severity = LOW only (Defect Type = Edge Case / UI/UX / Cosmetic per § 3 Defect Triage Matrix) | Release + monitor → validate Step 3 🟡 |
| **🔴 FAIL** | <100% | <100% | <90% | ANY bug with Severity ≥ MEDIUM (incl. Business Logic = HIGH) | Block release, retest 🔴 |

> **Deferred bug threshold (Step 2 only):** Maximum 2 bugs total (not per-type). Threshold is absolute — does not scale with TC count. Step 1 has a separate HIGH-severity check (see Step 1 table above).

### Step 3 — Risk Validation (🟡 PASS WITH RISK only)

**Check ALL of these for each deferred bug:**

| Check | Requirement | Merge Window |
|-------|------------|---|
| **Fix ETA** | Realistic & specific (e.g., "30 min", NOT "TBD") | Local: <30 min; Dev/Staging: <2h; CI: per CI SLA |
| **Owner** | Assigned + available during release window | Must acknowledge + confirm |
| **Re-test TC** | TC ID + verifier + date. Confirm main flow unaffected before merge. | Schedule before merge |

**If ANY check fails → escalate to 🔴 FAIL or escalate to PE for risk acceptance.**

---

## 5. Defect Registry Column Schema

**Columns vary by execution_mode:**
- **Manual mode** (ADO/JIRA owns RCA): `Defect ID | TC ID | Severity | ADO/JIRA Link`
- **Automation mode** (DEF-ID sequence, RCA in report): `Defect ID | TC ID | Severity | Root Cause | Owner | Fix ETA | Status | Fixed By Commit | Verified By | Date Verified | Notes`

**Infra vs feature defect:** classify per § 3 Infrastructure row → defer to Infra backlog (Ops). Does not gate merge.