# Full A-vs-B-vs-C Benchmark Scorecard

> **Note**: This document covers two benchmark runs. The **Opus run is the canonical baseline** going forward.
> The Sonnet run below is preserved as historical comparison data.

---

## Opus Run — Canonical Baseline

> **Benchmark date**: 2026-06-22
> **Rig**: `/design-bench` (#878)
> **Generation model (A & B)**: claude-opus-4-6
> **Judge model**: claude-opus-4-6 (independent)
> **Corpus version**: seed-v1
> **Products**: 5 (Cadence, Tender, Slipstream, Voltage, Plume)
> **Runs per product**: 3
> **Raw data**: `docs/design/fixtures/runs/full-abc-opus/`

| Metric | Value |
|--------|-------|
| **A vs B pairwise win-rate** | **90.0%** |
| **A rubric mean** | **4.11/5** |
| **B rubric mean** | 3.18/5 |
| **A slop count (avg)** | **1.2** |
| **B slop count (avg)** | 3.8 |
| **Judge calibration flags** | 4 (3 from Linear viewport artifact; 1 genuine) |

**Conclusion**: Opus produces tier-level better designs when guided by the harness. The 90% win-rate and 4.11 rubric mean against raw Opus output (arm B) confirm the harness itself is the improvement driver — Opus simply amplifies every layer of the harness more effectively than Sonnet. `claude-opus-4-6` is the validated generation model going forward.

---

## Sonnet Run — Historical Comparison

> **Benchmark date**: 2026-06-22
> **Rig**: `/design-bench` (#878)
> **Generation model (A & B)**: claude-sonnet-4-6
> **Judge model**: claude-sonnet-4-6 (independent)
> **Corpus version**: seed-v1
> **Products**: 5 (Cadence, Tender, Slipstream, Voltage, Plume)
> **Runs per product**: 3
> **Baseline**: [B-vs-C baseline](b-vs-c-baseline.md) (#879)

First full benchmark of the ForgeDock UI Taste Harness (arm A) against raw frontier model
output (arm B) and real reference pages (arm C).

## Headline Results

| Metric | Value | vs Baseline |
|--------|-------|-------------|
| **A win-rate vs B (clean)** | **54.2%** | New metric |
| **A rubric mean (clean)** | **3.40** | +0.30 vs B baseline (3.10) |
| **B rubric mean (clean)** | **3.31** | +0.21 vs B baseline (3.10) |
| **A slop (clean)** | **1.8** | -3.2 vs B baseline (5.0) |
| **B slop (clean)** | **2.9** | -2.1 vs B baseline (5.0) |
| **A vs C rubric gap** | **-1.13** | Closed from -1.69 (baseline B-C gap) |

**The harness works.** Arm A beats raw model output in 54% of head-to-head comparisons, has
64% fewer AI tells (1.8 vs 5.0 baseline), and closes the rubric gap to C by 0.56 points.
But the improvement is inconsistent — high variance across products reveals specific failure modes.

## A-vs-B Pairwise Results (the harness delta)

| Product | A wins | B wins | Ties | A win-rate |
|---------|--------|--------|------|------------|
| Cadence | 0 | 2 | 1 | 16.7% |
| Tender | 2 | 1 | 0 | 66.7% |
| Slipstream* | 3 | 0 | 0 | 100% |
| Voltage | 3 | 0 | 0 | 100% |
| Plume | 1 | 2 | 0 | 33.3% |
| **Clean total** | **6** | **5** | **1** | **54.2%** |
| **All** | **9** | **5** | **1** | **63.3%** |

*Slipstream excluded from clean metrics (arm C viewport artifact).

### Where A dominates

**Voltage**: A won all 3 runs. The harness's archetype commitment produced distinctly different
pages (technical-dense, bold-brutalist, minimal-luxury) where B defaulted to generic dark
developer templates. Run 2 (bold-brutalist, black+yellow) scored 5/5 on hierarchy, typography,
color, and originality — the single best arm A result in the benchmark.

**Tender**: A won 2/3. The harness produced more disciplined, less sloppy output with distinctive
editorial approaches (live request log hero, ledger-line financial doc aesthetic, SVG payment
topology diagram).

### Where A struggles

**Cadence**: B won 2/3. The judge found arm A pages had **sparse content and excessive whitespace** —
the harness's restraint doctrine over-corrected, producing pages that looked incomplete rather
than refined. B's more complete, content-dense pages read as more professional despite higher slop.

**Plume**: B won 2/3. Same failure mode — arm A run 1 was "nearly a stub page" per the judge.
The harness committed to restraint but under-delivered on content density.

**Root cause**: The harness doctrines (minimal-luxury archetype, whitespace rhythm, restraint)
can produce pages that are elegant in concept but empty in execution. The critique loop should
catch this ("is the page complete?") but didn't consistently.

## Rubric Comparison (1-5, clean means)

| Dimension | A | B | C | A-B delta | A-C gap |
|-----------|---|---|---|-----------|---------|
| Hierarchy | 3.50 | 3.67 | 5.00 | -0.17 | -1.50 |
| Typography | 3.92 | 3.50 | 4.80 | **+0.42** | -0.88 |
| Color | 3.42 | 3.25 | 4.80 | **+0.17** | -1.38 |
| Whitespace | 3.33 | 3.17 | 4.40 | **+0.16** | -1.07 |
| Originality | 3.42 | 2.83 | 4.40 | **+0.59** | -0.98 |
| Mobile | 2.83 | 3.42 | 4.20 | -0.59 | -1.37 |
| **Mean** | **3.40** | **3.31** | **4.53** | **+0.09** | **-1.13** |

### Key findings

- **Originality is the biggest A win (+0.59)**: The archetype sampling system works — committed
  design directions (bold-brutalist yellow, editorial serif, minimal sage-green) produce more
  distinctive output than B's template defaults. This was the widest gap in the baseline (-2.17)
  and is now partially closed.

- **Typography improved (+0.42)**: Non-default typeface requirement (Fraunces, Space Grotesk,
  Playfair Display, Syne) produces better typographic quality than B's Inter/system defaults.

- **Mobile is the biggest A loss (-0.59)**: The harness's focus on desktop composition (asymmetric
  layouts, editorial grids, signature moves) may not adapt well to mobile. B's simpler Tailwind
  layouts are inherently more responsive.

- **Hierarchy is flat (-0.17)**: Surprising — the harness should improve hierarchy. The sparse-page
  failure mode (Cadence, Plume run 1) drags this down significantly.

## Slop Comparison

| Arm | Mean (clean) | Baseline B | Improvement |
|-----|-------------|------------|-------------|
| A | **1.8** | 5.0 | **-64%** |
| B | **2.9** | 5.0 | -42% |
| C | **1.0** | 0.5 | — |

The harness nearly halved AI tells from the B baseline. The anti-slop negatives checklist
and archetype commitment together eliminate most template-like patterns. Notable: B also
improved from 5.0 to 2.9 — this is judge variance, not a real B improvement (same HTML files,
different judge agents).

## Per-Product Detail

### Voltage (A dominates: 3/3)

Best harness performance. Run 2 (bold-brutalist) is the benchmark's peak output:
- 5/5 hierarchy, typography, color, originality; 0 slop
- Judge: "Run 2's black/yellow newspaper-scale display type... genuinely distinctive"
- Judge: "A clearly closes the gap between B and C... meaningful work"

### Tender (A leads: 2/3)

Strong harness performance except run 2 (sparse rendering).
- Run 1 (technical-dense, live request log): 4/5 across the board, 1 slop
- Run 3 (editorial, SVG payment topology): 4/5, 1 slop
- Judge: "A closes the gap meaningfully in runs 1 and 3"

### Cadence (B leads: 2/3)

Harness over-indexed on restraint, producing sparse pages.
- Judge: "All three A runs suffer from incomplete content and excessive blank space"
- A slop was actually higher than B here (4.0 vs 3.3) — unusual

### Plume (B leads: 2/3)

Same sparse-page failure. Run 2 (bold-brutalist, black+yellow) was the exception:
- Run 2: originality 5/5, hierarchy 4/5 — genuinely competitive
- But run 1 was "nearly a stub page" — the worst arm A output

## Implications

### What works
1. **Archetype sampling** eliminates mode-collapse. Every A page has a distinctive identity.
2. **Anti-slop checklist** cuts AI tells by 64%. The negatives are mechanically effective.
3. **Non-default typefaces** lift typography quality measurably (+0.42 vs B).
4. **When the harness commits to density** (Voltage, Tender runs 1+3), it produces genuinely
   professional output.

### What needs fixing
1. **Content completeness gate**: The critique loop must verify pages have sufficient content
   density before passing. "Elegant but empty" is a failure mode the current loop doesn't catch.
2. **Mobile responsiveness**: The harness's desktop-first signature moves (asymmetric grids,
   editorial columns) need mobile adaptation in the spec.
3. **Consistency**: A's variance (stdev 0.82-0.99) is higher than B's (0.56-0.70). The harness
   needs guardrails against stub-page outputs.

### Target for next benchmark
- A win-rate vs B > 75% (currently 54%)
- A rubric mean > 4.0 (currently 3.40)
- A slop < 1.0 (currently 1.8)
- Zero stub-page outputs

## Raw Data

### Opus run (canonical baseline)
- Runs: `docs/design/fixtures/runs/full-abc-opus/runs.json`
- Scorecard: `docs/design/fixtures/runs/full-abc-opus/scorecard.json`
- Arm A HTML: `docs/design/fixtures/runs/full-abc-opus/{product}/arm-a/run-{1,2,3}.html`
- Arm B HTML: `docs/design/fixtures/runs/full-abc-opus/{product}/arm-b/run-{1,2,3}.html`

### Sonnet run (historical comparison)
- Runs: `docs/design/fixtures/runs/full-abc/runs.json`
- Scorecard: `docs/design/fixtures/runs/full-abc/scorecard.json`
- Arm A HTML: `docs/design/fixtures/runs/full-abc/{product}/arm-a/run-{1,2,3}.html`
- Arm B HTML: `docs/design/fixtures/runs/full-abc/{product}/arm-b/run-{1,2,3}.html`
- B-vs-C baseline: `docs/design/fixtures/runs/b-vs-c-baseline/`