# Evaluation Templates `evaluation` reads every `dimensions//_summary.md` (its **provenance frontmatter** + Findings) and reconciles them into **cross-dimension** documents under `research//evaluation/`. The value is the reconciliation — cross-checking views into one coherent picture, **weighted by how each claim was obtained**, with disagreements flagged. > This file governs the **four reconciliation rollups** (technology-architecture, product-features, data-model-api-surface, competitive-positioning) + the README. The three **experiential** docs that also live under `evaluation/` — `information-architecture.md`, `ux-flows.md`, `feature-coverage.md` — are produced by **Mode 4 (Cartography)** and governed by `cartography.md`; Evaluation's rollups are their inputs (the entity spine → IA surface cards; the feature map → the coverage matrix), and Cartography's coverage scorecard feeds Mode 5's experiential-coverage axis. **Three rules for every evaluation doc:** 1. **Anchor + tag every claim** with its dimension _and provenance_: `(api: raw/endpoint-catalog.md · openapi-verbatim · high)`. The method + confidence come verbatim from the source dimension's `_summary.md` frontmatter. 2. **Weight, don't average.** A claim's confidence = the **max** of its sources' confidence, **promoted one band only if corroborated by ≥2 _independent_ dimensions**, capped at source confidence when single-source. Conflicts are **flagged, never averaged**. **Tiebreaker — direct observation outranks inference.** When two sources conflict on the _same_ fact, a directly-observed value beats an inferred one regardless of dimension or stated confidence (a session-observed backend engine beats a DNS-inferred one; a live HTTP probe beats a single NXDOMAIN result). The inferred side is rendered tentative and never used to override the observation or to issue cross-dimension directives. 3. **Cross-check, don't concatenate.** Agree → state as fact. Disagree / single-source / low-conf → render hedged ("observed but unverified…") and flag in the doc's reconciliation section. **Independence (for the promotion rule):** `codebase`, `api`, `session`, `wire-capture`, `distribution-artifacts` are mutually independent and independent of marketing. **`website` and `docs` are NOT independent** for _feature claims_ (docs often restate marketing) — two of them agreeing is still one source. A bundle-declared route + a session-observed route ARE independent. **Same artifact ⇒ one source.** Two dimensions that string-mine the _same_ static asset (one bundle / one spec / one binary) are **one source** — never a band-bump — even when they report different counts; the divergent counts are a **reported range, not a flagged conflict** (`deployed-client-bundle` 278 vs `website` 361 over one bundle is one finding with a range, not corroboration and not a conflict). **OSS distribution clients are an independent corroboration lane.** An API-surface claim resting on a single _third-party_ wrapper (a community MCP server, an unofficial SDK) is tentative — but a **first-party open-source integration client** (`distribution-artifacts`: a Unity/Figma/CLI plugin or published SDK source) that calls the same routes is **independent of both marketing and the third-party wrapper**, and corroborates it → promotes. (A legacy/granular route family seen only in a third-party MCP becomes a real finding once the official OSS client calls it too.) **Open-core lineage as an absence oracle (independent corroboration of the open/closed seam).** When a target is an **open-core product with a closed cloud edition** (a clonable `codebase` lineage repo + a closed `session` runtime), a feature **verified ABSENT in the lineage repo** (grep/structure-map) **and observed PRESENT in the cloud session** is **two independent lanes agreeing on the _seam_** → promotes the open/closed-boundary claim to fact (the cloud-feature value itself stays a direct `session` observation). The lineage repo is as useful as an _absence map_ (what is NOT in the open floor) as it is a presence map; "verified-absent-in-CE + observed-in-cloud" is independent of marketing and of the session alone. (Origin: Akeneo — Reference Entities / DAM / Rules / Workflows observed in Serenity + grep-absent in Community Edition.) **Closed-feedback confidence cap.** When the `community` dimension finds **no first-party feedback loop** (no public issues / roadmap / changelog / forum — feedback funnelled to a gated channel), its user-pain / most-requested-feature claims are **inference-from-changelog only → capped at `low`/tentative**, never promoted on changelog frequency alone. The surface that _would_ corroborate user sentiment is `external-reputation` (a deferred dimension) — flag it as recommended-for-this-run when this cap fires (see discovery Part C deferred-dimension trigger). **Write-side-unobserved cap.** A `session` dimension carrying `write_side_observed: false` (a Pass-1-only run) has observed feature _existence/gating_ but **not** feature _execution behavior_. Any marketed generate/submit/launch _behavior_ claim therefore stays **tentative and is never promoted to fact** from marketing/docs alone — render the write-side gap **once** from the frontmatter flag, not re-hedged per rollup. > **Drift between a target's own sources is a FINDING about the target, not a defect in the capture.** > When the docs say one thing and the runtime does another — a documented field the API doesn't return, a > published limit the wire contradicts, a spec route that 404s — **do not treat this as an error to > reconcile away, and do not go hunting for which source is "really" right.** Both captures are faithful; > the product is incoherent. Record both verbatim (each anchored to its own `raw/` artifact), state the > divergence plainly, and **name it as a product finding** — documentation drift, an aspirational spec, a > deprecated-but-published surface, or a versioning gap between the docs and the deploy. > > This is often among the most valuable things a run produces: it is exactly what an integrator would hit > on day one and what no vendor advertises. The harness's job is **faithful capture with provenance**, not > adjudicating which of a company's own statements is true. A run that flattens a real inconsistency into a > single tidy claim has destroyed a finding and manufactured a false coherence. **Presentation mapping:** high → stated as fact; low/single-source → hedged as tentative; conflicted → flagged. (This is the "lighter" form of the provenance-weighting — done inline in the prose, not via a separate `claims.yaml`.) --- ## 1. `evaluation/technology-architecture.md` ```markdown # — Technology & Architecture ## TL;DR (3–5 sentences; the single most distinctive choice) ## Stack (languages/frameworks/build/runtime — table, each anchored+tagged) ## Architecture (client/server split, packages+deps, runtime/sync/data model, transport) ## How it's built (monorepo layout, public/private, codegen, CI, release cadence) ## Notable design choices (the 3–5 worth stealing + why) ## Cross-dimension reconciliation (agreements/disagreements + confidence per claim) ## Open questions ``` ## 2. `evaluation/product-features.md` ```markdown # — Product Features & Capabilities ## What it does (one paragraph) ## Feature map (every capability · user · maturity GA/beta/docs-only — anchored+tagged) ## User-facing surfaces (dashboard/editor/components/CLI/widgets) ## User journeys (the 2–3 primary flows end-to-end) ## Gaps & rough edges (from community signal) ## Cross-dimension reconciliation (marketing claims vs. what code/docs actually support — flag over-claims) ``` ## 3. `evaluation/data-model-api-surface.md` ```markdown # — Data Model & API / SDK Surface ## Core entities (domain objects + relations; ER-style — when the runtime exposes an RBAC verb taxonomy, derive the entity set from it: `X_CREATE/VIEW/EDIT/DELETE` ⇒ entity X) ## Integration surface (SDK/packages public API · REST/HTTP catalog · realtime/wire protocol) ## Published-vs-app-own API diff (emit WHENEVER the target publishes a developer/partner/MCP API — common for mature SaaS, but not universal. The app the SPA runs on is frequently a different, broader, differently-authed API than the published one: list ops the app uses that the public API omits, AND public ops the app never calls — both are findings. If there is NO published API, say so explicitly: "no published API — the app surface is the only surface" is itself the finding, not an empty section. REQUIRES paired inputs (ingestion §6): the pre-nav in-page tap [session app-own paths] + the unauth bundle/route-manifest [published paths]. Missing a lane → the diff is un-feedable; say which lane is absent rather than implying overlap.) ## Auth & tenancy model (token type, where it lives, scope expression) ## API-path diff (adaptive, N-way over the runtime sources present) (the seam rule: bundle-declared vs session-observed vs wire-observed. Title it for the sources ACTUALLY run — enumerate only those lanes; an absent lane [session blocked / wire folded] is a recorded gap, NOT a structural hole to apologize for. Dormant/unshipped routes are findings.) ## Cross-dimension reconciliation (published surface [api/packages/docs] vs observed wire [session/ wire-capture]; document drift — published=fact, observed-drift=tentative) ## Open questions ``` ## 4. `evaluation/competitive-positioning.md` ```markdown # — Competitive Positioning ## Positioning (their pitch + market — website/blog) ## Pricing & packaging (plans, what gates what, unit economics — website pricing + platform/limits) ## Differentiation (claimed moat vs the discovery's read of the real moat — often infra/data) ## Alternatives & comparison (named competitors + how it differs) ## Strategic read (direction from changelog/blog) ``` --- ## 5. `research//README.md` — the run story ```markdown # — Recon Run > One paragraph: what it is + why we ran recon. ## Access grade (the resolved vector — source/runtime/auth/presence — + matched case[s]) ## Dimensions covered (table: dimension | status | method | confidence | headline finding) ## Headline findings (5–10, each anchored+tagged) ## How to navigate (00-scope-verdict.md, 00-recon-plan.md, dimensions/, evaluation/ [4 reconciliation rollups + Cartography's information-architecture / ux-flows / feature-coverage], 05-self-correction-proposal.md) ## Lessons (candidate methodology learnings → feed self-correction) ## Reproduction ``` --- ## Evaluation method 1. Read every `dimensions/*/_summary.md` frontmatter first (the provenance index) — note method + confidence + status + gaps per dimension. 2. Read Findings; drop into specific `raw/` artifacts only when a claim needs a precise value. 3. Draft the four rollups + the README — each rollup may be drafted by its own sub-agent dispatched from the **main session** (overlapping reads, independent writes), then a single reconciliation pass keeps them consistent. 4. Tag every claim with provenance; weight per the rules above; flag every disagreement; record every gap. A confident doc built on one low-confidence dimension is worse than an honest "single-source, unverified." > **Classifying a raw observation** — to turn a storage-key prefix, an auth tell, an envelope shape, or a realtime frame into a _named_ stack finding, use the fingerprint tables in `tradecraft.md` (storage→stack, auth-provider, API-envelope, realtime-trap). They name what a finding _is_; this file governs how confidently to _state_ it.