# optbrew outcome measurement contract (evaluation 0.5.0)

Readiness scores and actual outcomes stay separate. `ScoreInput` contains the readiness observations of every registry family. `MetricObservation[]` contains owner-provided native search, analytics, and AI panel observations. Neither indexing, search traffic, citations, conversions, nor revenue is inferred from readiness. The official source IDs and URLs are in `docs/superpowers/specs/2026-10-13-optbrew-sources.md` in the source repository. The rules below follow approved design section 7.

## Comparable readiness inputs

`compare(before: ScoreInput, after: ScoreInput|null): ComparisonResult` uses Task 3 `validateEntity`, `freezeProtocol`, `compileUnits`, and `score`. It does not define a new protocol hash or scoring calculation. The result is `{status,score_deltas,reasons}`. Status is `comparable`, `not_comparable`, or `no_after`. `score_deltas` is empty unless status is comparable. The caller must keep a changed evaluation version as a separate result and re-evaluate preserved raw evidence under one version before making a comparison.

The Task 3 hash fixes target, environment, evaluation and protocol versions, selected URLs and page types, applicability conditions, engines, questions, surface, model, search use, locale, region, browser, and device. A changed hash returns `not_comparable` with field reasons. Even under the same hash, changing applicable unit keys or weights returns `applicable_units_changed`. Run ID, timestamps, and evidence paths do not change comparability. Evidence truth remains a collector/reviewer responsibility.

Each comparable profile delta is `{profile,before,after,raw_difference,percentage_point_delta,relative_change,coverage_percentage_point_delta,A_delta,M_delta,N_delta}`. `before` and `after` are **unrounded** `100N/M`; `raw_difference=after-before`. These are percentage-valued scores, so `percentage_point_delta` has the same numeric value. `relative_change=(after-before)/before` and is null when the previous value is zero or either side is unavailable. The deltas of `A`, `M`, and `N` retain Task 3 raw weighted units. Coverage uses unrounded `100M/A`. The displayed one-decimal `ScoreResult.score` is never subtracted for a delta. A comparable observation change is not proof that an edit caused it.

Run stored fixtures locally: `node skills/optbrew/scripts/cli.mjs compare --before tests/fixtures/scoring/before.json --after tests/fixtures/scoring/after.json --output .optbrew/test-compare.json`. The CLI validates before writing and does not call a provider.

## Native provider imports

`normalizeImport(provider, rows, metadata): MetricObservation[]` accepts `gsc`, `ga4`, `google_ai`, `bing_ai`, `naver`, or `se_ranking`. Rows retain their native `metric_id`, `value`, optional numerator and denominator, `provider_definition`, `definition`, display state and missing mask. The output names metrics as `provider:metric_id` so unlike definitions never collapse into one metric. GSC clicks, impressions, CTR and average top position are different native measures; CTR means clicks divided by impressions under the selected GSC filters. Page and property aggregations are distinct. GA4 conversions require the chosen event, property and date filters; clicks and citations do not estimate them. Google AI Overviews and AI Mode observations retain their report filter. Bing citations, cited pages and sample grounding queries retain Bing's scope. Naver exposes only owner-accessible native fields. SE Ranking SOV retains its provider definition and selected prompt population. No provider metric is converted to authority, answer accuracy, or rank.

`metadata` needs `timezone`, `window:{start,end,timezone}`, `collected_at` with offset, `sample_selection:{kind:'all'|'positive_only'|'unknown'}`, `source_ref`, `evidence_ref`, and a `provider_definition` unless each row supplies one. The output `filters` has `{provider,property,hostname,path,query,search_type,device,country,aggregation,collected_at,export_scope,...native_filters}`. Its top-level dimensions use `row.filters` first, then `metadata`, then the literal `unknown`. `export_scope` separately retains the original metadata `{property,hostname,path,query,search_type,device,country,aggregation}` with `unknown` for absent fields. For example, a mobile row inside an export labelled desktop is `filters.device:'mobile'` and `filters.export_scope.device:'desktop'`; consumers must not merge these cohorts. A row cannot contradict an explicit export `property` or `aggregation`; such input is rejected. Query scope differences remain visible in both fields. `window` uses local calendar dates and the provider timezone. `completeWindows(lastCompleteDate,timezone)` constructs two adjacent complete 28-day windows from a caller-supplied final complete local date. It does not decide when a provider's data became complete. Preserve campaign and seasonal context outside the rate itself.

Google's AI report may export displayed `~` or `-` as zero. A zero with no explicit nonmissing mask, or a row with those display states, becomes `value_state:'zero_or_missing',value:null`. A known unmasked zero sets `missing_mask:false` and remains measured. Missing or unsupported values are `unknown`; `not_applicable` requires an actual exclusion. `positive_only` exports cannot create an all-question citation response rate. They retain their selection bias in `sample_selection` for report rendering.

## Fixed AI response panel

`measurePanel(panel,responses,evaluations): MetricObservation[]` returns six metrics in order: `target_citation_response_rate`, `target_entity_mention_rate`, `citation_support_rate`, `fact_consistency_rate`, `observed_brand_share`, and `observation_completion_rate`. `panel.panel_hash` must be the existing `freezeProtocol(run_context).protocol_hash` from the owning run. This module never creates another score or hash format. `panel.questions` fixes distinct question IDs; `repeats` fixes planned slots. Each response identifies one `(question_id,repeat)` and a unique response ID. Valid responses provide actual citation URLs, evaluated entity mention, and brand mentions. Invalid or absent slots remain in the planned denominator for completion.

Citation and entity mention rates use valid responses as denominator. Citation means an actual cited target URL or domain; entity mention is a distinct evaluated flag. Each support evaluation has `{response_id,citations:[{citation_url,evaluable,source_accessible,supported}],claims?}`. `citation_url` must exactly match one URL in that valid response's `cited_urls`; duplicate response citation URLs and duplicate evaluation bindings are rejected. Citation support uses only bound, evaluable citations with accessible sources and a Boolean support verdict. Its `sample_selection` records `observed_citations`, `evaluation_scope` and `unknown_citations=observed_citations-evaluation_scope`, so inaccessible, unevaluable and unevaluated citations remain visible. Fact consistency uses evaluable important claims only when `panel.fact_list_ref` identifies an approved fact list. No accessible evaluable citation or missing facts produces `unknown` and a reason in `sample_selection`. Observed brand share counts each fixed-set brand at most once per response; its denominator is all fixed-set brand mentions, not all responses or a vendor's SOV. Completion uses valid responses divided by planned responses. For 10 questions × 3 repeats, 24 valid responses and 6 target citations yield 24/30 = 80% completion and 6/24 = 25% citation response rate.

Keep engine, surface, model, search-enabled state, locale and region in every metric filter. `claude_api`, `claude_web`, and search-disabled API calls are distinct conditions. Fewer than 10 unique questions yields `sample_selection.inference:'exploratory'`. Even a larger fixed panel is descriptive and does not establish population representativeness or causal impact. `clusterBootstrap({questionIds,valuesByQuestion,seed,iterations=2000,confidence=0.95})` resamples **whole question IDs**, carrying their repeats together. It returns `{interval,method,seed,iterations,unique_questions}`. A 95% interval describes variation under this fixed question sample; it is not a significance claim. Tests use fixed synthetic inputs; actual search, citation, and conversion outcomes remain unmeasured.