# Scout by OpenMerit

Scout is OpenMerit's model search agent. Describe your task and get model recommendations. Scout returns up to five model-and-endpoint candidates per task. Comparable measurements support performance ranking; narrow decision and vision tasks use an explicitly labelled task-fit shortlist when quality is unmeasured. The `/scout/` page and JSON API use the same index and ranking.

Scout is a separate hosted beta product. Npm 0.1.7 includes optional Claude Code setup for it; the hosted service does not change the Pi coordinator or Inference Terminal. Recommendations are evidence-based candidates for your own evaluation, not a guarantee of task accuracy or an independently validated state-of-the-art ranking.

**Website rebuild — published 24 October 2026:** the rebuild preserves the existing landing page at `/` and Scout at `/scout/`, and takes all hosted `/docs/` pages offline, including this guide. `/connect/`, the JSON API/MCP and the standalone `/scout/data-policy/` remain available. The guide remains maintained in Markdown; this does not remove the local CLI or its documentation from the package. See the [cutover notes](site-domain-migration.md) for publication and verification details.

**Small-screen handoff — source only, not deployed:** below 768 CSS pixels, the Scout page replaces its search interface with a note from Lazim asking visitors to use a larger screen. **Share link** attempts to copy the public Scout URL and opens the device’s native share sheet where supported. If clipboard access fails, the visible URL can be copied manually. At 768 CSS pixels and above, including full-width iPad layouts, the search interface remains available. This does not change JSON API or MCP access.

**Coding-agent setup — 24 October 2026:** stable npm 0.1.7 includes [device sign-in setup](#connect-a-coding-agent). Hosted Scout serves `/connect` and `/scout/mcp`; the button opens Claude Desktop with a setup prompt. This supersedes the 23 October web-placeholder reset. Existing older client configuration and authorization grants are not automatically removed or revoked. Scout serving, main-site documentation and npm publication remain separate releases; dated receipts below describe their original versions, not acceptance of every later build.

**Session restoration — unreleased source:** onboarding stays hidden while sign-in is checked, so returning signed-in visitors do not see the introduction flash on reload or reopening. Chat remains locked until an active session is confirmed. Signed-out visitors and failed checks reveal the introduction with sign-in still available; signing out restores it. No saved tour-completion flag grants access. This is not yet deployed and changes no homepage FAQ claim.

**Product wording — unreleased source:** the homepage, Scout page, onboarding and search/social descriptions call Scout “OpenMerit's model search agent,” with “model search” as the shorthand and “Find the right AI models for your task” as its promise. “Model recommendations” describes its output; the web and coding agents are access channels, not separate product categories. This copy is not yet deployed. Dated receipts below retain the wording and behavior of their original releases.

## Five-step introduction modal — deployed, 28 October 2026

**Deployment receipt:** the owner approved commit, push and deployment. [PR #28](https://github.com/laz-aslam/openmerit/pull/28) merged as [a70103a](https://github.com/laz-aslam/openmerit/commit/a70103a1c8eb8a701ee262106ae6e6c026cf04b4). All PR and merged-commit CI checks passed. The guarded Scout-serving deploy moved `openmerit-scout-alpha` from `2cf2c2d3-8aef-40f0-9c44-800bcab247bf` to `17ab0166-9a5c-43e9-932b-c13bee040db7` at 20:45 UTC. Live Scout HTML, chat/auth/onboarding JavaScript, styles and Claude screenshot matched the release build. `/` and `/connect/` stayed byte-identical; other Workers were not redeployed. The catalogue remained healthy on snapshot `588855a0bf28212fad48a9513dbf3d5b`.

Live Chrome desktop/mobile and WebKit mobile checks covered all five slides, locked future navigation, revisiting reached slides, real Google/GitHub sign-in controls with optional analytics blocked, dismissal keeping the page inert, signed-out reload returning to slide one and zero agent turns during the tour. Browser checks with simulated sign-in/replies covered all three example tasks, exact submitted messages, single submission after double activation, keyboard activation and New restoring examples; auth fixtures covered resumption after sign-in. Completed production OAuth, an authenticated production recommendation, native Claude installation and physical-phone behavior remain unverified.

Signed-out visitors see a five-step introduction modal over the actual Scout page, with the page dimmed and softly blurred behind it. The first is a personal note signed by Lazim and Praveen: Scout is OpenMerit's model discovery product in early beta, and feedback on what is useful, confusing or could improve is welcome. This is a text-only note, not a feedback form or a new feedback channel. The second introduces Scout with bundled lab/Hub marks and coverage from the same optional status request as the page: indexed models, benchmarked labs, benchmarks and distinct source-feed IDs. Hub repositories appear separately because they can overlap the catalogue. Labs count model publishers with benchmark results, not every inference host; source feeds include research records, not distinct organizations. Failed or invalid status responses leave a general coverage description, never fabricated counts, and do not block the tour or sign-in.

The third explains that Scout recommends rather than routes or resells inference tokens. Its illustration separates a recommendation from the visitor's app → provider/router → model runtime path. Recommendations can be constrained to the public endpoint inventory supplied through the [coding-agent project configuration](#using-scout-from-the-agent) or API; the slide does not import accounts, collect provider keys, connect endpoints or introduce a web configuration form. The fourth explains the task-to-shortlist workflow with an illustrative conversation; its model order, quality estimates and prices are example values, not a live recommendation or measured task accuracy. The fifth explains the optional Claude Code integration and shows cropped excerpts of an actual session, mentioning MCP, API and CLI availability. Its historical model recommendation is not a recommendation for the current visitor. The full-size capture opens separately. Viewing any step does not submit a task, run inference, install Scout or launch Claude.

**Continue** advances through the first four steps; **Get started** on the fifth opens the existing Google/GitHub sign-in flow above the introduction. The five progress controls revisit only slides already reached through Continue; future slides are disabled and excluded from keyboard navigation. Going back preserves access to all reached slides, including those ahead of the current slide, for the current page visit. There is no separate back button. The modal has no separate header sign-in or footer navigation, and the conversation/Claude previews have no supplementary captions. The conversation preview's title bar uses decorative red/yellow/green window lights instead of the Scout wordmark and example label; these do not close, minimize or maximize the modal. Its accessible image description still identifies the values as illustrative. The underlying page stays visible but inert until Clerk reports an active session, including after an OAuth return; a dismissed, failed or pending sign-in never enables it. The introduction cannot be dismissed to bypass this gate. Successful onboarding sign-in dismisses the introduction and focuses chat without automatically sending the example. Existing explicit draft-submission resumption remains separate. Returning active sessions bypass the introduction. Signing out restores the modal and makes the page inert again.

Tour progress stays in page memory: signed-out reloads and new page visits start at the first slide, even after opening or dismissing sign-in. The tour does not read or write a completion marker; old `scout-onboarding-v1` markers are ignored. Reloading with an active authenticated session still bypasses the introduction. Blocked storage does not prevent navigation or sign-in. Privacy controls and the data policy remain directly accessible without authentication at `/scout/privacy/` and `/scout/data-policy/`; their links stay in the main page footer, not the modal. No onboarding-specific analytics event is added; existing page/sign-in analytics and preferences remain effective.

After sign-in, the empty chat shows three task examples under **Try these**: support-ticket summarization, invoice extraction and product-photo classification. Selecting one immediately submits that task through the normal composer/auth flow and starts a conversation; no separate Send action is needed. If the session has expired, the task stays as a draft while sign-in opens and resumes through the existing authenticated-submit flow. The examples disappear when the first conversation turn appears; **New** clears the conversation and shows them again. They remain inside the chat surface, above the fixed composer, and cannot bypass the onboarding/auth gate. The homepage makes no affected example-task claim and needs no FAQ change.

Pointer navigation uses a short, non-looping opacity/position transition. Keyboard navigation and reduced motion change steps without movement. Keyboard focus stays inside the introduction, yielding to the sign-in modal while it is open. On narrow and short screens, the introduction scrolls within its rounded panel; the underlying page does not scroll. The homepage makes no affected onboarding or sign-in claim and needs no FAQ copy change.

Design references: [Linear's welcome screen](https://mobbin.com/screens/62773d0f-3574-4191-ac91-eb8a51697a95), [Linear's onboarding flow](https://mobbin.com/flows/cbaf58a9-fd84-4afe-b214-e65b24484013), [Notion's focused setup screen](https://mobbin.com/screens/99396b7b-c21d-4cea-8956-4942c2b9aae6), [Neon's integration walkthrough](https://mobbin.com/screens/8b8b58fe-86fd-4a2a-a5ad-51b84614d7b2), and [Torph's text-continuity examples](https://torph.lochie.me/). The implementation keeps Scout's own graphite palette and system typography; it does not embed Mobbin or generated mockup assets or add an animation dependency. The Claude Code image comes from the owner's 24 October recorded session: two unaltered content crops of `claude-automatic-expanded-full.png`, `(1330, 1029)–(2810, 1103)` and `(1330, 1552)–(2810, 1770)`, stacked with a 42-pixel neutral gap and encoded as WebP. OS chrome, account information and unrelated session content are excluded.

## Every model kind in the agent — source only, not deployed

This describes source changes that are not yet published or deployed. Until both the publisher and the serving Worker are deployed and a refresh has published a new catalogue version, the agent behaves as described in the next section.

Each published catalogue version will include every model in the merged catalogue, not only chat models: embedding, rerank, image, video, speech (text to speech), transcription (speech to text), decision, OCR, moderation, 3D and music models, plus models whose kind is unknown. Each model carries its kind, input and output modalities, context window, release date, open-weights status and up to 16 catalogue routes with their current base-tier prices. Models also carry their linked leaderboard results from the media, speech, decision and embedding leaderboards Scout imports (Arena, Artificial Analysis, TTS Arena, Open ASR, 3D Arena, VBench, MMEB and others), and from MTEB's final leaderboards: metric, whether higher or lower is better, score, rank out of every result on that leaderboard version, leaderboard version, date and source. Chat models keep Scout's reviewed routes and benchmark results unchanged. The agent reads only this published copy, so a refresh that is still running never changes what one conversation sees between search and the shortlist check.

The agent's tools change as follows:

- **search_catalogue** (new) searches models, benchmarks and leaderboards by free text. By default it combines keyword (BM25) and embedding results by reciprocal rank fusion; it can also use either alone, and filter by result type and model kind. Each result names the field that matched. Without embeddings (no OpenAI key, an embedding error, or a catalogue version published without them) it returns keyword results and says so. Its queries, written by the agent from your task, are embedded with OpenAI, as benchmark search already does.
- **search_models** filters by model kind and reports the cheapest eligible price in each unit a model is billed in: per million tokens, per image, per second of audio or video, per million characters, per page or per request. Sorting by price compares one unit only (by default the usual unit for the requested kind, otherwise tokens); models not billed in that unit follow, unsorted. A route that lists only zero prices, such as a local runtime or a free tier, is not counted as a price.
- **get_model_details** and **benchmark_leaderboard** include leaderboard results, and leaderboard IDs work wherever benchmark IDs do.
- **submit_recommendation** accepts models of any kind. Fewer than five are accepted only when fewer models meet the stated hard constraints, which can now include the model kind. The per-task workload is given in billing units (input and output tokens, images, seconds of audio or video, characters, pages, requests) and cost per 1,000 tasks is computed from catalogue prices on the eligible route that prices most of the workload, then the cheapest. Quantities the route does not price are listed, and such a partial cost never leads on cost. Citations must be the model's own benchmark or leaderboard results.

In the shortlist, the Input and Output columns keep per-million-token prices; a model billed in another unit shows that unit's price there instead, marked with its unit, and the expanded model shows the estimated cost per 1,000 tasks. Known limitations: per-second audio prices apply to audio seconds, but audio billed in tokens must be estimated as tokens; image prices are per generated image; prices are base-tier list prices from catalogue sources and can be wrong at the source.

## Conversational Scout agent replaces the search page — deployed, 28 October 2026

**Deployment receipt:** the owner approved the commit, push and Scout serving deployment. [PR #26](https://github.com/laz-aslam/openmerit/pull/26) merged as [332dda3](https://github.com/laz-aslam/openmerit/commit/332dda3d889a2da386f2d0bb59e60cc5170d29ce). After all PR and merged-commit CI checks passed, the guarded deployment wrapper moved `openmerit-scout-alpha` from `88e33d4c-4606-4eb1-be27-36a2b05309ff` to `2cf2c2d3-8aef-40f0-9c44-800bcab247bf` at 14:38 UTC (18:38 Dubai). The deployed Scout HTML, JavaScript, stylesheet and auth module matched the local release build. `/` and `/connect/` remained byte-identical; the publisher, router, terminal and memo service were not redeployed. The catalogue stayed healthy on snapshot `588855a0bf28212fad48a9513dbf3d5b`. Live desktop/mobile checks opened the real Google/GitHub sign-in controls with optional analytics blocked, preserved the task on dismissal and sent no agent turn before authentication. Mobile scroll checks reached the full-height chat before the footer. Completed OAuth, an authenticated production recommendation, native Claude installation and physical-phone keyboard behavior remain unverified. CI's first merged-commit attempt failed an existing assertion that mistook `900` inside random UUIDs for leaked error data; the failed jobs passed on retry without changing that test.

**Claude Code action — deployed, 28 October 2026:** **Add to Claude Code**, with the existing Claude icon, sits below the intro's coverage counts. It restores the secret-free setup handoff described [below](#connect-a-coding-agent): copying and requesting the prefilled Claude link happen independently, without waiting for clipboard permission or requiring a chat sign-in. Feedback appears below the button and distinguishes a copied prompt from a requested launch; neither proves installation. Current task text is never included. DNT/GPC, saved usage refusals and account unlinking propagate into the installer command. A fresh setup ID is included only when browser usage collection and account linking are both enabled; blocked optional analytics does not manufacture a saved installer opt-out or prevent the handoff. On stacked screens, the intro and setup feedback scroll above the full-height chat. Browser-platform fixtures cover prompt/deep-link equivalence, privacy and failure handling; native Claude launch and completed installation remain unverified.

**Header wordmark — deployed, 28 October 2026:** Scout uses the footer OpenMerit wordmark's 18px size, 550 weight and tight letter spacing, rendered lowercase. The **New** label aligns with the header divider's right end while retaining the button's padding; account controls precede it when present. Sign-in, account access and conversation reset are unchanged.

**Chat surface and footer — deployed, 28 October 2026:** the conversation and composer share a subtle theme-aware surface with no outline. Spacing separates responses and task-brief fields without horizontal dividers; the model comparison header also has no divider. Continue is the task brief's only action, using a compact filled button with no explanatory line below it. Buttons are 32 CSS pixels high for mouse input and at least 44 pixels when a coarse pointer is available. Scout responses show the OpenMerit icon beside their Scout label; user messages keep their You label. The OpenMerit home logo sits at the footer's right edge. Data policy, Privacy & analytics, then Scout by OpenMerit stay vertically stacked at its left edge with the same spacing. Link destinations and privacy controls are unchanged.

**Viewport layout — deployed, 28 October 2026:** above 900 CSS pixels, navigation, title/description/coverage and footer form a left column; the chat fills the right column's height, with its composer at the bottom and its conversation scrolling independently. At 900 pixels and below, navigation and intro scroll above a chat section that occupies the full dynamic viewport height. Native page scroll snapping stops at the chat's top; its transcript scrolls inside it while the composer stays at the bottom. Scrolling beyond the transcript's end can continue down the page to the footer, the only section below the chat. Focus the conversation to scroll with the keyboard, including long briefs and expanded model details. **New** stays in navigation instead of taking up a transcript row: a compact borderless button with a plus icon, labelled “New conversation” for assistive technology and in its tooltip. The intro and composer compact on short screens, and long drafts scroll inside a height-limited input instead of taking over the conversation. Touch-input text is at least 16 pixels to avoid focus zoom; safe-area padding protects the full-height chat's top and composer. Mobile viewport height follows browser chrome, with keyboard-driven layout resizing where the browser supports it. Physical-phone scrolling and keyboard behavior remain unverified.

**Intro coverage — deployed, 28 October 2026:** the heading reads “Find the best AI models for your work.” A small overlapping lab/Hugging Face icon stack sits above live coverage from the public status endpoint. The counts distinguish catalogue models (`availableModels`), Hub repositories (`specialistRepositories`), labs with benchmark results (`providers`), indexed benchmark series (`benchmarks`, including descriptions without results), and distinct source-feed IDs (including retained research records, not unique organizations). Catalogue models and Hub repositories may overlap and are not added into a unique total. If status is unavailable, counts stay hidden without blocking sign-in or sending. These are catalogue coverage counts, not proof of task performance or an optimal recommendation. The homepage makes no affected coverage claim and needs no copy change.

The conversational agent is the Scout page at `/scout/`, replacing the search box and its results table. The page stays open to everyone; sending a message requires sign-in, as searching did, and a message typed while signed out is kept while you sign in. The JSON search API (`/scout/api/recommend`), the MCP server (`/scout/mcp`), coding-agent setup at `/connect/` and the catalogue refresh are unchanged, so the CLI and installed coding agents keep their current behaviour. Sections below that describe the search page's prompt examples, results table and loading messages are historical; the Claude Code setup handoff remains available in the new page.

You describe the work; Scout writes a task brief with five components (input, work, output, criteria and constraints), each marked stated, assumed or missing. Each component is editable in place: correct anything Scout got wrong, or fill in a missing one. All five fields share the same editable background, including Work; the page does not show per-component question text. Once sent, the brief collapses. **Continue** sends only the components you changed and requests recommendations immediately, using stated assumptions for missing details. It also works without edits; there is no separate Skip questions button. Pressing Enter in a brief field does the same; Shift+Enter adds a line break. The message box stays at the bottom of the page. Each turn ends with an editable brief or a shortlist of five models, followed by Scout's summary. While a turn runs, its steps are listed with the current one last; afterwards they stay as a collapsed record of the work. The shortlist reports Scout's 0–100 quality estimate with the cited benchmark ranks, input and output prices per million tokens on its cheapest eligible route and release date. Expanding a model shows its rationale, benchmark ranks, route and reference latency. A missing release date is shown as unknown. Reply in the same conversation to change constraints or trade-offs; **New** starts over.

The agent runs in a Cloudflare Durable Object, one per conversation, using Pi Durable and the Agents SDK. Its model is set by `SCOUT_AGENT_MODEL`: `openai/<model>` calls OpenAI's Responses API directly with the Worker's `OPENAI_API_KEY` (default `openai/gpt-6-sol`), and any other value runs on Workers AI, such as `@cf/zai-org/glm-5.3`. Its tools search benchmark descriptions by embedding (OpenAI embeddings, as in search), read leaderboards and model records from the published catalogue, and use Workers AI web search when the catalogue cannot answer something decisive. Cost, latency and recency figures come from catalogue data, not from the model's text, and a shortlist that cites models or benchmarks outside the catalogue, or breaks a stated hard constraint, is rejected and resubmitted. A turn usually takes one to three minutes.

The page uses the same sign-in and rate limits as search:

- `POST /scout/api/agent` with `{"message": "…"}` starts a conversation and runs one turn. Continue it with `conversationId` plus `message`, `brief` (the edited components, such as `{"constraints": "Open weights only"}`; an empty string removes one) and/or `skip: true`. A client may supply its own UUID as `conversationId` with the first message.
- `GET /scout/api/agent/<conversationId>` returns `busy`, the turn's `progress` steps and its latest `answer`. The page polls it while a turn runs and reads the answer from it if the POST connection drops.

Conversations are keyed by the signed-in account, so one account cannot read another's. The conversation transcript is stored in its Durable Object with no expiry yet. With the default `openai/gpt-6-sol` agent model (not yet deployed), the task text and the conversation reach OpenAI, which also embeds benchmark and catalogue searches; web search runs on Workers AI. Before that change, the task text reached Workers AI and, for benchmark search, OpenAI. The [beta data policy](/scout/data-policy/) still needs updating to describe conversation storage. The deployed serving configuration includes the `SCOUT_AGENT` Durable Object (migration `agent-1`), the Workers AI binding and `nodejs_compat`.

## Repeatable searches and task-specific evidence first — deployed, 27 October 2026

**Repeatable results.** Scout's interpretation and evidence checks are AI calls whose answers vary between attempts, so an identical search could return a different shortlist: across 34 test cases run three times, repeats shared only 34–39% of their top five. A completed search now records each AI answer it received. For seven days, an identical search (same prompt, model/endpoint scope, catalogue generation, interpreter version and UTC day) replays those answers instead of asking again, so it returns the same shortlist. Failed or degraded searches record nothing. A new catalogue generation, interpreter version or day asks again. See [the data policy](/scout/data-policy/) for how the answers are stored. Relative dates in restrictions now resolve against the UTC day rather than the exact time. Interpreter version is 40.

**Task-specific evidence first.** When some candidates in the quality band have a score on a benchmark of the task itself (direct evidence), they are ranked and labelled strongest, best value and fastest among themselves first. A model estimated only from general benchmarks can no longer outrank them or remove them as dominated; it can still fill a remaining place, and a warning names it. The same applies to domain evidence when no direct benchmark was accepted. For example, a model with only a high general score no longer leads a medical coding search above models measured on MedCode.

The evidence assessment call now allows one retry within the search deadline (up to four seconds instead of 1.8), because a single slow answer previously failed the whole search.

## Twice-daily catalogue refresh — deployed, 25 October 2026

The hosted catalogue publisher refreshes twice a day, at 00:07 and 12:07 UTC (05:37 and 17:37 IST), instead of hourly. Each run reads the structured feeds and the D1 catalogue and applies the same publication checks. Since 26 October 2026 it no longer runs Parallel research jobs or the Hugging Face recently-modified listing discovery that fed them. Their 234 discovered benchmarks had no model results; the descriptions they collected are kept and are not refreshed, so they never mark the catalogue stale. Public feeds are reported stale 13 hours after their last successful check, which is one missed run plus an hour's grace. A single failed run can therefore leave up to roughly a day between source checks. The local `npm run scout` preview still refreshes its bundled feeds hourly. The owner-approved publisher and serving deployments went live on 25 October 2026 at 17:00 UTC.

## Beta evaluation logging — 22 October 2026 rollout

The beta default records submitted prompts alongside recommendations for matching evaluation and debugging, not foundation-model training. The serving configuration enables `SCOUT_BETA_EVALUATION_LOGGING=true`, alongside `POSTHOG_CAPTURE_CONTENT=true` and complete account-sync/deletion configuration. The [beta data policy](/scout/data-policy/) is generated from Markdown and published with Scout, independently of the main website. This rollout supersedes the dated default-off contract below. The operator remains responsible for the applicable privacy basis and restricted PostHog access; a beta label does not establish consent or another lawful basis.

With the rollout enabled, browser prompt sharing defaults on only when no saved content choice exists. Existing refusals, general analytics opt-out, DNT and GPC still win; unreadable preference storage keeps content off. The notice beside the prompt reflects the actual setting before submission. Footer controls can disable usage or prompt sharing. No draft text is collected. API/MCP content remains explicit opt-in through `X-Scout-Analytics-Content: on`; `off` or omission excludes it. CLI planning continues to send `off`.

In analytics version 4, the bounded, redacted prompt and up to three task excerpts/five-layer breakdowns exist **only** in `$ai_trace.$ai_input_state`, as `{prompt, tasks}`. The ordered recommendations appear in `$ai_output_state` and the existing content-free `scout_search_details` records. Public evidence, ranks, metric bases, quality-screen status, timings and interpreter/ranker/index versions remain available; they describe recommendations, not model executions. [PostHog Cloud documents 30-day retention](https://posthog.com/docs/ai-observability/data-retention) for these AI input/output fields in `posthog.ai_events`, stripped from ordinary events at ingestion. A synthetic live-project check confirmed this storage separation, email redaction, the recommendation payload and `retention_days=30`. Ordinary metadata follows normal project retention. No second prompt copy, custom expiry job or batch export is added. Actual expiry is not yet verified; old version-3 detail events and external exports are not retroactively erased by this change.

**Don't link this browser's activity to my account** sends `X-Scout-Analytics-Account: off`. The server substitutes fresh identifiers and excludes account identity/merging, ignoring old browser identifiers and client-supplied trace IDs that might already be linked. Unlinked responses omit `X-Scout-Trace-Id` so later account-linked interactions cannot reuse it. This setting applies to future requests from this browser, not other devices or copied coding-agent configuration; API/MCP callers can send the same header themselves. General account opt-out/deletion gates still apply. Separate Clerk profile and lifecycle analytics can continue while usage analytics is enabled. Unlinking does not erase old associations, make identifying prompt text anonymous, or enable automatic account-based deletion of unlinked historical events. Cross-request funnels and returning-user analysis are unavailable for unlinked activity.

## Example tasks — source only, not deployed

Three buttons below the prompt offer invoice extraction (one task), ticket triage and reply writing (two tasks), and feedback analysis, fix prioritization and release notes (three tasks). Choosing one replaces the draft with an editable example and focuses the prompt; it does not submit, open sign-in or consume a search. Use **Find models** or Enter when ready. The normal sign-in requirement applies.

Examples go through the same interpretation and ranking as typed prompts, with no fixed results or forced task counts. Each task returns up to five eligible candidates; the example does not guarantee five or establish that a recommended model solves it. When analytics is enabled, `scout_example_selected` records only one of three fixed example IDs, never the current draft. Existing privacy controls and separately opted-in submitted-prompt handling still apply.

## Product analytics

### Scout setup analytics default — version 0.1.10

Version 0.1.10 enables usage and installation diagnostics by default for Scout's `install`, `login`, `status`, `remove`, `logout`, `project` and `doctor` commands. The normal homepage and Scout handoff prompts use `npx -y openmerit@latest install --client claude-code` without an analytics-enabling environment prefix. Terminal and evaluation analytics remain opt-in. npm 0.1.9 uses the opt-in setup behavior below; publish 0.1.10 before deploying these prompts.

`OPENMERIT_TELEMETRY=0` or `DO_NOT_TRACK=1` disables setup analytics, strips outgoing correlation IDs and carries the refusal into the approval URL and installed MCP headers. Scout's usage-disabled handoff still includes `DO_NOT_TRACK=1`, and its unlinked handoff includes `OPENMERIT_ANALYTICS_ACCOUNT=off`. Browser DNT/GPC, saved browser choices and server-side account opt-outs remain effective. Analytics-enabled, linked installations retain the random installation UUID described below. The homepage creates no browser setup ID. MCP prompt-content capture stays off; beta status does not establish consent or another legal basis. Package and hosted publication are independent; these changes alone do not establish live event ingestion.

### Coding-agent lifecycle instrumentation — npm 0.1.9

Npm 0.1.9 includes these installer changes; browser/server instrumentation requires the matching independent Scout deployment. Release acceptance is recorded separately. Rerun the installer to update existing 0.1.8 installations; nothing backfills historical events. This contract supersedes the no-retry and process-only setup-identity descriptions below for lifecycle-enabled builds. Homepage/docs deployment remains separate.

| Touchpoint | Event and meaning |
|---|---|
| Add to coding agent | `scout_agent_setup_started` records the click. `scout_agent_launch_requested` records an attempted Claude deep link, **not proof the app opened**. `scout_agent_setup_copied` separately records clipboard success/failure. |
| Device authorization | Server-owned `scout_connect_step` records device creation, approval-page requests, pending/slow polling, approval or denial, expiry, key collection, device listing and revocation. Expected pending polls are not exceptions. Browser failures use `scout_connect_client_error` with a fixed stage/code. |
| Installation | Opt-in `scout_installation_step` records preflight, login/browser opening, configuration, verification, status, removal and completion. `applied`, component states and `connection` distinguish a plan, incomplete installation and a verified connection. A command exit of zero alone does not establish installation success. |
| MCP connection | `scout_request_completed` records every handled MCP request, including initialization/discovery and authentication failures. Allowlisted `mcp_method`, `mcp_tool`, reported agent and package version distinguish verifier probes from agent calls. Unrecognized names become `other`/`unknown`, never raw strings. |
| Each invocation/task | `scout_tool_invocation_started` and `scout_tool_invocation_completed` bracket authenticated tool dispatch. `scout_search_completed` counts an accepted recommendation attempt; `scout_search_details` retains each task's shortlist. All share the request trace. Protocol/tool errors are failures even with HTTP 200. |

When browser usage analytics is enabled, the copied setup prompt explicitly sets `OPENMERIT_TELEMETRY=1`; usage-disabled prompts instead set `DO_NOT_TRACK=1`. Account-unlinked prompts also set `OPENMERIT_ANALYTICS_ACCOUNT=off`. A linked setup gets a random `--setup-id` UUID that joins the click, resumed login, installation and later invocations. An opted-in installation stores a separate random `installation_id` in its existing ownership record and MCP headers; upgrades reuse it. These identifiers contain no account, hostname or project path. They are correlation hints, not authentication. Plans and status checks do not create an installation identity. Standalone CLI telemetry remains default-off, and `DO_NOT_TRACK=1` wins. Installed MCP content capture remains off.

The approval page initializes the same browser privacy controls and authenticated account preference synchronization as Scout, without inventing a Scout pageview. Installer URL refusals (`analytics=off`, `analytics_account=off`) remain effective without changing the browser's stored settings; a request-only refusal does not overwrite the account preference. A collected key is labelled **authorized**, not a verified installation. Account-wide opt-out still requires the authenticated preference endpoint and continues to gate linked and unlinked account requests. Unlinking removes setup/installation identifiers as well as browser/trace linkage. A privacy change in a browser does not rewrite an already installed agent's headers; rerun setup with the desired settings. Existing account opt-outs are not automatically reset.

First trigger and first task mean **first observed** tool call and task for that installation/account. They are derived from the same event stream, not additional search counts. Discovery is not a trigger. Recommendation completion is not downstream model execution or application-task success. No transcript is read, and this instrumentation cannot prove a startup rule loaded, a model decision occurred without calling Scout, or a native app actually opened. Missing consent, storage, connectivity or abrupt termination can leave a funnel incomplete; missing events must not be presented as a verified zero or a proven skipped call.

Initialization can record an allowlisted harness name and semver-shaped reported harness version; the installed headers report the npm installer version, not Claude's current version or invoking model. The stateless service cannot infer the coding agent's model from Scout's own provider generation events. Installer milestones include a fixed `command` category so status/doctor/project completion is never counted as a new installation. Setup milestone submission allows one transient retry within a hard 1.5-second total deadline; general CLI/terminal delivery retains the separately documented behavior below.

Browser event submission retries one transient failure with a stable event UUID. Server delivery retries transport failures, HTTP 429 and 5xx up to three total attempts, with the same event UUIDs/timestamps and a 1.5-second timeout per attempt. Permanent 4xx responses do not retry. Product responses do not wait for delivery. There is no durable product-event queue or exactly-once guarantee; bounded retries can still exhaust. The serving Worker configuration enables sanitized Cloudflare console counters (`scout_analytics_delivery`: delivered, failed, request-suppressed or account-suppressed), with query redaction and invocation logs disabled. Counters contain no identity, trace, device code, URL, content or credential and remain available independently of PostHog delivery. They are operational diagnostics, not PostHog service-log ingestion or session replay. `X-Scout-Analytics-Status` reports only whether the request permits capture, never a receipt of ingestion.

### Version-5 instrumentation

This contract applies to version-5 Scout serving builds. The serving rollout is independent of main-site publication and the npm package; npm 0.1.7 includes the opt-in CLI additions, while the website collector remains source-only. Prompt-content choices, redaction, retention and account deletion controls are unchanged. Main-site publication remains blocked by [source divergence](site-documentation.md#website-analytics-source-only-unreleased). Earlier dated receipts below describe earlier releases, not acceptance of version 5.

- First-party, validated web events map to native PostHog `$pageview`, `$pageleave`, `$web_vitals` and `$autocapture` events. The last represents only explicit allowlisted CTA IDs, not generic DOM autocapture. Page paths are restricted to public home, Scout and supported docs routes; queries, fragments, titles, text and raw URLs are excluded. A random pageview ID pairs each view with its leave. Leaves carry bounded duration, maximum scroll depth and the fraction of page content reached by the viewport.
- Acquisition fields are fixed referrer channels/domains and allowlisted `utm_source`, `utm_medium`, `utm_campaign` values; unknown campaign values are dropped. Browser, OS, device and viewport are categories, not raw user agents or exact dimensions. Native URL properties are reconstructed from a fixed public origin and allowed path, never the visitor's raw URL.
- Performance observations cover LCP, CLS, INP, FCP and TTFB through a self-hosted collector, where browser APIs support them. Treat these as sampled observations, not complete traffic coverage or a configured statistical sampling rate: unsupported browsers, privacy controls, delivery limits and lifecycle exits can omit measurements. No resource URL or attribution payload is sent.
- When account linking is enabled, shared same-tab identity supports website → Scout signup/search attribution; verified sign-in can identify that tab's anonymous activity. It is not cross-device anonymous tracking or a unique-person count. Unlinking instead uses fresh request-only server identities, ignoring supplied browser/session/trace correlation; no cross-request funnel is available.
- MCP outcomes inspect JSON-RPC errors and tool `isError`, including logical failures with HTTP 200. Server diagnostics add bounded stage, failure code and provider HTTP status, distinguishing timeout, transport/HTTP, rate/quota/billing, refusal, incomplete and validation/provider failures without raw messages.
- AI computation spans describe actual bounded pipeline attempts, including retries and provider-reported input/output/cached-input tokens when available. Cache hits/shared followers do not invent their own attempts. Missing usage stays unknown; no cost is fabricated. Recommended models and their estimated evidence remain recommendations, not executions or actual charges.
- Results offer `useful` / `not_useful` feedback, with an optional fixed reason (`wrong_task`, `missing_model`, `too_expensive`, `insufficient_evidence`, `other`), not free text. Linked feedback and result interactions use the request trace. A repeated failed prompt within five minutes in the same page can carry `retry_of`; the comparison stays in memory, not an analytics prompt/hash. Privacy changes invalidate prior feedback correlation. Delivery acknowledgement does not prove ingestion.
- Opt-in CLI diagnostics add sanitized package version, platform/runtime categories, command correlation and allowlisted application basename/line/column frames; see [the CLI contract](inference-terminal.md#optional-cli-product-analytics). No arguments, absolute paths, raw messages or environment contents are collected.

Replay is a separate spike and is **not activated or available**. No feature-flag/ranking experiment is active. Operational dashboards must distinguish the version-5 fields from older records with missing diagnostics. Alert cadence and recipients must be chosen and release ingestion acceptance completed before alerts can be enabled. See the [operator evaluation workflow](site-scout-architecture.md#reviewed-feedback-and-future-experiments).

**Release boundary, 22 October 2026:** hosted Scout uses the [unified PostHog integration](#unified-product-and-search-analytics-deployed-22-october-2026) below. New Cloudflare Analytics Engine search writes have stopped; historical records are unchanged. Main-site/docs instrumentation and npm publication remain pending.

Hosted Scout has PostHog product analytics enabled in the EU region. Other deployments remain **inactive without a PostHog project token, region and environment**. The latest rollout confirmed anonymous visits and rejected requests in PostHog storage, with three suppressed validation identities absent. Authenticated searches, profile synchronization and real Clerk lifecycle deliveries still require live acceptance; passing local tests or HTTP acknowledgements do not establish those outcomes. Npm 0.1.7 includes separately opt-in CLI telemetry.

The operator's PostHog dashboard has eight charts covering visits, the sign-in journey, recommendation attempts and outcomes, successful-search p50/p95 latency, model-result interactions, API/MCP request outcomes and opt-in CLI completions. Charts exclude test/development environments; early production activity includes rollout checks. Complete authenticated-session tracking, successful searches and result interactions still await real-usage verification and are labelled accordingly. Empty pending charts are not verified zero usage. Reusable metric definitions are saved as unapproved proposals, not canonical metrics. The separate main-site documentation deployment is unchanged.

When configured, explicit events cover visits, opening sign-in, authenticated sessions, rendered results, opening model evidence, successful model-ID copies, provider-link clicks, and bounded failure codes. Interaction events contain task position and model rank, **not the copied ID, link URL or task text**. `scout_search_completed` is server-owned: one event per accepted recommendation attempt, including cache hits and concurrent callers sharing computation, regardless of the number of tasks. It records outcome, task/candidate counts, server duration and cache status. `scout_request_completed` separately measures API/MCP requests, including authentication/validation failures and MCP tool failures returned with HTTP 200. Do not add request, rendered-result and search events together to count searches. A successful recommendation is not evidence of successful model execution.

Authenticated events use a namespaced SHA-256 pseudonym of the **verified** Clerk user ID. The configured profile sync below can associate that identity with a name and verified email after an eligible visit; it is not anonymous data. Raw Clerk user/session IDs, bearer tokens and cookies are never forwarded to PostHog. The same verified account has the same pseudonym across web/API/MCP. `client` (`web`, `cli`, `agent`, `unknown`) is an allowlisted caller hint; `transport` (`http`, `mcp`) comes from the actual route. Neither is an authorization signal. Browser events are client-reported, not a security audit.

The browser keeps a random anonymous ID and hashed account/session transition markers in this tab's `sessionStorage` so an OAuth return can join the visit-to-sign-in funnel. Sign-out/account changes rotate that anonymous ID. With storage disabled, these values remain in memory and redirect attribution may be lost. Anonymous users are not linked across separate tabs or visits; returning authenticated users can support account-level retention. CLI telemetry is separately [opt-in](inference-terminal.md#optional-cli-product-analytics), with a process-only identity and no implicit account linking.

There is no PostHog browser SDK, autocapture, session replay, DOM/text collection, feature-flag polling, geography lookup or IP forwarding. Browser events pass a typed allowlist at `/scout/api/analytics` before the server sends them to PostHog. Server search diagnostics can include bounded public source URLs and separately opted-in, best-effort-redacted prompt text as described below. Explicit exception handlers send sanitized categories and application frames, not raw messages or arbitrary stacks. Disabling analytics does not disable necessary provider processing or erase historical records.

Browser Do Not Track and Global Privacy Control suppress both browser and server-side PostHog events. To opt out for this browser independently, run `localStorage.setItem('scout-analytics-opt-out', '1')` on Scout and reload; remove that item and reload to restore collection. API/MCP clients can send `X-Scout-Analytics: off`, `DNT: 1` or `Sec-GPC: 1`. The event endpoint has a separate per-IP quota, so analytics clicks do not consume the search allowance. Delivery is best effort, without retries; the Worker uses background delivery with a 1.5-second timeout, and an outage cannot block recommendations. Receipt does not prove ingestion. PostHog retention, access, spend caps and any required consent are project/operator responsibilities; Cloudflare's three-month retention does not apply to PostHog.

### Unified product and search analytics — deployed, 22 October 2026

Hosted Scout now sends new search diagnostics to **PostHog only**, through the existing first-party ingestion boundary. It stops new Cloudflare Analytics Engine search writes; historical Cloudflare records are neither exported nor deleted. OpenAI processing, the short-lived result cache, catalogue storage and infrastructure logs are separate from analytics and unchanged. Anonymous event ingestion is confirmed; authenticated recommendation traces and errors have local contract coverage but still require live acceptance.

One `scout_search_completed` still counts one recommendation attempt, including cache hits/shared callers. A `$ai_trace` describes that request and `scout_search_details` records each task/slot's ordered shortlist, public evidence, metric bases, quality-screen status and diagnostic context. The server creates a unique `trace_id`, returns it as `X-Scout-Trace-Id`, and the browser includes it on result views/copies/provider clicks. Identities and traces never enter shared computations or cached results. These are **recommendation traces**, not model executions: candidate costs/latencies are reference estimates, not actual LLM charges or task-success measurements. Missing measurements remain null. Evidence is bounded to 20 items per list, depth 10 and a 48,000-character string budget per task/slot; `evidence_truncated` and original benchmark counts disclose omissions while all five compact ranks/metrics remain. Prompt-derived semantic-match prose is excluded.

**Historical initial content contract:** prompt text was off by default in this rollout; the beta revision above supersedes that browser default. API/MCP still require explicit `X-Scout-Analytics-Content: on`, the deployment content switch and complete account-sync/deletion configuration. CLI planning always sends `off`. General analytics opt-out, DNT or GPC overrides content sharing. Standard analytics and browser preference changes do not send input text; content is processed only when a task is submitted.

Opted-in text receives best-effort removal of common credentials, authorization values, private keys, emails, IP/phone patterns, URLs and local home/temp paths. This cannot recognize every personal name, proprietary passage or secret. Never submit confidential records. Prompt text is limited to 4,000 characters; layer/excerpt text to 1,000 each. Content is linked to the verified account and is identifiable when profile sync is active. Turning sharing off stops future capture, not in-flight work or historical retention. Profile deletion also requests deletion of these account-linked events. Set suitable PostHog retention, access controls and spending limits before activation; no automatic retention setting or old-data backfill is included.

Browser and server failures produce PostHog `$exception` events. Scout errors contain fixed categories, allowlisted exception classes, and up to 20 application-filename/line/column frames; messages, function names, absolute paths, source context, query strings, headers and arbitrary stacks are excluded. Browser exceptions are capped at three per page. Terminal/CLI and main-site errors currently carry categories/classes without frames. Deployment metadata labels serving releases. No session replay, DOM/text autocapture, fingerprinting or Sentry is added.

Source-only homepage/docs events cover page categories and explicit navigation destinations, with a separate anonymous tab ID and no Clerk linking; that separate site deployment is pending. Opt-in local terminal events cover navigation/map interactions, inspector kinds, bounded aggregate inventory counts and errors, never local model/agent/project names or source content. The same process-only ID links terminal events to its CLI launch. Authenticated Scout web/API/MCP events share the verified account pseudonym, but anonymous website/CLI identities are not automatically resolved to that account. See [terminal telemetry](inference-terminal.md#optional-cli-product-analytics). The existing production dashboard has not been updated for this release; new charts, retention changes and historical backfill are not included.

### Account profile synchronization — deployed, 22 October 2026

Clerk-to-PostHog synchronization is **deployed and configured, but not yet verified against real Clerk deliveries**. Its signing secret, account-state binding and project-restricted PostHog person-deletion credential are installed. Without those prerequisites, personal profile export stays off. Clerk remains the identity system; Cloudflare coordinates synchronization, rather than providing a second login provider. There is no Clerk Billing integration. Account deletion is covered locally, not proven against a production account.

After Scout receives an authenticated browser's enabled analytics preference, PostHog receives only the account's **name, verified primary email, signup time and connected Google/GitHub providers**. Values come from Clerk's current Backend API record, not browser-supplied profile data or an old webhook payload. Removed/unverified primary emails and cleared names are removed from the current PostHog profile. Existing accounts synchronize on their next eligible visit; there is no whole-directory backfill. Profiles and behavior use the same account pseudonym, so they are identifiable personal data. Neither profiles nor account IDs are added to the Cloudflare prompt archive.

Verified `user.created`, `session.created` and `session.ended` deliveries produce `scout_signed_up`, `scout_signed_in` and `scout_signed_out`. Session removal and revocation have separate events. Browser session observations become `scout_session_observed`, which links the anonymous visit to the authenticated account but is not a new login. Before this change, `scout_signed_in` meant a browser session observation; dashboards must distinguish the release boundary and must not add the two events together. Lifecycle events use the Clerk event time and stable delivery IDs. Provider attribution is `unknown` for webhook lifecycle events. The browser reports Google/GitHub on session observations only when a completed, verified sign-in attempt identifies that exact session; connected providers and the client's latest sign-in strategy are not proof. Signup or restored sessions can remain unknown.

The signed-in browser sends an authenticated boolean preference to `/scout/api/analytics/preference`, including when DNT, GPC or local opt-out is active. This privacy-control request contains no product event, anonymous ID or profile. Once received, an opt-out stops account-linked events and background profile/lifecycle forwarding and discards unsent lifecycle events. A later enabled preference from an authenticated browser can resume collection for that account; preferences are account-wide and the latest server-observed preference wins. Opt-out is not historical erasure. A browser that is offline, blocked or not signed in cannot update server-held preferences until it reconnects and authenticates. Request-local opt-outs still suppress that request immediately. These controls do not implement a consent banner or establish legal consent.

Per-account Cloudflare Durable Objects hold preference state, hashed profile fingerprints, up to 256 recent delivery receipts and deletion tombstones. Before a preference is known, at most 128 pseudonymous lifecycle events wait up to seven days for an eligible visit; alarms expire the pending buffer. No names, emails, raw Clerk IDs, tokens, webhook bodies or prompts are stored there. Signed webhooks retry through Clerk/Svix on delivery failure; failed preference initialization retries on a later visit. Ordinary product-event delivery remains best effort. Search never waits for Clerk profile retrieval or PostHog.

Deleting a Clerk account stores a permanent pseudonymous tombstone and requests deletion of its PostHog person, associated events and recordings, even if analytics is opted out or new capture is disabled. Late updates and still-valid session JWTs cannot send new account events through Scout. PostHog can still ingest an event accepted before deletion, so a Durable Object alarm repeats deletion daily, including after a response found no matching person. This reconciliation continues while the integration is configured and incurs background requests. PostHog deletion is asynchronous: an accepted request is **not proof that historical data has been erased**, and operators must check deletion status and replay failed webhook deliveries. Clearing a profile field alone does not remove old `$set` events. The separate Cloudflare prompt archive has no account identifier and is not erased by this sync. These changes do not authenticate or link anonymous CLI telemetry.

## Account sign-in — published, 21 October 2026

Hosted Scout uses Clerk authentication for **Google and GitHub only**, without Clerk Billing, subscriptions or payment collection. Production configuration checks confirmed both providers enabled, no additional required profile fields, and organizations and Billing disabled. The production domain and keys are configured and the serving Worker is deployed. Google/GitHub enablement is a Clerk dashboard setting, not something the source code alone establishes.

The page and index status remain public. **Sign in** opens Clerk's account UI; submitting a task while signed out opens that same flow without invoking matching. A prompt draft is temporarily kept in this tab's `sessionStorage` to survive the OAuth redirect, then removed on return or successful sign-in. Storage-disabled browsers can sign in but may lose their draft during the redirect. Signing out or changing accounts clears the visible prompt/results and cancels the browser's pending request; it does not guarantee cancellation of provider work already accepted by the server.

**Sign-in correction — deployed, 28 October 2026:** optional analytics downloads no longer block the conversational page's sign-in or submit handlers. Failed authentication initialization now displays an on-page error without clearing the task or starting an agent turn. Auth configuration and client initialization have 15-second limits; a sign-in modal that fails to render its controls within 30 seconds closes with a reload instruction. Closing a fully rendered modal does not count as an authentication error. Live desktop/mobile checks confirmed that the real Google/GitHub controls open with analytics blocked and preserve the draft on dismissal; these checks do not establish completed OAuth or an authenticated production conversation.

Submitting a task while signed out resumes that task automatically once an active session is available, including after a same-tab OAuth return. A separate temporary submission marker contains the submitted task in this tab's `sessionStorage` and is consumed before resuming, so session updates and later reloads do not replay it. Header **Sign in** only restores the draft; it does not submit it. Editing the draft or dismissing sign-in with its close button, backdrop or Escape cancels automatic submission. With storage disabled, same-page sign-in can resume in memory, but redirect return cannot restore the task or submission intent. These cases have simulated-session coverage, not proof of completed provider OAuth.

The Worker verifies Clerk session tokens before catalogue access for both `/scout/api/recommend` and `/scout/mcp`, including MCP discovery. Requests must send `Authorization: Bearer <session token>`; a cookie alone is not enough. Signature, expiry, issuer and authorized-party validation use Clerk's SDK, and pending sessions cannot search. Missing/invalid credentials return HTTP 401; missing server configuration returns 503 rather than enabling anonymous access. Status and `/scout/api/auth` remain public; the latter exposes only the publishable key and frontend API hostname. The secret key never reaches the browser.

For an authenticated development/API check, obtain a current token from `await window.Clerk.session.getToken()` in your own signed-in browser. Treat it as a credential, do not share it, and set it privately as `SCOUT_TOKEN`. Then send:

```sh
curl https://openmerit.ai/scout/api/recommend \
  -H "Authorization: Bearer $SCOUT_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"Extract six invoice fields as JSON"}'
```

Use the same header for MCP. These tokens are short-lived; clients must refresh them rather than store one as a permanent API key. This first integration does **not** implement agent OAuth discovery/registration, long-lived API keys, or unattended agent sign-in. Existing clients without a way to supply a current session token now receive HTTP 401 and need an update. Already-issued valid JWTs may remain usable until expiry after session revocation; this is not a per-request online session-status lookup.

The existing edge limiter applies separately to IP addresses and authenticated user IDs: 60 protected requests per minute for each key, including MCP initialization/discovery. It is not a durable monthly usage ledger or a billing cap; the local preview does not emulate Cloudflare's limiter. Clerk handles account/profile and session data and loads its browser SDK/UI from its own frontend API domain. Its optional SDK telemetry is disabled. Raw account IDs are used transiently for rate limiting, not attached to the historical Cloudflare prompt archive. The 22 October analytics release adds pseudonymous account-state storage and the profile sync above; no public saved-search history or billing integration is introduced. PostHog and Clerk have separate retention policies from the historical three-month Cloudflare archive.

Local development now also needs `CLERK_PUBLISHABLE_KEY` and `CLERK_SECRET_KEY` in the server environment. Configure a Clerk development instance with Google/GitHub sign-in and sign-up; disable other sign-in methods and unnecessary required profile fields. Use the production instance and its own OAuth applications for `openmerit.ai`. Never expose the secret key in client configuration or commit it. The deployed production modal was checked at desktop/mobile sizes, and both buttons reached provider authorization redirects requesting basic identity/email scopes only. Signed-out submissions opened sign-in without a recommendation request and preserved the tab's prompt draft. Hosted API/MCP checks rejected anonymous, invalid-token and cookie-only requests. Tests stopped before provider authentication: complete sign-in/sign-up, callback return and real-user authenticated searches remain unverified. Offline signature tests are not evidence of a complete Google/GitHub login.

## Canonical promotion — published, 21 October 2026

The [canonical Scout](https://openmerit.ai/scout/) now replaces the older interface and matcher with the former alpha: original prompt excerpts, one shared task-sequence table, automatic ambiguity resolution and cost-first selection after the relative quality screen. Canonical endpoints remain `/scout/api/status`, `/scout/api/recommend` and `/scout/mcp`. Existing `/scout/alpha/` page, asset and API/MCP URLs redirect to the corresponding canonical paths with HTTP 308, preserving query strings and POST methods/bodies. Clients should update their configured endpoint rather than rely on redirects. There is no separate alpha interface after promotion; retired `/scout/v1/` and `/scout/v2/` paths remain unavailable.

Only serving changes. The existing hourly catalogue publisher, research budget, data and secrets remain intact; the unreleased collection changes below are not activated. Search retains the alpha analytics dataset and existing three-month data-handling policy. The homepage, terminal and protected main application are outside this deployment. Candidate counts can remain below five per task, timing may be unknown, and searches can exceed five seconds. Promotion does not establish task accuracy or jointly evaluated workflows. This promotion supersedes the alpha-only scope in the dated release records below.

Fresh canonical browser searches returned five, nine and thirteen candidates for one, two and three tasks in 6.91, 6.44 and 6.71 seconds, using interpreter 37 without result-cache hits or clarification questions. A real MCP request through the old alpha URL redirected successfully to `/scout/mcp`. Redirects retained queries and POST bodies; untrusted origins remained rejected. Deployed page/script/style hashes matched the build, and desktop/mobile renders were inspected. The publisher's deployment and hourly schedule, main/router/terminal deployments and unrelated routes remained unchanged. Full verification passed 692 tests. These smoke checks establish the canonical routing and UI contracts, not a latency or task-accuracy guarantee.

## Task sequences — published alpha, 21 October 2026

The [alpha](https://openmerit.ai/scout/alpha/) UI uses a normal Model/Quality/Cost/Time table for one task. For two or three tasks, each task becomes a column in one shared table. Read each row left to right as a sequence of model choices, with quality, reference cost and time beneath each model. Column headings quote short excerpts from the submitted prompt rather than rewriting the work. The five-layer breakdown, role labels, frontier badges and capability tags remain hidden. The table scrolls internally when needed; narrow screens retain the side-by-side order instead of stacking separate task tables.

Rows align each task's independently ranked alternatives; they are **not jointly evaluated workflows**, a global ranking of model combinations, or verified model-to-model dependencies. The top row takes each task's first choice under its existing ranking policy. Five is still a maximum, not a quota: a shorter shortlist leaves an explicit gap, and no model is repeated to complete a sequence. The same model can legitimately appear in different task columns with different evidence and workloads. Expand any model for its task-specific evidence. **Assumptions & limitations** retains quality-screen status, workload, warnings and specialist discovery. Unknown values remain unknown; calibration accuracy is distinct from relative benchmark scores. Costs are USD per 1,000 reference tasks, not exact dataset charges, and are not added into a workflow quote.

Interpreter version 37 requests a short `tasks[].sourceText` excerpt in the existing decomposition call and checks that it occurs verbatim in the prompt. When every excerpt has a unique position, those positions establish task order while keeping restrictions and models attached; otherwise the interpreter's task order is retained. An invalid excerpt becomes `null`, without blocking the shortlist or asking a follow-up question; the UI then uses an unquoted interpreted title (or the original full prompt for a single task). Long headings are visually shortened, with their full text in the hover hint, accessible text and expanded model details. The existing five-layer data and `tasks[].slots[0]` remain; no additional model call, dependency analysis, recommended-model execution or ranking-policy change is introduced. This change is published only to alpha; stable Scout is unchanged.

Fresh hosted browser checks returned five, ten and fourteen candidates for one, two and three tasks in 7.66, 7.43 and 7.12 seconds. All three searches used interpreter 37 without result-cache hits, preserved original prompt excerpts and cost-ordered model lists, and displayed no clarification questions. A separate uncached MCP request returned five coding models. The deployed HTML, JavaScript and styles matched the checked build; desktop and 390px mobile captures were inspected, including horizontal scrolling and model evidence expansion, with no browser errors or page overflow. Full verification passed 692 tests before publication; deployment repeated 349 Scout tests and 13 documentation tests. Stable Scout, the main site, domain routing and the hourly publisher were unchanged. These checks do not establish recommended-model task accuracy, a fixed five candidates per task, or the under-five-second target.

## Newer document evidence — deployed, 22 October 2026

Document and image-reading tasks previously relied on document benchmarks that covered only models from OpenVLM's 17 September 2025 snapshot and four OmniDocBench entries, so receipts and invoices could return GPT-4.1 or GPT-4o. The hourly refresh now imports the [IDP Leaderboard](https://www.idp-leaderboard.org/) component series as three benchmarks. Fourteen of its 26 entries match catalogue models, including GPT-5.4, GPT-5.2, GPT-5 Mini, Gemini 3.1 Pro, Gemini 3 Flash and Claude Opus, Sonnet 4.6 and Haiku 4.5. Benchmark interpretation tags all three series as document and image evidence. Its newest entries still trail the latest releases.

A model with no document-reading result can now qualify for a document task on general image-understanding evidence. Such picks carry `missingEvidence: ["documents"]` and a warning naming them. Quality is still estimated only from accepted evidence; this widens eligibility, not the evidence itself.

The first refresh with this source published catalogue `4efa445856c7eec97819270e48c0e658` (checked 22 October 2026, 13:40 UTC). Replays of invoice and receipt prompts against it returned current models such as GPT-6.1 Sol and Claude Opus 5.5 rather than GPT-4.1 and GPT-4o, but only one or two candidates, both admitted on general image evidence.

In unreleased source, an accepted benchmark counts only for the ability the request-time applicability check confirmed for that task, not for every ability in its hourly interpretation. A spatial-reasoning benchmark accepted as image evidence therefore no longer counts as document evidence or carries document weight.

Also unreleased: models with accepted document evidence are ranked among themselves first, so stronger general image scores elsewhere cannot push them out of the quality band. Image-only fallbacks are ranked separately and fill only the remaining places, after every document-evidenced model. They receive no strongest, best-value or fastest label, and their warning says they are listed after models with document results. With this change alone, replays against the same catalogue returned three or four document-evidenced models for the invoice and receipt prompts, with Claude Opus 5.5 as the only fallback; the margin change below brings both to five document-evidenced models. A cost-priority search lists the cheapest document-evidenced model first.

## Writing, summary and analysis evidence — unreleased

Task interpretation (interpreter 39) distinguishes two narrower abilities: **summarization & synthesis** (a child of writing) and **analysis & recommendations** (a child of reasoning). Searches previously reduced summarizing feedback, prioritizing fixes and writing release notes to the same writing weight, and their only accepted evidence was one creative-writing benchmark covering 20 mostly older models, so all three received one shortlist. The hourly refresh takes core abilities from the released registry; abilities discovered by research are kept.

The hourly refresh adds two sources:

- [Vectara Hallucination Leaderboard](https://github.com/vectara/hallucination-leaderboard) (Apache-2.0): the share of summaries of supplied documents that stay consistent with their source. Models answering fewer than 90% of documents are excluded, because their rate covers only the documents they answered. It measures faithfulness, not summary quality; smaller models often score higher.
- [Arena leaderboard dataset](https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset) (CC BY 4.0): style-controlled human-preference ratings for eight request categories (creative writing; writing, literature and language; business, management and finance; instruction following; hard prompts; multi-turn; longer queries; expert) and the document arena. Ratings become an expected win rate against the median-rated model of the same leaderboard; ratings with confidence intervals wider than 60 points are excluded. Preference is not verified correctness.

A test refresh matched 59 catalogue models to Vectara, about 142 to each Arena text category and 22 to the document arena.

Three changes to quality qualification accompany them:

- **Disagreeing evaluations.** With three or more connected evaluations, one whose scores correlate below 0.3 with the estimate from the others (over at least five shared models) is left out of the common quality estimate. A narrow score range gives such an evaluation very small fitted noise, so it previously dominated the estimate: models with only a Vectara result ranked above every frontier model.
- **Margin and uncertainty.** The practical-equality margin is the larger of two points and 0.15 of the spread of model estimates on the task, so it means a similar share of model differences whichever evaluations a task uses. Each model's standard error also includes how much its own evaluations disagree about it, so one tightly measured evaluation cannot decide a result alone.
- **Agentic evidence.** When a task calls no tools and needs no coding, evaluations interpreted as agentic or tool-calling are not used, unless they are the only accepted evidence; a warning names them. Coding evaluations normally run in agent harnesses and keep them.

Replays of a three-task request (feedback themes, prioritized fixes, release notes) against a test refresh of catalogue `4efa445856c7eec97819270e48c0e658` gave three different shortlists. The summary task rests on Vectara alone and lists small, low-hallucination models. Invoice and receipt replays returned five document-evidenced models each.

## Shortlist corrections from user reports — deployed, 22 October 2026

Three reported results had separate causes. Image-reading tasks whose primary quality target was another ability, such as structured output, returned no candidates: evidence applicability accepted only benchmarks for the quality target, while ranking still required document or vision evidence for admission. Applicability now also offers the abilities the ranker requires (documents or vision, tools, memory), so a document benchmark can satisfy that requirement.

A stated preference such as "prefer the lowest-cost model that can extract these fields accurately" was lost, because ranking priority was judged from the extracted downstream work, which deliberately omits preferences. Structured extraction now also returns the stated tradeoff from the whole request (`balanced`, `quality`, `cost` or `latency`), and that value sets `workload.priority`. Requests for the cheapest model that is still accurate are `cost`; candidates still pass the relative quality screen first. Interpreter version 38; no extra model call.

Similar tasks can still receive identical shortlists when one benchmark is their only accepted evidence. The result now says so: a warning names the single benchmark, and later tasks note which earlier task has the same shortlist. These are interpretation and routing checks against the published catalogue, not executions of the recommended models.

## Common quality estimate and labelled tradeoffs — deployed, 22 October 2026

This change replaces the cost-first policy below. Quality screening previously compared a model only with models that had exactly the same set of accepted benchmarks. That split the catalogue into many two- or three-model groups, so the best of a small, weak group passed the screen and cost-first ordering promoted it.

Scout now fits **one latent quality estimate across every connected evaluation** for the task. Each benchmark has its own difficulty offset, score scale and noise, so a near-zero benchmark and a widely spread one contribute on comparable terms. Correlated series in one benchmark family share that family's weight. Copies of one evaluation from different publishers, such as GPQA Diamond reported by Epoch AI, Artificial Analysis and model developers, form one family and count as one evaluation; their scores stay separate series. Evidence from the task's own domain carries at least three quarters of the weight when present, and evidence that scores the task's own kind of work counts twice as much as other domain evidence. General proxies fill in the rest and connect models that lack domain results. Each model receives an estimate with a standard error; a benchmark run by only one model links nothing. Applicable calibration observations join as one further evaluation using effective quality. Models with no evidence connected to the main group receive no estimate and remain provisional; they are never compared numerically. The estimate is reported as `comparison.qualityEstimate` and `comparison.qualityStandardError`; `quality` is its relative percentile among the estimated models.

A configuration qualifies when it is not credibly worse than the strongest estimate: within two points plus one combined standard error. Supported Pareto selection over quality, reference cost and reference time follows. A balanced search keeps the strongest, lowest-known-cost and fastest frontier points, labelled in `pick` as `quality`, `value` and `speed`, then orders by estimated quality. An explicit `quality`, `cost` or `latency` priority orders the frontier accordingly. At most two models per provider are selected while other qualifying models remain; the cap never leaves a supported place empty. `ranking.version` is `10`, and `ranking.frontier` reports `version: 4`, the priority, the number of estimated models and benchmarks, and the tolerance.

Reference time previously required measurements bound to the selected endpoint. No published catalogue endpoint had such a binding, so time was unknown for every candidate and dominance never removed one. Endpoint-bound timing still takes precedence and never migrates to another route. Artificial Analysis model-level medians, which are not tied to any endpoint, now apply to each route of that model, marked `speed.scope: "model"`, with a warning that actual endpoint latency may differ.

These remain relative benchmark signals, not task accuracy. An offline replay of the bundled 15 October catalogue, with simulated evidence retrieval, returned frontier coding and reasoning models first instead of the lowest-cost members of small cohorts; it did not execute any recommended model. Ranking added about 6 ms per search locally.

## Cost-first selection — published alpha, 21 October 2026

The source change above supersedes this policy once deployed.

The [alpha](https://openmerit.ai/scout/alpha/) ranker puts the **lowest known reference cost first after a relative quality screen**. It keeps models within two proxy-score points of the strongest model with the same evaluation coverage, using the existing quality-indifference band. At least two distinct models must share that scope; duplicate routes do not create an independent comparison. Supported Pareto dominance still removes inferior configurations before the list is limited to five distinct models. Latency breaks price ties, followed by optional metadata preferences and quality. Extra quality, a soft preference or an inferred priority does not justify a higher price rank. Explicit metadata restrictions remain mandatory.

This is a conservative interim policy, **not a verified “good enough for your task” threshold**. A best-in-cohort benchmark score may still be inadequate for the task. Unrelated evaluation scales are not compared numerically, and the relative display score is not used as an accuracy floor. Even a user-stated target such as 95% accuracy remains unverified without matching task evaluations; Scout neither invents such a target nor claims that the proxy band satisfies it.

Missing or disconnected quality evidence cannot displace or pad a qualifying shortlist. If no comparable cohort exists, Scout retains an explicitly unverified discovery list rather than treating unknown quality as a pass. `ranking.frontier.qualityGate` reports `proxy-qualified` or `unverified`, the margin, and configuration counts. Unknown prices sort after known prices; unknown latency is not zero. Costs still use the disclosed reference token workload, not a quote for the user's exact images, execution loop or dataset. Native task-fit specialist lists remain unmeasured, not quality-qualified recommendations. No additional model calls, collection jobs or UI controls are introduced.

Local real-provider browser checks returned five, eight and thirteen candidates for one-, two- and three-task searches in 6.69, 4.70 and 7.57 seconds, all cache misses. The invoice task retained three distinct models after screening and dominance; it was not padded to five. Every returned benchmark shortlist was ordered by reference cost. A single-model inventory displayed an unverified shortlist, not a quality pass. Desktop and narrow light-theme renders were inspected. These tests use the local catalogue and do not execute the recommended models, establish task accuracy, validate hosted storage or meet the under-five-second target universally.

After deployment, hosted browser searches returned three, nine and fourteen candidates for one, two and three tasks in 5.50, 6.59 and 6.26 seconds. An additional MCP image-classification request returned four models in 5.08 seconds. All four requests used real OpenAI calls and production R2/D1/Vectorize without result-cache hits; every benchmark shortlist was cost-ordered and contained no clarification. A single-model inventory remained explicitly unverified. Desktop and mobile captures were inspected. The hosted catalogue and retrieved evaluation scopes differ from local checks; candidate counts and identities are not invariant across interpretations. Full verification passed 691 tests. Stable Scout, the main site and the hourly publisher were unchanged. These samples do not establish exact task accuracy, universal reliability or the five-second target.

## Direct model lists — published alpha, 21 October 2026

Alpha resolves ambiguity internally and returns one ranked shortlist per independent task, without clarification questions or answer buttons. One, two and three tasks target five, ten and fifteen candidates respectively. Each list contains distinct models; the same model may suit more than one task. Explicit restrictions and evidence requirements still apply, so fewer candidates can be returned when the verified frontier or selected inventory is too small. Scout does not invent models or measurements to fill a list.

Steps toward one deliverable remain one task. Scout chooses the most likely intended output when it is unstated, marks inferred task layers, and leaves optional handling policies unspecified. It preserves explicit labels, boxes, masks, formats, provider restrictions and exclusions. Missing runtime data does not trigger a conversation. More than three independent tasks still returns HTTP 422; provider failures remain errors rather than fabricated recommendations.

HTTP and MCP retain the existing response envelope: each `tasks[]` entry has its five layers and exactly one `slots[0]` containing the ranked models. The top-level compatibility shortlist remains available for a single task. `task.clarification` is always `null` in the evidence engine. Any residual semantic ambiguity is reconciled internally before evidence retrieval. The browser no longer displays model-role counts or clarification controls. The 30-second server and 35-second browser timeouts are unchanged; this is not a latency improvement.

Before the cost-first quality screen, local browser checks with real OpenAI calls returned five, ten and fifteen models for one-, two- and three-task requests in 4.77, 5.42 and 6.48 seconds, without result-cache hits. Explicit bounding-box and pixel-mask requests retained their requested output types. One of the first eleven real-provider checks returned HTTP 503; twenty subsequent fresh repetitions of that prompt succeeded, but the intermittent failure's cause is unconfirmed. These checks use the local catalogue, not hosted storage, and do not establish universal search reliability or the under-five-second target.

## Multi-task discovery — alpha, 21 October 2026

The `scout-one` interface and evidence matcher are published as an experimental [alpha](https://openmerit.ai/scout/alpha/). The stable `/scout/` service is unchanged; the dated hosted-beta records below describe its earlier releases. This section records the initial role-based alpha contract; the direct task lists and cost-first policy above supersede it.

Alpha serves `/scout/alpha/`, with `/scout/alpha/api/status`, `/scout/alpha/api/recommend` and `/scout/alpha/mcp`. It uses a separate Worker and reads the existing published catalogue; it does not run a second refresh or publish the expanded research policy. Alpha analytics uses a separate private dataset with the same data-handling policy. Initial publication checks confirmed the page, assets, status and MCP tool listing, but all three recommendation smoke checks failed: invoice and two-task HTTP requests returned unavailable in approximately 4.8 seconds, and the MCP summary request returned a tool error in 5.1 seconds. The subsequent timeout mitigation passed the four targeted hosted checks documented below; the latency target and nine-role limitation remain unresolved.

Submit text or Markdown, up to 4,000 characters and three independent tasks. Scout describes each task as **input**, **work/steps**, **deliverable**, **criteria/evals**, and **constraints/guardrails**. Each layer distinguishes explicit information, necessary inference and missing information. Missing details are not automatically hard requirements. Steps toward one deliverable normally remain one task. Only separate model roles or incompatible endpoint needs produce additional roles, up to three per task. Requests exceeding either cap return HTTP 422 rather than silently losing work.

The prompt field has no formatting toolbar or task-count hint; task decomposition happens automatically. The header shows catalogue counts without an hourly collection/status line. Source disclosures retain collection dates and refresh-failure details, and the status API still reports freshness.

Each role has its own shortlist of up to five distinct models. Two roles can therefore display ten candidates, with overlap between roles. These are independent alternatives, not ten jointly evaluated combinations or a ranking of every possible model pairing. Comparable measured objectives support Pareto selection across quality, cost and latency. Dominated or unsupported candidates do not pad the list to five; unknown quality, cost and timing remain unknown. Provisional frontiers and task-fit shortlists are labelled separately. Benchmark proxies do not establish exact task accuracy or a global optimum.

### API and MCP response

The input remains `{ "prompt": "...", "modelIds": [...], "endpointIds": [...] }`, with both inventories optional and enforced on every role. Read `tasks[].layers` and `tasks[].slots[]`. Each slot contains `label`, `task`, `recommendations`, `ranking`, `warnings`, `specialists`, and diagnostic timings. Layers contain `text` and `origin` (`explicit`, `inferred`, or `missing`); missing text is `null`. A single-role response also retains the previous top-level result. For multiple roles, top-level `ranking.mode` is `multi-task` and `recommendations` is empty; clients must read the nested lists. MCP limits evidence excerpts per candidate as before. An ambiguous role can request clarification without discarding the other tasks in the UI.

One Structured Outputs call decomposes the request and extracts scoped metadata restrictions. A single embedding batch and concurrent role-specific Decisions, eligibility and evidence checks follow. Keyword methodology retrieval supplements semantic vectors and prepared capability memberships; retrieval similarity never becomes a quality score. All roles share a 30-second server matching budget; the browser allows 35 seconds for transport overhead. A required provider failure or deadline returns unavailable, not a partially successful multi-task response. Exact-result caching remains bounded to 30 seconds and preserves evaluation dates.

The under-five-second end-to-end target is **not met**. Before mitigation, the first six-case local check with real OpenAI calls returned five HTTP 503 deadline failures at approximately 4.5 seconds; only the four-task overflow case returned its expected HTTP 422. Task decomposition consumed most of the budget. Separate decomposition-only diagnostics with a longer timeout took approximately 5.5 seconds for one invoice task and 17.9 seconds for three tasks with nine roles. Those initial diagnostics did not change the serving deadline. These are local observations against the bundled snapshot, not hosted acceptance or recommended-model accuracy measurements. Fixture-based UI and contract checks alone do not establish successful live search.

### Timeout mitigation — published 21 October 2026

Alpha now separates the five-second performance target from a 30-second total matching timeout, with a 35-second browser timeout to allow for transport overhead. Every role still shares one deadline, and required interpretation, eligibility and evidence checks remain unchanged. Slow valid work can finish instead of being rejected at 4.5 seconds; this does not make searches faster or satisfy the latency target.

The reported request to separate cats, dogs and snakes from 1,000 images returned HTTP 503 on alpha in 4.95 seconds. With the mitigation, the local browser returned five candidates in 6.23 seconds using real OpenAI calls and no result-cache hit. Local invoice and two-task checks returned candidates in 5.32 and 5.81 seconds; the same image task through MCP succeeded in 5.75 seconds. Four-task overflow still returned HTTP 422. A nine-role request still failed; a follow-up diagnostic found that a role's primary evidence capability could not be established after reconciliation. That semantic failure is not bypassed by the timeout change.

After deployment, the exact image-classification request returned five candidates in the hosted browser in 7.00 seconds. Hosted invoice and two-task HTTP checks succeeded in 5.59 and 6.08 seconds, with five candidates per task; the image task through MCP succeeded in 4.75 seconds. All four checks used the existing R2/D1/Vectorize catalogue and real OpenAI calls without result-cache hits. Stable Scout was unchanged. These targeted checks establish successful search for those requests, not task accuracy, a universal success rate or compliance with the under-five-second target.

### Hourly collection and coverage

The collection changes below remain **unreleased**; alpha reads the existing stable catalogue. The branch retains the `7 * * * *` UTC trigger, six-way refresh concurrency and 24-job Parallel budget. Pending paid jobs resume first. Up to one quarter of the budget is reserved for six discovery categories: gateways (OpenRouter and Vercel AI Gateway); lab launches and updates; independent evaluations (Artificial Analysis and Arena); editorial leads (SemiAnalysis and Citrini); Hugging Face, hosted inference and self-hosted engines; and industry-specialist labs. Smaller budgets rotate categories. These are research starting points, not dedicated integrations or guaranteed hourly coverage of each site.

Research requests cited lifecycle/expiry information, capability updates, deployment regions, fine-tuning scope and documented hardware requirements. Standard rates remain separate from promotional offers, subsidies and expiry conditions. Editorial claims supply leads rather than numerical benchmark observations. Provider-specific facts must retain their scope; OpenRouter expiry is stored on its route rather than on every deployment of a model. Unknown or unsupported facts remain unavailable. No new paid collection run has validated this expanded coverage yet.

OpenAI receives the submitted request for decomposition and derived work descriptions for interpretation and embedding. Private hosted analytics records each model role separately with the original prompt, role position and allowlisted supporting evidence; the existing sampled three-month retention applies. Do not put secrets or personal information into prompts. Keys stay server-side.

For development, `npm run scout` serves the page on port 45833 and requires `OPENAI_API_KEY` for matching. It refreshes structured feeds locally each hour; it does not run the hosted R2/D1/Vectorize publication pipeline. Local evidence retrieval uses lexical matches and prepared memberships without a Vectorize binding. Production-like validation needs an isolated Worker with the storage bindings and `OPENAI_API_KEY`. Never put its value in browser assets, Git or a chat message.

## Evidence architecture — hosted beta, 16 October 2026

Scout separates semantic interpretation from exact metadata checks and numerical selection. The hosted service uses this evidence architecture after publication of its validated serving generation. The prompt and results interface is unchanged; no npm release is included.

| Stage | Hosted implementation |
| --- | --- |
| Hourly discovery | Structured feeds and incremental Parallel research; resumable Hub discovery archives public listings without inventing usable endpoints or scores. |
| Hourly evidence | Source-backed identities, deployment facts, benchmark methodology and specialist profiles; cached summaries fall back to the original source when support is uncertain. |
| Hourly publication | Private R2 records, an indexed D1 fact/observation projection and versioned Vectorize benchmark embeddings. The active pointer changes only after the complete generation is ready. |
| Task interpretation | Structured Outputs extracts metadata restrictions, alternatives, preferences and the downstream work. Decisions interprets that work for capabilities, operations and output serialization. No prompt keywords or provider/family exception lists are used. |
| Eligibility | Generic typed equality, membership and range comparisons against documented facts, intersected with explicit inventories. Missing or incorrectly typed facts cannot establish a restriction. |
| Evidence retrieval | A query embedding retrieves benchmark methodology, supplemented by hourly capability memberships. Semantic similarity supplies evidence candidates, never a model's score. |
| Selection | All eligible verified catalogue models are considered before selecting up to five distinct models. One quality estimate spans connected evaluations; Pareto comparisons require known quality, cost and latency. |

Mandatory restrictions use alternatives of complete condition groups. Soft preferences affect ordering, not eligibility. Relative dates use a supplied current UTC day; arbitrary numeric bounds are retained, including impossible requirements. An unsupported mandatory condition cannot silently disappear. A no-match result describes the available verified catalogue and inventory, not every model in existence.

Quality remains a relative benchmark signal, not predicted accuracy on an arbitrary task. Correlated benchmark families are grouped. Incomplete or incomparable objectives remain provisional and cannot establish a global three-objective frontier. Narrow typed tasks can still use a task-fit shortlist with unknown quality. Comparable direct calibration observations supplement quality evidence without excluding other eligible models. Calibration episode charges and timing do not replace the common token-workload reference costs and endpoint-bound timing; unknown values remain unknown.

The UI and HTTP/MCP inputs remain unchanged. This path has no request-time web research, catalogue refresh, evidence preparation or recommended-model execution. Interpretation, query embedding and bounded Decisions checks are external calls; latency is measured end to end rather than assumed instantaneous. The matching budget is six seconds. A provider failure returns an unavailable response instead of silently relaxing restrictions.

OpenAI receives the submitted task for structured interpretation, Decisions and query embeddings. Query vectors are used transiently; only public benchmark embeddings enter Vectorize. Public model/endpoint facts and observations enter private D1, with source records in private R2. Parallel receives public research subjects, not visitor prompts. Existing private prompt-and-evidence analytics and three-month sampled retention remain unchanged. Credentials remain server-side Cloudflare secrets.

The canonical paths remain `/scout/`, `/scout/api/recommend`, `/scout/api/status` and `/scout/mcp`. There are no versioned preview pages or API aliases. The dated sections below retain historical implementation and validation records; this architecture and the current Ranking section supersede conflicting earlier policies.

Acceptance retained the earlier provider failures rather than counting them as successes. The final remote filter/control run initially passed 18 of 27 cases; nine reached the six-second deadline. Each of those nine passed one focused recheck without changing the matcher. Ten additional ambiguous-output cases passed on the revised interpreter. These checks establish interpretation, eligibility and API/MCP behavior for the tested prompts, not universal reliability or actual recommended-model accuracy. The isolated real-feed refresh completed R2/D1/Vectorize publication with new Parallel purchases disabled; the normal production workflow was then exercised separately. The production refresh published successfully after a staging-memory correction. Public HTTP/MCP checks initially passed six of eight cases; the provider and invoice timeouts both passed one focused recheck. Successful live observations were approximately 2.4–6.4 seconds. Six research jobs remain queued for later enrichment. These are separate attempts, not an eight-case clean first pass.

## Task relevance corrections — hosted beta, 17 October 2026

These corrections are deployed as a limited hosted beta. Hourly evidence documents retain verified research descriptions and directly selected methodology alongside feed metadata; embedding caches change when that evidence changes. The already verified descriptions are indexed directly, without a second model-generated summary; original citation excerpts remain in the source records rather than the semantic search text. Structured extraction separates the downstream work from model restrictions before Decisions interprets its capabilities and output contract. Name resolution can run alongside Decisions. This avoids interpreting model selection itself as a downstream capability; it does add a sequential interpretation dependency within the existing six-second budget.

The initial task-relevance beta used query vectors to retrieve up to sixteen methodologies, then required both a general-relevance predicate and a scored-ability predicate for each. Although intended to admit component proxies, these overlapping questions could disagree and reject a benchmark because its domain differed from the requested task. The matching repair below replaces that assessment path; earlier acceptance results remain historical records.

Classification and judgment searches consider applicable benchmark evidence before an unmeasured native endpoint contract. Without comparable candidates they retain an explicitly unmeasured task-fit fallback; contract compatibility alone does not demonstrate accuracy. Spatial pipelines keep their separate compatibility checks. Ambiguous output representations can use the existing clarification response, with model restrictions preserved. No task-name, object-name or model-specific exception is added.

The Artificial Analysis parser locates serialized records by their data shape rather than a component-property name and never executes downloaded scripts. Omitted historical observations are retained only for models still listed and unchanged benchmark definitions, keeping their original provenance and historical status. Missing endpoint-specific timing and incomparable quality scopes remain unknown or provisional; restoring a feed does not authorize copying model-level timing to an unverified route. Matching retains its six-second deadline, and the hourly schedule and UI are unchanged. Default selection orders supported frontier points by quality and known cost/time, without reserving slots for the cheapest or fastest point. Explicit cost and latency priorities and metadata preferences still apply.

The initial release was withheld after real-task validation; the owner subsequently approved a limited beta release with the evidence and accuracy limitations retained. Earlier development attempts exposed source-relevance, retrieval and interpretation errors; a passing interpretation/filter run alone did not authorize deployment. A fresh execution set made **38 OpenRouter requests**, with **$0.223226** in reported charges and no unknown-charge calls. Five shortlisted models per task received two public examples each; bird and invoice tasks also included two comparison models from the earlier shortlist. The source images, answer keys, scorer and selected pool were frozen before dispatch. These executions used recorded OpenRouter routes, which do not establish latency or cost on a different recommended host.

| Fresh task check | Shortlisted models | Comparison models | Release gate |
| --- | --- | --- | --- |
| Bird species names in alphabetical order | 19/20 required items correct | 8/8 correct | Lower sampled accuracy |
| Book versus poster labels | 20/20 labels correct | No separate comparison pool | Passed |
| Invoice number and total, strict JSON | 18/20 fields correct | 8/8 correct | Lower sampled accuracy |

One model confused two bird species; two invoice candidates added an extra zero to a printed total. All outputs parsed successfully, so schema compliance did not establish accuracy. These failures remain recorded; no model-specific exception, held-out score or answer key was added to ranking. The final downstream-interpretation correction passed 11/11 matching cases, including exact provider/family/context/date filters, known matches, no-match and ambiguous-output cases. Observed client durations were approximately 2.2–5.9 seconds, with no percentile guarantee. That correction changed some bird/category shortlist membership, so the frozen execution results do not certify the final pool; the invoice pool remained the same and failed the original accuracy-only gate. That gate compared five shortlisted models with two comparison models rather than evaluating quality, cost and latency together. In this sample the comparison models had approximately 4.1 times the mean request charge for birds and 3.4 times for invoices. Lower mean accuracy alone does not establish a worse Pareto shortlist. The beta makes no claim of improved task accuracy or globally optimal five-model membership. Broader coverage, comparable deployment measurements and independent validation of all three objectives remain follow-up work. The held-out outcomes remain excluded from ranking.

Publication retained the latest completed hourly research and a healthy refreshed feed set. The final rollout check passed 9/11 cases on its first attempt: bird matching reached the deadline and explicit SVG generation returned no shortlist. Both passed one focused recheck without a matcher change. After deployment, the public HTTP suite again passed 9/11: the explicit SVG request returned no shortlist, and a Qwen-family code-authoring request admitted tool-calling evidence. Creator/family/context/date eligibility checks and impossible-context no-match passed. The live browser bird prompt displayed five recommendations without overflow. These are retained semantic and latency limitations of the beta, not a clean first-pass acceptance or proof of best-five performance.

## Matching reliability repair — 17 October 2026

The task Decisions batch now distinguishes the primary quality target from endpoint requirements such as image input or serialization. Retrieval and evidence assessment use that same target. Verified hourly memberships backstop the vector results even when some primary evidence was already retrieved, so an incomplete vector result cannot suppress other prepared evidence that fits the selected inventory. The existing sixteen-methodology request bound remains; this is not exhaustive methodology retrieval.

If the primary target contradicts the required abilities, Scout makes one bounded Decisions reconciliation using the quoted task and previous answers. It has at most 750 milliseconds within the existing matching deadline. An unresolved contradiction returns unavailable instead of being presented as a verified no-match. This is a general consistency check, without task-name or format keywords.

One Decisions batch identifies each proposed benchmark's deliberately scored ability from its own methodology and the registry's evaluation definitions. It does not require a general component benchmark to reproduce the task's exact domain, labels or output artifact. Only a recognized ability with sufficient confidence contributes proxy quality; incidental operations, unknown scope, rejected answers and unavailable checks do not. A primary-target proxy does not establish the quality of every component in a combined task or exact task accuracy. Endpoint modality, format and tool restrictions still apply independently. No model, provider or task-object exceptions are used.

Exact metadata selection and query embedding start after extraction and canonical-name resolution, alongside task Decisions. Model evidence loads alongside the benchmark assessment; unaccepted observations are removed before ranking. These changes reuse the existing hourly catalogue, D1 projection and vectors, add no background service and retain the six-second matching deadline. The UI and API/MCP request inputs are unchanged. No new npm publication is included.

Diagnostic stage timings overlap and must not be added together. `interpretationMs` includes the concurrently prepared eligibility/embedding result needed to complete interpretation; individual Decisions, extraction and database timings remain available in `runtime`. Cache hits retain the original evaluation time and report zero new interpretation time as before.

Before deployment, full repository verification passed, including 336 Scout tests. Eight filter/ambiguity checks passed against the prepared metadata oracle: provider, family, context, date, impossible context, an incidental provider mention, an ambiguous diagram and explicit SVG source generation. The isolated Ubuntu VM run passed 203/203 assertions against staging. All seven workloads passed through both HTTP and the official MCP SDK; installed CLI email and SQL plans contained only the declared connections. An earlier staged run exposed the remaining email retrieval gap. A subsequent test setup reused an initialized directory and correctly failed `eval init`; the final isolated run used a fresh directory. Those attempts remain recorded separately. Final production acceptance also passed 203/203 VM assertions and all eight filter/ambiguity cases after the consistency correction. The HTTP workload median was 3.665 seconds versus the earlier 4.924-second sample; no final request timed out. These checks establish tested matching and integration contracts, not exact task accuracy or a latency guarantee. The [testing guide](testing.md#hosted-matching-repair-17-october-2026) retains the earlier failed rollout and separate final results.

## Latency improvements — hosted beta, 17 October 2026

The hosted implementation keeps the existing task and format Decisions, exact metadata comparisons and numerical selection. Numeric and date vocabularies are omitted from interpretation input because their bounds are arbitrary; small canonical vocabularies remain in the first call. Larger vocabularies are deferred until an extracted restriction references them; a model-assisted resolution checks against the complete published vocabulary without changing unrelated conditions. Unknown mandatory names remain unresolved and cannot widen the result set. Stable reference data has an explicit provider prompt-cache boundary separate from the prompt and UTC reference time. Cache availability is measured rather than guaranteed.

Hourly staging prepares content-addressed specialist packs alongside profile indexes. Searches select the same scoped profiles and use those packs instead of many repository reads. A version-checked derived pack index can also be prepared for an existing immutable catalogue without repeating research. Existing generations remain readable through the original shard path when no matching pack index is available. Source research and the hourly trigger are unchanged.

Successful searches may be reused within a Worker isolate for up to 30 seconds, bounded to 32 entries and approximately 4 MiB. Exact prompt text, model/endpoint inventories, publication, interpretation version and UTC calendar date identify a cache entry. Cache hits preserve the original `evaluatedAt` time; date conditions remain evaluated as of that time for at most 30 seconds, and results are not reused across UTC midnight. Identical concurrent requests share computation. Errors and results missing a specialist or applicability check are not cached. This temporary memory cache is not semantic similarity caching. Since 26 October 2026, a separate seven-day answer cache in D1 replays the AI answers of an identical completed search; see the repeatable-searches section above.

Private search records now include fixed stage durations, pending stages at a deadline, cache status and allowlisted provider status/error codes. They do not store raw errors, stack traces, headers or credentials. API results include `runtime` timings/cache status; cache hits report their current serving duration and zero new interpretation time. The page remains unchanged. A six-second matching deadline remains a reliability boundary; these changes do not establish a latency guarantee. The full repository release checks passed, including 318 Scout tests. An initial candidate timed out on books/posters; that failure was retained, and the interpretation was then reduced further. The revised semantic suite passed 17 of 17 cases, including cold repetitions and exact-cache reuse, and four final cold checks passed after the independent database reads were parallelized. These small samples do not establish a production percentile or a latency guarantee. Post-deployment HTTP/MCP checks passed five of five on their first attempts: supplied-label vision, creator restriction, impossible context, JSON invoice extraction through MCP and an exact repeat. Observed end-to-end durations were approximately 2.0–4.3 seconds for new searches and 146 ms for the repeated prompt served from the same isolate cache. The 127 UI source files and five live scripts/styles were unchanged. These are individual observations, not percentile guarantees.

## Default architecture — hosted beta, 16 October 2026

Scout maintains separate model, endpoint, benchmark and observation records. Structured feeds, resumable discovery and bounded Parallel research prepare source-backed public metadata each hour. Unreviewed discovery listings remain outside the executable catalogue. New or changed methodology, metadata and embeddings use source-addressed caches. An incomplete generation cannot replace the active catalogue; searches continue against the last committed version.

A search runs Decisions task interpretation, Structured Outputs metadata extraction and a transient query embedding. It applies exact restrictions in D1, retrieves prepared evidence and relevant specialist profiles, and compares verified eligible configurations locally. There is no request-time web research, source preparation or model execution. The model pool has no pre-frontier top-K cap; evidence retrieval and specialist checks remain bounded. Missing objectives, incomplete benchmark coverage and approximate evidence retrieval prevent a claim of a globally verified optimum.

The trigger remains hourly at `7 * * * *` UTC, with up to 24 Parallel jobs, refresh concurrency six and seven-day research rechecks. Discovery and enrichment are incremental; the trigger does not promise complete world coverage or a revisit to every source each hour. Relevant specialist checks remain bounded to 64 prepared profiles. Matching has a six-second budget and includes external provider/storage calls. Successful focused remote checks were approximately 2.4–5.4 seconds; earlier timeouts are disclosed above. Instantaneous results are not guaranteed.

## Matching corrections — hosted beta, 15 October 2026

The hosted beta includes the following corrections. This update does not publish a new npm package:

- Benchmark, measured and task-fit candidates share the same endpoint requirements for image input, structured output and tools. Reviewed typed endpoints must support the requested operation. General catalogue entries require declared text-only output; text accompanying audio, image or video generation does not establish a general text contract. Mixed-output endpoints need a separately reviewed operation contract. Image inputs with text outputs remain supported. These are metadata eligibility checks, not task-accuracy guarantees.
- Unknown input/output prices and the compatibility blended price remain `null`, including when only one rate is known. Actual advertised zero rates remain zero. The existing page renders unknown prices with a dash.
- Either `modelIds` or `endpointIds` suppresses separate specialist discovery, including empty and unknown-only endpoint inventories. Explicitly selected compatible specialist endpoints can still enter a task-fit shortlist.
- When selecting among routes for the same model, an explicit latency priority compares known endpoint-bound completion times first, then cost. Without comparable timing, known timing precedes unknown timing; if neither route has timing, known cost and stable endpoint IDs decide. Other priorities retain cost-first route selection. Task-fit candidates retain their contract-fit and deployment tiers. Timing is never borrowed from another route.

Output metadata is normalized during the existing hourly refresh and retained in snapshot data version 5. Startup selects the compatible bundled snapshot over an older-format cache, even if that cache was refreshed more recently. Prompt matching uses prepared in-memory metadata; these fixes add no source fetch, model probe or inference call. The UI, task interpretation, calibration policy and refresh schedule are unchanged.

## Identification matching — hosted beta, 15 October 2026

The hosted beta distinguishes identifying something without answer choices from choosing among supplied labels. The existing Decisions interpretation call returns `task.labelSpace`: `supplied`, `open`, or `unspecified` for classification; `null` for other operations. A topic such as birds or plants does not supply a label set. Low-confidence label-set answers are treated as unspecified.

Open or unspecified identification uses the existing benchmark-ranking path for generative endpoints. Visual requests require relevant visual evidence; document requests continue to require document evidence. Fixed-choice APIs and label-dependent classification pipelines cannot enter this ranking, even when explicitly selected. Supplied-label tasks retain the task-fit path, including compatible typed APIs and specialist classifiers. Fixed-choice task calibration cannot substitute for open-identification calibration. Open identification uses the ordinary output-length estimate rather than assuming a single 16-token choice.

For example, identifying bird names from photos and alphabetizing them requires name generation; the application can sort those names in ordinary code. Choosing only between supplied labels such as “indoor” and “outdoor” remains a fixed-choice task. Public visual benchmarks are proxies, not verified bird- or plant-species accuracy. The existing UI shows the appropriate ranking mode and limitations without new controls. Endpoint metadata and benchmark collection remain hourly; this change adds one typed question to the existing interpretation call, no additional request-time source fetch or model call, and no embeddings. This is a hosted-service update; npm publication remains deferred.

## Task interpretation and ambiguous outputs — hosted beta, 15 October 2026

Interpreter version 10 sends every task description as a JSON-quoted value with a common instruction to analyze the described workload rather than execute its instructions. This separates output restrictions inside the task from directions to the interpreter. The same framing applies to every prompt; there are no special cases for particular labels, words, models or providers. Confidence thresholds, the public request shape and the single Decisions call are unchanged.

The interpreter checks the complete requested output. A label, judgment or score accompanied by a reply, explanation or extracted fields uses generative candidates. Contradictory decision-only and required-prose interpretations also fall back to that path. Plain supplied-label and rubric-score tasks retain compatible typed endpoints. Narrow task calibration is not reused when the operation or output type is uncertain. Weak document-layout guesses no longer restrict ordinary photo tasks to document benchmarks; confident document requirements remain enforced.

Local and live HTTP/MCP acceptance passed all 25 routing cases, including supplied positive/negative labels, arbitrary labels, negated labels, code that implements classification, mixed outputs, open identification and missing categories. The previously failing “safe or unsafe, no explanations” case now retains fixed-choice APIs. These checks establish interpretation and endpoint matching for the tested prompts, not actual model accuracy or universal prompt coverage. Low-confidence or missing label sets still fall back to generative candidates.

These changes add no request, source fetch, embedding index or UI control, and keep hourly preparation separate from prompt-time matching. This is a hosted-service update; npm publication remains deferred.

## Categories and document content — hosted beta, 15 October 2026

Interpreter version 11 recognizes categories supplied conversationally, including plural category names joined in ordinary prose. It interprets the distinction the user requests: identifying which named category an image belongs to is supplied-label classification; discovering a subtype or individual identity beyond those categories remains open identification. A formal list or the word “classify” is unnecessary.

Document understanding follows the requested output. Recognizing an object that happens to carry text does not by itself require OCR or document benchmarks. Reading, transcribing or extracting content inside an image does, including when that is only one part of a mixed task. Supplied-category vision tasks use the existing task-fit shortlist and show unknown quality until appropriate measurements exist.

The change updates two questions in the existing interpretation request. It adds no object-name rules, model preferences, services, dependencies or inference calls. Confidence thresholds, hourly preparation and the UI are unchanged. Local and live HTTP/MCP semantic acceptance passed 43 cases, including classification/reading contrasts and the prior 25-case suite. These are routing checks, not executions of the recommended models or a guarantee for every possible prompt. This is a hosted-service update; npm publication remains deferred.

## Semantic formats and SQL interpretation — hosted beta, 15 October 2026

Interpreter version 13 includes removal of the English keyword scans for strict output formats and structured-data capability. The existing Decisions questions now supply those judgments directly: a prescribed record or delimited layout for software to parse can require structured output even without a named format. The capability question distinguishes extraction and categorization from merely organizing prose or reading already formatted input. The same 21 questions run in one request; no UI control, source fetch, new dependency or additional inference call is added. Hourly catalogue preparation is unchanged.

The response decoder consumes typed answers without rereading the prompt. It retains response validation and general capability-consistency rules. The existing 0.4 confidence threshold now also guards structured-data capability and strict-format requirements. A low-confidence format answer in either direction remains marked uncertain, does not impose a hard format requirement, and cannot match narrow calibration evidence. This threshold is a heuristic, not a calibrated probability of correctness. Semantic interpretation can still misread a request, including unfamiliar formats or languages; catalogue format support is not proof that a model will satisfy an exact output schema.

The operation question recognizes writing, revising or debugging a read-only SQL query even when concrete schema details are omitted from the task description. Those details can be supplied when the model performs the task. Expressions that produce records or serialized values remain query-writing work; they do not make the model's answer a serialized data result. Executing a query, modifying stored data, explaining a query and generating another program remain distinct operations. This changes the general Decisions definition, with no query-text matching, function-name exceptions, dialect lookup or extra question.

This hosted update combines the format changes with SQL interpretation. Real Decisions interpretation passed all 91 cases locally and through live HTTP/MCP: the prior 75 cases, including the previously failing SQL request, twelve additional SQL boundaries and four fresh holdouts. Tests include query source versus serialized output, omitted/runtime schema details, recursive queries, execution loops, data modifications and multilingual requests. npm publication remains deferred. Semantic checks establish tested interpretation and endpoint matching, not query execution, exact-schema compliance or universal prompt accuracy.

## Semantic specialist matching — hosted beta, 15 October 2026

Interpreter version 14 and snapshot data version 6 replace Hugging Face prompt keyword dictionaries, language-name lookup and partial label overlap with Decisions compatibility checks. The hourly refresh prepares each repository's specialist operation and subject scope from public metadata, caches that analysis by revision and relevant metadata, and groups equivalent profiles. Changed repositories are analyzed in batches of up to twenty; unchanged profiles are reused. Preparation uses the server's Decisions API key and incurs provider usage. If preparation fails, Scout retains the previous complete specialist snapshot and its original timestamp while other sources can update.

A search still starts with the existing 21-question task interpretation. When eligible Hub profiles are relevant, a further batched Decisions request checks their subject, explicit language requirements and complete fixed-label coverage. Shared checks are evaluated once; all applicable checks must pass with the existing 0.4 confidence threshold. Requests contain at most 180 compatibility checks plus one language-requirement question; larger sets are split into bounded batches. Open identification, unsupported work, tool/memory tasks and inventories without eligible Hub routes skip this work. There are no repository fetches, model executions or per-repository calls during matching. A failed compatibility request omits Hub candidates for that search and leaves ordinary eligible results available with a warning. There is no keyword fallback.

Compatibility is eligibility, not predicted accuracy. Scout still enforces endpoint contracts, preserves unknown quality/cost/time, and orders discovery components by metadata completeness and diversity. Metadata and semantic interpretation can be wrong, and the existing bounded catalogue can omit suitable specialists. The UI and hourly schedule are unchanged. This revision is deployed on the hosted beta; npm publication remains deferred. All 107 semantic cases passed locally and through live HTTP/MCP: the prior 91 regression cases plus twelve specialist cases and four fresh holdouts. They check domain paraphrases, background context, negation, language requirements, multilingual instructions, full versus partial fixed-label coverage and open identification. These are interpretation and matching checks, not executions of recommended models.

## Semantic benchmark interpretation — hosted beta, 15 October 2026

Snapshot data version 7 prepares benchmark capabilities and correlated benchmark families with Decisions during the hourly refresh. It replaces benchmark-name regexes, manually assigned capability tags and the automatic reasoning fallback. Source adapters retain metric identifiers, score units and parsing contracts. Epoch descriptions and methodology are read from its published catalogue; Artificial Analysis and BFCL methodology are read from their public pages. Unavailable definitions remain explicit limitations, and names alone do not establish an opaque benchmark's capabilities.

Each changed benchmark receives an isolated interpretation request so another benchmark's description cannot supply its capabilities. For shared publisher methodology, a preceding Decisions request selects the directly applicable definition for that benchmark and evaluated subset; unrelated sections are omitted from capability interpretation. A section titled with the benchmark's own name, such as "GPQA Diamond" on Artificial Analysis's methodology page, is always kept and listed first, ahead of shared sections such as the Intelligence Index overview. The publisher's methodology is used even when an earlier research description exists. Publisher-declared suite identities are retained as source facts; other family relationships are interpreted by Decisions. Capability choices and family references are validated, and positive choices below the existing 0.4 confidence threshold are not used. Family links group related evidence; numerical comparisons still require the exact same benchmark series. Unknown scopes receive no capability tags and cannot contribute capability-ranking evidence. This is semantic metadata interpretation, not a measurement of model quality or a guarantee that the classifications are correct.

Interpretations are cached by definition, relevant publisher methodology, family inventory and interpreter version. Score, model, price and collection-time changes alone do not invalidate that cache. Family inventory changes trigger rechecking its references. If an interpretation fails, unchanged cached records remain usable, affected new or changed series stay unclassified, and source failure status is exposed until a later refresh succeeds. A methodology download failure retains the prior methodology when available and marks its source degraded. There is no keyword fallback.

Searches read the prepared tags and family IDs without fetching methodology or making benchmark interpretation calls. The existing prompt interpretation and optional specialist compatibility calls remain. `benchmarkInterpretation` in status/recommendation responses reports prepared, classified, unclassified and unavailable series counts; each returned benchmark observation includes its capability tags, family and interpretation version/time. The UI and hourly schedule are unchanged. npm publication remains deferred.

The initial deployed snapshot processed all 146 benchmark series: 136 have supported capability tags and ten remain unclassified. Individual weak capability judgments are also omitted, even when other capabilities for that series are supported. A repeat preparation with unchanged metadata made zero Decisions calls. All 24 semantic benchmark cases and 107 task-interpretation/matching cases passed both locally and through live HTTP/MCP, alongside 546 automated checks. These checks cover interpretation, caching, source failure and matching; they do not execute recommended models or establish universal semantic accuracy.

## Use Scout

**Source only, not deployed:** results use **Model Choices** in their count and no longer display rank numbers, the Rank column or the “Read left to right” instruction. The Assumptions & limitations disclosure, including its supplementary specialist list, is replaced with “Results could be wrong. Please verify before choosing a model.” Model evidence remains expandable; ranking, API evidence and the methodology documented here are unchanged. Feedback's reason selector and action buttons share a 40px height.

**Source only, not deployed:** the pending search state places rotating search messages in the results-title position, preceded by a tiny pulsing orb. It does not repeat the submitted prompt or show a separate loading heading. Reduced-motion preferences disable both the orb pulse and the 180ms opacity/3px message transition.

Loading messages describe search perspectives, not reported backend stages or completion percentages. Live timing starts at submission, including authentication and network wait, and freezes as **Took …s** when the response arrives. A cached response may legitimately be much faster. The existing 35-second request timeout still starts after authentication; replacing a search, signing out, completing or failing stops its loading updates.

**Source only, not deployed:** focusing the task field changes its background without drawing an outline around the input container. Keyboard focus outlines on buttons and links are unchanged.

**Source only, not deployed:** example prompts appear as rounded pills numbered 1, 2 and 3 beside **Try**, inside the bottom-left of the task field. They disappear as soon as the field contains any text, including whitespace, and return when it is cleared. Selecting a pill fills an editable draft without submitting it. The visible input-helper sentence is removed; Shift + Enter still adds a line. The Claude Code logo uses its existing transparent image without a white background in either theme.

Open [Scout](https://openmerit.ai/scout/), type a task, and select the arrow button (**Find models**) or press Enter. For example: “I want to upload an invoice and get the invoice number, date, vendor, subtotal, tax, and total as JSON.” The task field grows and shrinks as you type or paste. Very long prompts scroll within the field once it reaches its height limit. Shift + Enter adds a line. When the field is empty and unfocused, its placeholder cycles through detailed example tasks every ten seconds with a brief fade. Rotation pauses while you type, while the field is focused, or while the page is hidden. With reduced motion enabled, the example stays still. Placeholders never fill the input or send a request. The heading shows live counts of API catalogue models, Hugging Face repositories, benchmark series and sources. API identities and Hub repositories are separate counts and can overlap; neither count promises a runnable endpoint for every entry. The evaluation guide remains available in the fixed bottom footer, with space reserved beneath the page content.

The comparison table keeps recommendations in Scout rank order. Benchmark mode shows a relative quality signal and reference cost/time estimates. Applicable calibration contributes quality evidence within its recorded scope; its episode charges and timing do not replace the common reference workload. **Candidates to evaluate** denotes task-fit mode: quality is shown as unknown, alongside known reference costs and deployment requirements. When image separation is ambiguous, Scout first offers labels, bounding boxes or pixel masks/cutouts as output choices. **Assumptions & limitations** contains the sample size, workload and measurement limits. Neither mode executes your prompt. Click a model row to expand or collapse its details, beginning with its ID and followed by measured evidence, benchmark scores, evaluation configurations, separate input/output prices, and source links. With a keyboard, focus the model name and use Enter or Space. On narrow screens, scroll the table horizontally; the model column stays visible. Provider marks appear beside provider names, without icon tiles. They are bundled and served from the same origin; Hugging Face results resolve the repository publisher, using a known lab mark or its bundled organization avatar; otherwise a Hugging Face mark identifies the repository platform. Missing or failed marks leave the provider name visible. No third-party favicon service is contacted. The page follows the device’s light or dark appearance with the Inference Terminal’s compact typography and neutral surfaces. Complete JSON responses are available through the API. **JSON API**, **Capabilities**, and **Sources** expand beneath the search and results. **Sources** uses a compact list; expand a source for its coverage, methods, limits, collection times and refresh failures.

**Source only, not deployed:** the Scout page no longer exposes the JSON API setup, capability index or source catalogue disclosures, including the header API link. Privacy & analytics has its own [settings page](/scout/privacy/), linked from Scout's footer alongside the data policy. The task-logging notice and sharing controls live on that page; existing choices are preserved, and the former settings fragment forwards there. This is a presentation change, not API access control: the JSON API, status endpoint and MCP remain available under their existing contracts, and recommendation rows retain their evidence links and limitations.

**Provider page** in each model's evidence opens its provider's official pricing, model documentation, or website. Model links never fall back to a router. Each quote identifies its source: a router route, independent provider metadata, or a reviewed provider contract. Routes have separate prices. Self-hosted infrastructure cost and incomplete quotes remain unknown. Hugging Face deployment links open the publisher’s model repository and pinned card. Benchmark links lead to the original evaluation sources.

## Connect a coding agent

**Device sign-in setup — npm 0.1.8:** use the stable `latest` tag for the commands below. Version 0.1.8 adds the startup model-decision rule to the device setup introduced in 0.1.7. Hosted Scout serves `/connect` and `/scout/mcp` independently of npm. Claude Code is the only supported client; other agents are not yet supported. The older `0.1.7-preview.5` package does not contain these commands.

Setup takes four steps, and you act in each one only once:

1. **Click Add to Claude Code** on the Scout page. It copies a short setup prompt, shows **Copied to clipboard** above the button and opens a `claude://code/new?q=…` link. Your browser may ask once whether to open the Claude desktop app. Desktop opens a new Code session with the prompt filled in; it never sends the prompt for you.
2. **Send the prompt in Claude**, then allow the command it asks to run. The prompt asks it to run one command, with a 10-minute timeout:

   ```sh
   npx -y openmerit@latest install --client claude-code
   ```

3. **Click Approve** on `https://openmerit.ai/connect`, which the command opens for you. Sign in first if needed. The page shows the client, the device name, the scope, an eight-character code that matches the command's output, and the only permission granted: Scout recommendations through MCP. Only approve a request you started yourself.
4. **Done.** The command has kept waiting, so nothing needs rerunning. It stores the device key, adds the `openmerit` MCP server with `claude mcp add-json`, writes the Scout skill and startup rule, and checks the connection with a real MCP `initialize` and tool discovery (not a search). The approval page switches to *Claude Code is connected* once the command has collected its key, and Claude Code reports the result. **Start a new Claude Code session** to load the startup instructions; reconnecting `openmerit` in `/mcp` alone is not enough.

The prompt contains no key, account data or task draft. If the Claude desktop app isn't installed or your browser blocks the link, paste the copied prompt into Claude Code instead. The command needs `npx` (Node.js 22.19 or newer) and the `claude` CLI on the PATH of that session; the installer runs `claude mcp add-json`. Having the desktop app alone does not establish that those commands are available. See [Claude Desktop](https://code.claude.com/docs/en/desktop).

When the command can open a browser, it waits up to nine minutes for the Approve click. When it can't, for example over SSH, it prints the link and code and returns after 30 seconds with exit code 10 so the agent can show them. Rerunning the same command resumes the same request; it does not issue a new code. Codes expire after ten minutes and can be used once. `OPENMERIT_LOGIN_WAIT` sets the wait in seconds.

`--client` defaults to the agent running the command (Claude Code sets `CLAUDECODE=1`). `--scope` defaults to `user`, which makes Scout available in every project. `--scope project` writes the skill to `.claude/skills/` and startup rule to `.claude/rules/` in the current project and stores the MCP server in Claude Code's private per-project (`local`) configuration, so the key never enters the repository. `--plan` lists the changes without making them. `--json` prints one result object for agents.

| Exit code | Meaning |
|---|---|
| 0 | Done |
| 10 | Approval pending; open the link, approve, rerun |
| 11 | Denied, expired or rejected key; rerun for a new code |
| 20 | Conflict; nothing was changed |
| 30 | OpenMerit unreachable |
| 1 | Other error |

**Conflicts are never overwritten.** Install stops if Claude Code already has an `openmerit` server it did not create, if an installed file was edited, or if `openmerit` exists in another Claude Code scope or `.mcp.json`. An install record stores hashes of what it wrote, so rerunning repairs missing files and updates unchanged ones. An unchanged rule file from the earlier source preview (`rules/openmerit.md`) is removed; an edited one is left alone.

### Device keys

Each approval creates one key for one device and client, limited to `scout:recommend`. It is accepted only on `/scout/mcp`; the JSON API and browser still require a Clerk session. The server stores only SHA-256 hashes of keys and device codes. A key expires after 90 days without use; each use extends it. The usual per-user rate limits and analytics preferences apply.

Missing or invalid MCP credentials return HTTP 401 with `device_key_required`. The recovery command is `npx -y openmerit@latest install`; the older preview package does not include the installer.

On the device, the key is stored in `~/.config/openmerit/credentials.json` (mode 600; `OPENMERIT_CONFIG_DIR` or `XDG_CONFIG_HOME` override the location) and in Claude Code's MCP header configuration, which Claude Code stores in plain text. The CLI never prints the key and scrubs it from any failing `claude` output. Do not paste keys into a chat.

Revoke devices under **Connected devices** on `/connect`. `openmerit remove` and `openmerit logout` also revoke this device's key.

### Commands

```sh
openmerit install [--client claude-code] [--scope user|project] [--plan] [--json] [--no-open]
openmerit status  [--client claude-code] [--scope user|project] [--check] [--json]
openmerit remove  [--client claude-code] [--scope user|project] [--plan] [--json]
openmerit login   # replace an expired or revoked key, then rerun install
openmerit logout  # revoke and delete this device's key
openmerit project --policy existing-only|existing-first [--endpoint-id <id> ...] [--plan]
openmerit doctor  # Node.js, claude, reachability, key, MCP check and install state
```

Removal deletes the MCP server, the unchanged skill and startup rule, and the install record, and revokes the key. It keeps edited files and `.openmerit/scout.json`.

### Using Scout from the agent

Claude can load the `openmerit-scout` skill when choosing an application inference model, even if you do not mention Scout. The skill-only setup in npm 0.1.7 can miss implicit choices while building an app, including selecting pretrained browser models. Availability in the skill list does not prove invocation. Unrelated edits, ordinary inference calls and the coding agent's own model are outside its scope. On first use in a project, the skill instructs Claude to ask once whether the project is **existing-only** or **existing-first** and which public catalogue endpoint IDs it can reach, then save them with `openmerit project`. Every search passes the saved `endpointIds`, including `[]`. Existing-only never broadens; existing-first asks before an unscoped search. Task descriptions go to Scout and OpenAI; do not send source files, secrets or customer records.

The agent compares model and endpoint, evidence, reference cost/time and unknowns, and helps configure the chosen route. New providers, credentials, spending and deployments need separate approval. A skill is a default, not enforcement. The MCP tool, `scout_models`, is read-only and does not execute models.

### Automatic instruction updates — version 0.1.10

Version 0.1.10 adds default-on updates for the installed Scout skill and startup rule. **npm 0.1.9 does not contain the updater.** After 0.1.10 is published, existing installations need one explicit installer rerun in their original scope to acquire it. A website or Scout deployment cannot bootstrap files on someone else's machine. Compatible hosted Scout changes already reach agents through their next MCP request.

Version 0.1.11 also recognizes updater entry points reached through symbolic links. Existing installations need an installer rerun to acquire this updater-code fix; automatic instruction updates do not replace the installed updater itself.

Setup discloses automatic updates and installs asynchronous `SessionStart` and `Stop` command hooks. They check npm's stable `latest` release at most once per 24 hours per installation. Offline or failed checks retain the installed version and retry at the next eligible hook after 15 minutes. `Stop` means Claude finished responding, not that the user's task succeeded. No background service watches connectivity while Claude Code is closed, and checks never ask Claude to keep working. A one-shot `claude -p` process can exit before an asynchronous Stop hook finishes; startup checks and later sessions provide the retry path. Managed policies or disabled hooks can prevent automatic checks; an explicit manual update remains available.

Completed failed checks emit a non-blocking Claude Code notice: the installed version was retained, automatic checks retry after 15 minutes during use, and the user can say **“update openmerit”** to retry manually. Hooks during that backoff stay silent; another eligible failed check emits the notice again. Current-version checks and disabled automatic checks stay silent. Delivery depends on Claude Code processing the completed asynchronous hook output; an exiting one-shot process may not show the notice.

The updater lives beside the install record, independent of the npx cache. User scope adds hooks to Claude's `settings.json`; project scope uses `.claude/settings.local.json`. Existing settings and other hooks are preserved. It downloads only from the public npm registry, verifies the package's SHA-512 integrity against registry metadata, and reads a fixed instruction JSON bundle in memory. It does not install the downloaded package or execute its scripts. npm sees the normal network request, but receives no device key, installation ID, task, project path or transcript. Registry integrity protects against mismatched bytes, not a compromised publisher or registry.

Only bundles declaring the supported instruction/permission contract update automatically. Releases that change provider permissions, authentication, analytics, hook code or MCP configuration need a reviewed installer upgrade; the updater leaves those components alone. The contract is a release-maintainer assertion, not an automated security review of prose. Maintainers must change the contract when those boundaries change. Updates preserve credentials, project policy and user-edited instructions, hooks and updater files. Writes are serialized and journaled so an interrupted instruction update can recover the previous complete pair; intervening user edits stop recovery instead of being overwritten.

Start a **new Claude Code session** after an update to reliably load the new instructions. A running session may retain earlier instructions; reconnecting `/mcp` does not reload the startup rule. The installed rule tells Claude how to handle “update openmerit”; this natural-language trigger is guidance, not an enforced command.

Commands in version 0.1.10 and later:

```sh
openmerit update                         # check immediately, even if auto updates are off
openmerit update --auto off              # persistently disable automatic updates
openmerit update --auto on               # enable them again
openmerit update --scope project --json  # update this project's installation
openmerit install --client claude-code --auto-update off  # opt out during setup
```

Use the same scope as installation. `OPENMERIT_AUTO_UPDATE=off` suppresses automatic checks in that process without disabling an explicit manual update. Installer reruns preserve a saved opt-out. `status` includes the automatic-update preference, hook readiness and instruction version; `remove` removes unchanged owned hooks/updater files and retains personal settings and edited files. Manual updates report conflicts, retryable failures or a required installer upgrade instead of claiming success. If lock recovery itself was interrupted, inspect the PID in `openmerit-update.lock.recovery` beside the install record and confirm that process has stopped before removing that recovery lock and retrying.

### Startup model-decision rule — npm 0.1.8

The 0.1.8 installer adds `rules/openmerit-scout.md` alongside the skill. Unlike an on-demand skill, an unscoped [Claude Code rule](https://code.claude.com/docs/en/memory#organize-rules-with-claude/rules/) loads at session startup. The rule requires loading the skill and consulting Scout before any application-model choice, even when the user only asks for a feature. It explicitly includes pretrained/local/browser models, weight downloads, SDK defaults, embeddings, OCR, speech, classification, detection and segmentation. An explicit user model choice must be assessed, not silently replaced. Unchanged approved models and ordinary inference calls do not require another search.

Rerunning `npx -y openmerit@latest install --client claude-code` with version 0.1.8 or later upgrades an unchanged npm 0.1.7 installation without another device approval. User scope writes the rule under `~/.claude/rules/` (or `CLAUDE_CONFIG_DIR`) and covers new projects and Desktop's no-folder sessions. Project scope writes `.claude/rules/openmerit-scout.md` and applies only in that project. The existing skill, private MCP entry and key locations are unchanged. Start a **new Claude Code session** after installation or upgrade; reconnecting `/mcp` alone does not reload startup instructions.

The installer records rule ownership, repairs missing rules, resumes interrupted updates and refuses to overwrite edited or unowned rules. `--plan` includes the rule without writing it; `status` reports its state and `doctor` treats a missing rule as incomplete. `remove` deletes only the unchanged owned rule and retains edited rules and other user instructions.

If policy is missing, Claude must ask before searching. If Scout is unavailable or has no suitable evidence/candidate for the required deployment, the instructions require disclosing the gap and pausing model selection, not choosing from memory or an SDK default. Model-independent work may continue. Catalogue coverage is not guaranteed for local/browser models. These are behavioral instructions, **not hard enforcement**; Claude can still fail to follow them. Older 0.1.7 installations remain skill-only until upgraded; website deployment alone does not update an installed client.

## JSON API

```sh
curl https://openmerit.ai/scout/api/recommend \
  -H "Authorization: Bearer $SCOUT_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"Upload an invoice and extract six fields as JSON"}'
```

`POST /scout/api/recommend` requires a current Clerk session bearer token and `prompt`: a non-empty string of at most 4,000 characters. The JSON body is limited to 18,000 bytes. Unknown properties are rejected.

The JSON API additionally accepts optional `modelIds`, a unique list of up to 100 legacy OpenRouter catalogue IDs. This preserves the existing-project inventory contract; a model ID alone does not grant a direct-provider endpoint. To select specific routes instead, use `endpointIds`, a unique list of up to 100 IDs returned in `recommendations[].endpoint.id`. Both filters intersect if supplied. This filters eligible candidates before ranking; an empty or unknown-only list returns no models. The response echoes `selection.modelIds` and `selection.endpointIds` (`null` for unscoped). Scoped requests omit specialist discovery because it is outside the declared inventory. No endpoint URL or credential is accepted. The [agent evaluation runner](agent-evaluations.md) uses this for existing-project scope; the hosted API supports it as of 15 October 2026. The npm evaluation commands are available in `openmerit@0.1.7-preview.3` under the `preview` tag, separate from stable `0.1.6`.

Recommendation `id` preserves legacy router IDs; a new protocol receives a distinct catalogue ID. `modelId` is the canonical model identity, while `endpoint.modelId` is the identifier to send to that provider. Clients must use the returned protocol and deployment requirements rather than treating every option as a chat-completions model. Existing npm clients using legacy `modelIds` remain scoped to their router inventory.

The response contains:

- `recommendations`: up to five distinct model IDs with rank, provider, `endpoint` (provider model ID, protocol, configuration, deployment, provenance and limits), reason, quality signal, evidence coverage, input/output pricing, reference timing when available, supported inputs, and source-linked evidence. `reference` gives cost in USD and completion seconds (`null` when unavailable). Benchmark mode includes assumed input/output tokens; measured mode sets those token counts to `null` and supplies `measured` with observed field accuracy, the reliability-capped `effectiveQuality`, cross-task `availability`, valid outputs, sample size, charge coverage, provider routes and evidence provenance. `comparison` gives the number of comparable candidates and average shared-capability coverage. The compatibility `pricing.blendedPerMillion` field retains its old 3:1 calculation but is no longer scored.
- `task`: interpreted capabilities and weights; image-input, structured-output, tool-calling and persistent-memory requirements; `requirements.tools` (`yes`, `no`, `unspecified`); `requirements.memory` (`none`, `session`, `persistent`, `unspecified`); consequential `uncertainty`; `workload.input` and `workload.output` (`short`, `medium`, `long`), `workload.priority` (`balanced`, `quality`, `cost`, `latency`), `kind` (`extract`, `classify`, `dispatch`, `code`, `act`, `remember`, `sql`, `calculate`, `retrieve`, `judge`, `detect`, `segment`, `visual_unspecified`, `other`), `outputContract` (labels, decision, boxes, masks, text, or unspecified), `outputFormat` (`json`, `other`, `none`, or `unspecified`; only specifically established formats can match format-specific calibration), `outputCardinality`, optional `clarification`, and `interpreter: "openai-hybrid"`. `keywords` is retained as an empty compatibility field. A general-purpose fallback means no specific capability was identified, not that the classifier was unavailable.
- `specialists`: a separate discovery object with up to three Hub repositories, metadata-based task fit, declared licenses, pinned model-card revisions, live hosting metadata when supplied, and original reported evaluation records. `mode` is `discovery`; quality, cost and latency are `null`. These candidates are not additional ranked places. Compatible vision deployments may instead enter a task-fit shortlist; they are not counted twice. Unsupported task types can omit this field.
- `specialistRepositories`: bounded Hub catalogue count, separate from the evidence-backed `models` count.
- `calibration`: released task profiles, scopes, sample sizes, measurement dates and expiry status.
- `checkedAt`, `stale`, `degraded`, `sources`: index and source health, feed purpose, limitations and coverage. Each source has its own last successful check. A failed source is marked degraded; any hourly source unchecked for more than two hours makes the index stale. Versioned endpoint contracts disclose their manual review date separately; a scheduled metadata refresh does not renew those price claims.
- `benchmarks`, `benchmarksWithResults`, `models`, `providers`, `availableModels`, `eligibleModels`: indexed series, series with matched results, evidence-backed models, distinct indexed model providers (including alias consolidation), catalogue models, and eligible candidates for this task. `endpoints` counts hosted route listings, including reviewed typed contracts; `independentModels` counts catalogue identities with metadata independent of OpenRouter. Neither number implies credentials or verified account access.
- `wordCloud`: capability vocabulary with benchmark and model coverage counts.
- `ranking`: algorithm version, `mode` (`measured`, `benchmarks`, `task-fit`, or `clarification`), workload and tradeoff weights. A measured `profile` gives the task type, sample size, eligible tested pool, protocol hash, collection and expiry times, and public evidence URL. `benchmarkEligibleModels` reports the separate public-benchmark pool. Rank is the final shortlist order. `ranking.frontier` reports compared configurations, selected distinct models and whether the objective scope is comparable or partial. The compatibility `score` can be `null`; there is no weighted quality/cost/time composite in frontier selection.
- `warnings`, `methodology`, `interpretationMs`, `elapsedMs`: limitations, ranking method, Decisions request time, and measured server ranking time when the runtime provides a sufficiently precise clock (`null` otherwise). Cloudflare freezes its clock during synchronous work, so the hosted page reports measured HTTP round-trip time instead of claiming zero-millisecond computation.

`pricing.url` is the official provider destination, not the source of the quoted rates. It is `null` for a provider without a reviewed destination; the page then omits the link. `pricing.basis` identifies the price source and calculation assumptions.

`GET /scout/api/status` returns index coverage, source timestamps, freshness, and scheduler state without submitting a prompt. API responses are `no-store`. Errors return JSON with `error`: 400 for invalid input, 413 for an oversized request, 415 for a non-JSON body, 405 for an unsupported method, 404 for an unknown endpoint, 429 when shared admission limits are reached, and 503 if interpretation is unavailable. A failed, malformed, refused, or timed-out Decisions response never silently falls back to keyword ranking.

Scout recommends models; it does not process an uploaded invoice, execute a task, install a model, or select a provider account.

## Real sources and their limits

The twice-daily updater fetches eight independent public sources, including Models.dev and fourteen bounded Hugging Face pipeline queries, and adds hosted routes and models from OpenMerit's merged model catalogue. Two additional versioned provider-contract sources have explicit manual review dates. It prepares the index before requests arrive. Queries use batched Decisions questions for intent classification, a bounded verification pass for weak answers, and relevant specialist compatibility checks; they do not browse benchmark sites or ask a model to invent rankings.

| Feed | Used for |
| --- | --- |
| [Epoch AI](https://epoch.ai/benchmarks) | Benchmark definitions, published results, source links, and evaluation configurations. |
| [Artificial Analysis](https://artificialanalysis.ai/leaderboards/models) | Benchmark results for 32 evaluations (including HealthBench, HealthBench Professional, Humanity’s Last Exam Diamond, CritPt v2 Hard and FrontierCode 2.0, each its own series beside its earlier version), measured first-answer delay and reference generation speed. Elo-style ratings (GDPval-AA, AA-3D and Code2Video-AA) become the expected win rate against the median-rated model, as for Arena; AA-Briefcase is not read. Archived evaluations remain labelled when their models are still available. |
| [OmniDocBench](https://github.com/opendatalab/OmniDocBench#end-to-end-evaluation) | Document parsing scores from the explicitly labelled `v1.6_full` leaderboard, including newer document models than the OpenVLM snapshot. |
| [IDP Leaderboard](https://www.idp-leaderboard.org/) | Nanonets' MIT-licensed document processing leaderboard: OlmOCR, OmniDocBench v1.5 and IDP Core (key information extraction, handwriting, tables, visual QA) component scores. The overall average is not imported; per-model dates are not supplied. |
| [OpenCompass / VLMEvalKit](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard) | Visual understanding and OCR benchmark results. Its feed currently identifies itself as 17 September 2025. |
| [Berkeley Function Calling Leaderboard (BFCL v4)](https://gorilla.cs.berkeley.edu/leaderboard) | Native function-calling single-turn, live, and multi-turn scores; separate memory scores for key-value, vector, and recursive-summarization stores. |
| [Arena leaderboard dataset](https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset) | Style-controlled human-preference ratings (CC BY 4.0) from four boards: text request categories (including medicine and healthcare, legal and government, the sciences, and entertainment, sports and media), document prompts, vision (overall, OCR, diagrams, homework and creative writing about images) and Code Arena WebDev (overall, HTML, React and image-to-WebDev). Each becomes the expected win rate against the median-rated model of its category; ratings with confidence intervals wider than 60 points are left out. Each category keeps its own publication date. Human preference, not verified correctness. A separate text **factuality** board (overall, medicine and healthcare, legal and government, business and finance) blends those votes with Arena's automated web verification of each response's factual claims at 25% weight; it is converted the same way and covers web-verifiable claims only. Search and Agent Arena are not read: they rate models together with a search system or agent harness. |
| [Vals AI](https://www.vals.ai/benchmarks) | Overall accuracy on 27 benchmarks Vals AI runs, many on private test sets: healthcare (MedQA, MedCode, MedScribe), legal (LegalBench, Legal Research Bench, HLAB), finance (CorpFin, TaxEval, MortgageTax, Finance Agent, Excel modelling, ClaimBench), public benefits, education, multilingual math, science, code migration, programming and security. Composite indexes, archived versions and evaluations other feeds already supply are not read. Vals AI publishes no data licence; Scout keeps only overall scores with links to each page. Per-model evaluation dates are not supplied. Agent products such as a model inside Claude Code or Codex are not matched to the model. |
| [Scale Labs (SEAL)](https://labs.scale.com/leaderboard) | Published scores from 11 current boards Scale runs, mostly on private sets: Professional Reasoning (legal and finance), MCP Atlas, MultiChallenge, the Remote Labor Index, SWE Atlas (refactoring, test writing and codebase questions), HiL-Bench, DrugDiscoveryBench and Humanity's Sixth Sense (visual reasoning). Each result keeps the date Scale added it. Rows Scale flags as possibly contaminated or deprecated are left out, as are deprecated boards, safety boards, audio, Humanity's Last Exam Diamond (read from Artificial Analysis) and SWE-Bench Pro V2. Agent boards rate a model inside a named harness, such as Codex or Claude Code; the harness is kept as the result's configuration. Scale names configurations inconsistently, so some rows stay unmatched. |
| [LiveBench](https://livebench.ai) | LiveBench 2.0 category scores (long agentic, reasoning, coding, browser, data analysis, research and design) from the newest release's published table, about 30 frontier models per release. No per-model dates; the overall score is not read. |
| [τ-bench by Sierra](https://taubench.com) | pass^1 on τ²-bench airline, retail and telecom and τ-bench banking, from Sierra's own text runs on harness 1.x. Developer-submitted and custom-agent entries are left out, and banking keeps only the AllTools retrieval setup. Few models per domain. |
| [OpenCompass](https://opencompass.org.cn) | Text-only knowledge, abstraction, reasoning, interaction and agent ability averages for about 20 models. The overall score and the safety board are not read. |
| [HELM Arabic](https://crfm.stanford.edu/helm/) (Stanford CRFM) | Mean score across HELM's Arabic scenarios from the newest release, mostly multiple choice. Models missing a scenario are left out; no per-model dates. |
| [BRIDGE Medical](https://huggingface.co/spaces/YLab-Open/BRIDGE-Medical-Leaderboard) (YLab) | Zero-shot average across clinical-text tasks in several languages, mostly open-weight models. Average only; no per-model dates. |
| [SuperCLUE](https://www.superclueai.com) | Chinese general-board category scores: math and science reasoning, hallucination control, precise instruction following, agentic coding and planning, for about 20 models. Boards older than six months are not read. |
| [LLM Stats](https://llm-stats.com/benchmarks) (vendor-reported) | Scores model developers report in announcements, papers and model cards, compiled by LLM Stats and reused with attribution under its terms. Scout reads LLM Stats' 100 most-reported benchmarks and keeps only results it marks self-reported, on 0–1 or 0–100 scales, as separate `· vendor-reported` series that never merge with third-party measurements of the same benchmark. Vendor-reported results count toward ranking. They are not independently verified, and settings such as tools, attempts and reasoning effort follow each vendor. Base (pretrained) checkpoints are never matched to instruct models. A benchmark that leaves the 100 keeps its last results. |
| [OpenRouter](https://openrouter.ai/api/v1/models) | Available model IDs, input modalities, declared tools and structured-output support, context limits, and advertised token prices. |
| [Models.dev](https://models.dev) | Independent native model IDs, declared capabilities and prices via ten direct-provider adapters: OpenAI, Anthropic, Google, DeepSeek, Mistral, xAI, Cohere, Moonshot, MiniMax and Z.AI. Not a performance benchmark or account-availability check. |
| OpenMerit model catalogue | Hosted routes and advertised prices, merged field by field from [Models.dev](https://models.dev), [LiteLLM](https://github.com/BerriAI/litellm), OpenRouter's complete output-modality feed, [Vercel AI Gateway](https://vercel.com/ai-gateway/models), [DeepInfra's own model API](https://deepinfra.com/models), [genai-prices](https://github.com/pydantic/genai-prices) and the [TrueFoundry models registry](https://github.com/truefoundry/models) (MIT). [Eden AI's expert-model list](https://www.edenai.co/docs/v3/expert-models/listing-models) adds the gateway's own OCR and document-parsing, translation, text-to-speech, transcription, moderation and AI-content detection routes, including vendors no other source lists, such as Mindee, Veryfi, Affinda, DeepL, ModernMT, Gladia and Sightengine. Its prices are per character (per million), page, document file, image, request or second of audio or video, exclude Eden AI's platform fee, and come with no stated data licence. Eden's token-priced routes that do not say input or output, free quotes and its LLM catalogue are left out; only speech and OCR models with an existing catalogue ID link to it. TrueFoundry adds per-provider configs for Bedrock, Azure and Microsoft Foundry, Vertex, Gemini, OpenAI, Mistral, xAI, Cohere, Databricks, Deepgram, ElevenLabs and others, including speech, transcription, image, video, rerank, OCR, moderation and decision routes, with batch, priority and flex rates and regional prices where they differ from the base region; retired routes and context-tier surcharges are left out. OpenRouter contributes chat, decision, embedding, rerank, image, video, speech and transcription routes. DeepInfra contributes its chat, decision, embedding, rerank, image, video, speech, music and transcription inventory. Token, character, image, frame and input/output-time rates keep their provider-defined units; conditional overrides and unsupported multipliers remain in the archived source payload instead of being flattened into misleading base prices. Novita is not ingested in this revision: its public OpenAI model list mostly overlaps routes already present and does not supply source-specific pricing or modalities that this importer can normalize. A route is attached only through an exact model identity. Models with such routes that no other source lists become catalogue models, so benchmark results naming them are kept; a catalogue model whose name another listed model already uses is left out. Benchmark rows are also matched through the catalogue's other names for a listed model (host model IDs, listing names and Hugging Face repositories), and after removing run settings such as reasoning effort, thinking budgets and trailing parentheticals. A name shared by distinct catalogue models matches nothing, and a row under Scout's own name for a model outranks loosely matched rows for the same benchmark. Listings are metadata, not verified account access or a performance test. |
| [OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions) | Reviewed typed-choice, predicate and score endpoint contract, inputs and endpoint-specific rates. No general chat timing is inherited. |
| [TypeSafe Jev](https://typesafe.ai/blog/jev-1-14) | Reviewed System One contract. Image and single-box output are experimental; headers, version and incomplete total pricing are disclosed. Jev is a candidate for compatible tasks, not Scout’s ranking engine. |
| [Vercel AI Gateway decision routes](https://vercel.com/docs/ai-gateway/modalities/decision) | Decision models Vercel AI Gateway routes (Laya, Liquid d1, Perplexity Deciders, Jev, Microsoft Decision-1, Baseten Elephant and Mammoth and others), read from the OpenMerit catalogue each refresh with their input-token rates, context and image input. One reviewed contract covers them: `POST https://ai-gateway.vercel.sh/v1/evaluate` with supplied choices, ordered rubric scores and boolean probabilities. They join classification and judging shortlists as task-fit candidates; promotional and free-tier rates are shown as listed and can end. A free-tier ID with the same name as a paid one is listed once. Many decision rates are identical, so between routes with the same reference cost, the model's standing on [JevBench](https://benchmarkheaven.com/jev-models) (capability: intelligence and calibration, ranked systems only) and the [Artificial Analysis Decision Index](https://artificialanalysis.ai/evaluations/artificial-analysis-decision-index) breaks the tie: the share of each board's other entries it beats, averaged over the boards it appears on. Models without results come after those with them at that price; a cheaper route always comes first. Results link by exact model name. These boards measure general typed decisions, not accuracy on your task; the reason and `decisionBenchmarks` show each result. |
| [Hugging Face](https://huggingface.co/models) | Public repository metadata, model-card results and structured evaluation records for specialist discovery. Separate from performance ranking. |

Over 100 benchmark **series** are indexed, including source-specific implementations and versions. This is not 100 independent publishers, and coverage is not uniformly current. Some definitions have no results matched to an available model. Both counts are exposed. Missing metrics and ambiguous or unmatched model versions are omitted. Model discovery merges OpenRouter with independent native-provider records and catalogue host routes before matching public benchmark results. An OpenRouter outage does not prevent an independent catalogue from bootstrapping. Distinct endpoints retain prices and parameters; duplicate hosting entries do not create additional model votes or consume multiple places in the five. Endpoint listings are metadata, not verified access in the caller’s account.

Collection time is not measurement time. A successful refresh does not make an old result new. Some feeds omit measurement dates; those remain unknown. Model release dates are never substituted. OmniDocBench's repository version and leaderboard version differ; Scout uses the version actually printed on the score table. Document parsing is a proxy for invoice extraction, not an evaluation of your exact six fields. New models without comparable evidence may be absent from a task's shortlist.

Epoch AI attribution: “Capabilities & benchmarking,” published at epoch.ai, normalized and matched by Scout under the source's [CC BY 4.0 terms](https://creativecommons.org/licenses/by/4.0/). Other sources are attributed in the source panel and individual evidence links. These are data-source attributions, not a statement about OpenMerit's licensing.

### Pending embedding benchmark and media-price collection

**Not yet deployed:** the publisher source adds [MTEB's final leaderboard summaries](https://huggingface.co/spaces/mteb/leaderboard), including historical/off-menu suites and text, image, audio and video embedding evaluations. It collects only published aggregate and per-task scores plus their model, experiment, suite and training-disclosure metadata—not benchmark datasets, embeddings, predictions or per-example outputs. Different suite versions and experiment settings remain separate; missing aggregates remain unknown. Results are source-reported, not independently verified by Scout. The results dataset declares CC0-1.0; that does not change the individual task datasets' licences.

This is catalogue ingestion only. MTEB results do not yet enter the published chat benchmark index or recommendations, and embedding/rerank workload pricing and recommendation paths remain unfinished. The summaries do not supply evaluated checkpoint revisions or measurement dates; collection and release dates do not substitute for either. An unavailable or malformed suite retains its previous complete results and original collection timestamp.

Also **not yet deployed:** three supplemental [Portkey pricing feeds](https://github.com/Portkey-AI/models) contribute reviewed Deepgram Aura, ElevenLabs speech and Stability AI image quotes. Speech generation rates are per input character, Scribe transcription rates per input-audio second, and image rates per image with any published quality/size tier. Portkey's token-like field names do not establish token billing. Unknown denominators, service aliases, custom calculations and zero-only media placeholders are excluded rather than advertised as free generation. Each response is archived with its source attribution; listed prices are not an account-specific quote or proof of availability. These supplemental prices do not add a media recommendation path.

The pending publisher also discovers Hugging Face inference-provider routes across all tasks, including repositories from personal accounts. This is independent of the admitted-organisation crawl and does not admit an author's entire collection. Bounded pages and repository batches resume from durable cursors; incomplete collection never withdraws existing routes, and source-wide removals wait for completed application. Application batches can become visible incrementally, while the source retains its previous successful-check timestamp until completion. Only exact/reviewed identities link to canonical models, non-token task prices remain unknown, and provider listings do not establish working credentials or task accuracy. This does not widen the executable recommendation contracts.

## Specialist models and open weights

Some specialist models have useful evaluations outside general LLM leaderboards. Scout reads both legacy model-card `model-index` results and the Hub’s [structured evaluation records](https://huggingface.co/docs/hub/eval-results). It retains the original dataset, task or metric, units, configuration, split, reported date, setup notes and source link when supplied. Scores are not averaged across unrelated datasets. A repository revision pins the metadata collected; it does not establish which checkpoint produced every reported score. Reported dates can also default to a commit date. Scout does not validate verification tokens or independently reproduce these scores.

Each refresh's discovery sample takes up to 30 repositories sorted by downloads for each of fourteen pipelines: document question answering, token classification, text classification, sentence similarity, text-to-text generation, text generation, image-to-text, image-text-to-text, image classification, zero-shot image classification, object detection, zero-shot object detection, image segmentation and mask generation. The two image-to-text queries remain OCR-filtered. Bounded pinned configs add label vocabularies for up to six repositories in each fixed-label vision pipeline. Private/disabled repositories, adapters and quantization packages are excluded; fine-tunes remain separate. Downloads select coverage only, never quality. This bounded sample can miss capable low-download specialists.

For document, code and retrieval components, **Specialist models** expands beneath the five recommendations when compatible prepared profiles match. For classification, detection and segmentation, compatible native pipelines may enter the five **Candidates to evaluate** instead. Fixed-label pipelines need complete semantic coverage of the requested categories; one overlapping label is insufficient. Zero-shot pipelines accept supplied labels; masks can require points or boxes. Pipeline metadata is not an accuracy guarantee. Up to three components are ordered by metadata completeness and component diversity after compatibility checks. Domain and language requirements come from Decisions, with no prompt keyword dictionary. Tool execution and persistent-memory tasks do not receive unvalidated specialist matches. Entity taggers, OCR and embedding models are components requiring integration, not interchangeable chat APIs. Repeated repository names are de-emphasized for display diversity without merging their identities. See the release status in [semantic specialist matching](#semantic-specialist-matching-hosted-beta-15-october-2026).

Cards show declared licenses, access gates, language, library, parameter count when known, and live hosting routes listed by the Hub. Missing hosting metadata does not prove that no provider exists. Size is not a hardware-fit estimate. Public weights do not establish an open-source license or permission for a particular use; check the pinned model card and actual license. Scout neither downloads weights nor executes repository code, and it does not send task descriptions to Hugging Face.

Missing evaluations mean **unknown**, not poor performance. Task-fit candidates do not receive quality scores through download counts, self-reported results, or general reasoning benchmarks. A performance ranking requires comparable task checks, full workload cost and timing under a recorded deployment configuration. Self-hosted cost depends on hardware and utilization; a zero token price would be misleading. The same quality/cost/speed comparison can then be used without a provider preference. Model-specific exceptions and inferred free inference are not used.

## Ranking

[OpenAI Decisions](https://developers.openai.com/api/docs/guides/decisions), using `gpt-6-luna`, interprets the work, capability requirements, operation, answer space, serialization and tradeoff priority. It distinguishes writing source from executing it, supplied categories from open identification, text-bearing objects from reading their contents, and session state from persistent memory. Typed answers are validated and checked for consistency. Uncertain answers remain disclosed; they cannot authorize narrow calibration evidence. No prompt keyword rules or model-name bonuses enter selection.

A Structured Outputs call first extracts downstream work, explicit model/endpoint restrictions and preferences using the published metadata definitions and canonical vocabulary. Mandatory conditions use alternatives of complete AND groups; exclusions and arbitrary numeric/date bounds are preserved. Relative dates use a supplied current UTC day. Generic typed comparisons enforce those conditions and intersect optional `modelIds`/`endpointIds` inventories. Missing facts cannot satisfy a restriction, including an exclusion. Unexpressible mandatory restrictions produce no verified matches rather than silently broadening the search. Names mentioned as task data, clients or comparison subjects are not automatic filters. Interpretation is probabilistic and can still misread a prompt.

Image tasks require declared image input and relevant visual evidence in performance ranking. Document reading needs document evidence. JSON/schema tasks need declared `response_format` or `structured_outputs`; required tool execution needs tools support and relevant tool evidence. Persistent memory requires external-memory evidence rather than a large context window. [BFCL memory evaluations](https://gorilla.cs.berkeley.edu/blogs/16_bfcl_v4_memory.html) do not establish built-in memory or performance on the visitor's storage system. Recognized image/video/audio generation and transcription remain outside the supported output index. Ambiguous spatial output produces the existing clarification choices.

Classification and judgment tasks first use applicable benchmark evidence. With no supported benchmark candidates, supplied-label classification and judgments retain the explicitly labelled **task-fit** fallback. Boxes and masks use their compatible task-fit path. Documented operation/output fit, deployment readiness and known reference costs/times determine unmeasured shortlists; quality on the task stays unknown. Between decision models with the same reference cost, their standing on two general decision leaderboards breaks the tie, as described in the Vercel decision routes row of the source table. Open or uncertain identification instead needs name-generating endpoints and relevant evidence; fixed-choice APIs cannot satisfy it. Specialist compatibility uses hourly-prepared public profiles. Unknown compatibility omits a repository with a warning, rather than assuming task accuracy.

For performance ranking, a query embedding retrieves benchmark methodologies, supplemented by capability memberships prepared at each refresh. Bounded independent Decisions predicates verify relevance and a deliberately scored required ability before a methodology contributes proxy evidence. A second Decisions question compares each methodology with the task's description: *direct* (the same kind of work in the same domain, such as MedCode for ICD-10 coding), *domain* (the task's subject domain or profession, but a different form of work), *generic* (a general ability the task uses) or *unrelated*. When direct and domain together hold at least 70% of that judgement, the benchmark is accepted as task-specific evidence, whatever the generic ability check says. A general proxy judged unrelated with at least 80%, such as competition mathematics for clinical questions, is withheld. Direct and domain evidence are listed first on each recommendation. The four nearest methodologies are always assessed, whatever their prepared capability tags, because a vertical benchmark such as TaxEval is often tagged for a different ability than the one the task names. When no single ability clearly determines quality, as with security review or tax checking, Scout uses the required ability the interpreter favours most and adds a warning, instead of failing the search. Similarity never supplies a quality score. Rejected, unknown and unavailable evidence is withheld. If some checks fail, verified evidence can still be used with an explicit warning; if no applicability can be verified because checks are unavailable, the search returns unavailable rather than reporting a false no-match.

Quality comparisons use shared benchmark results, grouped by correlated family and task capability. Different cohorts are not compared through unrelated percentiles. BFCL uses native function-calling configurations with their original labels; correlated series are one family. Missing scores remain unknown and actual zero remains a score. Model/version matching remains conservative. A "+" in a model name is significant, so Command A+ and Command A are separate models rather than one ambiguous name. A catalogue model's dated ID (such as Cohere's `command-a-03-2025`) counts as a copy of the undated model. A relative quality signal is not predicted accuracy or a statistical confidence interval.

Matching calibration observations can supplement quality for their exact operation, format, modality, tools, retention and workload scope. Each admitted model needs at least six calibration cases, and evidence expires after 30 days (seven days before 22 October 2026). An applicable profile past that age stays excluded and the result carries a warning naming the profile and its expiry date, rather than silently using benchmarks only. A match no longer requires five tested models or excludes all untested eligible models. Observations remain tied to their tested router deployment. Repeated calibration request failures still cap effective quality. Calibration episode bills and durations remain source evidence; they are not substituted for only some candidates' reference costs or times. Held-out outcomes and answer keys never enter runtime ranking.

All eligible verified catalogue configurations with relevant evidence enter numerical comparison before model deduplication. A configuration can dominate another only when both have a common quality estimate and known cost and time: it must be at least as good in all three and strictly better in one. Missing objectives and incomparable coverage remain provisional. This cannot establish a global three-objective frontier over every model in the world.

Up to five distinct models represent the supported tradeoffs. Balanced searches keep the strongest, lowest-cost and fastest qualifying frontier points; explicit priorities order the frontier instead. Soft metadata preferences break exact ties. At most two models per provider are selected while alternatives remain. Preferences do not broaden eligibility. There is no weighted quality/cost/time composite. Fewer than five points can remain even when more models are eligible; the API reports eligibility separately from the selected count. Stable IDs resolve exact ties.

Reference input sizes are **1,024 / 4,096 / 32,768 tokens**, and visible output sizes are **128 / 512 / 2,048 tokens**, for short / medium / long. Supplied-label choices use a 16-token output reference. Reference cost is input tokens × advertised input rate + output tokens × advertised output rate. This excludes hidden reasoning, cache effects, image-specific tokenization and other modality charges. Decisions has no output-token charge. Unknown or self-hosted operating cost is not inferred to be zero.

Reference completion time is **time to the first answer token + visible output tokens / throughput**. Endpoint-bound measurements take precedence; otherwise an unbound model-level median applies and is labelled `speed.scope: "model"`. Multiple published configurations retain paired timing before taking a median. Timing bound to one route is not borrowed by another, and early reasoning tokens are not confused with the first answer. Missing timing stays unknown. The page's request duration measures Scout matching, not a recommended model executing the visitor's task.

The shortlist requires evaluation on the visitor's own examples. Public benchmarks, small calibration samples and semantic applicability checks do not prove the optimal membership or order of five models for an arbitrary task. The hosted release uses Clerk accounts for sign-in but adds no public prompt history or additional filter controls. Public methodology embeddings are stored in Vectorize; visitor query vectors are transient. Private prompt-and-evidence analytics are described below.

## Current validation limits

The evidence architecture is validated against the published catalogue, not a million-model production load. Synthetic 10,000-configuration checks cover comparable and disconnected evidence scopes; they do not establish request latency with dense, mixed benchmark coverage. Discovery remains incremental and neither coverage counts nor a shortlist establish a globally verified optimum.

The identification correction was checked on the original bird-name prompt, plant-species identification, supplied indoor/outdoor choices and supplied ticket categories with strict JSON, using real prompt interpretation. Regression checks also cover missing/uncertain label sets, explicit endpoint scopes, specialist discovery, visual evidence and classification calibration boundaries. These checks verify interpretation and eligibility, not the accuracy of model-generated bird or plant names.

The earlier 15 October endpoint-catalogue revision passed **20/20 fresh task-interpretation cases** with expectations frozen before dispatch. They covered image/text labels, predicates, rubric scores, single/all-instance boxes, masks, ambiguous separation, visual/text extraction, code, SQL, tool loops, memory, prose and unsupported generation. A prior 18-case development check exposed two routing errors; those cases were retired before the final set. Unit checks cover output-contract eligibility, exact endpoint scoping, name invariance, duplicate endpoints and unknown prices. This validates tested routing boundaries, not best-five task performance.

A separate execution check used the same twelve 32×32 images from the public [CIFAR-10 test split](https://huggingface.co/datasets/uoft-cs/cifar10), rows 80–91, with ten category labels. Decisions `gpt-6-luna` returned valid choices for 12/12 and matched 5/12 labels, with 0.344s median request time and $0.0002148 total estimated input-token charges at the published rate. This is an estimate, not reconciled billing. A local CPU run of `Xenova/clip-vit-base-patch32`, ONNX q8 through Transformers.js, matched 7/12, with 0.022s median warm inference and about 8.6s initial loading. Local hardware and operating costs were not measured and are not zero. The deployment configuration and cold-start difference matter; these timings are not a universal speed comparison. Public test images may have appeared in training.

These small, low-resolution results are **not ingested into production ranking** and do not establish the correct order of five endpoints. They also do not validate the original full-precision CLIP weights or other checkpoints. Jev is indexed from its published task/endpoint contract and was not executed without its separate credential. Task-fit mode keeps quality unknown instead of promoting cheap or specialized options through an invented accuracy score.

The latest 15 October coding/extraction follow-up made **132 requests**: six new calibration cases and six disjoint new holdout cases per category, with six coding candidates and five extraction candidates. These are authored application scenarios, not private production records. Predictions, scorer and evidence were frozen before holdout dispatch. The v4 quality/cost/speed policy was unchanged. Candidate evidence used only the new calibration split; no model-specific rule or held-out result entered it.

| Latest proposal | Top-five membership, before → candidate | Correct ordered pairs, before → candidate | Release |
| --- | --- | --- | --- |
| Small JavaScript functions | 4/5 → 5/5 | 7/10 → 8/10 | Withheld |
| Text field extraction | 5/5 → 5/5 | 8/10 → 7/10 | Withheld |

The per-category gate required the exact five in the same order as the unchanged policy applied to held-out quality, cost and time, without selected-quality regression. Neither passed. The proposed coding five all passed their six fresh cases, but cost and timing changed the first three positions. Extraction had valid JSON in all 30 held-out responses, yet two models missed fields; a perfect schema did not establish accurate extraction. All rankings remain on their previously released evidence path. This result does not validate exact-five ordering or whole-catalogue membership.

Reported charges were $0.581873295 across calibration and holdout, excluding Decisions. Two empty, content-filtered calibration responses supplied no charge; their cost remains unknown and they scored zero. The harness reserved $0.30 for unknown charges inside its $2 guard. Free-text case, whitespace and terminal punctuation normalization was fixed from calibration before holdout; identifiers, dates and numbers stayed exact. Each holdout order is based on six cases per model and one endpoint run, so it remains sensitive to task mix and timing.

The 15 October revision tested four proposed measured profiles on **72 fresh episodes: six models × three cases × four categories**. The earlier broad audit was explicitly retired to calibration data, alongside additional cases. The candidate policy improved pair agreement only from **24/40 to 25/40** and membership from **17/20 to 18/20**; none of its four complete lists matched the measured order. Coding membership and selected quality regressed; tool ordering regressed; extraction ordering regressed. Those three initial profiles were **not published**, and the experimental quality-smoothing rule was also withheld. The action pool was subsequently expanded from six to nine models by running the three unmeasured candidates from the corrected benchmark shortlist on the same six calibration cases. This adds comparable evidence rather than a model-specific score adjustment; a further fresh test checks the resulting shortlist.

Of the initial four proposed profiles, only external memory passed the release gate: pair agreement **5/10 → 7/10**, membership **4/5 → 5/5**, and mean selected task-check quality **75.8% → 82.5%** within its six-model test pool. This is a validation-gated release decision, not an unbiased performance estimate after selection. The independent tool classification fix remains: a required external-action loop now receives tool and agent evidence instead of being treated as reasoning-only. The same pairwise policy and existing profiles remain otherwise unchanged. A separate two-case check of the memory shortlist matched all ten ordered pairs. Its quality checks include an ambiguous “table only” preference phrase that all five stored literally; accepting that interpretation leaves the order unchanged. This small follow-up is not a general accuracy guarantee.

The expanded nine-model tool profile's final check used **two new cases × five selected models**. All ten ordered pairs matched the measured policy order, and all ten episodes took the permitted actions despite misleading tool-result instructions. This tests order within the five, not the optimal membership of the whole catalogue. Two episodes paused at the local evaluation spend guard; successful responses were replayed locally to continue them without repeating provider calls. Reported test timing sums active provider-call wall time for every model, excluding that harness pause. Production calibration timing remains full-episode wall time.

The candidate action grader was corrected from calibration outputs after test dispatch began but before any held-out outputs were inspected. Final predictions and grader hashes were frozen before inspection; no test outcomes entered calibration. This was not a fully preregistered experiment. Tool action scoring does not assess final prose precision. The release still does **not** establish the correct order of all five models across tasks.

The older results below describe ranking version 3 at that release, not a fresh validation of the current version.

The 14 October development audit used 438 calibration observations, an initial 358-call test, then 159 fresh validation calls after a general reliability correction. The unchanged visual shortlist retained its original 40-call test; it was not rerun. The final validation has three fresh cases per text task and five document photos. The candidate pools were selected before testing and do not cover the whole catalogue.

| Task | Top-five membership, before → after | Ordered pairs, before → after | Field accuracy, before → after |
| --- | --- | --- | --- |
| Text extraction | 5/5 → 4/5 | 6/10 → 6/10 | 43/45 → 42/45 |
| Classification | 4/5 → 4/5 | 4/10 → 9/10 | 30/30 → 30/30 |
| Single-tool dispatch | 1/5 → 5/5 | 6/10 → 8/10 | 15/15 → 15/15 |
| SQL generation | 4/5 → 4/5 | 3/10 → 9/10 | 15/15 → 15/15 |
| Numerical reasoning | 3/5 → 3/5 | 3/10 → 10/10 | 15/15 → 15/15 |
| Long-text retrieval | 3/5 → 4/5 | 8/10 → 6/10 | 45/45 → 43/45 |
| Document photos | 4/5 → 4/5 | 8/10 → 7/10 | 135/150 → 134/150 |

Membership compares against the measured-policy top five within each tested pool. Ordered pairs compare all ten pairs within each shortlist, using the same 60/20/20 policy on actual test accuracy, charges and request time. Overall membership improved from 24/35 to 28/35 and pair agreement from 38/70 to 55/70, but none of the seven final lists matched the entire measured order. Text extraction, long-text retrieval and visual extraction still showed small accuracy regressions. These results do **not** establish that the text-extraction regression is fully resolved or that all five recommendations are optimal.

The first test exposed cross-task request failures; those test outcomes were not ingested as calibration. The reliability correction uses calibration observations only. Provider rate limits and timeouts count as failures, and latency can change between runs. The final sample is small and the visual test contains four payment slips and one invoice, including missing identifiers and tax fields. Reported improvements are development evidence, not a SOTA claim.

## Refresh and data handling

The task description is sent to Scout and then to OpenAI for Decisions interpretation, Structured Outputs metadata extraction, a transient query embedding and, when relevant, specialist compatibility/applicability checks. Hourly preparation also sends public repository metadata and benchmark definitions/methodology to Decisions; no user prompts are involved in that refresh. New search analytics use [PostHog](#product-analytics), with the separate browser/API content choices and beta defaults described above. Runtime request logging remains disabled. The public index and published calibration aggregates are persisted; prompts are not sent to benchmark sites or recommended-model providers. [OpenAI's API data controls and retention policies](https://developers.openai.com/api/docs/guides/your-data) apply to those provider calls. Scout does not claim zero retention by OpenAI. Hosting infrastructure can have separate request-metadata policies. Clipboard actions occur only when selected. Describe the task rather than pasting actual invoices, credentials or private customer records.

**Historical Cloudflare archive:** before the 22 October consolidation, Scout stored submitted prompts with task categories, ordered shortlists, metrics and supporting public evidence in private Analytics Engine records. New writes have stopped. Historical prompt text is unredacted and may contain personal information or secrets; no separately collected account/visitor identity was attached. These records remain accessible only through authenticated Cloudflare account queries and are neither exported nor erased by the PostHog integration.

Cloudflare [retains Analytics Engine events for three months](https://developers.cloudflare.com/analytics/analytics-engine/limits/) and may sample them; that policy does not apply to PostHog. Earlier records are preserved as collected: version 1 has no prompt, and versions 1–2 have no attached evidence snapshot. Missing historical fields cannot be reconstructed. Neither historical analytics nor current recommendation traces prove task success. Scout has no public history/query endpoint.

Each query starts with batched Decisions questions and can make one verification pass for weak answers, followed by relevant specialist compatibility batches. Task interpretation and specialist comparison share a four-second request budget; catalogue reads and ranking add their own time. Optional verification has at most 750 milliseconds and leaves weak answers uncertain if unavailable. Independent specialist batches run in parallel, with at most 100 questions per call. Transient connection, timeout, rate-limit and server failures can receive one retry within the remaining deadline; valid `Retry-After` delays are honored. Authentication, exhausted-credit and quota failures are not retried. Failed interpretation returns an explicit unavailable response; failed specialist checks omit unverified specialist candidates. No keyword fallback is used. These bounds reduce avoidable waits and failures; they do not guarantee a successful search or a particular latency. In-memory admission allows 32 active requests and 600 starts per minute per Worker isolate; a separate edge limiter allows 60 POST requests per minute per IP. Excess requests fail promptly instead of queuing. These are operational limits, not an account-level billing cap or a throughput guarantee. Credentials remain server-side.

Public sources refresh twice daily and independently; reviewed typed-endpoint contracts retain their manual review dates. Freshness follows each source's prepared policy: public feeds have a 13-hour freshness allowance (one twelve-hour cycle plus an hour), while incremental research uses the configured seven-day recheck interval. A source does not become stale merely because it follows a longer research interval. Missing or invalid check dates remain overdue, and actual collection or preparation failures remain degraded. A failed download, changed parser format, or unexpected loss of coverage preserves that source's previous evidence and timestamp while healthy sources and prices continue updating. Fresh catalogue removals still remove unavailable models. All fourteen Hub queries and required semantic preparation must succeed for its discovery snapshot to replace the previous one. A complete outage preserves the previous index. Failure and stale status are exposed in both interfaces.

Source content is parsed as data, never executed. Downloads, decompression, and request bodies are size-bounded; source downloads time out. Links are restricted to HTTP(S). The publisher's OpenVLM data URL uses HTTP and lacks transport authentication; the remaining feeds use HTTPS. Scout does not independently authenticate benchmark submissions.

## Run locally

Requires Node.js 22.19 or newer. Make `OPENAI_API_KEY` available in the server environment before starting; never put it in browser code. It must have access to the Decisions API:

```sh
npm ci
npm run scout
```

Open `http://127.0.0.1:45833/scout/`. This lightweight local preview uses the shared interpreter and matcher with bundled/source-feed data; it does not run the hosted refresh workflow or R2 publication system. Local API and MCP endpoints use that same origin. `SCOUT_PORT` changes the port. The server immediately uses the newer of its bundled index and `.openmerit/scout/snapshot.json` cache, refreshes when older than an hour, and continues hourly while running. Stopping the process stops its local schedule.

`npm run scout:refresh` fetches the eight public sources and atomically updates `scout/snapshot.json` after validation. Successful sources replace their previous data; failed sources retain labelled prior data when available. Source downloads need no provider inference, but preparing new or changed specialist profiles and benchmark interpretations requires `OPENAI_API_KEY` and incurs Decisions usage. Unchanged metadata reuses cached analysis. Without a usable key, required specialist preparation retains the previous specialist snapshot; affected new or changed benchmark definitions remain unclassified. Both failures are reported on the affected source. It does not run model evaluations or update the separate calibration file. Calibration refreshes are an explicit offline evaluation and deployment; no paid scheduled evaluation job is installed. Recommendation queries require the server key and incur OpenAI interpretation/embedding usage. The local source-refresh command updates the bundled feeds; the hosted twice-daily Workflow additionally prepares and publishes its D1/Vectorize serving generation.

## Hosting and verification

`wrangler.scout.jsonc` defines the Scout Worker, a private R2 catalogue, a twice-daily Workflow and a SQLite Durable Object coordinating publication. The cron runs at 00:07 and 12:07 UTC (`7 0,12 * * *`). Configure `OPENAI_API_KEY` and `PARALLEL_API_KEY` as Worker secrets. Refreshes build immutable catalogue versions and prepared candidate pools; searches read those pools directly and never pass through the publication coordinator. A running refresh does not block searches against the active catalogue. The first hosted publication must be validated before serving searches; a missing catalogue fails explicitly. The Worker is `openmerit-scout`, the private catalogue bucket is `openmerit-scout-catalogue`, and the Workflow is `openmerit-scout-refresh`. Both API keys are stored as Cloudflare Worker secrets and read from server-side bindings; they are not shipped to the browser. Separate workers.dev and preview URLs are disabled.

Publication records a durable completion receipt after updating the active pointer, pending-research queue and health status. Interrupted publication resumes unfinished work; a lost success reply can be acknowledged again after the lease is released. Replaying a completed publication cannot roll back a newer catalogue or clear a later refresh's lease. Completed research receipts leave the queue only when publication commits; unfinished research retains its original paid job for a later hourly run.

Research rediscovery updates an existing record with the same stable primary key, even when semantic interpretation does not link it to another catalogue record. Different source/version identities remain separate unless the evidence supports a semantic link. Duplicate primary keys fail publication validation; searches continue against the last committed catalogue while a later refresh reuses durable research receipts.

The 17 October post-deployment recovery reused all 24 saved research jobs and published successfully after this identity correction, with a matching completion receipt and empty backlog. Memory-limit and vector-index readiness retries occurred during that run. Its fresh npm VM acceptance also exposed unavailable books/posters and invoice HTTP responses, empty SQL and safety-rewrite shortlists, and empty installed-CLI plans. These remain beta limitations; the detailed results are in the [release validation record](testing.md#preview4-release-checks-17-october-2026).

Prepared capability pools can overlap. Searches use their prepared ID lists to choose a smaller covering set of pools, preserving the selected candidates and complete evidence while avoiding repeated downloads of the same model records. Explicit model and endpoint inventories retain their existing scope checks. These metadata reads do not fetch source pages or redo hourly research.

`demo/domain-router.mjs` routes the landing page, Scout and `/connect/` to the Scout serving service. Hosted docs, demos, engineering articles and terminal pages/APIs are offline during the rebuild. The separately protected memo was restored through the router on 25 October 2026; see the [cutover record](site-domain-migration.md). There is no Scout admin/report route. Do not deploy the older root `wrangler.jsonc` to publish Scout. `npm run build:scout` packages the landing page, Scout, `/connect/`, the standalone Scout data policy and required shared assets; generated docs, memo and demo applications are excluded.

```sh
npm run check:scout
npm run verify
npx wrangler@4.136.3 dev --config wrangler.scout.jsonc
```

Tests cover measured-profile matching, held-out data isolation, ordered top-five behavior, false-positive regressions, response validation, memory/tool eligibility, provenance, missing metrics, source outage recovery, catalogue removals, provider failure/timeout, admission limits, shared API/MCP interpretation, compact evidence and routing isolation. Real API diagnostic cases are separate opt-in checks that incur provider usage. These establish implementation behavior, not task-level model accuracy. Local checks and CI do not publish a deployment. Physical mobile hardware requires separate verification.