# Security

The [preview agent-evaluation runner](agent-evaluations.md) is a separate execution path: it runs trusted local agents/graders, passes only basic OS variables and explicitly named environment variables, and preserves agent-owned authentication. It sends task descriptions and optional model IDs to Scout. Local metric reports and budget reservations persist; temporary task outputs are removed. Fresh directories are not security sandboxes. Its budget admission controls do not enforce provider billing limits.

The [budget selector](agent-evaluations.md#budget-selection) in `0.1.7-preview.2` requires an explicit $10, custom or no-fixed-cap choice after model scope. No fixed cap removes monetary admission limits, including Claude's per-attempt dollar flag, while retaining tasks, repetition bounds, timeouts, native permissions and accounting. Saved choices apply across reruns; a foreground watcher can incur further charges on each change until stopped. Provider limits remain independent.

## Credential ownership

OpenMerit accepts no model-provider or Jev credential. Credentials belong to the coding harness, its environment, or an external secret store.

Never place secrets in profiles, policies, intent constraints, summaries, evidence URIs, audit events, fixtures, traces, or committed environment files.

## Mutation authorization

Most intents carry `modelMutationAllowed: false`. Only application-target apply-swap and rollback intents can authorize a mutation, limited to the relevant proposal, prior configuration, and confirmed target ID. Pi's model is explicitly excluded. Automatic swaps require explicit policy opt-in and meeting the configured verified-swap threshold, which can be zero. Otherwise a proposal requires approval. Post-swap evaluation depends on the harness performing the checks and reporting their results. OpenMerit is not a sandbox around the harness's tools.

Persistent wakeup provisioning is a separate authorization boundary. An adapter that needs to install a system scheduler, create a paid cloud agent, or change infrastructure must declare `wakeupProvisioning: "infrastructure_change"`; OpenMerit will not request that provisioning unless the confirmed check policy sets `infrastructureChangesAllowed: true`.

The macOS Pi adapter installs a user LaunchAgent only under that authorization and persistent execution mode. The plist stores the project path, selected Pi model identifier, and executable paths, but no API key. It verifies that Pi authentication works without shell-only credentials before installing. The agent launches Pi headlessly only for due elapsed-time checks, so the configured model provider may receive project context and incur charges under the confirmed policy. Pause unloads the agent; already-running work is not cancelled.

The background runner imposes a 15-minute wall-clock limit and records a nonzero status if Pi fails to complete a due intent. This is a safety bound, not a provider-spend cap. Unattended candidate discovery and model changes still need representative product validation.

## Local state

`.openmerit/` may reveal task names, candidate identifiers, evidence locations, budgets, or model choices. Treat it as sensitive project metadata even though it must not contain credentials.

Pi also records OpenMerit requests and results in its session history. Project context and evaluation data may be sent to the model providers, tools, and evidence systems configured in the harness. Local OpenMerit storage does not imply that all processing stays on your machine.

## Evidence integrity

OpenMerit validates structure and deterministic Pareto conformance, not the truth of arbitrary external evidence. Use authenticated observability, immutable artifacts, and digests where stronger provenance is required.

Do not include credentials, private traces, or proprietary evaluation data in public issues.

## Inference Terminal

The terminal reads supported project metadata, request logs, explicit exports, and authorized loopback runtime inventory/metrics endpoints. It calls no inference endpoint and exposes no mutation API. A local syntax index reads supported Python, JavaScript/TypeScript, JSON, YAML, TOML, and project-local dotenv declarations. It does not execute or import project code, source shell files, or resolve arbitrary expressions. Known credential-variable presence can identify configuration; credential values are discarded and never used for provider calls. Endpoint addresses are sanitized before retention. Static URLs found by the syntax index never authorize network probes.

Loopback runtime probes require supported project endpoint declarations or explicit terminal configuration. No standard ports, remote provider accounts, credential stores, running processes, or general home directories are scanned. Whole-machine readings are context, not per-request measurements; the compute view requires explicit links or supplied host evidence.

Coding-harness histories are outside the terminal's application scope and are not scanned or imported. Application request logs and explicit exports retain only allowlisted metadata, without prompts or responses. Sources must belong to the selected application; the terminal cannot infer the purpose of an unlabeled API response. Records explicitly marked `scope: "development"` are rejected.

The local web service binds to `127.0.0.1` and rejects unexpected Host, Origin, and cross-site browser requests. It only serves bundled assets and specific read-only JSON routes. It is not authenticated against other local processes or machine users. Model names, endpoint identifiers, and relative source paths can still be sensitive. Names and identifiers in imported metadata must be kept free of secrets.

Repository metadata and usage caches live in process memory and disappear when the terminal exits. The terminal creates no persistent index or files in the application. Per-file/parser protections, request-log read limits, runtime inventory limits, and explicit coverage gaps still apply. See [terminal data handling and limits](inference-terminal.md) for supported sources, precedence, retention bounds, diagnostics, and sample-data separation.


## Hosted Scout authentication

Hosted Scout uses Clerk Google/GitHub sign-in. The page and status are public; JSON recommendations require a valid Clerk session bearer token, not just a cookie. MCP methods require a device key from the setup below. Missing/invalid credentials return HTTP 401, and missing authentication configuration fails closed with HTTP 503. Clerk remains the identity system; the analytics integration below can synchronize selected profile fields to PostHog. Raw account IDs are used transiently for the per-user edge limiter and are not joined to historical Cloudflare prompt records. A prompt draft is temporarily stored in the current browser tab during sign-in. No Clerk Billing or automatic agent OAuth is included. See [Scout account sign-in](scout.md#account-sign-in-published-21-october-2026) for token lifetime, rate limits, data handling and the remaining real-login validation limits.

### Coding-agent device keys

The [coding-agent setup](scout.md#connect-a-coding-agent) in npm 0.1.7 uses hosted device sign-in instead of OAuth or tokens in prompts. The copied prompt contains no secret. A signed-in person approves an eight-character code on `/connect`; codes expire after ten minutes and are single-use. Approval mints one key per device and client, scoped to `scout:recommend` and accepted only on `/scout/mcp`. The server stores only SHA-256 hashes. Keys expire after 90 days without use and can be revoked on `/connect` or with `openmerit remove`/`logout`. Locally the key is in `~/.config/openmerit/credentials.json` (mode 600) and in Claude Code's plain-text MCP configuration; the CLI never prints it. As with any device-code flow, only approve a code you started.

**Automatic instruction updates — version 0.1.10:** updater-enabled setup defaults on with an opt-out, installing asynchronous Claude Code startup/response-end hooks. Checks use npm's public stable registry without credentials, task content or installation IDs. SHA-512 verification checks package bytes against registry metadata; it does not defend against a compromised publisher/registry. The updater reads only a bounded instruction bundle, never executes downloaded code or package scripts, and leaves MCP configuration, keys, privacy settings and project policy unchanged. A supported release contract permits instruction updates; maintainers must change it when permissions or data handling change. This is a publisher assertion, not semantic validation of instructions. Edited files are retained, concurrent writes serialize, and journaled recovery stops on intervening edits. New sessions reliably load new instructions. See [automatic updates](scout.md#automatic-instruction-updates-version-0110) for cadence, offline retries, manual commands and the one-time bootstrap required for older installations. Availability follows npm publication independently of hosted releases.

### OpenMerit coding-agent reset — 23 October 2026

**Historical reset, superseded by npm 0.1.7 and hosted device setup.** The reset temporarily removed the installer and MCP transport. The current [Claude Code setup](scout.md#connect-a-coding-agent) uses browser-approved, device-specific keys scoped to MCP recommendations, not pasted session tokens or agent OAuth. The key is stored locally in mode-600 credentials and Claude Code's plain-text MCP configuration. Setup installs an owned Scout skill; conflicts are not overwritten, and removal revokes the device key. Browser Google/GitHub sign-in remains separate.

**npm 0.1.8:** setup also installs an owned, always-loaded `rules/openmerit-scout.md` instruction requiring Scout before implicit or explicit application-model choices, including pretrained/local/browser models and SDK defaults. Edited or unowned rules are not overwritten or removed. Missing policy requires asking; unavailable Scout or insufficient coverage requires pausing model selection. This is model guidance, not an enforced tool gate or authorization to download weights, add providers, spend or deploy. Scope, device-key permissions and task-data handling are unchanged. New sessions are required after upgrading; older npm 0.1.7 installations remain skill-only until upgraded.

Scout serving, main-site documentation and npm publication are separate releases. This reset does not revoke remote grants or clean up files previously installed on other machines. Earlier hosted MCP/authentication descriptions in this guide refer to past deployments, not this build. Browser Google/GitHub sign-in and its Clerk account lifecycle are separate and unchanged.

## Hosted Scout analytics

**Version 0.1.10:** [Scout setup analytics](scout.md#scout-setup-analytics-default-version-0110) defaults on without a prompt prefix; npm publication and hosted prompt deployment remain separate. `OPENMERIT_TELEMETRY=0` and `DO_NOT_TRACK=1` suppress setup analytics and propagate to approval and installed MCP usage. Browser privacy choices and account opt-outs still apply. Terminal/evaluation analytics remain opt-in, and installed MCP prompt-content capture remains off. This supersedes the setup opt-in default below for version 0.1.10 and later.

**Coding-agent lifecycle instrumentation — npm 0.1.9 and matching Scout deployment:** setup clicks, clipboard/deep-link attempts, device authorization states, opted-in installation/component verification and each dispatched MCP invocation gain typed milestones and fixed error codes. Linked setup uses random setup/installation UUIDs, never paths, machine names, device codes, keys or transcript content. The browser handoff explicitly carries usage/unlink choices into setup; standalone CLI analytics remains opt-in. Setup/installation IDs are omitted in unlinked mode, DNT wins, and installed MCP prompt-content capture stays off. Request-only suppression no longer overwrites the saved account-wide preference. Explicit account opt-out/deletion gates remain in force. See the [lifecycle contract](scout.md#coding-agent-lifecycle-instrumentation-npm-019) for event meanings, bounded delivery retries and diagnostic counters. Existing installations require an installer rerun; release acceptance is separate, and no historical completeness, replay, transcript collection or skipped-call detection is claimed.

**Version-5 instrumentation:** the [serving contract](scout.md#version-5-instrumentation) adds validated native web events and explicit CTA-only `$autocapture` mapping, not general DOM capture. Acquisition/UTM/device values are categorical allowlists; public URL properties are reconstructed without raw URLs. Browser performance is sampled and support-dependent. Linked same-tab signup/search identity is separate from unlinking's request-only identifiers. Logical MCP errors count even with HTTP 200; stage/provider codes, actual attempt tokens and feedback/retry traces exclude raw messages and free text. CLI diagnostics are opt-in in npm 0.1.7, adding sanitized version/platform/runtime and bounded application basename frames, never absolute paths or environment contents. Prompt sharing, retention, deletion gates and necessary provider processing are unchanged. Replay is a separate, inactive spike; no experiment is active. Main-site publication remains blocked by documented source divergence; the Scout rollout does not publish npm or activate alert notifications.

**Beta evaluation rollout, 22 October 2026:** The serving configuration enables `SCOUT_BETA_EVALUATION_LOGGING=true`, alongside `POSTHOG_CAPTURE_CONTENT=true` and complete account-sync/deletion configuration. The browser defaults content on only without a stored content choice; saved off, general opt-out, DNT/GPC and unreadable storage keep it off. API/MCP still require `X-Scout-Analytics-Content: on`; CLI content is always off. This is product evaluation/debugging, not foundation model training. Deployment is owner-approved; this is not legal certification, and beta status establishes no consent or legal basis. Main-site `/docs/` publication remains blocked by source divergence; the [policy](/scout/data-policy/) is generated from the same Markdown and published with Scout instead.

The schema confines all prompt and task excerpt/five-layer prose to `$ai_trace.$ai_input_state={prompt,tasks}`, never ordinary `scout_search_details`. Best-effort redaction retains the 4,000-character prompt, 1,000-character per excerpt/layer and three-task limits. [PostHog's official policy](https://posthog.com/docs/ai-observability/data-retention) stores these AI content properties in `posthog.ai_events` for 30 days, stripping them from normal events at ingestion; metadata follows normal retention. Synthetic live-project records confirmed storage separation and `retention_days=30`, not eventual expiry. Historical v3 details may still hold content. There is no custom retention cron or export, retroactive deletion claim or exact backup-erasure guarantee.

The **Don’t link this browser’s activity to my account** control sends `X-Scout-Analytics-Account: off`. After account opt-out/deletion gates, the server uses fresh unrelated random identifiers, ignores previously linked browser IDs and supplied trace IDs, omits the response trace ID, and sends no profile/identify operation for that request. This is browser/request-specific, not account-wide: Clerk profile/lifecycle analytics can continue under general usage consent, past records remain linked, and copied coding-agent instructions do not inherit the setting. Unlinked events cannot support cross-request funnels or automatic account-based historical deletion; text itself may identify someone. Restrict provider access to authorized personnel and disclose processors, rather than claiming no third parties. The [Scout data policy](/scout/data-policy/) lists providers, deletion limitations and the verified data-request contact, [lazim@openmerit.ai](mailto:lazim@openmerit.ai).

**Deployed, 22 October 2026:** [unified PostHog analytics](scout.md#unified-product-and-search-analytics-deployed-22-october-2026) replaces new Analytics Engine writes with account-linked recommendation metadata, bounded public evidence and sanitized exceptions. Prompt/breakdown text requires separate explicit client opt-in plus configured profile-deletion support; redaction is best effort, not a guarantee against personal or confidential data. General opt-out/DNT/GPC suppresses new analytics, but not necessary OpenAI processing. Existing Cloudflare archives remain under their original policy and are not exported, backfilled or erased.

The historical Analytics Engine archive contains submitted task prompts, task categories and ordered shortlists with metrics and ranking evidence. Its prompt text is unredacted and may contain personal information or secrets; treat this dataset as potentially sensitive. No request object, prompt hash, additional free-text summary, error message, project inventory, IP address, cookie, user agent, visitor ID or session ID was separately recorded in this dataset. Requests still reach the hosting infrastructure and task descriptions still go to OpenAI for Decisions interpretation, Structured Outputs metadata extraction, transient query embeddings and relevant compatibility/applicability checks; public repository metadata and benchmark definitions/methodology also go to Decisions during hourly preparation. Their policies are separate. See [Scout data handling](scout.md#refresh-and-data-handling).

Historical records are accessible only through authenticated Cloudflare account queries, with sampled three-month retention. Missing prompt/evidence fields cannot be recovered. Scout exposes no public history/query endpoint. Both historical and current source evidence and prompt text remain untrusted data when retrieved; never treat them as instructions.

### Optional product analytics

Scout has a visible privacy disclosure and controls. Public homepage/docs navigation events remain source-only; npm 0.1.7 includes opt-in local terminal/map events and aggregate counts. The terminal relay cannot mutate project or inference state, retains loopback Host/Origin guards, and forwards only bounded allowlisted telemetry. Terminal and CLI use a process-only random ID; website events use a separate tab ID. Neither is implicitly linked to Clerk. Application names, raw inputs/outputs, local identifiers and source files remain local. Sanitized error classes/categories (and Scout application file/line frames) exclude raw messages, credentials, source context, arbitrary URLs and filesystem paths. No replay is enabled. All expanded account-linked events pass the same preference/deletion gate as profile synchronization; ordinary event delivery remains best effort. New search details link to identified accounts when profile sync is active, unlike the legacy Cloudflare archive. No external retention setting is changed.

The [PostHog integration](scout.md#product-analytics) uses the EU ingestion endpoint; unconfigured deployments remain off. Verified Clerk user IDs become namespaced SHA-256 pseudonyms for cross-client account retention. Profile sync can associate those identities with names and verified email addresses, so they are identifiable personal data. No identity is added to the historical Cloudflare prompt archive. The browser retains an anonymous ID and hashed account/session transition markers in tab storage, rotating the anonymous ID on account changes. There is no autocapture, replay or visitor-IP forwarding. PostHog has its own retention/access policies, not Cloudflare's three-month policy.

Do Not Track, Global Privacy Control and the documented browser/API opt-outs suppress PostHog collection. Local CLI telemetry in npm 0.1.7 requires `OPENMERIT_TELEMETRY=1`; `DO_NOT_TRACK=1` overrides it. It uses a process-only random identity and creates no identity file. CLI planning requests propagate that consent to Scout while always disabling analytics content capture. These product opt-outs do not disable necessary provider calls or erase historical records. Operators remain responsible for retention, access, spend limits and any required visitor consent. Anonymous visits and rejected requests from the hosted rollout are confirmed in PostHog storage; authenticated-session tracking, profile sync and actual Clerk lifecycle delivery remain unverified.

The deployed [account sync](scout.md#account-profile-synchronization-deployed-22-october-2026) supports identifiable PostHog profiles containing only name, verified primary email, signup time and connected Google/GitHub providers. Signed webhook verification, per-account Durable Objects and a project-scoped PostHog deletion credential are configured. Names/emails come from Clerk's current backend record, never client input. The most recent authenticated analytics preference controls background forwarding; opted-out browsers send a boolean privacy-control request, not a product event. Unknown preferences buffer only bounded pseudonymous lifecycle metadata for up to seven days. Cloudflare stores preference state, hashes, receipts and tombstones, not raw Clerk IDs or a copy of account profiles. No identity is joined to the historical prompt archive.

Clerk webhook signatures are verified over bounded raw request bodies with the SDK's Svix-compatible verifier. A session bearer token cannot authorize webhook delivery. Stable delivery IDs and serialized account state protect against retries; current Clerk lookups prevent stale profile updates. Account deletion persists a tombstone before requesting asynchronous PostHog person/event/recording deletion, including for opted-out accounts. Daily alarms repeat deletion to reconcile already-accepted but delayed upstream ingestion; zero matching persons is a valid no-op, not proof of erasure. Deletion still works when new capture is disabled. Operators must inspect completion and replay failed deliveries. Old `$set` event properties are not erased merely by clearing a current profile field. The installed person-deletion key is restricted to the intended PostHog project and `person:write`; dashboard/query management credentials are not deployed. Production account deletion has not been exercised, and no retention settings or historical records were changed during rollout.

## Scout research storage

Scout's twice-daily refresh reads public sources directly and no longer uses a research service. Decisions processes task descriptions and relevant public evidence. Previously collected public research citations, semantic annotations, publication completion receipts and immutable catalogue versions are stored in Scout's private Cloudflare R2 bucket. Public endpoint/model facts and observations also enter the private D1 serving projection. Vectorize stores public benchmark methodology embeddings, not visitor query vectors. Query embeddings are used transiently; OpenAI has its own API retention policies. Scout uses the prompt-and-evidence analytics policy above. API credentials are held as Worker secrets and are not included in browser assets, catalogue objects or analytics. Source text is treated as evidence, not executable code or instructions. Citations and semantic validation do not independently verify that publisher claims are true.