<p align="center">
  <a href="https://github.com/dogfood-lab/testing-os">
    <img src="https://raw.githubusercontent.com/dogfood-lab/testing-os/main/assets/logo.png" alt="testing-os" width="280">
  </a>
</p>

# @dogfood-lab/verify

> Central verifier for testing-os. Validates submissions against schema and policy, produces persisted records.

Part of the [`testing-os`](https://github.com/dogfood-lab/testing-os) monorepo — the operating system for testing in the AI era.

The verifier sits between dispatch and persist: every dogfood submission passes through here before it's written to `records/`. Returns a structured verdict (`ok` / `rejection_reasons[]`) so callers — including the `@dogfood-lab/ingest` pipeline — can decide whether to persist the record or surface the rejection to the operator.

## Install

```bash
npm install @dogfood-lab/verify
```

## Usage

```js
import { verify } from '@dogfood-lab/verify';

const result = verify(submission, {
  policy,
  schemas,        // from @dogfood-lab/schemas
  provenance: 'github',
});

if (!result.ok) {
  // rejection_reasons is an array of STRINGS with stable prefixes (see
  // "Error shape" below). Operators discriminate failure class via the
  // prefix; the rest of the string carries the human-readable detail.
  for (const reason of result.rejection_reasons) {
    console.error(reason);
  }
  process.exit(1);
}

// result.record is the persistable artifact
```

## Validators

`@dogfood-lab/verify/validators/*` ships discrete validators that can be composed or called directly:

| Validator | Purpose |
|---|---|
| `validators/schema.js` | JSON Schema check against `@dogfood-lab/schemas` (SHAPE gate) |
| `validators/schema-version.js` | `schema_version` VALUE gate — refuses an incompatible MAJOR against `SUPPORTED_SCHEMA_VERSIONS` |
| `validators/policy.js` | Per-repo policy compliance (prototype-pollution-safe deep merge) |
| `validators/provenance.js` | GitHub Actions run-ID confirmation via API (with timeout guard) |
| `validators/steps.js` | Step-by-step contract checks (gate accumulation, ordering) |
| `validators/verdict.js` | Final verdict synthesis from upstream validator results |

Import a single validator:

```js
import { validateSchema } from '@dogfood-lab/verify/validators/schema.js';
import { validateProvenance } from '@dogfood-lab/verify/validators/provenance.js';
```

## CLI (`dogfood-verify`)

The package ships a `dogfood-verify` bin with two verbs.

**Verify** a submission (local dry-run / explain — never writes):

```bash
dogfood-verify --file submission.json --explain   # human verdict, reasons classified by who-fixes-it
dogfood-verify --file submission.json --json       # machine-readable
# exit: 0 accepted · 1 rejected · 2 operator error
```

**Lint** a policy file (VERIFY-F3, author-time — no submission needed):

```bash
dogfood-verify lint policies/repos/<org>/<repo>.yaml        # or global-policy.yaml
dogfood-verify lint policies/global-policy.yaml --json      # for CI
# exit: 0 clean or warnings-only · 1 errors · 2 operator error
```

`lint` runs the structural schema gate **plus** the data-independent predicate checks (`unknown_field`, `max_depth`, `node_budget`) over every `when` predicate, and emits an **advisory** warning on the `[]` footgun (a negative operator over a `[]` path fails open) with the fail-closed `not(any(...))` rewrite as a suggestion — never auto-applied, never a hard error. It is the `opa check` analogue. It **cannot** statically catch a `type_mismatch` or a fan-out overrun (both are data-dependent) and says so. Full contract + coverage boundary: [`docs/policy-lint.md`](https://github.com/dogfood-lab/testing-os/blob/main/docs/policy-lint.md).

## Submission envelope

The full envelope shape is defined by `@dogfood-lab/schemas` (`dogfood-record-submission.schema.json`). Minimum required fields:

```json
{
  "repo": "org/repo",
  "commit": "<git-sha>",
  "submitted_at": "2026-05-14T15:00:00Z",
  "records": [/* one or more dogfood-record envelopes */]
}
```

Provenance fields (`github_run_id`, `github_workflow_ref`) are required when `provenance: 'github'` is set. The verifier confirms the run ID against the GitHub Actions API before accepting.

## Error shape

`rejection_reasons[]` is an array of **strings** — the persisted-record schema (`dogfood-record.schema.json` → `verification.rejection_reasons`) enforces `items: { type: 'string' }`. Machine-readable discrimination happens via **stable string prefixes**.

### Prefix taxonomy

The verifier emits rejection-reason strings under stable prefixes, each mapping to one of four routing classes:

Discrimination happens by **class**, surfaced by `parseRejectionReason` (below). Every prefix maps to one of four classes: **submission-bad** (the submitter fixes the payload), **operational** (the verifier/tooling faulted), **ingest** (an ingest-side load fault), or **unknown** (unrecognized prefix).

**Retryable** (F-f8952a50, wave 10) is a SEPARATE, narrower per-prefix flag, orthogonal to `class`: may a same-`run_id` resubmission whose ONLY prior rejection carries this prefix still reach an acceptance, once corrected? `packages/ingest/persist.js`'s duplicate guard (`isDuplicate` → `isRetryableRejection`) reads this flag — never `class` or `prefix` directly — to decide whether a stale `_rejected` record blocks a same-`run_id` retry that is now headed to acceptance. Every `operational` / `ingest` / `unknown` prefix is `retryable: false` (an ops fault or an unrecognized signal is never the submitter's to retry past). Within `submission-bad`, the taxonomy splits further — **shape/addressing** prefixes ("we could not even read/place/shape your submission") are retryable; **content-verdict** prefixes ("we read your submission and rendered a verdict against its own reported content") are not, so a submitter cannot launder a genuinely-bad run into an accepted one by resubmitting different self-reported content under the same `run_id`. The table below marks each `submission-bad` prefix's retryable value explicitly.

**Submission-bad** — `class: 'submission-bad'` (the submitter's payload failed a validator gate; fix the submission and resubmit):

| Prefix | Retryable | Source | Meaning |
|---|---|---|---|
| `schema:` | Yes — shape | `validators/schema.js` | JSON Schema check on the submission/record envelope failed. The rest of the string carries the AJV path + message. |
| `policy:` | **No — content verdict** | `validators/policy.js` | Per-repo policy gate failed (forbidden tags, missing required fields, surface evidence/CI requirements, or a declarative `when`/`custom_rules` predicate matched — see the [policy DSL](https://dogfood-lab.github.io/testing-os/handbook/policy-dsl/)). Judges the run's OWN reported content (tags, evidence, scenario results); consuming the run_id is the deliberate anti-gaming behavior. |
| `policy-config:` | Yes — shape | `validators/policy.js` | **VERIFY-F1.** A REPO custom-rule predicate hit an eval-time semantic fault the schema could not catch — an unknown leading field, a numeric operator against a non-number, or a depth/width/fan-out budget. The repo authored the bad RULE (config), not the run's content, so the fix belongs to the submitter and is retryable. (A malformed GLOBAL predicate is `VALIDATOR_FAULT_POLICY:` operational instead — see below.) |
| `steps[<id>]:` | Yes — shape | `validators/steps.js` | Step-level contract check failed on a specific step id (gate accumulation, ordering, evidence shape — a completeness/structure mismatch, not a verdict on whether the steps passed). |
| `provenance:` | **No — content verdict** | `validators/provenance.js` | The run was genuinely **absent / not confirmable** — a 404 from the provider API, or the run head did not match the submitted commit/repo. The provider could not confirm THIS specific run happened as claimed; resubmitting different self-reported content under the same run_id to "become confirmable" is exactly the laundering the anti-gaming doctrine blocks. (Operational provider faults — 429/5xx/401/403 — are NOT this class; see `provenance-fault:` below.) |
| `repo:` | Yes — shape | `index.js` cross-field guard | `submission.repo` does not match the owner/repo encoded in `source.run_url` (anti-forgery guard). Emitted as `repo:mismatch: …`. Pure identity/addressing — the run happened, only the repo/run_url pairing was mis-stated. |
| `submission-contains-verifier-field:` | Yes — shape | `index.js` | The submission carried a verifier-owned field (`policy_version`, `verification`, or an object `overall_verdict`) it must not author. |
| `CONTRACT_SCHEMA_TOO_OLD:` | Yes — shape | `validators/schema-version.js` | The submission's `schema_version` declares a MAJOR **below** the supported floor. **The submitter must re-emit** against the current contract. A patch/minor delta inside the supported major range is NOT rejected. (NOT symmetric with `CONTRACT_SCHEMA_TOO_NEW:` below — see that row for why.) |
| `unsafe-record-path:` | Yes — shape | `packages/ingest/run.js`, `writeRecord()` catch | The record passed schema validation but `computeRecordPath()`'s traversal guard (`isUnsafeSegment`, stricter than the schema's `repo` pattern — e.g. `../etc`) still refused to place it on disk. The submitter's own `repo` string is the problem; nothing is persisted (there is no safe path to write to). |

**Operational** — `class: 'operational'`, always `retryable: false` (the validator itself threw an internal error; investigate the verifier, do NOT bounce to the submitter):

| Prefix | Source | Meaning |
|---|---|---|
| `VALIDATOR_FAULT_SCHEMA:` | `runValidator('schema', …)` catch | Internal exception inside the schema validator. The rest of the string carries the thrown `.message`. |
| `VALIDATOR_FAULT_POLICY:` | `runValidator('policy', …)` catch | Internal exception inside the policy validator — including a **GLOBAL** declarative-rule predicate fault (VERIFY-F1): a broken `policies/global-policy.yaml` is an ops incident (the studio's own config), so its predicate fault throws here rather than bouncing to the submitter. The repo-authored counterpart is `policy-config:` submission-bad. |
| `VALIDATOR_FAULT_STEPS:` | `runValidator('steps', …)` catch | Internal exception inside the steps validator. |
| `VALIDATOR_FAULT_CONTRACT_SCHEMA_VERSION:` | `runValidator('contract_schema_version', …)` catch | The version gate was called with an unknown contract key (a programmer error at the call site, not a submission fault). |
| `CONTRACT_SCHEMA_TOO_NEW:` | `validators/schema-version.js` | **F-be0deacd (wave 20).** The submission's `schema_version` declares a MAJOR **above** what this build supports (see `SUPPORTED_SCHEMA_VERSIONS` in `@dogfood-lab/schemas`) — THIS BUILD is behind a schema major its own submitters have already adopted. No resubmission, corrected or not, can ever satisfy a `major > maxMajor` comparison until testing-os itself ships an upgrade — **the operator must upgrade testing-os**. Page ops; do NOT bounce it back to the submitter. Unlike every other row in this table, `validators/schema-version.js` RETURNS this as an ordinary rejection string rather than throwing, so (unlike the `VALIDATOR_FAULT_*`/`provenance-fault:`/`scenario-fetch-fault:` rows) it genuinely persists to `records/_rejected/` and is reachable by `packages/ingest/persist.js`'s `isRetryableRejection()` — **F-51780da9 (wave 22):** rather than trusting the frozen `retryable: false` this prefix's own classification carries, `isRetryableRejection()` re-derives retryability for this ONE prefix against the CURRENT build's `SUPPORTED_SCHEMA_VERSIONS` ceiling, so a stale TOO_NEW rejection unblocks once the operator upgrades testing-os past the declared major. Distinct from the submission-bad `CONTRACT_SCHEMA_TOO_OLD:` above — the two prefixes are asymmetric despite the shared emitter. |
| `submission-malformed:` | `index.js` null/non-object early-return | The submission itself was `null` or not an object — a malfunctioning **dispatcher** sent garbage, not a submitter who authored a bad-but-shaped payload. Page ops / inspect the dispatch pipeline; do NOT bounce it to a submitter. |
| `provenance-fault:` | `index.js` provenance catch | The provenance adapter THREW an operational error confirming the run — a provider **429 rate-limit, 5xx outage, or 401/403 token** fault (`validators/provenance.js` throws these on purpose for non-404 responses). The submitter's payload is fine; the verifier could not reach a verdict. Page ops / retry; do NOT bounce it to a submitter. Distinct from the submission-bad `provenance:` (genuine absence/404). |
| `scenario-fetch-fault:` | `packages/ingest/load-context.js` | The scenario fetcher THREW after exhausting its retry budget (**5xx/429 outage, transport reject**) or hit a **401/403 credential** fault loading a scenario definition. The submission may be perfectly good — the fetch infrastructure faulted. The ingest CLI lets this propagate (exit 2, nothing persisted); a true missing file is the ingest-class `scenario-load: … (reason: not_found)` instead. |

Any future `VALIDATOR_FAULT_<NEW>:` prefix is classified `operational` by family — `parseRejectionReason` matches the `VALIDATOR_FAULT_` head, so a new validator class needs no parser edit. The `submission-malformed:` prefix is matched literally (it is not part of the `VALIDATOR_FAULT_` family).

**Ingest** — `class: 'ingest'`, always `retryable: false` (an ingest-side load fault, not a verifier gate):

| Prefix | Source | Meaning |
|---|---|---|
| `scenario-load:` | `packages/ingest/run.js` | A scenario referenced by `scenario_results` could not be loaded from the source repo (typed-reason: `timeout` / `not_found` / `parse_error` / `invalid_id` / `too_large` / `schema_invalid`). Outages and credential faults are NOT this class — they throw `scenario-fetch-fault:` (operational, above) instead of rejecting the submission. |

### Operator hygiene

Discriminate by **class**, not by hand-rolled `.startsWith()` chains. `parseRejectionReason(reason)` returns `{ class, prefix, detail, retryable }`:

```js
import { parseRejectionReason } from '@dogfood-lab/verify';

for (const r of result.rejection_reasons) {
  const { class: cls, prefix, detail, retryable } =
    parseRejectionReason(r);
  switch (cls) {
    case 'operational':
      // Verifier-side fault. Page ops; do NOT bounce
      // back to the submitter as "fix your payload".
      notifyOps(prefix, detail);
      break;
    case 'submission-bad':
      // The payload failed a gate — surface to the
      // submitter so they fix it and resubmit.
      surfaceToSubmitter(prefix, detail);
      break;
    case 'ingest':
      // Ingest-side scenario fetch. The typed reason in
      // `detail` (timeout vs not_found/…) decides triage.
      triageScenarioLoad(detail);
      break;
    default: // 'unknown'
      // Unrecognized prefix — log + surface raw text.
      log.warn('unknown rejection_reason', r);
  }
}
```

Persistence note: every entry above is round-tripped verbatim through `verification.rejection_reasons` in the persisted-record JSON; the schema enforces `array of string` so any consumer of the audit-DB ground truth sees the same prefix vocabulary.

### Warnings channel (accepted-with-warning)

Not every policy signal is a rejection. A policy rule declared `severity: warn` produces a `policy: <id>: <message>` entry on `verification.warnings` (an optional `array of string` on the persisted record) **without** flipping the verdict to `rejected` — the submission is accepted and recorded, the warning rides alongside it. (`severity: info` rules are logged only and never persisted; `severity: reject` rules go to `rejection_reasons` as above.) Consumers that want advisory signals read `verification.warnings`; the routing decision (`parseRejectionReason`) only concerns `rejection_reasons`. A clean accepted submission carries no `warnings` key at all.

## Docs

📖 Full handbook: **<https://dogfood-lab.github.io/testing-os/handbook/>**

## License

MIT © 2026 mcp-tool-shop