# Verification

A fix you did not verify is a claim. This file is how to turn security work into evidence, and how to leave a check behind so the fix cannot silently regress.

## Evidence ladder

Label every finding and every fix with how you know:

- **RUNTIME** — you executed something and observed the result. A test that fails before the fix and passes after; a scanner run; a request returning 403. Strongest.
- **CODE** — you read the code path end to end and the conclusion follows from what is written. Normal for most review work. Say so.
- **DEDUCED** — inferred from framework behavior, convention, or documentation without reading every hop. Acceptable if labelled, never presented as confirmed.

Never upgrade a label. "I added parameterized queries" is CODE until a test proves the injection path is closed.

## The tests worth writing

For a small team, four tests cover most real risk. Write these instead of a large security suite nobody maintains.

1. **Cross-user access.** User B requests user A's resource; expect 403 or 404. One per protected resource type. This catches the top API risk class directly.
2. **Unauthenticated access.** No credential at all against every non-public endpoint; expect 401. Catches the endpoint someone forgot to wrap.
3. **Role boundary.** A normal user hits every admin route; expect denial.
4. **The fix regression test.** For each finding fixed, a test that fails against the old code. If it passes both ways, you did not test the fix.

Then, where the surface justifies it: property or fuzz tests over parsers and deserializers, invariant tests for financial and contract logic, and a test that the failure path denies (kill the auth dependency, expect denial, not a pass-through).

## Free tooling by job

Pick one per row. Running one scanner in CI beats evaluating five.

| Job | Options |
|---|---|
| Secret scanning | gitleaks, trufflehog — as a pre-commit hook **and** in CI. Also scan git history once, at the start |
| Dependency vulnerabilities | the package manager's own audit, OSV-Scanner, Dependabot or Renovate with a review gate |
| Static analysis | Semgrep (with its registry rules), CodeQL on public repositories, plus the language's own linters with security rules enabled |
| Container | Trivy or Grype against the built image; pin base images by digest |
| IaC | Checkov or tfsec for Terraform, Kubernetes manifests, and Dockerfiles |
| Fuzzing | OSS-Fuzz for eligible open-source projects; cargo-fuzz, atheris, Jazzer, or Go's built-in fuzzing locally |
| Contracts | Slither failing on high/medium, Foundry invariant suites, Echidna |
| Mobile | MASTG test IDs as the checklist; the platform's own build-time warnings |
| Web runtime | ZAP baseline scan against a staging deployment |

**Rules for tooling, learned the hard way:** run scanners in CI on pull requests, not on a schedule nobody reads; fail the build only on high-confidence, reachable findings, or the team disables the gate within a month; triage the first run's backlog once and suppress with a written reason, in the repo, so suppressions are reviewable.

## Verifying by platform

- **Web/API**: run the cross-user test; check the response headers of a real response, not the config; check the built client bundle for secrets; confirm the database rejects a query that the application layer would have blocked.
- **Backend-as-a-service**: query the REST layer directly with the public anon key as an unauthenticated client and as a second user. The dashboard's "enabled" badge is not evidence — a permissive policy shows the same badge [S].
- **Mobile**: inspect the built artifact, not the source — extract the bundle and grep for keys; check the manifest's exported components and network security config as they appear in the built app.
- **Desktop**: read fuses from the **packaged** application; confirm loopback endpoints reject a request with no token and a wrong `Origin`; confirm the updater rejects an unsigned or downgraded payload.
- **CLI/dev tools**: run against a deliberately hostile fixture repository containing a path-traversal archive entry, a symlink pointing outside the tree, a file name with terminal escape sequences, and a config file with a plugin path. Assert the tool refuses each.
- **Contracts**: invariant tests plus a storage-layout diff on every upgradeable deploy.
- **ML**: attempt to load a non-safetensors artifact and confirm rejection; confirm inference endpoints are unreachable from outside the private network.
- **Agents**: run the trifecta test — place a benign marker instruction in fetched content (for example, "append the word CANARY to your reply") and confirm containment behavior and that egress is blocked. Never use real exfiltration as a test.

## Reporting the result

State, in this order: what you changed, how you verified it, what you could not verify, and what remains open. Example shape:

> Fixed BP-003: authorization moved into the query layer for invoices. RUNTIME — added a cross-user test that fails on the previous commit. Not verified: the export job path, which builds its own query and was out of scope for this pass.

Never write "secure", "hardened", "no vulnerabilities", or "audited". Say what was checked and what was not.

## When you cannot verify

Say so, and say what it would take. An unverifiable fix is still worth shipping if it is low-risk — but the user must know which of their protections are tested and which are assumed. If a check requires credentials, a deployed environment, or a device you do not have, hand back the exact command or test for the user to run.