# Agent Experience validation

Run from the package root on Node.js `>=22.19.0`.

## Complete source suite

```bash
npm run check
npm audit --omit=dev
```

`npm run check` includes:

1. setup/default/privacy behavior;
2. SQLite storage and migrations;
3. capture/redaction;
4. consolidation/proposal validation;
5. explicit review/CAS/law gates;
6. selector behavior plus muted per-answer steering provenance, privacy, ordering, and fail-closed visibility;
7. selector adapters and legacy CLI behavior;
8. nonblocking manual Analyze that drains the action-start queue through sequential bounded calls, excludes mid-run appends, preserves committed progress on later failure/cancellation, and emits one final summary;
9. separate-field semantic duplicate routing, method reconciliation, and candidate restoration;
10. future-schema and online backup/journaled restore hardening;
11. bounded observation tail/index/snapshot reads plus safe Analyze rotation/retention;
12. managed local embedding unit checks;
13. two-connection semantic activation and atomic scan adversarial checks;
14. token-lock stale/dead/malformed/ownership recovery;
15. conversational draft/confirm, direct declaration, numbered review, privacy, stale-state, and no-orphan semantic failure;
16. source/import bundling and generated CLI drift;
17. scheduled Analyze no-work/model-call gating, outer/inner lock discipline, sanitized bounded receipts, durable TUI-only transcript delivery across startup/reload, idempotent retained/unreadable receipt handling, stale-session and renderer/append failure retention, fixed local systemd rendering, injected systemctl lifecycle, and config-state preservation.

The suite must perform no real systemd mutation, hosted embedding request, model download, or live scheduled model call. Phase 17 uses fake systemctl/model adapters and temporary unit/state roots.

## Real pinned local-model integration

The normal suite deliberately does not redownload about 150 MB. To exercise the exact pinned assets from an already downloaded fixture:

```bash
AX_LOCAL_MODEL_FIXTURE_DIR=/path/to/pinned/model-directory \
AX_LOCAL_ORT_WASM=/path/to/ort-wasm-simd-threaded.wasm \
node --experimental-strip-types ./scripts/test-agent-experience-phase13-local-embedding.mjs
```

The model fixture directory must contain:

```text
config.json
tokenizer.json
tokenizer_config.json
onnx/model_int8.onnx
```

This integration test verifies:

- exact size/SHA-256 manifest;
- 0700/0600 cache permissions;
- streamed install and idempotent cache reuse;
- network-blocked offline inference;
- 384-dimensional normalized output;
- separate condition/behavior field scoring with a 5,500-bp minimum-field review threshold;
- eight-of-nine precision-first English/Russian/Spanish/German/French/Chinese and cross-language paraphrase routing, including one documented conservative false negative;
- same-topic/different-action and same-action/different-situation rejection;
- 128-token rejection;
- worker idle unload;
- corruption rejection;
- cancellation cleanup;
- managed removal.

Run the same test with an actual Node 22.19+ binary before release, not only a newer development Node.

## Packed artifact

The authoritative gate packs and installs the package itself:

```bash
node scripts/verify-isolated-package.mjs
```

The verifier first runs the source gate, creates one `mkdtemp` root outside the repository, live npm cache, global npm prefix, and live Experience state root, and then:

1. runs `npm pack --pack-destination <temp>/pack` with a temporary npm cache;
2. performs a fresh `npm install --prefix <temp>/install` of that exact tarball with lifecycle scripts disabled;
3. resolves direct `@earendil-works/pi-agent-core ^0.83.0` and supported `@earendil-works/pi-coding-agent >=0.83.0` peers inside the isolated prefix;
4. checks package version, peer contracts, no install/postinstall/prepare hooks, the complete setup/Advisor source allowlist, CLI help/status, package-relative worker resolution, managed dependency size, and the installed skill with Pi's actual packaged skill/frontmatter loader;
5. runs grouped setup and Advisor PTY smokes from the installed copy with explicit package/transcript paths and child-only `AX_STATE_ROOT`;
6. prints transcript/screen artifact paths, byte counts, and SHA-256 digests before unconditional cleanup.

The tarball must include `CHANGELOG.md`, current executable `dist/experience-consolidate.mjs`, setup and all Advisor modules, phase 23–25 checks, both PTY harnesses, the public skill, local worker, and pinned runtime glue. It must contain no model weight/WASM asset, private state, credentials, Python cache, install hook, or source-map leakage.

Do not substitute `npm pack --dry-run`, a repository import, a global skill loader, the shared npm cache, or a legacy-peer-deps install for this gate.

## Real Pi skill/frontmatter loader

Run Pi's actual skill loader against the installed package and require zero diagnostics. A YAML parser alone is insufficient.

## Isolated installed-package TUI

`node scripts/verify-isolated-package.mjs` launches real Pi from the fresh isolated dependency tree with empty isolated Pi agent/session directories, offline mode, and defaults/context/templates/themes/approval disabled. It exercises the grouped `/experience setup` screen first:

- **Learning from conversations**;
- **Guidance and Advisor**;
- **Manage habits**;
- **Automation and privacy**;
- **Status and help**;
- both authenticated model controls in their expected fail-closed state when the isolated agent has no credentials, Analyze/review failure states, duplicate prevention, retention, schedule/review-prompt explanations, status, all-off, and Done.

The companion Advisor driver is test-only, imports `buildAdvisorCustomMessage` from the installed package, and calls Pi's public `sendMessage(message, { triggerTurn:false })` API. It emits exactly one fixture per command/update and never changes production runtime behavior. Preserve installed-package artifacts proving:

- collapsed and expanded approved-habit concern and blocker cards;
- exact `When:` / `Do:` provenance and next-step authority;
- wide and narrow wrapping without terminal overflow;
- silence when no supplied approved habit is violated;
- one card per update;
- no IDs, aliases, scores, checksums, raw output, transcript excerpts, or private paths;
- unchanged live Experience state and no live/global install destination.

The verifier must pass explicit installed-package and transcript arguments to both Python smokes, use only its child `AX_STATE_ROOT`, print the artifact manifest before cleanup, and remove its single temporary root in `finally`.

Also exercise conversational tools separately when changing them: exact draft display, same-turn confirmation rejection, later-turn save, corrected-draft replacement, numbered suggestion/duplicate listing, explicit decision application, stale-list refresh, and idempotent retry.

For direct approved-habit reminder changes, retain the separate steering smoke: the prompt renders first, then `◇ Steered by habit · <exact selected condition>`, then response work/answer; expansion shows exact approved `When:` / `Do:` pairs and unrelated/unsupported cases remain silent.

All visible UI/tool results must contain no habit IDs, checksums, duplicate thresholds, local-model identifiers, provider endpoints, source refs, private paths, API-key instructions, audit fields, or required advanced subcommands.

## Adversarial acceptance matrix

Release evidence must include:

- future `user_version=999` unchanged after rejected open/init;
- populated transactional/idempotent v5→v6 migration;
- hot-writer online backup, logical restore, and `PRAGMA integrity_check`;
- symlink/traversal/tamper/unknown-artifact rejection;
- injected restore interruption resulting in complete old or restored state;
- concurrent approval, re-enable, and promotion barriers;
- approved waiting visibility and unchanged-identity promotion;
- semantic scan batch/cancel/progress/snapshot/write-failure rollback;
- approved-only default scans and zero candidate-to-candidate proposal/same-batch/scan routing;
- obsolete pending method cleanup with precise audit reason;
- restore-after-last behavior for candidates with multiple pending relations, including approved-waiting identity;
- new wording-hash and validated legacy-cache keep-separate continuity, plus changed/corrupt fail-closed cases;
- live/expired/dead/empty/malformed/foreign/mismatch lock behavior;
- bounded append and disjoint Analyze watermarks;
- generation-aware Analyze schemas after source rotation, while wrong-generation source refs still fail closed;
- cross-batch learning from compact structured context;
- 7/14/30-day journaled rotation/retention;
- exact embedding payload privacy probes;
- full disabled/all-off/no-implicit-timer/no-law-write regressions;
- direct declaration creates no row before later-turn confirmation, bypasses repetition only, and rechecks law/conflict/semantic gates;
- semantic-unavailable declaration creates no candidate/relation; clean activation and duplicate-block routing are atomic;
- conversational review exposes numbered sanitized wording only, revalidates hidden snapshots, and rejects stale or same-turn mutation;
- retry/correction/expiry/session-isolation behavior creates no duplicate or replaced-draft habit;
- `before_agent_start` returns synchronously without embedding/model work; the packed TUI smoke requires the submitted message to re-render within 1.5 seconds, then verifies order: triggering user message → one post-render assessment → one `agent_experience.habit_steering` entry → assistant response;
- collapsed rendering identifies every exact selected condition; expanded/malformed rendering stays safe; no-selection emits no marker;
- every enabled selection embeds the request locally and ephemerally, validates the cached condition vectors for the deterministic top 100 eligible habits it considers (confidence descending, then id; beyond-cap habits are excluded deterministically, not failed closed), and uses no lexical-only or vector-only fallback;
- the strict schema-v3 judge receives only a bounded redacted current request, at most four role-tagged prior visible user/assistant messages, and retrieved conditions labeled with deterministic short process-local aliases; it covers every alias exactly, rejects schema-v2, unknown/missing/duplicate/rewritten/original-ID output, context-only applicability, mention/quotation/negation/generic wording and possible later triggers, maps accepted aliases back before all downstream use, and treats a present request about a future-dated subject (for example planning next summer) as current applicability;
- current message remains the sole trigger: assistant/user context may resolve an explicit follow-up such as “yes, do that” or “make it two weeks,” while assistant-only relevance, unrelated topic changes, negation, low confidence, ambiguity, malformed/partial output, timeout, cancellation, missing auth, and state drift fail closed;
- context extraction excludes system/developer/custom/tool/tool-result/thinking/tool-call/image-only/hidden entries, applies 4-message/300-character/1,200-total redacted caps, snapshots once per response, and degrades invalid context to the current-only path;
- contextual retrieval is current-first/newest-context-first and Unicode-safe within 120 UTF-8 bytes while the judge keeps the full bounded context; current-only and compact-context vectors normally use one local embedding batch, primary order wins, duplicate IDs retain the primary result, and secondary-only candidates append within the unchanged cap; empty context preserves current-only behavior;
- confirmed lexical false positives stay silent, while status/code/release/decision, multilingual, and contextual true positives pass vector retrieval plus judgment;
- missing/corrupt vectors fail before the judge; setup guarantees the current-law reply-time top 100 and adds a best-effort law-agnostic fill up to 500 as a law-change buffer (a future law exposing candidates beyond the buffer fails closed with no injection until re-preparation — safe degradation, not silent wrongness), and post-activation maintenance failure never rolls back approved habit state;
- any non-cancellation compact-context batch failure retries current-only embedding exactly once; cancellation never retries, current-only failure remains fail-closed, and the mandatory judge runs exactly once after successful fallback;
- long internal habit IDs never enter runtime judge prompts or adapter candidate lists; aliases and their exact map remain ephemeral, while returned selections, post-judge revalidation, steering provenance, and selected/skipped logs use restored original IDs; latency probes use the same alias protocol;
- selector guidance has no daily quota; repeated eligible messages continue receiving guidance, while hit logs remain audit/provenance only and persist no prompt/context text, derivative, vector, similarity, raw error, or judge rationale/confidence; sanitized failure rows contain only closed reason/stage/mode/model/retrieval-mode values;
- durable provenance stores approved wording/count/time only and never enters LLM context; separate transient guidance enters only the marked response through the provider's system-instruction field, never user/custom conversation content;
- every entry in the package's explicit ten-entry provider-payload allowlist receives immutable, idempotent system-level adaptation; unknown APIs, malformed allowlisted shapes, conflicting sentinels, and marker failure return the original payload with no new marker/guidance;
- tool-loop context receives the same snapshotted context/result/guidance without duplicate extraction, embedding, assessment, markers, or system blocks; no-selection and provenance-failure tombstones also prevent retries, and a new user message cannot inherit old steering;
- non-TUI, renderer/build/append failure produces no habit guidance and only a static sanitized diagnostic;
- judge missteering regressions in the automated suite are a prompt/protocol-layer faithful-judge simulation only; model-level judge precision is validated out-of-band with a real configured model and is deliberately kept outside `npm run check` and release claims. Recorded evidence: the original 8-case probe passed 8/8, and an expanded EN/Russian/paraphrase/negation/quotation/mixed-intent set passed 18/18;
- capture persists at `agent_settled` (deferred across Pi's automatic retries): the last non-empty run's answer is saved; a run whose terminal assistant message ends in `error`, `aborted`, or `length` (truncated) is dropped in full — including its earlier same-run text — so partial/error/truncated output is never saved and an exhausted sequence of failed runs persists nothing; next-input/session-shutdown are backstops.

## Release gate

Before version bump or tag/push, update `CHANGELOG.md` first with verified user-facing changes. Keep work under **Unreleased** until release preparation, then move it under the exact release version/date. Do not reconstruct or guess unsupported historical entries.

Before tag/push:

```bash
npm run check
npm audit --omit=dev
git diff --check
git status --short
```

Then obtain independent DeepSeek, an available model-diverse reviewer, and constitution review of the actual diff plus test evidence. Skip a reviewer that stalls or reports exhausted usage rather than blocking release evidence. Reviewer verdicts do not replace test evidence.

After final commit and an explicitly approved tag push, verify local `HEAD`, `origin/main`, and the new immutable release tag resolve to the same commit while all historical tags remain unchanged. npm publication is a separate manual action and is outside this release scope.

Rollback for this unreleased provider-guidance change is explicit and reversible: first turn **Approved-habit reminders** off in `/experience setup`; for package rollback, pin the prior known-good immutable release with `pi install npm:pi-experiences@0.1.47`, then start a new Pi process (or reload the current one). Source rollback must revert the eventual corrective commit; never rewrite `v0.1.47`.