# Testing Pi Codecks The test surface is intentionally split so public validation is credential-free while live Codecks validation remains explicit and maintainer-controlled. ## Public-safe checks From a clean checkout: ```bash npm ci npm test npm run pack:validate npm run pack:smoke npm run pack:dry-run ``` `npm test` runs unit, fixture, registration, schema-lifecycle, rendering, transport, and package-metadata tests. Built-in 1Password coverage uses an inert fake manager to prove the exact `op run --no-masking -- ` path, executable override validation, sanitization, and no-fallback behavior; it never contacts 1Password or Codecks. These checks use local fakes and must not contact Codecks even when credentials happen to exist in the caller's environment. The repository-only external-provider launcher has an injected-fetch test that verifies the normal credential-provider selection and one fixed exact-read identity query without opening a socket. `npm run pack:validate` checks the npm dry-run manifest against the public allow-list, verifies required Pi resources, rejects private/local paths, and scans packed text for high-confidence sensitive-content patterns. `npm run pack:smoke` creates a tarball in an operating-system temporary directory, removes Codecks variables from the child environment, installs the tarball into a neutral temporary project in offline mode, and verifies the source entrypoint, skills, prompt assets, and direct Codecks tool registration. The temporary files are removed afterward. Public GitHub Actions run only these safe checks. Forked pull requests never receive Codecks secrets. ### Credential-efficiency evidence `card-get-tool.test.ts` compares equivalent full-card retrieval using an injected credential provider and mocked Codecks transport. With reuse disabled: | Cards | Individual credential resolutions / HTTP attempts | Sequential batches (maximum 25) | | --- | --- | --- | | 17 | 17 / 17 | 1 / 1 | | 28 | 28 / 28 | 2 / 2 | | 37 | 37 / 37 | 2 / 2 | These are client credential-resolution and Codecks-fetch counts, not measured 1Password internal HTTP counts or proof of live API support. The state-machine suite separately verifies enabled in-flight coalescing, TTL, cancellation, bounded cooldown, configuration changes, and late-completion precedence. Provider tests use fresh subprocesses to confirm independent process-local lifetimes. The inert fake-`op` fixture drives the actual bundled adapter, asserting helper execution rather than merely accepting an error string; transport tests distinguish its trusted envelope from identical third-party bytes. The bounded-reader suite checks exact limits, absent/misleading Content-Length, abort, stalled cancellation, and timeout cleanup. ### Cooperating-orchestrator scenario (manual guidance check) This scenario checks the instructions in the card-operations reference, not package-enforced agent behavior: 1. Assign a private configuration the public workflow label `credential-scope-1`; schedule one batch (or one necessary single read), leaving later groups and parent verification unscheduled. 2. Substitute a safe child `credential_rate_limited` result. Verify the example workflow stops future same-label groups, including fresh children and parent verification, while retaining earlier successes and already-dispatched outcomes. Report unknown provider retry timing; do not interpret local backoff as a reset estimate. 3. Repeat with `credential_helper_unavailable`: stop and diagnose rather than blindly retry, but do not create or claim rate-limit evidence. Confirm the report contains no token, reference, account identity, private digest, or raw diagnostic. Package cooldown and cache tests cannot establish that arbitrary deployed agents obey these instructions. There is no cross-process circuit breaker; cooperating parents remain responsible for scheduling. ## Optional external-provider live validation `npm run validate:external-provider-live` is a repository-only launcher for optional, separately authorized maintainer work or a trusted adapter wrapper. It is not normal package validation or public-CI work. It reads configuration only from the process environment and makes one fixed authenticated exact-read identity query only when **both** `CODECKS_CREDENTIAL_PROVIDER=external-helper` and the non-secret acknowledgement `PI_CODECKS_ALLOW_LIVE_VALIDATION=1` match exactly. Any missing or different value returns the fixed `invalid_configuration` category before it invokes a helper or fetch; the launcher never selects or falls back to the ambient `environment` provider, even if Codecks tokens exist. It accepts no request, model, or command-line configuration surface. Its only stdout is one redacted JSON line with fixed `status`, `category`, and `durationMs`; `durationMs` is clamped to `0..60000`. HTTP `401`/`403` and `_root.loggedInUser` explicitly `null` or the literal empty string in an otherwise valid identity response report `authentication_rejected`. A missing identity property, nonobject/missing root, whitespace-only string, incompatible shape, or other nonempty unresolvable identity reports `malformed_response`; missing is intentionally conservative because it is not the API's explicit unauthenticated convention. It never prints caught errors, stacks, account/profile/helper paths, tokens, references, API bodies, or vendor diagnostics. It exits `0` only for `authenticated`. Do not run it from public CI or with production credentials. Configure an absolute trusted helper path as described in the [external helper protocol](external-credential-helper-protocol.md); deterministic tests use only injected fake fetch and helper implementations, never a live request. The coverage includes direct and map-resolved users, `null`/literal-empty identities, missing/incompatible structures, unresolved references, whitespace-only strings, and HTTP authentication rejects. ## Explicit live integration validation Run live validation only when you control the target account and understand which checks can mutate data: ```bash npm run test:integration ``` The local command has three configuration outcomes: 1. **Credentials absent:** reports `skipped` and exits successfully. This is not live evidence. 2. **Credentials present, `CODECKS_TEST_DECK` absent:** runs read-only validation and reports that mutation coverage is disabled. 3. **Credentials and `CODECKS_TEST_DECK` present:** runs mutation coverage in the explicitly selected disposable fixture deck, temporarily updates and restores that deck's description, and attempts safe cleanup. Provide credentials through `CODECKS_ACCOUNT` plus `CODECKS_TOKEN` (or their documented aliases). Profiles may be selected with `CODECKS_TEST_PROFILE`. Do not put values in repository files or shell history, and never target a production deck. Additional optional settings enable narrowly scoped checks: - `CODECKS_TEST_VISION_BOARD_CARD` - `CODECKS_TEST_ATTACHMENT_PATH` - `CODECKS_TEST_RUN` The integration script applies conservative request-rate and timeout bounds. A query/dispatch shape change requires successful live maintainer validation before release. `npm run test:all` runs unit checks and then invokes the integration command. Because absent credentials produce a local skip, `test:all` alone does not prove that live validation ran; inspect its reported outcome. ## Focused local bulk-clear validation `npm run test:integration:bulk-clear` exercises the local package's bulk-update preview/apply path and reads back milestone, effort, priority, tags, and assignee removal. Deck and combined deck/assignee removal are unavailable and are not dispatched; credential-free tests verify their whole-batch rejection before network access. It requires explicit `CODECKS_TEST_DECK=Test` and `CODECKS_TEST_MILESTONE` selecting an existing milestone for temporary fixture membership, and uses the package's configured credential provider. It is never part of credential-free `npm test`. The test creates one uniquely named fixture assigned to the authenticated creator and keeps it in Test. Cleanup restores its creator assignment, clears its milestone, and marks it Done; it does not delete the card. The script reports the fixture reference and cleanup outcome. It stops on failed readback or an uncertain write rather than replaying it; inspect the reported fixture and artifact before another run. Credential failures stop subsequent requests, including cleanup. Other-user note visibility is not tested. ## Protected GitHub workflow The separate integration workflow runs automatically after pushes to `main` and remains available through manual dispatch from `main`. It checks out `main` explicitly and uses a protected `codecks-integration` environment containing credentials for a dedicated limited CI user and disposable fixture deck. Configure the environment without a reviewer gate and retain its deployment-branch policy limited to `main`; repository YAML is not a substitute for that external protection. Missing configuration fails before the test starts. Concurrency prevents two mutation runs from using the shared fixture at once. Public pull-request CI and the publish workflow do not receive Codecks credentials or run live validation. Before release, verify that the independent integration run for the exact `main` commit completed with mutation-enabled success.