--- name: deploy-test description: "Test a deployed Introspection agent (Recipe) version on the platform: run its cases against that version, hold one real conversation with it, turn the turns that matter into cases, and report to the owner, saying what was faked. Use after deploy-connect, when the version is ready and each server it needs is connected, faked or recorded missing. Needs `introspection login`; the last step of deploy." --- # Deploy: test Show that the deployed version does the owner's job on the platform, not just on this machine, and tell the owner what's left: before the pull request lands after a build, or before real users rely on it for a Recipe deployed from `main`. ## Before you start - **Read only what you need.** Every `introspection` command takes `--query` (JMESPath), so a step reads the fields it uses rather than whole records: `tasks create … --query id`, `tasks get --query "[status, is_archived]"`, `runtimes get --query image_build_status`, `bindings mcp list --runtime-id --environment staging --query "[].[mcp_server_id, connected]"`. Read a conversation whole with `conversations get` only when you judge it. A single string prints quoted: strip it with `| tr -d '"'` when you capture it. - You need the `introspection` CLI (`npm i -g @introspection-ai/cli`, 0.68.0 or later) and `introspection login` (`introspection whoami`). - **Check the stage:** the record names a deployed version (runtime, version id, commit) whose image is ready (`introspection runtimes get `), and deploy-connect has recorded each server as connected, faked or missing. If not, go back. - **Check the connections first:** `introspection bindings mcp list --runtime-id --environment -o table` compares what the Recipe declares with what's connected for that version and environment. Never run a task in an environment missing a connection the Recipe requires. A server deploy-connect faked reaches only a development task `introspection dev --mcp` serves while it stays attached; one recorded missing means its cases are recorded as not run. A Harbor trial binds the servers its own task declares, so it needs neither. - **Say what you'll run before the first paid run,** and get the owner's yes: the environment, the runtime and version, each agent and its model, which cases on which runner, and the number of trials and retries. More trials, reruns or a different model need a new yes. ## Steps 1. **Run the case set against the version** ([running cases on a version](references/cases-on-version.md)). Harbor tasks run on the platform with `introspection eval run -p -r --runtime-version `: `evals/harbor`, or wherever the Recipe keeps it (`harbor/`, say). Before asking to run one, check what it needs on this machine and put any gap in the same question: the Harbor adapter (`introspection setup --target harbor`, which the owner runs); for `--runtime-version`, `INTROSPECTION_HARBOR_ENVIRONMENT=introspection_harbor.task_environment:IntrospectionEnvironment` and a project API key with the `evals` and `runtimes` capabilities in `INTROSPECTION_DP_URL` and `INTROSPECTION_TOKEN`, set by the owner in their own terminal and never pasted into the chat. Offer to record the trial as not run when the owner would rather not set these up now. `--commit-sha` with the owner's login fetches the commit from the registered repository on this machine, which fails for a private repository this machine's git can't read (git exits 128): report it as a blocker, not a failing case, and offer `--runtime-version` with a project API key the owner sets in their own terminal. Vitest suites run only the local Recipe today, so each Vitest case runs as an eval task on the version, and you read its conversation against the case. Read results as `eval-run` says, and compare each case with its local result as `eval-compare` says. 2. **Hold one real conversation on the version** ([the conversation](references/conversation.md)), in staging when every server it needs is connected there, or in development when deploy-connect faked one through `dev`; say which before you start. Play the owner first: start a task on the exact version with a realistic request that makes the agent use each server it needs, follow it, then send a follow-up and a correction. Then hand the chat to the owner. When their browser is on this machine (a local session, not a cloud sandbox), use `introspection open --task --wait`: it opens the chat, where they can keep talking to the agent, and returns when they click "Return to agent" or close it, so give that call a long timeout (an hour). Use `--print` only where there's no browser on this machine. To chain commands on the task in one call, capture its id: `T=$(introspection tasks create … --query id | tr -d '"')`. Read the whole conversation with `introspection conversations get `. 3. **Turn the turns that matter into cases.** For each turn where the agent went wrong, or did something the owner wants kept, write its case with `eval-case` and get the owner's approval, including that the turn may be committed: it's a real conversation, and may hold their customers' data. Then add its Vitest row with `introspection eval add --from-conversation --turn --case-name `, and add `case` to the row naming the case's id (`eval add` doesn't write it yet). A turn from a local session is the same with `--from-session --turn `. Run the new rows (`introspection eval run -p evals/vitest`); a new case that fails goes back to build-draft with the turn that showed it. 4. **Report to the owner in plain words:** what passed, what failed and why, and what's left for a person to judge. Name every server that was faked, and the environment each result is from. After a build, write it into the pull request description and Part B ([the report](references/conversation.md#the-report)), commit to `pr/`, and push once the owner says to; the pull request is ready to land when the cases pass, or each failure has a reason the owner accepts, and landing it is the owner's decision. For a Recipe deployed from `main` with no pull request, give the report in your message and commit it only if the owner asks: a push to `main` builds a new version. 5. **Turn on monitoring.** Improve works from real conversations, so check that the project watches them for patterns: `introspection automations list` shows whether the observation automation is enabled. If it isn't, ask the owner whether to turn it on (`introspection automations update --enable`); if the login can't read automations, give them the link the command prints and say what to turn on there. 6. **Close the deploy** in one short message: the agent is deployed and tested, where it runs, and what comes next, for example: "Your agent is deployed and its test conversation passed on staging. Once it has some real conversations, we can look at improving it." Then stop: improve starts from real conversations, not from here. ## Guardrails - **Keep the four claims apart.** A version is created when it exists, active when an environment serves it, deployed when it's the one selected there, and verified only when a conversation on it showed the work done from the commit you meant. A version existing, an environment serving it or a task's status alone is not verified. - **A pass in one place proves only that place.** A local run proves the package, a pull request version tested in staging proves that version in staging, and production serves what lands on `main`. Say which one you tested. - **A failure before the agent runs is a blocker, not a failing case.** Setup, credentials, connections, the runner and the platform failing are reported as what blocked the run, never scored as the agent's. - **Network behaviour is proven here, not locally.** A local run had open network; on the platform the agent reaches only the hosts its connections allow. A tool call that worked locally and fails here on network is a missing connection, not an agent fault. - **Never change an approved case to make it pass,** and never tune the agent here: a fix goes back to build-draft on the same branch, and deploy runs again on its version. - **No credentials, no connecting, no merging.** The owner signs in, connects and lands. ## Done when - Every case has run on the version, or has a written reason it didn't (a missing connection, a Harbor-only setup). - Every result says where it ran and what was faked. - One real conversation on the version has been read in full, and the owner has had the chat. - The turns that matter are approved cases with rows, and they pass or have a reason. - The report is in the pull request description and Part B (or in the message, for a Recipe deployed from `main`), and the owner has it. - The owner knows whether conversations are monitored for patterns, and has had the closing message.