--- name: deploy-test description: "Test a deployed Introspection agent (Recipe) version on the platform: run its Harbor cases against that version, hold one real conversation with it, turn the turns that matter into Harbor cases, and report to the owner, saying what was faked. Use after deploy-create, when the version is ready and each server it needs is connected, faked or recorded missing. Needs `introspection login`; the last step of deploy." metadata: min-cli-version: "0.73.0" --- # Deploy: test Show that the deployed version does the owner's job on the platform, not just on this machine, and tell the owner what's left: before the pull request lands after a build, or before real users rely on it for a Recipe deployed from `main`. ## Before you start - **Read only what you need.** Every `introspection` command takes `--query` (JMESPath), so a step reads the fields it uses rather than whole records: `tasks create … --query id`, `tasks get --query "[status, is_archived]"`, `runtimes get --query image_build_status`, `bindings mcp list --runtime-id --environment staging --query "[].[mcp_server_id, connected]"`. Read a conversation whole with `conversations get` only when you judge it. A single string prints quoted: strip it with `| tr -d '"'` when you capture it. - You need the `introspection` CLI (`npm i -g @introspection-ai/cli`, 0.71.0 or later) and `introspection login` (`introspection whoami`). - **Check the stage:** the record names a deployed version (runtime, version id, commit) whose image is ready (`introspection runtimes get `), and deploy-connect has recorded each server as connected, faked or missing. If not, go back. - **Check the connections first:** `introspection bindings mcp list --runtime-id --environment -o table` compares what the Recipe declares with what's connected for that version and environment. Never run a task in an environment missing a connection the Recipe requires. A server deploy-connect faked reaches only a development task `introspection dev --mcp` serves while it stays attached; one recorded missing means its cases are recorded as not run. A Harbor trial binds the servers its own task declares, so it needs neither. - **Say what you'll run before the first paid run,** and get the owner's yes: the environment, the runtime and version, each agent and its model, which cases on which runner, and the number of trials and retries. More trials, reruns or a different model need a new yes. ## Steps 1. **Run the Harbor cases against the version** ([running cases on a version](references/cases-on-version.md)). Harbor tasks run on the platform with `introspection eval run -p -r --runtime-version `: `evals/harbor`, or wherever the Recipe keeps it (`harbor/`, say). Before asking to run one, check what it needs on this machine and put any gap in the same question: the Harbor adapter (`introspection setup --target harbor`, which the owner runs); for `--runtime-version`, `INTROSPECTION_HARBOR_ENVIRONMENT=introspection_harbor.task_environment:IntrospectionEnvironment` and a project API key with the `evals` and `runtimes` capabilities in `INTROSPECTION_DP_URL` and `INTROSPECTION_TOKEN`, set by the owner in their own terminal and never pasted into the chat. Offer to record the trial as not run when the owner would rather not set these up now. `--commit-sha` with the owner's login fetches the commit from the registered repository on this machine, which fails for a private repository this machine's git can't read (git exits 128): report it as a blocker, not a failing case, and offer `--runtime-version` with a project API key the owner sets in their own terminal. Read results as `eval-run` says, and compare each case with its local result as `eval-compare` says. The Vitest cases don't run on the version: run them locally at the commit the version was built from, as [the Vitest cases](references/cases-on-version.md#the-vitest-cases-at-the-versions-commit) says, every time, and report them as local and faked. When the Recipe has no Harbor cases, the version ran none: say so in the report rather than skipping the step silently. 2. **Hold one real conversation on the version** ([the conversation](references/conversation.md)), in staging when every server it needs is connected there, or in development when deploy-connect faked one through `dev`; say which before you start. Play the owner first: start a task on the exact version with a realistic request that makes the agent use each server it needs, follow it, then send a follow-up and a correction. Then give the owner a link to what you played in the app's chat view, from `introspection open --task --print`, embedded in your message; don't open it, since it needs no answer. Ask, with the question tool, whether they want to try it themselves. If they do, start a new task on the same version for them rather than handing over the one you played: the chat they open then holds only their own conversation, not your invented request and data. Give them its link the same way, `introspection open --task --print`, so nothing opens or blocks while they talk to the agent, and ask with the question tool for them to say when they're done. To chain commands on the task in one call, capture its id: `T=$(introspection tasks create … --query id | tr -d '"')`. Read each whole conversation with `introspection conversations get `, and keep apart the turns you wrote and the owner's: a turn you played shows what the agent does with your request, not what the owner would ask. 3. **Turn the turns that matter into cases.** For each turn where the agent went wrong, or did something the owner wants kept, draft its case with `eval-case` in the conversation and get the owner's approval, including that the turn may be committed: it's a real conversation, and may hold their customers' data. Don't write the case file yet: `eval add` refuses a case whose file exists. A case from a turn on the version is a Harbor case, as improve's cases from production are: add it with `introspection eval add evals/harbor --runner harbor --case-id --from-conversation --turn `, which writes the draft case file and the Harbor task `evals/harbor//` naming it, then write the approved draft into that file and the task's verifier for what the case asks (`eval-author`). When the Recipe has no Harbor suite yet, this creates it. A turn from a local session is the same with `--from-session --turn `. Record the owner's approval, naming them and the message they approved in, as `eval-case` says. Run the new tasks on the version, as step 1 runs the others; a new case that fails goes back to build-draft with the turn that showed it. Once they pass, promote a full run of the Harbor suite, the new cases included, as its baseline in a commit of its own (`introspection eval baseline promote `, `eval-compare`): a run on the version measures the commit it was built from, so step 1's run, repeated with the new cases, is that commit's baseline, and a local run with Docker serves as well. Write every commit message as a Conventional Commit: a type (`feat`, `fix`, `docs`, `test`, `chore`) and what changed for the agent's users, as in `feat: ask for the budget before planning`. improve's regression check compares with it. After a build the commit goes on `pr/` with the report; for a Recipe deployed from `main`, ask first, since a push to `main` builds a new version. 4. **Report to the owner in plain words:** what passed, what failed and why, and what's left for a person to judge. Name every server that was faked, and the environment each result is from. After a build, write it into the pull request description and Part B ([the report](references/conversation.md#the-report)), commit to `pr/`, and push once the owner says to; the pull request is ready to land when the cases pass, or each failure has a reason the owner accepts, and landing it is the owner's decision. For a Recipe deployed from `main` with no pull request, give the report in your message and commit it only if the owner asks: a push to `main` builds a new version. 5. **Turn on monitoring.** Improve works from real conversations, so check that the project watches them for patterns: `introspection automations list` shows whether the observation automation is enabled. If it isn't, ask the owner whether to turn it on (`introspection automations update --enable`); if the login can't read automations, give them the link the command prints and say what to turn on there. 6. **Close the deploy** in one short message: the agent is deployed and tested, where it runs, and what comes next, for example: "Your agent is deployed and its test conversation passed on staging. Once it has some real conversations, we can look at improving it." Then stop: improve starts from real conversations, not from here. ## Guardrails - **Keep the four claims apart.** A version is created when it exists, active when an environment serves it, deployed when it's the one selected there, and verified only when a conversation on it showed the work done from the commit you meant. A version existing, an environment serving it or a task's status alone is not verified. - **A pass in one place proves only that place.** A local run proves the package, a pull request version tested in staging proves that version in staging, and production serves what lands on `main`. Say which one you tested. - **A failure before the agent runs is a blocker, not a failing case.** Setup, credentials, connections, the runner and the platform failing are reported as what blocked the run, never scored as the agent's. - **Network behaviour is proven here, not locally.** A local run had open network; on the platform the agent reaches only the hosts its connections allow. A tool call that worked locally and fails here on network is a missing connection, not an agent fault. - **Never change an approved case to make it pass,** and never tune the agent here: a fix goes back to build-draft on the same branch, and deploy runs again on its version. - **No credentials, no connecting, no merging.** The owner signs in, connects and lands. ## Done when - Every Harbor case has run on the version, or has a written reason it didn't (a missing connection, the Harbor setup). - Every result says where it ran and what was faked. - Every Vitest case has run locally at the version's commit, reported as local and faked, or the report says the Recipe has none. - One real conversation on the version has been read in full, the owner was offered a chat of their own, and the report says which turns you wrote. - The turns that matter are approved Harbor cases, their approval recorded as `eval-case` says, and they pass on the version or have a reason; the Harbor suite's local baseline includes them. - The report is in the pull request description and Part B (or in the message, for a Recipe deployed from `main`), and the owner has it. - The owner knows whether conversations are monitored for patterns, and has had the closing message.