--- name: deploy-test description: "Test a deployed agent version on the platform: run its cases on the version, hold one real conversation, turn the turns that matter into cases and report to the owner. The last step of deploy." metadata: min-cli-version: "0.76.0" next: "" --- # Deploy: test Show that the deployed version does the owner's job on the platform, not just on this machine, and tell the owner what's left: before the pull request lands after a build, or before real users rely on a Recipe deployed from `main`. ## Before you start deploy-connect has checked the CLI, the login and the scopes. Check the stage: the record names the version (runtime, version id, commit), its image is ready, and each server is recorded as connected, faked or missing. If not, go back. ```sh introspection runtimes get --query image_build_status ``` **Check the connections** for the version and environment you'll test in: ```sh introspection bindings mcp list --runtime-id --environment staging -o table ``` Never run a task in an environment missing a connection the Recipe requires. A faked server reaches only a development task while `introspection dev --mcp` stays attached; a missing one means its cases are recorded as not run. A Harbor trial binds the servers its own task declares, so it needs neither. **Say what you'll run before the first paid run,** and get the owner's yes: the environment, the runtime and version, each agent and its model, which cases on which runner, and the number of trials and retries. More trials, reruns or a different model need a new yes. ## 1. Run the cases on the version How: [running cases on a version](references/cases-on-version.md). - **Harbor cases** run as a plane trial of the version (`eval-run`'s `hosted`). Before asking, check what this machine needs (its "Harbor tasks" section) and put any gap in the same question; the owner can choose to record the trial as not run. - Read results as `eval-run` says, and compare each with its local result as `eval-compare` says. - **Vitest cases** run locally at the version's commit, every time, reported as local and faked ([the Vitest cases](references/cases-on-version.md#the-vitest-cases-at-the-versions-commit)). - No Harbor cases: say so in the report rather than skipping the step. ## 2. Hold one real conversation How: [the conversation](references/conversation.md). In staging when every server it needs is connected there, or in development when deploy-connect faked one; say which before you start. 1. **Play the owner:** start a task on the exact version with a realistic request that uses each server, follow it, then send a follow-up and a correction: ```sh T=$(introspection tasks create … --query id | tr -d '"') introspection conversations get "$T" ``` 2. **Show them what you played:** embed the link from `introspection open --task "$T" --print` in your message. Don't open it; it needs no answer. 3. **Offer them their own:** ask with the question tool whether they want to try it. If they do, start a new task on the same version, so their chat holds only their conversation, give its link the same way (`--print`), and ask them to say when they're done. 4. **Read each conversation whole,** keeping your turns and the owner's apart: a turn you played shows what the agent does with your request, not what the owner would ask. **Once a change lands in production,** verify the active commit the same way: a fresh task in the change's situation, read whole, then the judgement trend against the one before. A drop is evidence for improve-collect. ## 3. Turn the turns that matter into cases For each turn where the agent went wrong, or did something the owner wants kept, draft its case with `eval-case`, get the owner's approval (including that a real conversation may be committed), add it as a Harbor case, run it on the version and promote the new baseline, as [turning turns into cases](references/cases-from-turns.md) says. A new case that fails goes back to build-draft. ## 4. Report to the owner In plain words: what passed, what failed and why, what's left for a person to judge, every server that was faked, and the environment each result is from. - **After a build:** write it into the pull request description and Part B ([the report](references/conversation.md#the-report)), commit to `pr/`, and push once the owner says to. The pull request is ready to land when the cases pass or each failure has a reason the owner accepts; landing it is their decision. - **A Recipe deployed from `main`** with no pull request: give the report in your message, and commit it only if the owner asks. ## 5. Turn on monitoring Improve works from real conversations, so check the project watches them for patterns: ```sh introspection automations list introspection automations update --enable # with the owner's yes ``` If the login can't read automations, give the owner the link the command prints and say what to turn on there. ## 6. Close the deploy One short message: the agent is deployed and tested, where it runs, and that improving it starts once it has real conversations. Then stop. ## Guardrails - **Keep the four claims apart.** A version is created when it exists, active when an environment serves it, deployed when it's the one selected there, and verified only when a conversation on it showed the work done from the commit you meant. A version existing, an environment serving it or a task's status alone is not verified. - **A pass in one place proves only that place.** A local run proves the package, a pull request version tested in staging proves that version in staging, and production serves what lands on `main`. Say which one you tested. - **A failure before the agent runs is a blocker, not a failing case.** Setup, credentials, connections, the runner and the platform failing are reported as what blocked the run, never scored as the agent's. - **Network behaviour is proven here, not locally.** A local run had open network; on the platform the agent reaches only the hosts its connections allow. A tool call that worked locally and fails here on network is a missing connection, not an agent fault. - **Never change an approved case to make it pass,** and never tune the agent here: a fix goes back to build-draft on the same branch, and deploy runs again on its version. - **No credentials, no connecting, no merging.** The owner signs in, connects and lands. ## Done when - Every Harbor case has run on the version, or has a written reason it didn't. - Every Vitest case has run locally at the version's commit, reported as local and faked, or the report says there are none. - Every result says where it ran and what was faked. - One real conversation on the version has been read in full, the owner was offered a chat of their own, and the report says which turns you wrote. - The turns that matter are approved Harbor cases that pass on the version or have a reason, and the Harbor baseline includes them. - The report is in the pull request description and Part B (or the message, for a Recipe deployed from `main`), and the owner has it. - The owner knows whether conversations are monitored, and has had the closing message. Deploy ends here; improve starts from real conversations.