--- name: improve-collect description: "Collect the evidence for improving a deployed Introspection agent (Recipe), read-only: the owner's test chats on a branch's version, then production, or a sweep over a period when the owner asks for one. Flags failures, samples ordinary conversations as controls and counts prevalence before anything changes. Use when the owner asks to fix or improve an agent that has traces. The first step of improve; improve-classify follows." --- # Improve: collect Gather what the agent actually did before anyone decides what is wrong with it. This step reads conversations, feedback, patterns and metrics of one deployed version, flags the failures, and hands improve-classify a body of evidence it can group and diagnose. It changes nothing. ## Before you start - You need the `introspection` CLI 0.69.0 or later (`npm i -g @introspection-ai/cli`), logged in to the project that runs the agent (`introspection login`). `introspection whoami` names the project, its Data Plane (`dp_url`) and the login's scopes; a read the scopes don't cover is a gap to report, not one to work around. - Improve starts from traces of a deployed version: first the owner's tests of a branch's version, then production's. An agent with no deployed version has no traces yet; go back to build, or deploy, to give it one. - Read [evidence surfaces](references/evidence-surfaces.md) before the first evidence command, [conversation evidence](references/conversation-evidence.md) before judging a task or a conversation, and [metrics query](references/metrics-query.md) before writing a metrics document. ## What this step covers Resolving the version the evidence comes from, collecting in the loop, on production, or as a sweep the owner asks for, flagging failures, choosing controls, counting prevalence and reviewing a large body of evidence in parallel. ## What it does not cover Grouping the flagged conversations, finding where each went wrong and which layer owns it, labels from the domain owner, and the confirmation gate are improve-classify's. Writing a case, a branch or a fix is improve-propose's, after the owner approves. ## Read-only until the gate Gather every read-only piece of evidence you can reach without asking: inspecting needs no permission. The owner's confirmation comes after the diagnosis, at improve-classify's gate, and covers the proposed edits and the pull request, never the reading. Nothing in this step writes: no branch, case, file in the repository, annotation or pull request. Keep exports and notes in a temporary folder outside the repository (`mktemp -d`), and record evidence by its identifiers. A task awaiting human input is live work, even when its execution has ended. Never cancel it, prompt it or resume it while collecting; report the ones that need someone's attention to their owner. Never disable, roll back, resample or otherwise change a live judge, experiment, runtime or binding while collecting. Those are live state, and changing them stays with the docs site's `operate` workflow (`introspection skills operate`). Speak only from evidence you read in this session. When you can't reach the evidence (no login, a missing scope, a version you can't resolve), or the owner asks for a dry run, stop at that boundary: say what is unverified and which evidence would settle it. Never write "I inspected" for work you didn't do, and never fill a finding with invented counts, traces, conversations, causes or results. ## Steps 1. **Read the request.** A feature (a tool, a capability, something the agent should do and can't) starts from the behaviour the owner asks for and acceptance cases they approve; evidence that shows the need helps but isn't required. A failure or a drop in quality starts from the traces. What the owner says is wrong is the scope of the search or a hypothesis to test, never the proof of a cause. A request to generate and evaluate many candidates, raise the overall eval score, or make a fixed model match a reference is optimize's, not improve's: say so to the owner rather than turning each candidate into its own confirmation. Handing such a request to optimize is not designed yet. Plan the evidence to falsify the owner's hypothesis as well as support it, and include controls for a failure the hypothesis would only move elsewhere. Tailor the plan to the focus: for instructions, how they load and what else in them conflicts; for tools and configuration, their contracts, permissions and deterministic failures; for an eval or a judge, its labels, calibration and variance; for a goal, the conversations that reach it and the ones that don't. 2. **Resolve the version and the commit it was built from.** Resolve the project, the runtime group, the version that served the conversations, the Recipe's repository and the commit that version was built from, and confirm that the evidence and the code you read describe that same version before concluding anything: ```sh introspection runtimes get -o json # environments, runtime_group_id, recipe_id introspection runtimes versions -o json # every version; production's lists "production" in environments introspection recipes get -o json # repository_id, git_ref, git_commit_sha, sub_path ``` Read the Recipe at that commit (`git show :`, or a checkout of it), not at the tip of a branch that has moved on. Conversations carry their `recipe_git_commit_sha`; a conversation from another commit is evidence about another version. Keep runtime-group slugs, runtime ids, runtime-version ids, task ids and conversation ids apart, carry the canonical value a listing returns into each later command, and confirm which project each command reads (`-p` on the runtime and recipe commands; the login's project otherwise) rather than assuming the one selected at login. 3. **In the loop: the owner's tests of a branch's version.** When the owner tests a branch's version (deploy gives each pull request one), follow their test chats. `introspection open --runtime --print` prints the page where they chat with that version; `introspection open --task --wait` hands them an existing task's chat and returns when they close it. Then read what they did: ```sh introspection tasks list --runtime-id -o json introspection conversations list --filter runtime_id= --lookback 1d -o json ``` Read each test chat in full ([conversation evidence](references/conversation-evidence.md)). At a turn where the owner corrected the agent, or where the result looks wrong to you, ask the owner what should have happened there, quoting the turn, in one question. Their answer is the strongest signal there is: write it down with the conversation id and the turn, in their words, for improve-classify. It becomes a case only after the gate. 4. **Then production.** Once the owner's tests are read, or when the request is about what production does, read production's version the same way: its conversations in a bounded window, filtered by `runtime_group_id` and `environment=production`, with the commit each one ran checked against the version you resolved. 5. **A sweep, only when the owner asks for one.** A sweep reads one environment's traces over a period and proposes failures to look at. Improve never sweeps on its own initiative or on a schedule. Choose an explicit window that fits the traffic, state it, and widen it once if it is too sparse; never search unbounded history without saying so. Bound the window (since the last sweep, or yesterday in the project's time zone) and compare it against a credible baseline, such as the same window seven days earlier. Read representative conversations before concluding anything from the comparison. With no narrower direction, go from recurring patterns to the observations under them to the exact conversations, including their feedback, judgements and model and tool failures, and build that population before choosing examples from it ([evidence surfaces](references/evidence-surfaces.md#patterns-and-observations)). 6. **Flag failures with the four signals,** most trustworthy first: 1. **The owner says so:** a correction in a test chat, or negative feedback (`introspection.feedback` events). A user's rating in production is evidence for review, not a verdict. 2. **Mechanical signals:** failed tool calls, refused edits, errors, runs that were abandoned or ended with nothing (a task cancelled mid-turn, one torn down by the idle window, a question never answered). 3. **Recurring problems the platform already groups:** patterns and their observations, read under a lens. 4. **A judge model,** only once enough owner-labelled cases exist to calibrate it. A judge whose definition hasn't been validated on owner-labelled cases (improve-classify calibrates them) gives leads, not flags. A zero pattern or observation count is not proof that nothing is wrong: analysis covers conversations partially, and conversations with real tool failures can carry no observation. Check the analysis ran over the window and read raw conversations before calling a run clean. 7. **Read each flagged conversation in full,** from the task row outward ([conversation evidence](references/conversation-evidence.md)): why the task ended, then every turn, tool call and result around the failure, the files it produced, and its feedback and annotations. 8. **Sample controls.** Sample on purpose: the flagged incidents, plus random ordinary traffic (successes included), the important cohorts, boundaries and refusals, and rare paths with a high cost of failure. Pair what clustering surfaces with a random sample, and draw controls from the same window and cohort as the failures. Size a production sample by how well it covers the cohorts and how much results vary, never by an arbitrary percentage. Stratify when results are unstable or miss a segment, and sample again when the traffic, the agent's contract or the judge model changes. Keep the set a person will read small, usually five to ten conversations, mixing recurring or emerging behaviour, high-consequence outcomes, disagreement between feedback and judgements, tool failures, coverage across patterns and versions, and ordinary controls. Prefer conversations nobody has reviewed yet, and say why when you keep one already annotated. Don't stack near-duplicates to make a pattern look stronger; when one conversation stands for a larger population, say how that population was measured. Review complete traces (input, context, tool calls and results, output, errors, latency and cost), and keep the stable identifiers (conversation, trace, span, task) so feedback and outcomes join back to the trace. 9. **Count prevalence on the aggregate surface.** Call something common, rare, rising or regressing only from an aggregate metrics query over the window ([metrics query](references/metrics-query.md)), never from the handful of conversations you read. Take traffic, quality and cost from metrics; never work out a rate for the project from a partial list of records. 10. **Review a large body of evidence in parallel,** when there is more than one pass reads well. Decide whether to split by how much evidence there is, how coupled it is, and what access, cost and rate limits allow; where the host offers no subagents, review the slices one after another yourself under the same rules. Give each reviewer subagent the same resolved version and commit, the same boundaries (read-only, no live changes, no invented evidence) and its own slice: a group, a period or a signal. Each reports what it inspected, the evidence, and the earliest point where the conversation diverged from what should have happened, not a fix of its own. Keep the synthesis in the main agent: compare the reports, settle where they disagree by reading the evidence yourself, and be the one who talks to the owner. 11. **Hand the evidence to improve-classify:** the version and commit, the window, each flagged conversation with its signal, identifiers and turn, the owner's own words about what should have happened, the controls, the prevalence counts with their queries, and every gap you couldn't close. Go on with improve-classify straight away. ## Evidence that needs care ### MCP servers: how the agent's calls fail To see how the agent calls a server the Recipe declares and how those calls fail (`error_classes` by origin), run `introspection bindings mcp observe --runtime --out /observed.ts --examples /examples`, without building a fake. Keep `--out` and `--examples` outside the repository: the examples are raw customer calls, and the committed `observed.ts` is changed only after the gate. Observe reads the server's captured `tools.ts` (`eval-fakes`). When the Recipe has a fake, `introspection bindings mcp capture --check` and `observe --check` compare it with the live server and traces and write nothing. Record any drift as evidence: a changed input schema lets evals pass while real calls fail, and a changed description changes the agent's prompt where no eval sees it. Recapturing, type-checking and running the checks and evals again is the change improve-propose makes. A fake built before any runtime called its server is approximated. Once live traffic exists, flag it: improve-propose runs `observe` and `checkRecipeFakes({ recording })` and rewrites the answers they show to be wrong. ### Experiments When evidence comes from an experiment (`introspection experiments get `), read each arm's judged count beside the posterior: an arm with no judgements is unmeasured, never losing. Don't call a comparison decided without per-arm evidence, matching judge definition hashes and routing you verified. ### Who changed what Who changed something, or what changed when, is answered by the audit log, not by the resource's own row; pair it with the resource's lineage (its versions and the commits they were built from) for both who and what. The CLI doesn't read the audit log yet, so report who-changed as unverified rather than inferring it from a row's `updated_at`. ### Trace references and retention A case keeps a reference to the conversation it came from, so its eval can read the trace again. Trace references that outlive the platform's retention are not built yet: note the date of each conversation you hand on, since a case written much later may no longer be able to read it, and today `eval add` copies the turn into the repository instead (`eval-case`). ## Done when - The version and the commit it was built from are resolved, and every piece of evidence is checked against them. - Every flagged conversation was read in full, names its signal, and carries its conversation id, turn and trace id. - The owner's corrections are recorded in their words, the controls are sampled from the same window and cohort, and every prevalence claim has the metrics query behind it. - Nothing was written, changed or cancelled, and every gap is named with the evidence that would close it. Then go on with improve-classify.