--- name: improve-classify description: "Diagnose evidence about a deployed agent (Recipe): with nothing flagged, the owner picks a pattern; open-code and group failures, find the earliest divergence and owning layer, weigh them, and bring a proportionate remedy to the owner's gate. Use after improve-collect." metadata: min-cli-version: "0.79.0" next: "improve-propose" --- # Improve: classify Turn improve-collect's evidence into a diagnosis the owner approves: where it first went wrong, which layer owns it, how much it matters, and the smallest remedy. It writes nothing to the repository or the platform: its records go in `runs/classify/` (a note per conversation, the diagnosis), the owner's answers in the conversation and the gate's files in the outputs folder, and it ends at the confirmation gate. ## Before you start You need the `introspection` CLI 0.79.0 or later, logged in, and improve-collect's hand-off. Without it, or when a group needs more evidence, go back to improve-collect. Read this file to its end first: the gate (step 11) and "Done when" decide what your reply to the owner holds. ## Boundaries A diagnosis runs up to the gate and stops there. Hand on a grounded, small change the Recipe owns; never treat the diagnosis, or the owner's request to look into something, as permission to edit. Reading needs no permission, and an edit waits for the gate ([working rules](references/working.md), which also say which identifiers may select a target, how to resume, and what stays with `operate`). **Every number names its source:** a slice count its query or its ids file's line count, a prevalence its metrics query, a group's size the conversation ids under it. ## Nothing flagged: start from the patterns With nothing flagged, let the owner choose a pattern they don't want, as [classification from patterns](references/from-patterns.md) says, then carry on from step 1. ## 1. Read the Recipe at the served commit Telemetry shows what happened, not why the agent was allowed to do it. Before writing any diagnosis, read the Recipe at the commit improve-collect resolved (`git show :`): its agents, instructions, tools, skills, judges and evals as they were for those conversations. ## 2. Open-code each flagged conversation Read each trace in context to its earliest meaningful divergence. Write one self-contained, plain note per conversation read, flagged or not, to `runs/classify/notes/.md`, that someone could act on without opening it. Don't standardize early: revisit earlier traces as later ones sharpen the search, until new traces stop producing new kinds of note (often around a hundred). Patterns, scores and the agent's own explanations are leads, never proof. Read [open coding](references/coding.md) for what a note may rest on, and to split the coding across `evidence-analyst` subagents. ## 3. Group the notes into failure modes Group the notes into failure modes someone can act on, never "bad quality". Each mode keeps its definition, its notes in their original wording, examples and counterexamples, its owner and its definition's version; notes that fit none go under "none of the above". New signals restart the grouping ([open coding](references/coding.md#when-the-grouping-changes)). ## 4. Find the earliest divergence and the owning layer Before any remedy, find where the behaviour first left what was intended: later mistakes are often its consequences, and fixing them leaves the cause. Then name which of the four causes owns it, with the first failing step's evidence, as [failure layers](references/failure-layers.md) says. Record both per group in `runs/classify/diagnosis.md`, quoting the turn (for a Recipe file, its path and lines at the served commit), with the notes it rests on. A step in doubt is replayed ([replay](references/replay.md)); one that doesn't reproduce stops the diagnosis there. A failing eval case, or a regression improve-propose sent back, is read as [eval results as evidence](references/eval-evidence.md) says. ## 5. Weigh prevalence, severity and importance apart Prevalence (from improve-collect's metrics queries), severity (for the person it happened to) and business importance (for the owner) are separate: write each per group. Rank by them, the users affected, detectability and the fix's cost; never let frequency bury a rare catastrophic failure. Match the evidence to each group's exposure, and treat business outcomes as pressure on coverage ([exposure and outcomes](references/weighing.md)). ## 6. Get the domain owner's labels where only they can judge When what the product should have done is unclear, ask the domain owner; never encode a guess or infer it from a task's status or feedback. Ask in the conversation, never by writing an annotation, label or reviewer assignment to the platform, and record the answer in the owner's words with the conversation id, trace id and turn, as [asking the owner](references/labels.md#asking-the-owner) says. ## 7. Choose a proportionate remedy In this order: a deterministic failure gets a deterministic fix (code, a tool, configuration) with an ordinary test; an ordinary test whenever one protects the behaviour; an eval case only for a recurring, important risk a test can't measure; an experiment only when offline evidence can't decide. When offline evidence can't decide, read [remedies](references/remedy.md). ## 8. Ask each product decision the remedy leaves open How far a ban goes and what the agent does instead are the owner's choices. Ask them together with `ask_user_question`, each answerable on its own, before you draft a case ([product decisions](references/remedy.md#product-decisions-a-remedy-leaves-open)). Then check every comparison conversation against the answers: one they make wrong is a failure, not a control ([the check](references/remedy.md#check-the-comparison-conversations-against-the-answers)). ## 9. Draft the cases a remedy needs Draft each case's title, situation, good result and bad result, naming its conversation and turn, as [drafting cases](references/cases.md) says, including whether its content may go into Git. At most two or three cases per iteration: the clearest failure, the highest-consequence boundary, one control. Keep other ideas as a backlog of situations. ## 10. Draft an online judge only when one is needed Draft one only when a group needs watching in production: its question and labelled rows for the gate, calibrated on the branch after approval with `introspection judges add`, `judges eval` and `judges validate` ([judges](references/judges.md)). ## 11. Bring each group to the owner at the gate For each group worth fixing, write one file per drafted case, each stating its own proof, and a `change.md` in a draft folder `improve-.draft/` in the outputs folder, never the repository, as [the gate's files](references/gate.md) says. Ask for one review per group with `review_artifacts`, `change.md` last, with a one-line message. Once the owner approves every file, rename the folder to `improve-/`: that folder is the approval. ## Done when - From patterns, the owner picked from options labelled with the patterns' own names, and each pick is its own group. - `runs/classify/notes/` has a note for every conversation read, each in a failure mode or "none of the above"; `diagnosis.md` names each group's earliest divergence and owning layer, with prevalence, severity and importance apart. - Every comparison conversation was checked against the owner's decisions, and each case's kind matches what it expects. - Every number the owner saw names its source. - What only the domain owner could judge was answered in their words, or is reported as undecided. - Each group worth fixing has a proportionate remedy, at most two or three drafted cases, and any judge with labelled rows. - The owner answered each product decision before cases were drafted, then reviewed the case files and `change.md`; nothing reached the repository before approval. - Each approved group has a folder `improve-/` holding exactly its approved case files and `change.md`; a dropped case has no file, a declined group no folder. When the owner approves, go on with `improve-propose`, giving it each approved folder's path.