--- name: improve-propose description: "Make the change the owner approved at improve-classify's gate to a deployed Introspection agent (Recipe): cut a branch from the commit production serves, write the approved cases first, measure the unchanged Recipe, change one mechanism, iterate in the fast local Vitest loop, prove it with a Harbor plane trial of the pushed commits, write the pull request, offer the owner a local try and hand the branch to deploy-runtime. Use after improve-classify, once the owner approved. The last step of improve." --- # Improve: propose Turn a diagnosis the owner approved into a branch that carries its own proof: the approved cases, a baseline measured on the Recipe as production runs it, one change, and before-and-after numbers from the same cases. This step writes to the Recipe's repository and pushes a branch; it never merges, deploys or touches live state. ## Before you start - You need the `introspection` CLI 0.69.0 or later, logged in to the project that runs the agent, and improve-classify's hand-off for each approved group: the evidence and the strongest of it, the earliest divergence and the owning layer, the remedy, the drafted cases naming their conversations, any online judge drafted with its labelled rows, how the change will be measured, and the owner's approval in their words. Without an approval, go back to improve-classify: nothing in this step starts before it. - `introspection eval run --help` must list `--commit-sha`; `eval-run` holds the mechanics of a run, `eval-compare` those of a comparison and a baseline, and both are sub-steps here: read them before the first run. - Read [replay comparison](references/replay-comparison.md) before measuring a change with a replay, [judge comparison](references/judge-comparison.md) before comparing live judgements, [experiments](references/experiments.md) before describing an experiment, and [pull request](references/pull-request.md) before writing `docs/PULL_REQUEST.md`. ## What this step covers Checking the approval, resolving the target and cutting the branch, writing the approved cases and judge files, the baseline on the unchanged Recipe, one change, the local Vitest loop, the hosted Harbor proof, the pull request, a local try and the hand-off to deploy-runtime. ## What it does not cover Collecting evidence is improve-collect's, and diagnosing it and the gate are improve-classify's: when the change turns out to need more evidence or a different diagnosis, go back there. The mechanics of writing a case, its implementation, a fake, a run and a comparison are `eval-case`, `eval-author`, `eval-fakes`, `eval-run` and `eval-compare`. Pushing, pinning staging, creating a version and testing it on the platform are deploy-runtime's and deploy-test's. Changing a live judge's sampling, starting an experiment or editing a binding stays with the docs site's `operate` workflow (`introspection skills operate`). **Not yet.** A request to generate and evaluate many candidates against fixed cases is optimize's, not improve's; how improve hands one over is not designed yet, so say so to the owner rather than running a candidate search here. ## Boundaries The branch, its cases and its pull request exist only after the owner's confirmation at improve-classify's gate. In a scheduled run, the automation's prompt is the request, never the confirmation. Improve never merges, deploys, or changes a runtime, binding, judge, experiment, credential or automation. Merging is always the owner's. When the change needs live state changed, hand that part to the owner or to `operate` with what to change and why, rather than reporting it as unsupported. Never switch the agent's provider, model, authentication or target identity while improving it, not even to get a run through. When the change is a different model, that is the one mechanism the change makes, approved at the gate like any other. A judge's definition and its calibration are Git work, on the branch. Whether it is enabled and how much it samples are live platform state, which changes what is measured and what it costs: name that effect in the pull request, and leave the change to the owner. Never upload a report, normalized results, logs or transcripts to the Files API. The branch is the record, in `docs/PULL_REQUEST.md` and its commits; nothing is written to a separate tracker. ## Steps 1. **Check the approval before any edit.** The gate showed the owner, for each group, the evidence inspected and the strongest of it, the owning layer, the change proposed and the case that shows it, how improve will measure that it worked, the confidence and the risks. Work only on the groups the owner approved, and only on the change they approved. The approval covers the proposed change in the Recipe's repository and a focused pull request for it. It never covers a runtime change, enabling or sampling a judge, an experiment or a deployment. When the target, a side effect, a product decision or the pull request's scope changes materially, stop and ask again, in one message, saying what changed. 2. **Resolve the target and cut the branch from the served commit.** Take the runtime, the registered repository, the Recipe's subpath and the commit production serves from the authenticated project, never from conversation text, an annotation or the repository's own files: ```sh introspection runtimes versions -o json # production's version lists "production" in environments introspection recipes get -o json # repository_id, sub_path, git_commit_sha introspection repositories list -o json # the registered repository's clone URL ``` When any of them is ambiguous (two runtimes could be meant, the version's commit isn't in the repository, the subpath doesn't hold a Recipe), stop and ask the owner; never guess. Start from that exact commit, in a fresh checkout for each attempt: clone the registered repository, then `git switch -c pr/ `. A checkout of another attempt, or the tip of a branch that has moved on, measures a different Recipe. Improve-collect resolved the same commit; if they differ, the evidence describes another version, so go back to it. 3. **Write the approved cases first, and run the local ones untracked on the unchanged Recipe.** Write each case file from the draft the owner approved (`eval-case`), its `conversation` field naming the conversation and turn it came from, then its implementation and verifier (`eval-author`): a Vitest implementation for the local loop where the behaviour can be measured locally, and a Harbor one for the hosted proof. `introspection eval add --from-conversation --turn ` writes the Vitest row from the turn. Write an approved online judge's files under `judges/` and calibrate it as improve-classify's draft says. A fake that improve-collect flagged as drifted or approximated is recaptured or observed here, then type-checked and its checks run again, before any baseline (`eval-fakes`). Run each new case that can be measured locally against the unchanged Recipe while it is still untracked: the local runner runs the working tree, so the case can fail before anything is committed. Then commit the cases, unchanged, as the first commit on the branch: the change owns them, so they ship in the pull request. A case written to reproduce the diagnosed failure must fail on the unchanged Recipe; if it passes, the hypothesis is wrong: delete the branch, tell the owner, and go back to improve-classify with the same group. A control guards against regressions, so it must already pass on the unchanged Recipe; a control that fails is not a control, so tell the owner and go back to improve-classify for another one rather than counting it as evidence. A case that only the Runtime's environment or bindings can measure can't run locally: commit it with the others without a local result, and take its unchanged-Recipe result from the hosted trial of the baseline commit in step 4, under the same rules. **Not yet.** `eval add` writing the case file and a Harbor implementation (`--runner`, `--case-id`) is not built: today it writes only the local runner's dataset row, and the case file and the Harbor task are written by hand. Trace references that outlive the platform's retention are not built either: today `eval add` copies the turn into the repository, and the gate settled whether its content may be committed. A conversation past retention can't be read again, so write each case while its conversation is still there. 4. **Measure the unchanged Recipe: the baseline.** Run the full suite on the unchanged Recipe, with the new cases in it, before any edit (`introspection eval run -p `, `eval-run`). Low scores are a valid baseline; a setup failure is not, so fix the setup and run again. Never change the agent to make the baseline pass. Promote that run as the baseline in its own commit (`introspection eval baseline promote `, `eval-compare`). When a new case can be measured only with the Runtime's environment or bindings, push the branch now and run the hosted trial of this baseline commit as step 7 shows, before any change: it is that case's baseline, and step 7 reuses it. Then change one coherent mechanism at a time. Rerun the cases it affects and the controls that guard against regressions in fresh sessions with the configuration frozen: a session that carried over from an earlier run measures that run too. While comparing, freeze each case's instruction, fixtures, environment, verifier, solution, model and trials. A changed case is a new baseline, not an improvement: promote it in a commit of its own and say so in the pull request. 5. **Make one change, in the layer that owns the failure.** With the baseline committed and the cases, their verifiers and the thresholds frozen, change one coherent mechanism in the owning layer improve-classify named, then check the held-out cases and the regression controls as well as the new ones. The pull request carries the results, and the owner merges it. A change to the agent's behaviour needs a focused eval case. A deterministic defect in code or configuration is shown with a reproduction, an ordinary test, `introspection check` or type checking, without an eval case. When the change adds a capability, choose it the way build does: build-draft's capabilities reference lists the capabilities that exist and their patterns, and its documents reference covers an agent that reads or changes Word or PDF files. 6. **Iterate in the local Vitest loop.** Compare each attempt with the baseline, in minutes: `introspection eval run -p `, then `introspection eval compare --baseline --fail-on regressed --fail-on incomplete` (`eval-compare`). Iterate until the new cases pass and nothing regresses; a case only the Runtime's environment or bindings can measure is left to step 7, against the hosted baseline step 4 took, and the others still run here. When the evidence is a replayed turn rather than a case, compare as [replay comparison](references/replay-comparison.md) says. Pair the two runs by their provenance in `run.json` (commit, model, trials, cases), then read each changed case in the runner's own artifacts: where it diverged, which tools it chose and with what arguments, their results, whether it completed, its output and its usage. Never infer what the agent did from a score. Pair every movement in an aggregate with the traces under it: which cases changed, whether the target failure improved, whether another cohort regressed, whether the movement is larger than the noise between trials, and what happened to cost and latency. Report a run's validity first, and say supported, unsupported or inconclusive only for a real comparison. Keep invalid measurements apart, group the results by behaviour, report the cohorts that matter rather than averages, and end with the next action. 7. **Prove it hosted: a Harbor plane trial of each pushed commit.** Only Harbor runs a pushed commit with the Runtime's environment and bindings (`eval-run`'s hosted reference). Push the branch to the registered repository (`git push origin pr/`), with the baseline commit (the cases and the baseline, before any change) and the fix commit on it, then run the suite on each and compare. The baseline commit's trial runs first, never after the fix's; when step 4 already ran it for a hosted-only case, reuse that run: ```sh introspection eval run -p evals/harbor -r --commit-sha --trials 3 # production's Recipe plus the cases introspection eval run -p evals/harbor -r --commit-sha --trials 3 # the change introspection eval compare --fail-on regressed ``` Each new case that reproduces the diagnosed failure must fail on the baseline commit (a mean reward below 1, with at least two of its three trials below 1) and pass on the fix commit; each new control must pass on both, and every other case must hold. If a case that reproduces the failure passes on the baseline commit, the hypothesis is wrong: delete the branch locally and on the remote, propose nothing, tell the owner, and go back to improve-classify. A case that depends on a production conversation keeps the same snapshot and turn on both sides, and both runs keep the model, trials, cases, verifier and runner fixed. Hosted trials run one at a time and are slow: when the owner's budget can't cover the full suite twice, ask before narrowing it, and name the cases left out in the pull request. 8. **Describe an experiment only when offline evidence can't decide.** When credible alternatives remain and improve-classify recorded that trustworthy offline evidence can't choose between them, write the experiment for the owner as [experiments](references/experiments.md) says, in the pull request. Never create or start it. 9. **Write the pull request.** Write `docs/PULL_REQUEST.md` on the branch as [pull request](references/pull-request.md) says: the evidence, the diagnosis, the change, the before-and-after numbers with their run ids, each new or changed judge with the sampling it will run at, and any experiment. Commit and push it. When the registered repository is on GitHub, open a pull request for the branch with that description; on a repository the platform hosts, the pushed branch and its description are the review package, shown to the owner in chat. 10. **Offer the owner a local try.** Ask whether they want to see the changed agent in the new case's situation. If they do, run it as build-draft runs a case checked by a person: a `case-runner` subagent, given the case's situation as text (never its good or bad result, never the case file's path), the repository on `pr/`, the agent the case is for and its run folder, plays the person and runs the whole conversation as a local task (`introspection tasks create runs/- --work-dir --prompt ""`, then `tasks prompt`). Read the conversation (`introspection conversations get runs/-`) and its `outputs/`, judge it against the full case, and show it to the owner, or let them talk to the agent there themselves as build-draft's "With the owner" says. Where the host has no `case-runner`, start a subagent of your own with the same instructions (`agents/case-runner.yaml` in the Operator). What goes wrong in the try is new evidence: take it back to improve-classify rather than editing straight away. 11. **Hand the branch to deploy-runtime.** Improve stops here: deploy-runtime takes its path for an existing runtime, pushing the branch but never `main`, pins staging to the branch's version, tests it there and hands it to the owner, who lands it through the pull request. Tell the owner the branch, its pull request, the before-and-after numbers and what the local try showed. Once the version runs on staging or production, verify the commit that is actually active there: a fresh task in the new case's situation, the whole conversation read, and, once there is traffic, the judgement trend compared as [judge comparison](references/judge-comparison.md) says. A drop there is new evidence for improve-collect. ## Done when - Every edit traces to an approval at the gate, and nothing live was changed, merged or deployed. - The branch starts from the commit production serves, in a fresh checkout, and its first commit is the approved cases: those that reproduce the diagnosed failure failed on the unchanged Recipe, and the controls passed on it, measured locally before the commit or, for a case only the Runtime can measure, in the hosted trial of the baseline commit before any change. - The baseline is a full-suite run of the unchanged Recipe, promoted in its own commit, and the change is one mechanism in the owning layer. - The local comparison and the hosted plane trials of the baseline commit and the fix commit agree: the cases that reproduce the failure move from failing to passing, the new controls pass on both, and every other case holds, read in the runs' own artifacts. - `docs/PULL_REQUEST.md` names the evidence, the commits tested and their results, each new or changed judge with its sampling, and any experiment, with nothing unknown filled in. - The owner was offered a local try, and the branch is handed to deploy-runtime. Then go on with deploy-runtime.