--- name: optimize-quality description: "Find a version of a deployed agent that passes more of its approved cases, and set the quality floor. Use when the owner wants a better agent overall, a higher eval score, or a cheaper model matching a stronger one." metadata: min-cli-version: "0.73.0" next: "optimize-cost" --- # Optimize: quality Search candidate changes against the agent's approved cases, held fixed, and keep the one that beats the noise, accepted on cases the search never saw: the quality floor the later optimize steps keep. It writes only to its own branch and to worktrees outside the delivery tree. ## Before you start - The `introspection` CLI 0.71.0 or later, logged in to the project that runs the agent. A Harbor suite also needs Harbor and Docker here (the owner runs `introspection setup --target harbor`). - Read `eval-run` and `eval-compare` before the first run, [the contract](references/contract.md) before freezing it, and [the search](references/search.md) before the first candidate. - A proposer (`agents/proposer.yaml`) and a fresh reviewer (`agents/reviewer.yaml`) subagent per candidate. In Claude Code or Codex, start your own with the same instructions. ## Not this step - A production failure is improve's. A broken verifier or a missing case goes to improve or `eval-case` before a search starts. - Deploy pushes, pins and tests. Runtimes, bindings, judges and experiments are the docs site's `operate` workflow (`introspection skills operate`). ## Boundaries This skill is the explicit request for subagents. You coordinate and never write a candidate yourself. When the host can't start a subagent, or a start fails, stop before generating any candidate and say so; never fall back to candidates of your own. An explicit request to optimize approves the whole bounded search, confirmed at most once, with its budget in money. Report the end of the reference run, a batch or a round as progress and carry on: what ran, what passed, its tokens and cost, the money spent against the budget and what runs next, so the owner can stop the search at any point. Stop at the budget, never past it. Ask only about an objective or a constraint you can't infer safely, never about routine settings such as candidate names, worktree layout or the default budget. Never merge or change live state, and never switch the agent's provider or model outside a search axis the contract declares. Keep credentials out of prompts, the ledger and commits. ## 1. Check that optimizing is worth it Check the entry criteria and audit the suite, as [the contract](references/contract.md#when-optimizing-is-worth-it) says. - **The suite can't measure a gain** (too few cases, unknown noise, no held-out set, a verifier to fix): say what is missing and how to get it, and stop. - **Every case already passes every trial:** there is no quality headroom. Say so, record the passing set as the floor, and go on with optimize-cost. - **Only cheaper or faster was asked for:** run steps 2 to 5, take the reference's pass rate per case as the floor, and go on with optimize-cost or optimize-latency. ## 2. Pin a clean seed Find the runtime, repository, Recipe subpath and served commit, as improve-propose does: ```sh introspection runtimes versions -o json introspection recipes get -o json ``` Cut `pr/` from that commit in a fresh checkout. The search folder, `` in the references, is `runs/optimize-/` in the build folder, outside the repository. Never evaluate from a dirty tree. When the owner wants uncommitted changes in the search, commit only those as the seed in a worktree of its own, leave unrelated changes untouched, and run the reference and every candidate from worktrees descended from that seed. When the intended changes can't be separated safely, ask before spending any of the budget. ## 3. Price the search Run the focused batch once as production serves the Recipe, one trial, and read each case's tokens, cost and time from its `run.json`. This proves the harness and prices a run. Project the search from it as [pricing the search](references/contract.md#pricing-the-search) says. ## 4. Freeze the contract Write the objective (the pass rate over the selection set, as pass@k or pass^k), everything [the contract](references/contract.md#what-the-contract-freezes) freezes, and the budget into `/contract.md`. State it to the owner in one message, as [pricing the search](references/contract.md#pricing-the-search) says, and get the single confirmation the search needs. When the projection exceeds the owner's budget, cut the plan or ask for more before spending. ## 5. Measure the reference Run the reference once over the full selection set with the contract's trials (`eval-run`'s "Run at a commit"); a small batch proves the harness, never a score. Reuse an earlier complete run only when its whole contract matches. When the candidates run under another fixed configuration than the reference (a cheaper model meant to match a stronger one), also run the unchanged seed under it, as the candidates' baseline; it takes no candidate slot. Trust only valid scores ([invalid runs](references/search.md#invalid-runs)). Record each case's spread between trials: a candidate must beat it to count. ## 6. Run rounds Run each round as [a round](references/search.md#a-round) says. Stop when the success condition is met, the budget is spent, two rounds in a row don't improve the best candidate, a blocker stops valid work, or the owner changes the contract. When the budget ends first, report what the search didn't establish; never turn the stopping rule into a claim about the best the agent can do. ## 7. Choose one and accept it on the held-out set Choose one candidate from selection-set results only, passing every gate. When none reaches the target, take the best valid one only if it improves the objective; otherwise leave the Recipe unchanged. Run the held-out cases and the critical controls once, on that candidate only, to accept or reject it. Never compare candidates on them, choose a replacement, reflect or tune from them. A failure ends the search with the Recipe unchanged: say that no candidate earned promotion. Apply the accepted commit to `pr/` exactly: ```sh git cherry-pick ``` Confirm the tree matches what was evaluated, then promote its full-suite run as the baseline in its own commit (`eval-compare`), for later phases and improves to compare against. Rerun each surprise that would change the choice or a finding first, as [surprises](references/search.md#surprises) says. ## 8. Clean up On every ending, a blocker or an invalid run included, remove the search's seed and candidate worktrees (`git worktree remove`), any linked `.env`, fixture state and ports. Keep the delivery tree, the candidate commits, the ledger and the artifacts the ledger points to. ## 9. Report and hand over the floor Tell the owner what the search found, and write it as the quality section of `docs/PULL_REQUEST.md`, as [handing over the floor](references/handoff.md) says. The winner's pass rate per case is the quality floor: optimize-cost and optimize-latency hold it as a hard gate. ## Done when - The suite met the entry criteria, or the owner was told what was missing and nothing was searched. - The contract was confirmed once, and every ledger result was measured under it. - Every candidate came from a subagent in its own worktree, was reviewed before any eval and evaluated in stages; invalid runs never counted. - One candidate was chosen on the selection set and accepted once on the held-out set, or the Recipe is unchanged and the owner knows why. - The change is on `pr/` with its baseline promoted, the worktrees are gone, and the pull request's quality section names the floor for `optimize-cost`.