--- name: optimize-cost description: "Find the cheapest version of a deployed agent that holds the quality floor, and set the cost ceiling. Use when the owner wants an agent cheaper without losing quality, or after the quality search." metadata: min-cli-version: "0.73.0" next: "optimize-latency" --- # Optimize: cost Find the cheapest version that still passes every case the quality floor requires. The search is optimize-quality's, run with cost as the objective and quality as a hard gate; its result is the cost ceiling optimize-latency must keep. It writes only to its branch and worktrees, and never touches live state. ## Before you start - What optimize-quality hands over: the contract in `/contract.md`, the ledger, the branch `pr/` with its accepted change and promoted baseline, and the quality floor. When the owner asked only for a cheaper agent, run optimize-quality's steps 1 to 5 first (the entry check, the seed, the price, the contract and the reference), and take the reference's pass rate per case as the floor. - Read optimize-quality's references to the contract and the search before the first candidate, [the knobs](references/knobs.md) before writing a hypothesis, and [starting points](references/starting-points.md) before choosing which candidates to try first. ## Not this step Raising quality is optimize-quality's, and latency optimize-latency's. The search's mechanics (worktrees, proposers, the integrity review, the stages, choosing and applying the winner, cleaning up) are optimize-quality's; a run and a comparison are `eval-run`'s and `eval-compare`'s. ## Measuring cost Cost, elapsed time, paid attempts and repeated actions are evidence only under fixed conditions, and a cheaper or faster run is no improvement if it is less correct, safe or useful. - **Per completed task, never per request.** Sum each case's `usage.agent.cost_usd` from `run.json` over the same cases and trials (`eval-run`'s "Read run.json"), and divide by the cases that passed. A cheaper request that needs more turns or retries isn't cheaper, and a run that fails more tasks costs more per task done. - **Price, not tokens.** `cost_usd` prices cache writes and cache reads at their own rates. A setting that reads less from the cache can cost more while sending fewer tokens, so never rank candidates by token counts. - **Several trials, all three measures.** Each candidate runs the contract's trials; one trial per setting is noise. Record quality, argument errors and latency beside cost for every candidate, as optimize-quality's contract reference says: fuller tool descriptions or argument signatures add tokens to every request but can cut argument errors, so fewer retried calls and turns can make the same change cheaper and faster per task, and fewer errors is quality too. ## 1. Add the cost objective to the contract Optimize cost with quality held as a hard gate: every case the floor passes must still pass across the contract's trials, and candidates are compared by the summed per-case `cost_usd` over the same selection set, per completed task. Add the objective, the gate and this phase's budget in money to `/contract.md`. The edit surface, the sets, the runner and the trials stay as frozen; the model or a setting becomes a search axis only when the contract declares it, and the owner's request to optimize cost is what declares the model one. ## 2. Price the search before running it Start from optimize-quality's end-to-end run at the default configuration, or run the focused batch once that way. Read each case's tokens from `run.json` (input, output, and the turns it took), multiply them by each candidate model's list prices for a projected cost per case, then by the cases, trials and rounds the search plans. Models differ in how many turns and how much output a task takes, so carry a margin (2× until a trial on that model says otherwise) and correct each projection after its first trial. When the projection exceeds this phase's budget, cut the plan before spending: fewer candidates, fewer cases in the focused batch, or the expensive models only once a cheaper one fails the gate. Bring any remaining overrun to the owner. ## 3. Measure the reference: the quality winner Its full selection run from optimize-quality is the reference when the contract still matches; otherwise run it again. Read where its cost goes before proposing anything: which cases cost most, how many requests each takes, how much is cached, which tool results are largest, and where calls failed and were retried. ## 4. Choose the hypotheses From that evidence and [the knobs](references/knobs.md), give each proposer one change that should cost less for the same work: a cheaper model or lower effort, how tools reach the model, caching and retention, trimming context or the size of a result, fewer skills loaded. Order the first candidates by [starting points](references/starting-points.md), and let the suite decide. ## 5. Search, choose and accept Run the rounds, choose one candidate and accept it on the held-out set exactly as optimize-quality's steps 6 to 8 say, with this phase's objective and the quality gate. Spend trials where the decision is, as [spending trials](references/spending-trials.md) says: cheap axes first, the model by the intelligence it needs, one trial before deciding. Apply the winner to `pr/`, promote its baseline in its own commit, and clean up. ## 6. Attribute cost once a version runs `recipe_git_commit_sha` is the Recipe's version in the traces: group spans or conversations by it to attribute a cost or latency change to a Recipe change (improve-collect's metrics reference has the query document). After deploy-test, compare the branch's version with the served one this way, on the same kind of traffic, and say in the pull request that the eval numbers came first. ## 7. Check the evidence will carry a report Before you draw anything, check that the runs can show why the winner won: the recommended setting compared with at least two others on its model, and the winning model with at least two others on its setting, each with enough trials to rank. When they fall short, the data is thin: don't build the page. Go back to step 4 and form new hypotheses from what the runs showed (the settings that failed on one model tried on the model that passed, another way tools reach the model, the next model along the index's Pareto line), run them as step 5 says, and check again. Stay within this phase's budget; when it can't buy the comparisons, tell the owner what is missing and what it would cost, and ask with `ask_user_question` whether to spend it or report what there is. ## 8. Report and hand over the ceiling Add the cost section to `docs/PULL_REQUEST.md`, with its charts, and build the owner's results page, as [handing over the ceiling](references/handoff.md) says. When nothing beat the reference, say so and keep the Recipe as optimize-quality left it. The winner's cost per completed task is the cost ceiling optimize-latency holds. ## Done when - The cost objective and the quality gate were added to the contract, and every candidate was measured under it with the contract's trials. - Candidates were compared by cost per completed task over the same cases, with quality, argument errors and latency recorded beside it. - The winner held every case of the quality floor and was accepted once on the held-out set, or the Recipe is unchanged and the owner was told why. - The pull request's cost section names the ceiling and every change that needs the owner's review. Then go on with `optimize-latency`.