--- name: build-draft description: Write a new Introspection agent (Recipe) from its approved cases, locally, and run the cases until they pass. Use when the cases in evals/cases/ are approved and the Recipe isn't written or doesn't pass yet. metadata: min-cli-version: "0.73.0" next: "build-review" --- # Build: draft Write the Recipe, its design and its checks on `pr/`, locally, and run the approved cases until they pass or each remaining failure is explained. The owner isn't needed here, except when a case itself turns out wrong or ambiguous: then propose the change and review it with the owner, as `eval-case` says. Never edit an approved case or clear its approval yourself. ## Before you start - **Tools:** the `introspection` CLI and Pi. No login. - **Work in brainstorm's repository,** on its `pr/` branch. Read `docs/DESIGN.md` Part A and every case in `evals/cases/` first. If a case isn't approved, stop and go back to `build-brainstorm`. - **Build only the agents with approved cases.** Another agent the brief names goes back to brainstorm for its own cases, or waits; don't ask about it or draft it here. ## 1. Choose capabilities For each case, pick what it needs from [the capabilities that exist](references/capabilities.md); cases needing no tools, documents or connections skip this and step 5. Write custom code only where no capability covers a case, and say so in the design. A missing tool is fixed with a tool, not a longer prompt. ## 2. Write Part B of the design Write Part B of `docs/DESIGN.md` with [the design template](references/design-template.md): each behaviour or capability cites the cases that need it, anything no case needs is cut, and every case says how it's checked. ## 3. Write the Recipe Write what the design calls for as [the Recipe layout](references/recipe-layout.md) and [what every Recipe gets right](references/writing.md) say, including the owner's exact values and standing knowledge; for Word or PDF, [documents](references/documents.md). - **The model:** replace the template's placeholder with the model chosen in brainstorm, one this machine has credentials for, and record it in Part B. - **Every tool you list must exist in the installed runtime.** `introspection check` doesn't verify tools; only a run does. If a capability isn't installed, use its fallback and say so in Part B. - **Keep test details out:** nothing specific to a sample input (a party name, a test file path) goes into the Recipe. ## 4. Smoke-test it once `check` validates the files; only a run catches a missing model key or credentials, or a listed tool that doesn't exist. From the build folder: ```sh introspection check introspection tasks create runs/smoke-1 --work-dir --prompt "hello" ``` Confirm the agent answers on the model you chose, not the template's. Skip the run when the case runs follow straight away, since their first turn does the same. - **`check` still fails after one fix:** tell the owner what fails and what you tried. Never get past it by removing something the design approved (a skill, a tool, a case). - **The run fails:** stop and tell the owner what failed and what you need (Pi signed in, network allowed for `introspection`, an extension conflict fixed). Never fix the machine yourself: don't run `introspection setup` or `upgrade`, and don't change Pi's settings or point Pi at another settings folder. ## 5. Give the agent its data without real credentials Decide which servers to fake, how much, and what Part B reports, as [build's decisions about fakes](references/fakes.md) say. Build the fakes as `eval-simulate` says, write each as its `write` reference says, and bind them as `eval-author`'s `vitest-fakes` or `harbor-fakes` says. Case inputs are read by hash; never commit a client's file. ## 6. Run every case in subagents, all at once Once the Recipe is written, in the same turn start the eval runner (`agents/eval-runner.yaml`) for the `code` cases and a `case-runner` for each `person` case (the owner's sample among them, if they gave one), up to four at a time, then wait for them all. [Running the cases](references/running-cases.md) says what each is given, how a run is laid out, and what counts as proof. - **Don't run a case yourself.** A smoke test isn't a case run, and scaffolding an eval doesn't run it. - **Never produce the agent's output yourself.** A script you run is not the Recipe, and its result proves nothing about the agent. - **Tell the owner in one line before anything over a minute** ("running your sample through the agent, about five minutes"). ## 7. Fix what fails Read what a run did before changing anything. **Name the cause before each fix,** from the run's events, and fix the part that owns it rather than adding instructions on top ([common failures](references/failures.md)). Don't reword instructions after each output to steer the next run. - **A case that passes only right after its instructions were changed for it** counts as passing when it passes again in a second run with nothing changed. - **Never weaken the agent to get a run through:** keep the designed model, thinking level and instructions. Speed and cost are optimize's job. - **A partial result beats none.** When a tool refuses part of the work, the agent does the rest and reports what it couldn't do. Stopping with no output is a failure, unless a case says to stop. - **A missing capability:** stop and say so in Part B and the pull request (for example "editing Word needs the `documents` tool; the fallback doesn't finish in time") instead of spending fixes on settings. - **Three fixes to the same problem** and the Recipe still can't run: stop and record why in Part B. **The owner's real document must work;** synthetic excerpts lack its structure (fields, bookmarks, tracked changes, tables). Ask for a sample by naming its case and the input that case brings, in the owner's words. A different kind of input: say how it differs and ask for the right one; it becomes a new case only if the owner asks, approved as in brainstorm. ## 8. Show the owner Show each result ("here's its result on your sample; anything you'd change?"). Run what else they want to try with `introspection tasks create` and show the replies; never ask them to run a command. To talk to the agent, see [with the owner](references/running-cases.md#with-the-owner). What they say becomes cases. For `person` cases, review the runs' transcripts with `review_artifacts`, labelled by case, before build-review: an approved transcript is the person's check. Ask only about a run of the Recipe as it is now; after a change, run the case again first. ## 9. Commit and hand over Commit to `pr/` with the case results in the commit message: passed, or failed with its reason. ## What not to do - Don't change a case to make it pass. If a case looks wrong, note it and raise it with the owner in build-review's summary; the owner decides. - Don't push, deploy or connect real data here; those belong to build-review and deploy. ## Done when - Every case has a result from its subagent: it passes, or has a written reason in Part B (for example, it needs a real connection deploy will make). A case that hasn't run is recorded as planned, never as passing. - The case on the owner's own sample produces a usable result; if it can't, that blocks the build, whatever the other cases show. - Then hand over to `build-review`.