--- name: build-draft description: Write a new Introspection agent (Recipe) from its approved cases, locally, and run the cases until they pass. Use after build-brainstorm, when the cases in evals/cases/ are approved. Not for a repository without approved cases; that's build-brainstorm. Needs no login; build-review follows. --- # Build: draft Write the Recipe, its design and its checks on the `pr/` branch, working only on this machine, and run the approved cases until they pass or every remaining failure is explained. The owner isn't needed in this step. ## Before you start - You need the `introspection` CLI (`npm i -g @introspection-ai/cli`) and Pi. No login: everything here is local. - Work in the repository build-brainstorm created, on its `pr/` branch. Read `docs/DESIGN.md` Part A and every case in `evals/cases/` first. If a case isn't approved, stop and go back to build-brainstorm. Build only the agents that have approved cases: another agent the brief names (a reviewer for a second audience) goes back to build-brainstorm for its own brief and cases, or waits for a later build; don't ask about it or draft it here. ## Steps 1. **Choose capabilities before writing code.** For each case, pick what the agent needs from [the capabilities that exist](references/capabilities.md); an agent whose cases need no tools, documents or connections needs none, so skip it and step 4. Write custom code only where no capability covers a case, and say so in the design. A missing tool is fixed with a tool, not a longer prompt. 2. **Write Part B of `docs/DESIGN.md`** with [the design template](references/design-template.md): each behaviour or capability cites the cases that need it, anything no case needs is cut, and every case says how it's checked. 3. **Write the Recipe, then smoke-test it once:** after setting the model, run it as a local task from the build folder, `introspection tasks create runs/smoke-1 --work-dir --prompt "hello"`, and confirm the agent answers on the model you chose, not the template's. `introspection check` validates the files; only a run catches a missing model key, a model Pi has no credentials for, or a listed tool that doesn't exist. One run is enough, and none when the rehearsal follows straight away, since its first turn does the same. **If it fails, stop there and tell the owner what failed and what you need** (Pi signed in, network allowed for `introspection`, an extension conflict fixed): an agent that can't run can't be drafted or reviewed. Never fix the machine yourself: don't run `introspection setup` or `upgrade`, and don't change Pi's settings or point Pi at another settings folder. **What the Recipe holds:** instructions, skills, tools and the `package.json#pi` declarations the design calls for ([what the template gives you and how the pieces compose](references/recipe-layout.md); for Word or PDF files, [documents](references/documents.md)). Put instructions every agent shares in `SYSTEM.md`, role-specific ones in each agent's `system_instructions`, and detail the agent needs only sometimes in skills; `tools`, `skills` and `subagents` lists replace rather than merge when one agent builds on another. Choose the model and provider for local runs deliberately, one the owner's machine has credentials for, and record them in Part B; replace the template's placeholder model. - **Every tool you list must exist in the installed runtime.** `introspection check` doesn't verify tools; only a run does. If a capability isn't installed (for example `pi.documents` before its release), use its fallback and say so in Part B. - **Make standing knowledge reachable at run time.** Put the owner's rules, positions and situation (a playbook, a scoring guide, a brief) in a skill: `skills//SKILL.md` with the rules and facts written out as text, the source files beside it, named by paths relative to the skill's folder so the agent can open them in a run and after deploy, and the skill listed in the agent's `skills`. Tell the agent never to search outside its workspace for them. A file no skill or instruction names is invisible to the agent, and Word and PDF files can't be read at run time without a tool. Put the exact values the owner gave (author name, output naming, side) in the instructions. - **Write two to four conversation starters** as Pi prompt templates in `prompts/.md`, listed with `"pi": { "prompts": ["./prompts/*.md"] }` in `package.json`. People see them when they open a new conversation with the agent: in Pi as its first question, and as chips in the Introspection chat. Each is a short request a user would really type, **no more than 10 words**, in the owner's language, taken from an approved ordinary-work case so it's something the agent has been tested on (for an interviewer, "Interview me for the data engineer role"; for contract review, "Redline the contract in files/"). Give each a `description` frontmatter line that reads as the label, the same short phrase, and `suggest: true`, which marks a template as a starter; other templates stay `/` commands only. No starter for something the cases don't cover. - **Files for people go in `outputs/`** (`/workspace/outputs` when deployed): the platform keeps them for whoever may see that task. Name the folder in the instructions, never "the workspace". - **Keep test details out of the instructions:** nothing specific to a sample input (a sample's party name, a test file path) goes into the Recipe. 4. **Give the agent its data without real credentials.** Decide which servers the cases need and how much of each to fake, and what Part B reports about them ([build's decisions about fakes](references/fakes.md)). Build the fakes as `eval-fakes` says, write each one as its `write` reference says, and bind them in an eval as `eval-author`'s `vitest-fakes` or `harbor-fakes` says. Case inputs are read from the local machine by hash; never commit a client's file. 5. **Turn the cases into checks with the eval runner subagent (`agents/eval-runner.yaml`).** Start one for the build, given the repository and the ids of the cases marked `code`: it writes their evals from the approved case files ([which runner and how they're scored](references/evals.md)), runs them, and reports each case's result, never changing the Recipe or a case. Keep the same one for the whole build, messaging it to run again after you change the Recipe. Cases marked `person` are rehearsed (step 7). The eval runner reads `eval-author`, `eval-run` and their references; you don't need to. 6. **Prove it works, then show the owner.** Draft runs every case once, locally; deploy runs them again against the deployed version. Run the cases as step 7 says: the `code` ones in the eval suite, the `person` ones as rehearsals (the owner's sample among them, if they gave one). Tell the owner in one line before anything that takes more than a minute ("running your sample through the agent, about five minutes"). When it produces a result, show it ("here's its result on your sample; anything you'd change?") and offer them `cd && introspection local` to talk to it themselves; what they say becomes cases. 7. **Run and iterate.** - `introspection check` validates the Recipe. - **A case counts as run only when the Recipe produces the result itself,** run the way a real request runs: from a folder of its own in the workspace, beside the agent's repository (`runs/-/`), never inside the repository and never in `/tmp`, with the case input under its `files/`, writing the output where the request says. Run every case the same way, in every host, as a local task named by its folder: `introspection tasks create runs/- --work-dir --file @… --prompt ""` lays the folder out as a sandbox workspace (`files/`, `outputs/`, `memories/`, `repos/`), puts the inputs in its `files/` and runs the first turn. Each further turn of a conversation is `introspection tasks prompt runs/- --prompt ""`, in the same Pi session. Both print the agent's reply; a turn still running after their wait says so, and `introspection tasks get runs/-` follows it: look at what it's doing rather than starting it again. Setting up a run: from the build folder, the one holding the agent's repository, run `tasks create` with the new folder's path; don't make the folder first or `cd` into it, since `create` makes and lays it out. Pass every input the request brings with `--file`, so it lands in the run's `files/`. A second attempt is a new folder, `runs/-`: never reuse, empty or delete an earlier run's folder, which is the record of what that version did. The documents tool opens files only inside the workspace, so that's where you can read what the run produced; never copy a run's output into the repository to read it. Proof is a fresh run whose prompt holds only what a real request brings; a run given anything more, or started inside the repository, proves nothing. - **Use the `introspection` CLI for anything Recipe-related:** `tasks get` and `conversations get` to read a run, `eval create` and `eval run` for evals, `local` to run the agent; never a script over a run's session or spans files. - **Change the Recipe's own files with `edit` and `write`,** one file at a time; a script that rewrites files you wrote is a sign you need a different edit, not a regex. Scripts are for data you can't otherwise read. - **Refer to the Recipe's own files relative to the skill that holds them,** never to the repository root: a real run starts in the request's folder, so `python python/helper.py` fails there. - **The owner's real document must work.** Synthetic excerpts you wrote can't stand in for it: they lack the structure real files have (fields, bookmarks, tracked changes, tables). Draft isn't done until the case on the owner's own sample produces a usable result; if it can't, that blocks the build, whatever the other cases show. When you ask for a sample, name the case it's for and describe the input that case's request brings, in the owner's words, so they send the right kind of file, not just any example of their work. If what they share is a different kind of input, say how it differs and ask for the right one; a new kind of input becomes a new case only if the owner asks for one, approved as in brainstorm. - **A partial result beats none.** When a tool refuses part of the work, the agent does the rest and reports what it couldn't do. Stopping with no output is a failure, unless a case says to stop. - **Never produce the agent's output yourself.** A script you run is not the Recipe, and its result proves nothing about the agent. - **Run every case in subagents, all at once:** once the Recipe is written, start the eval runner for the `code` cases and a `case-runner` for each `person` case in the same turn, then wait for them all. Don't run a case yourself. A smoke test isn't a case run, and scaffolding an eval doesn't run it: draft isn't done until every case has a result from its subagent. - **Cases marked `code` run in the eval runner's suite, not by hand.** Start it alongside the `case-runner`s. When a case fails, read its report: if the agent got it wrong, fix the Recipe and ask it to run that case again; if it says the check misjudged a run that did what the case asks, check that against the case before agreeing a change. Ask for the whole suite once the Recipe is done, before build-review. Slow cases wait for deploy-test. Don't also rehearse a `code` case: its eval already plays the conversation. In Claude Code or Codex, start a subagent of your own with the same instructions (`agents/eval-runner.yaml` in the Operator). - **Rehearse the cases marked `person` with `case-runner` subagents, all at once.** When the agent's cases are conversations (an interview, a support chat), one prompt tests only the opening, so each runs as a whole conversation. Start one `case-runner` per case, up to four at a time, each given the case's situation as text (never its good or bad result, and never the case file's path), the repository, the agent the case is for (its file name in `agents/`, or "the default agent"; never the word Recipe), its run folder `runs/-` and any inputs the request brings. Each plays the person and runs the conversation itself with `introspection tasks create` and `tasks prompt`, then returns its folder. Wait for them all, then read each conversation (`introspection conversations get runs/-`) and its `outputs/`, and judge it against the full case, which only you have seen. Never write or edit the person's messages yourself. In Claude Code or Codex, where there's no `case-runner`, start a subagent of your own with the same instructions (`agents/case-runner.yaml` in the Operator), or play the person yourself, one case at a time. - **With the owner**, when they want to talk to the agent or the case turns on what only they know: either relay (they write each message in this chat, you send it with `tasks prompt` word for word and show the agent's reply word for word, adding nothing), or they talk to it themselves in a terminal: `pi --recipe --session-dir runs/-/.introspection/session --session-id -` opens the same conversation, a rehearsal included. Don't send turns while they have it open. A conversation they had is evidence: offer to turn a turn that went wrong into a case (`eval-case`, `--from-session`). - **Show the owner the conversations.** For cases checked by a person, ask for the review with `review_artifacts` on the runs' transcripts, labelled by case, before build-review: an approved transcript is the person's check. Only ask about a run of the Recipe as it is now: after you change the Recipe, run the case again before asking, rather than asking about a run from before the change. - **Read what a run did before changing anything.** `introspection tasks get runs/-` says whether a turn is still going (wait for it), finished, or failed, and what it was doing; the folder holds the session for the detail. A failure with the model's error is the provider (access, credentials, quota): fix that or tell the owner; never switch the agent to another model to get it through. - **Never weaken the agent to get a run through:** keep the designed model, thinking level and instructions. Speed and cost are optimize's job, with measurements; a proof on a weakened agent proves nothing. - **When the cause is a missing capability, stop and say so** in Part B and the pull request (for example "editing Word needs the `documents` tool; the fallback writes very large scripts and doesn't finish in time") instead of spending fixes on settings. Fix the part that owns a failure rather than adding instructions on top ([common failures](references/failures.md)). **Name the cause before each fix,** from the run's events, and change only what that cause needs; don't reword instructions after each output to steer the next run. A case that passes only right after its instructions were changed for it counts as passing when it passes again in a second run with nothing changed. If the Recipe still can't run after three fixes to the same problem, stop and record why in Part B. Stop when every case passes, or when each remaining failure has a written reason in Part B (for example, it needs a real connection deploy will make). A case that hasn't been run is recorded as planned, never as passing. 8. **Commit to `pr/`** with the case results in the commit message (passed, failed with reason), and hand over to **build-review**. ## What not to do - Don't change a case to make it pass. If a case looks wrong, note it and raise it with the owner in build-review's summary; the owner decides. - Don't push, deploy or connect real data here; those belong to build-review and deploy.