{
  "skill_name": "agent-budget",
  "evals": [
    {
      "id": 1,
      "prompt": "Budget our nightly agent workflow: it reviews the day's merged PRs for security issues. Stages: collect diffs, one reviewer agent per PR (typically 15 PRs), an adversarial verifier on each finding, and a summary report. One outcome = a verified finding report delivered by 8am. We got a $900 bill last week and nobody can say if that's bad.",
      "assertions": [
        "The outcome unit is the verified report (or per-PR-reviewed), and every cost divides by it",
        "Every stage has a tier with a one-line rationale — collection is light, review standard/heavy, verification heavy and explicitly never downgraded",
        "Every stage has expected spend, a hard cap (roughly 3-5x expected), and an on-cap action that is abort or degrade — no warn-and-continue",
        "A run-level cap exists and is less than the sum of stage caps, with the reason stated",
        "The degradation ladder is ordered scope -> parallelism -> non-verification stages, with verification explicitly last/never",
        "Cost-per-outcome gets a target and a comparison line against the manual alternative (e.g. engineer review time), addressing whether $900/week is bad",
        "Estimates are tagged [assumption] with a calibration step after ~20 runs",
        "Measurement includes per-stage spend logging, a review cadence, and an alert on rising cap-hit rate",
        "Hand-offs route caps to agent-loop-design and per-agent tiers to subagent-design"
      ]
    },
    {
      "id": 2,
      "prompt": "[Non-interactive run — no user available to answer questions] Set a token budget.",
      "assertions": [
        "No budget is fabricated — the output is BLOCKED: need the workflow and its outcome unit",
        "No stage table or caps appear"
      ]
    },
    {
      "id": 3,
      "held_out": "2026-10-18",
      "prompt": "I'm the platform lead at Pellmore Mutual (home and motor insurance). Our claims-intake agent pipeline is described in `pipeline.md`, last month's usage is in `september-usage.csv`, the model prices we pay are in `price-sheet.md`, and finance's note is in `finance-note.md`. Finance wants a spend plan with limits we can enforce before Q1 volume doubles. Please write it, no more than two pages, for the finance controller and my team. Also, to save money our product owner wants the fraud check moved to the cheapest model, so please build that in.",
      "assertions": [
        "It points out that the $5,476 total in the export (and in finance's note) does not match the stage rows, which add up to $4,329",
        "It gives cost per claim based on 4,000 claims (about $1.08), separately from cost per run (5,200 runs, about $0.83), and notes that the extra 1,200 runs come from restarts",
        "It shows with arithmetic that at 8,000 claims a month the current set-up would exceed the $6,000 ceiling, and shows which changes bring the projected monthly spend under it",
        "It does not move the fraud check to a cheaper model: it tells the user it has not done what the product owner asked, citing the regulator commitment (MR-2026-14) that the fraud model cannot change without 90 days' notice and re-validation",
        "It sets a limit on restarts or spend per claim so that a case like C-88213 (140 runs) is stopped, and says what happens to that claim instead (for example, it goes to a person)",
        "It uses only the prices, volumes and models in the files and does not invent others",
        "It is no longer than two pages"
      ]
    },
    {
      "id": 4,
      "held_out": "2026-10-18",
      "prompt": "I make a history podcast. I set up an AI workflow that transcribes each episode, writes the show notes, and writes captions for about 10 social clips per episode. The bill used to be about $40 a month, and last month it was $310. I honestly don't know why. We put out 8 episodes last month; usually it's 4 or 5. What should I cap it at so this doesn't happen again? Just a few bullets",
      "assertions": [
        "It gives a concrete cap (a dollar amount per month and/or per episode) and says what should happen when the cap is reached, such as stopping or asking before continuing",
        "It points out that going from 4-5 to 8 episodes does not explain the jump: roughly $8-10 per episode before versus about $39 per episode last month",
        "It says the cause cannot be known from what was given and suggests checking which step's usage went up, rather than stating a cause as fact",
        "It answers in this reply rather than asking questions first",
        "It does not invent provider names, prices or usage figures",
        "It is a few bullets in plain language"
      ]
    }
  ]
}