{
  "skill_name": "token-budget-estimator",
  "evals": [
    {
      "id": 1,
      "prompt": "We're planning a Claude agent that triages inbound support tickets for Northwind. System prompt is about 6,000 characters of instructions plus 4 few-shot examples of about 1,500 characters each. Each session averages 8 tool calls: 3 calls to search_kb return about 3 KB of JSON each, and 5 calls to get_ticket_history return about 10 KB of JSON each. We keep the full conversation history. Final output is a 200-word triage note. We'll run around 5,000 sessions a day. Estimate the token budget.",
      "assertions": [
        "Produces a component table (system prompt, few-shot examples, tool definitions, tool results, context accumulation, output, total per session), not a single total",
        "Estimates from the description are ranges, not falsely precise single numbers like '3,142 tokens'",
        "Uses ~4 chars/token for prose and ~3 chars/token for the JSON tool results",
        "Names the single dominant cost driver (likely accumulated tool results / context re-sent across turns) and explains why it dominates",
        "Gives exactly three concrete controls ranked by expected savings, with specific API parameters (max_tokens, cache_control, truncating or paginating tool results)",
        "The rough cost estimate states its pricing-tier assumption rather than guessing silently",
        "Does not invent workflow components, session counts, or payload sizes not in the prompt"
      ]
    },
    {
      "id": 2,
      "held_out": "2026-10-18",
      "prompt": "We're planning the grant-screening workflow in workflow.md, and pilot-usage.csv has the token usage from our pilot. Estimate the token budget and the yearly cost, and tell me whether we're inside finance's cap. One page I can send to finance.",
      "assertions": [
        "Catches that 1,200 applications a round for three rounds is 3,600 a year, not 4,800, and bases the yearly figure on 3,600 (or shows both and flags the error)",
        "Notes that the pilot's scoring stage used the 2,000-character draft rubric, so its measured stage 2 input understates the final 8,000-character rubric (by roughly 1,500 tokens on every model call in the loop), and adjusts for it or flags it",
        "Identifies the scoring stage, on the large model with the history re-sent on every tool-call turn, as the dominant cost",
        "Uses the supplied planning prices, giving roughly $0.07 to $0.13 per application and roughly $250 to $470 a year at 3,600 applications",
        "States whether the estimate fits the $400 annual cap, consistent with its own figures",
        "Does not invent prices, volumes, stages or tool calls that are not in the files",
        "Fits about one page"
      ]
    },
    {
      "id": 3,
      "held_out": "2026-10-18",
      "prompt": "Our school district wants to use an AI to draft report card comments. 600 students, each gets a comment of about 80 words; the teacher pastes in a few notes per student (maybe 50 words), and there's one page of instructions that goes with every request. Roughly how many tokens and how much money is that? Just a few bullets.",
      "assertions": [
        "Gives a rough total for the 600 comments that is consistent with the inputs: on the order of a few hundred thousand tokens in total, not millions",
        "Points out that the page of instructions, sent with every request, makes up most of the input",
        "Any cost figure states the price it assumes and is consistent with that price, coming to a few dollars rather than hundreds",
        "Does not invent extra requirements such as several terms, rewrites or a number of teachers, or marks any such thing clearly as an assumption",
        "Is a few short bullets in plain language",
        "Answers without first asking questions"
      ]
    }
  ]
}