{
  "skill_name": "capacity-cost-model",
  "evals": [
    {
      "id": 1,
      "prompt": "Model the run cost for our new document-search service at Northwind on AWS. Demand: 1,200 active tenants today, growing about 8% a month; average 15 searches/sec, peak 60/sec weekday mornings. Each tenant indexes about 2 GB. We'll use OpenSearch, ECS Fargate for the API, and CloudWatch for logs. There are dev and staging environments too. What will it cost now and at 3x and 10x?",
      "assertions": [
        "Sizes from the demand driver (tenants and searches/sec, with the 8%/month growth curve), not from a chosen instance count",
        "Every cost line is a unit price x quantity, with the unit price's source and date tagged",
        "Expresses unit economics (e.g. cost per active tenant per month), not just a bare monthly total",
        "Egress, observability (CloudWatch logs), and non-prod (dev and staging) lines are present or explicitly marked n/a",
        "States average and peak load separately and makes the headroom/target-utilization choice explicit with reasoning",
        "Projects 1x/3x/10x and names at least one scaling cliff (e.g. an OpenSearch tier or shard step) rather than extrapolating linearly",
        "Outputs are low/expected/high ranges with a lean, and the cost-per-tenant trend at scale is checked and flagged if it rises",
        "Does not invent demand figures or services beyond those in the prompt; unknown inputs are stated as assumptions"
      ]
    },
    {
      "id": 2,
      "held_out": "2026-10-18",
      "prompt": "I'm on the platform team at Brackwater Water. The programme manager wants a cloud cost estimate for the smart meter data platform for the finance committee. Her request is in brief.md, field ops' rollout plan is in rollout-plan.md, and our provider's prices are in price-sheet.md. Can you put the estimate together?",
      "assertions": [
        "Sizes ingestion and database writes for the synchronised upload window (about 4,800 readings per second at 120,000 meters), not for the daily average (about 130 per second)",
        "Points out that the brief's hourly readings and 'about 1 million readings a day' do not match the rollout plan's 15-minute readings, which give about 11.5 million readings a day at 120,000 meters",
        "Names the database Standard tier's 5,000 writes-per-second limit as a cliff: the year-2 peak sits just under it with little headroom, and the 3x scenario crosses it into the $2,400-a-month Enterprise tier",
        "Keeps the regulator's 7-year retention as a fixed requirement and does not propose shortening it to cut cost",
        "Does not give a payback period or savings figure, and says the material has no benefit or old-contract figures to compute one from",
        "Uses only prices from the price sheet; any cost not on it is marked as an assumption, not presented as a Nimbus price",
        "Fits in about two pages, as the committee asked"
      ]
    },
    {
      "id": 3,
      "held_out": "2026-10-18",
      "prompt": "hi, i run a bakery (one shop right now) and my nephew set up our online ordering website. the hosting bill was $45 last month and we had roughly 1,200 orders. we're opening two more shops next spring. how much will the website cost us then? also christmas week we get about 4x the normal orders. just a few bullet points please, i'm not technical",
      "assertions": [
        "Gives an estimated monthly cost or range for after the new shops open, rather than only asking questions",
        "Bases the estimate on order volume and states the assumption about how orders grow with two more shops",
        "Does not simply triple the $45 without saying that part of a hosting bill is often a fixed plan fee that does not grow with orders",
        "Flags the Christmas peak (about 4x normal orders, so roughly 12x today's volume if orders triple) as the thing to check against the hosting plan's limits before December",
        "Suggests what to ask the nephew or the hosting company, such as what the $45 plan covers and what its limits are",
        "Does not invent a hosting company's prices, plan names or limits",
        "Stays at a few plain-language bullet points with no unexplained jargon"
      ]
    }
  ]
}