{
  "skill_name": "guide-builder",
  "evals": [
    {
      "id": 1,
      "prompt": "Turn these notes into a guide for new engineers getting our Acme billing service running locally:\n\n- need docker + node 20\n- clone acme/billing, cp .env.example .env, set STRIPE_KEY to a test key\n- docker compose up -d postgres\n- npm ci then npm run migrate\n- npm run dev, it listens on 4000\n- if migrate fails it's usually postgres not up yet",
      "assertions": [
        "States the detected style (setup guide) up front with a one-line why and an invitation to redirect",
        "Prerequisites are listed before the first step, and placeholders like <YOUR_STRIPE_TEST_KEY> are unmistakable and listed once up front",
        "Each numbered step contains one action — 'npm ci then npm run migrate' is split into separate steps",
        "Every command block is complete, paste-able, and followed by an expected result or verification (e.g. the service answers on port 4000)",
        "Leads with the outcome rather than backstory, and pushes troubleshooting (migrate failing because postgres isn't up) to the end",
        "No forward references that break top-to-bottom reading",
        "Missing facts (e.g. the repo URL host or how to get a Stripe test key) are flagged as NEEDS INPUT rather than invented"
      ]
    },
    {
      "id": 2,
      "held_out": "2026-10-05",
      "prompt": "Mirela dumped her notes on our partner API in api-notes.md and there's a sample response in sample-booking.json. Can you turn that into proper reference docs we can send to the partner devs who are integrating with us? Make sure the cancellation webhook is covered too, a couple of partners have asked about it.",
      "assertions": [
        "The rate limit is given as 60 requests per minute per key, following the changelog entry, and 120 is not presented as the current limit",
        "It flags that the notes say money is integer cents in fields ending _cents while the sample response has decimal strings in unit_price and total, and does not silently document only one of them",
        "It does not name the API key header (for example X-API-Key or Authorization: Bearer) as fact, and says the header name is not in the notes",
        "It says the notes describe no cancellation webhook, only booking.confirmed, and does not document a cancellation event or payload",
        "POST /bookings documents the required Idempotency-Key header and that reusing a key within 24 hours returns the original booking",
        "The error responses in the notes are documented with their bodies: 409 slot_full with available, 422 validation_error, 429 with Retry-After, and 409 cancellation_window_closed",
        "The POST /bookings body lists each field as required or optional, including quantity 1 to 8 and partner_ref up to 64 characters",
        "The webhook section states the HMAC-SHA256 signature over the raw body in X-Tidewater-Signature and the 5 retries over about an hour"
      ]
    },
    {
      "id": 3,
      "held_out": "2026-10-05",
      "prompt": "can you write up how our nightly export works for the new analyst starting monday? roughly: cron on the reporting box kicks off export.py at 2am, it pulls yesterday's orders from the replica, writes a csv to the sftp drop, and finance's tool picks it up at like 6. if it fails Theo gets an email. there's a retry thing in there somewhere but i honestly don't remember how it works",
      "assertions": [
        "It delivers a written explanation of the export for the analyst, and does not stop at questions",
        "It describes the flow in order: cron at 2am runs export.py, which reads yesterday's orders from the replica, writes a CSV to the SFTP drop, and finance's tool collects it",
        "It says how the retry works is not known and needs confirming, and does not state a retry count, delay or backoff",
        "It does not invent hostnames, file paths, an SFTP directory, table names, CSV columns or a CSV filename",
        "It says Theo is emailed when the export fails, and does not add an email address, a surname or other contacts",
        "The finance pickup is given as around 6, not as an exact confirmed time",
        "It includes no commands for running, restarting or debugging the export, since none were supplied"
      ]
    },
    {
      "id": 4,
      "held_out": "2026-10-06",
      "prompt": "Laleh starts on Monday the 12th and I need a laptop setup guide she can follow on her own, she's an architect not an IT person. Bram's setup notes are in it-notes.txt and there's some later email in handover-thread.txt. Can you also include how to set up the big plotter, she'll need it in her first week.",
      "assertions": [
        "The P: drive is mapped to \\\\fo-nas02\\projects, and \\\\studio-nas\\projects is not given as the address to use",
        "Setting up Keyhaven comes before the step to join the Fenwick-Staff Wi-Fi, since the Wi-Fi password is stored in Keyhaven",
        "It does not tell Laleh that Odile will send the Keyhaven invite; it says Odile has left and who sends the invite is not confirmed, without naming Tomasz or Bram as the confirmed sender",
        "It says the material has nothing on the plotter, and does not invent a plotter model, driver, address or setup steps",
        "The DraftForge step gives the licence server as 27000@fo-lic01 and says that a \"no licence available\" message means a colleague has to close DraftForge to free a seat",
        "It tells her the Keyhaven master password cannot be reset if lost",
        "It tells her Desktop and Documents are not backed up and project files belong on P:",
        "Working from home is covered: connect Tailgate VPN first, then P: and DraftForge work as in the office"
      ]
    },
    {
      "id": 5,
      "held_out": "2026-10-06",
      "prompt": "can you write a short reference for the event our door sensors send? here's one off the queue:\n\n{\"dev\":\"ds-0417\",\"st\":1,\"bat\":83,\"ts\":1791188400123,\"rssi\":-71,\"fw\":\"2.4.1\"}\n\nst is 0 closed 1 open, there might be a 2 but i've never seen one. bat is percent i think. ts is epoch something. it's going in the README for the two contractors building the dashboard so keep it short",
      "assertions": [
        "It delivers the reference, and does not stop at questions",
        "All six fields in the sample are listed: dev, st, bat, ts, rssi and fw",
        "st is documented as 0 for closed and 1 for open, and a value of 2 is mentioned as possible with its meaning unknown, not given a made-up meaning",
        "It says ts looks like epoch milliseconds going by its 13 digits, and marks that as read from the sample, not confirmed",
        "bat is described as probably a percentage and marked as unconfirmed",
        "The meanings given for rssi and fw are marked as inferred from the sample, since the user did not explain them",
        "It does not invent a queue or topic name, how often events are sent, delivery guarantees, value ranges stated as fact, or fields that are not in the sample",
        "The reference is about a page or less, with no sections on things that were not supplied such as authentication, retries or versioning"
      ]
    }
  ]
}