{ "skill_name": "guide-builder", "evals": [ { "id": 1, "prompt": "Turn these notes into a guide for new engineers getting our Acme billing service running locally:\n\n- need docker + node 20\n- clone acme/billing, cp .env.example .env, set STRIPE_KEY to a test key\n- docker compose up -d postgres\n- npm ci then npm run migrate\n- npm run dev, it listens on 4000\n- if migrate fails it's usually postgres not up yet", "assertions": [ "States the detected style (setup guide) up front with a one-line why and an invitation to redirect", "Prerequisites are listed before the first step, and placeholders like are unmistakable and listed once up front", "Each numbered step contains one action — 'npm ci then npm run migrate' is split into separate steps", "Every command block is complete, paste-able, and followed by an expected result or verification (e.g. the service answers on port 4000)", "Leads with the outcome rather than backstory, and pushes troubleshooting (migrate failing because postgres isn't up) to the end", "No forward references that break top-to-bottom reading", "Missing facts (e.g. the repo URL host or how to get a Stripe test key) are flagged as NEEDS INPUT rather than invented" ] }, { "id": 2, "held_out": "2026-10-05", "prompt": "Mirela dumped her notes on our partner API in api-notes.md and there's a sample response in sample-booking.json. Can you turn that into proper reference docs we can send to the partner devs who are integrating with us? Make sure the cancellation webhook is covered too, a couple of partners have asked about it.", "assertions": [ "The rate limit is given as 60 requests per minute per key, following the changelog entry, and 120 is not presented as the current limit", "It flags that the notes say money is integer cents in fields ending _cents while the sample response has decimal strings in unit_price and total, and does not silently document only one of them", "It does not name the API key header (for example X-API-Key or Authorization: Bearer) as fact, and says the header name is not in the notes", "It says the notes describe no cancellation webhook, only booking.confirmed, and does not document a cancellation event or payload", "POST /bookings documents the required Idempotency-Key header and that reusing a key within 24 hours returns the original booking", "The error responses in the notes are documented with their bodies: 409 slot_full with available, 422 validation_error, 429 with Retry-After, and 409 cancellation_window_closed", "The POST /bookings body lists each field as required or optional, including quantity 1 to 8 and partner_ref up to 64 characters", "The webhook section states the HMAC-SHA256 signature over the raw body in X-Tidewater-Signature and the 5 retries over about an hour" ] }, { "id": 3, "held_out": "2026-10-05", "prompt": "can you write up how our nightly export works for the new analyst starting monday? roughly: cron on the reporting box kicks off export.py at 2am, it pulls yesterday's orders from the replica, writes a csv to the sftp drop, and finance's tool picks it up at like 6. if it fails Theo gets an email. there's a retry thing in there somewhere but i honestly don't remember how it works", "assertions": [ "It delivers a written explanation of the export for the analyst, and does not stop at questions", "It describes the flow in order: cron at 2am runs export.py, which reads yesterday's orders from the replica, writes a CSV to the SFTP drop, and finance's tool collects it", "It says how the retry works is not known and needs confirming, and does not state a retry count, delay or backoff", "It does not invent hostnames, file paths, an SFTP directory, table names, CSV columns or a CSV filename", "It says Theo is emailed when the export fails, and does not add an email address, a surname or other contacts", "The finance pickup is given as around 6, not as an exact confirmed time", "It includes no commands for running, restarting or debugging the export, since none were supplied" ] }, { "id": 4, "held_out": "2026-10-06", "prompt": "Laleh starts on Monday the 12th and I need a laptop setup guide she can follow on her own, she's an architect not an IT person. Bram's setup notes are in it-notes.txt and there's some later email in handover-thread.txt. Can you also include how to set up the big plotter, she'll need it in her first week.", "assertions": [ "The P: drive is mapped to \\\\fo-nas02\\projects, and \\\\studio-nas\\projects is not given as the address to use", "Setting up Keyhaven comes before the step to join the Fenwick-Staff Wi-Fi, since the Wi-Fi password is stored in Keyhaven", "It does not tell Laleh that Odile will send the Keyhaven invite; it says Odile has left and who sends the invite is not confirmed, without naming Tomasz or Bram as the confirmed sender", "It says the material has nothing on the plotter, and does not invent a plotter model, driver, address or setup steps", "The DraftForge step gives the licence server as 27000@fo-lic01 and says that a \"no licence available\" message means a colleague has to close DraftForge to free a seat", "It tells her the Keyhaven master password cannot be reset if lost", "It tells her Desktop and Documents are not backed up and project files belong on P:", "Working from home is covered: connect Tailgate VPN first, then P: and DraftForge work as in the office" ] }, { "id": 5, "held_out": "2026-10-06", "prompt": "can you write a short reference for the event our door sensors send? here's one off the queue:\n\n{\"dev\":\"ds-0417\",\"st\":1,\"bat\":83,\"ts\":1791188400123,\"rssi\":-71,\"fw\":\"2.4.1\"}\n\nst is 0 closed 1 open, there might be a 2 but i've never seen one. bat is percent i think. ts is epoch something. it's going in the README for the two contractors building the dashboard so keep it short", "assertions": [ "It delivers the reference, and does not stop at questions", "All six fields in the sample are listed: dev, st, bat, ts, rssi and fw", "st is documented as 0 for closed and 1 for open, and a value of 2 is mentioned as possible with its meaning unknown, not given a made-up meaning", "It says ts looks like epoch milliseconds going by its 13 digits, and marks that as read from the sample, not confirmed", "bat is described as probably a percentage and marked as unconfirmed", "The meanings given for rssi and fw are marked as inferred from the sample, since the user did not explain them", "It does not invent a queue or topic name, how often events are sent, delivery guarantees, value ranges stated as fact, or fields that are not in the sample", "The reference is about a page or less, with no sections on things that were not supplied such as authentication, retries or versioning" ] }, { "id": 6, "held_out": "2026-10-17", "prompt": "Can you turn setup-notes.md and thread.txt into a setup guide for Wren? She's a new analyst starting Monday and needs to get Larkspur running on her laptop on day one without pinging anyone. She's fine in a terminal but has never seen our stack. Something she can follow in one sitting, under 600 words.", "assertions": [ "The steps run in an order that works: lark-analysts access is requested up front, `lark runtime start` comes before `lark db init`, the database is initialised before `lark seed`, and the config file is created before `lark login`, which comes before `lark sync`, even though the notes put seeding first", "The local database port is given as 6543, the 2.4 default from the thread, and 5433 is not presented as the current port", "`lark deploy --prod` does not appear as a step, and the lunch chat is left out", "If the guide says the login can stop working and need `lark login` again, it is marked as one colleague's experience rather than stated as documented behaviour", "It does not invent a region or workspace value, a VPN requirement either way, or any command, flag, file path, error message or timing not in the two files", "The guide is delivered without stopping to ask first, and the unanswered questions (which region Wren should use, whether sync needs the VPN off-site) are raised to the requester outside the guide rather than answered by guesswork inside it", "The guide is under 600 words" ] }, { "id": 7, "held_out": "2026-10-17", "prompt": "hi, I do the welcome stuff for Brackenhill Allotment Association. can you make this into a short welcome guide for new plot holders? half a page max, they're mostly retirees and young families, not techy at all.\n\nshed key is in the lockbox by the gate, code is on the welcome sheet. water troughs get turned on in april by whoever's on the rota. sign the tool book when you borrow anything. bonfires NOT allowed, council letter last yr. compost bays - green bin is for weeds, the wooden ones for everything else? not sure tbh ask Derek. new folk do their induction walk with a committee member before they get a key", "assertions": [ "The induction walk with a committee member comes before getting the key or using the lockbox, even though the notes mention the key first", "It does not invent the lockbox code, a date for the water going on, rota names, contact details for Derek or anyone else, rent, opening hours, penalties or any rule not in the notes", "The compost-bay split is not stated as a settled rule in the guide, and the requester is told outside the guide that it needs confirming with Derek", "The bonfire ban and signing the tool book are both included as clear rules", "It is written in plain everyday language a non-technical newcomer can follow, with no jargon or committee shorthand left unexplained", "The guide fits in about half a page (roughly 250 words or fewer) and is delivered without stopping to ask questions first" ] } ] }