{
  "skill_name": "release-notes",
  "evals": [
    {
      "id": 1,
      "prompt": "Write release notes for v2.4.0 from this log (range v2.3.0..HEAD, 2026-08-12 to 2026-09-28):\n\na1b2c3d|feat: bulk CSV import for contacts (#412)\ne4f5a6b|fix: NPE in ReportExporter.flush() when >10k rows (#418)\nc7d8e9f|feat!: remove legacy /v1/contacts endpoint (#420)\n0a1b2c3|chore(deps): bump lodash 4.17.20 -> 4.17.21\n4d5e6f7|refactor: extract ExportService\n8a9b0c1|fix stuff\n2d3e4f5|perf: cache tenant settings, report export ~3x faster (#425)",
      "assertions": [
        "Delivers the notes from the log without stopping to ask who they are for, stating any assumption about the audience in a line",
        "The removal of /v1/contacts comes first, with the step API users must take",
        "Customer notes contain no commit-speak — no ReportExporter.flush(), class names, ticket numbers, or author names",
        "The lodash bump and the ExportService refactor do not appear in customer-facing notes, not even as 'various improvements'",
        "The notes carry no working notes: no commit range, commit count, hashes or review lists in customer-facing text",
        "The vague 'fix stuff' commit is not given an invented meaning in the notes, and is named to the user outside them",
        "No marketing adjectives; the performance line states the measured change (~3x faster export) rather than 'blazing fast'",
        "Does not invent features or fixes not present in the log"
      ]
    },
    {
      "id": 2,
      "held_out": "2026-10-05",
      "prompt": "We're cutting 3.2.0 of Tidewater Booking tomorrow. The log since 3.1.0 is in commits.txt. I need two things: the \"what's new\" text for the customer email (goes to clinic owners and to the devs who integrate with our API and webhooks), and the entry for CHANGELOG.md in the repo. Marketing wants the headline to be SMS reminders and the Google Calendar sync, and wants to say exports are 5x faster. We're a hosted service, nobody runs our code themselves.",
      "assertions": [
        "Google Calendar sync is not announced as shipping in 3.2.0; the output says it was reverted (commit a4e8b60 / #225) and is not in this release",
        "The export speed-up is given as 41s to 14s for 50k bookings, or roughly 3x, and the requested '5x faster' claim is not used or is called out as unsupported by the log",
        "The webhook field rename (start/end to starts_at/ends_at in booking payloads) is called out as a breaking change that integrators must act on, even though its commit is labelled refactor",
        "The removal of X-Tidewater-Key header auth is called out as breaking with the instruction to switch to Bearer tokens, and breaking changes appear before the new features",
        "The 'wip' (3b7c5e2) and 'more fixes' (71f9d3b) commits are flagged as unclear and needing someone to check, not described as specific features or fixes",
        "The duplicate-calendar-events fix (#218) is not listed as a fix customers receive, because it was reverted along with the sync",
        "The customer email text does not mention the twilio or vite bumps, the BookingRepo rename, CI changes or the Node 20 server requirement",
        "Each line of the CHANGELOG entry carries a commit hash or PR number from the log"
      ]
    },
    {
      "id": 3,
      "held_out": "2026-10-05",
      "prompt": "need release notes for pressroom (our static site CLI, the people using it are our clients' developers), going out as 0.9.0 today. this is what's gone in since 0.8.2, half from git log --oneline and half from memory, sorry:\n\nd41c0aa add --watch flag\n77e2b19 fix crash on empty frontmatter\nbump deps\n3a9f6c1 config: `output_dir` now `outDir` (old key ignored)\nc08e512 misc cleanup\n9b1d7e4 faster builds\n\nalso Priya fixed something with image paths on windows, I don't have the commit",
      "assertions": [
        "The output_dir to outDir config change is flagged as breaking, with the action to rename the key in existing config files because the old key is now ignored",
        "'Faster builds' is reported without any invented figure or percentage, and the output says the size of the speed-up is not known",
        "The Windows image-path fix is included only as far as described and marked as unconfirmed or missing its commit; no specific bug behaviour is invented for it",
        "'misc cleanup' (c08e512) and 'bump deps' are not turned into user-facing claims",
        "The --watch flag and the empty-frontmatter crash fix both appear as user-facing items",
        "It does not invent dates, PR numbers, a commit count for commits it was not shown, or features not in the pasted list",
        "It delivers usable release notes in the reply rather than only asking questions, and lists the items that need confirming before sending"
      ]
    },
    {
      "id": 4,
      "held_out": "2026-10-06",
      "prompt": "Portside WMS 5.0.0 goes out to customers on Thursday. The log since 4.9.2 is in git-log.txt. Two things please: (1) upgrade notes for admins. Our customers run Portside on their own servers, so their IT people need to know what to do before they upgrade. (2) A short paragraph for the customer newsletter, three or four sentences. Sales want the newsletter to lead with batch scanning and to say the duplicate pick list problem is finally fixed. Also remind people that 4.x support ends on 31 December.",
      "assertions": [
        "The PostgreSQL 15+ requirement is called out as something admins must deal with before upgrading (the installer aborts on PostgreSQL 13 and 14), even though its commit is labelled chore",
        "The removal of /api/v1/stock is called out as a breaking change with /api/v2/stock-levels as the replacement, and the breaking items come before the new features in the admin notes",
        "Batch scanning is described as off by default (flag scan.batch.enabled, missed reads on older handhelds), not as a feature that works out of the box after upgrading",
        "The duplicate pick list problem is not described as fully fixed; the output says split shipments can still produce a duplicate line (#1893) or flags that the 'finally fixed' claim is not supported by the log",
        "The end of 4.x maintenance is given as 31 March 2027, or the conflict between the requested 31 December date and the log is flagged; 31 December is not repeated as fact",
        "The wms.mail.host to wms.smtp.host rename is described as a deprecation where the old key still works, not as a change that breaks existing configuration on upgrade",
        "The admin notes include the security fix for session tokens not being invalidated on password change (LSA-2026-04)",
        "The newsletter paragraph does not mention the spring-boot bump, the StockLedger refactor or the CI test shards"
      ]
    },
    {
      "id": 5,
      "held_out": "2026-10-06",
      "prompt": "need the what's new text for the app stores for Tallybird 1.6 (habit tracker, it's just me and Sam). short, people don't read these. stuff that's in: dark mode, home screen widgets (iOS only i think? Sam would know), fixed the bug where streaks reset if you logged right around midnight, a few crash fixes, Sam did something with sync, and we dropped iOS 15. iOS goes out tomorrow, android build is still stuck in review",
      "assertions": [
        "It delivers store text that could be pasted in as it stands, not only a list of questions",
        "Dark mode and the fix for streaks resetting around midnight both appear in the store text",
        "The end of iOS 15 support is mentioned in the iOS text or raised explicitly as something users on iOS 15 need to be told",
        "Widgets are not promised to Android users; the output flags that whether widgets are iOS-only needs confirming with Sam, or gives separate iOS and Android text with widgets only in the iOS one",
        "The sync change is not described as a specific improvement (such as faster or more reliable sync); it is left out or flagged as needing detail from Sam",
        "It does not invent crash counts, percentages, a release date for Android, or features that are not in the list",
        "The store text itself is short: about 100 words or fewer per store"
      ]
    },
    {
      "id": 6,
      "held_out": "2026-10-17",
      "prompt": "Our next Quillmoor release goes out tomorrow. Quillmoor is our invoicing app for freelancers and small studios, and some of our customers use our API. The git log since the last release is in `git-log.txt`. Write the customer-facing release notes for our public changelog page: Markdown, 200 words max. I want them ready to paste.",
      "assertions": [
        "The webhook payload change (`amount` now an integer in minor units, `client_name` renamed to `client.display_name`) is in the notes and comes before every other change, worded so a customer with a webhook integration can tell they must update their handler",
        "The Swiss QR-bill feature is not presented as part of the release, since it was reverted in the same range",
        "No internal-only work appears in the notes: no CI changes, dependency bumps, test fixtures, the auth/session refactor, feature-flag removal, release checklist, or typo fixes in comments or CONTRIBUTING",
        "The 'misc fixes from Henrike's QA pass' and 'sync stuff' commits are either left out or mentioned no more specifically than general small fixes; no particular behaviour, such as a sync feature, is attributed to them",
        "Does not invent anything absent from the log or request: no version number, release date, PR or ticket number, count of changes, or figure such as how much faster the invoice list is",
        "Covers at least six of these seven customer-visible changes: duplicating an invoice, receipt photos from a phone's browser, the client-portal ZIP download, no more reminders for invoices marked paid offline, the VAT rounding fix, the due date column in CSV export, and the faster invoice list",
        "The pasteable notes are 200 words or fewer and in Markdown, are delivered without first stopping to ask questions, and contain no working notes about what was left out or why"
      ]
    },
    {
      "id": 7,
      "held_out": "2026-10-17",
      "prompt": "I run the Bramblegate Tool Library, it's a volunteer thing where members borrow tools through our little app. My nephew does the app and he did a bunch of stuff this month. From memory:\n\n- you can reserve tools a week ahead now instead of 3 days? I think a week\n- the login code by text message, he fixed that, people were complaining it didn't arrive\n- something about late fees, he changed how they work, I need to check with him but basically you get a reminder before it's late now\n- everyone has to re-accept the borrowing agreement next time they log in because the committee updated it\n- new photos on the tool pages\n- also a lot of backend stuff I don't understand\n\nCan you write the text for the \"What's new\" pop-up members see when they open the app? It's a small box, so 75 words max.",
      "assertions": [
        "Re-accepting the updated borrowing agreement at next log-in comes first in the pop-up text and is stated as something the member will have to do",
        "Does not invent details: no late-fee amounts, no number of days before the due date that the reminder arrives, no reservation window other than a week, no cause for the login problem, no date, and no change that is not in the notes",
        "The late-fee item goes only as far as described, a reminder before an item is late, and does not claim fees were raised, lowered or removed",
        "The 'backend stuff' is not in the pop-up text, and nothing specific is claimed about it",
        "Mentions reserving further ahead, the fixed text-message login code, and the new tool photos",
        "The pop-up text is 75 words or fewer and in plain language for library members, with no developer jargon such as 'backend', 'OTP' or 'SMS verification flow'",
        "Delivers the pop-up text without first stopping to ask for confirmation; any suggestion to check details with the developer sits outside the pop-up text"
      ]
    }
  ]
}