{
  "skill_name": "release-notes",
  "evals": [
    {
      "id": 1,
      "prompt": "Write release notes for v2.4.0 from this log (range v2.3.0..HEAD, 2026-08-12 to 2026-09-28):\n\na1b2c3d|feat: bulk CSV import for contacts (#412)\ne4f5a6b|fix: NPE in ReportExporter.flush() when >10k rows (#418)\nc7d8e9f|feat!: remove legacy /v1/contacts endpoint (#420)\n0a1b2c3|chore(deps): bump lodash 4.17.20 -> 4.17.21\n4d5e6f7|refactor: extract ExportService\n8a9b0c1|fix stuff\n2d3e4f5|perf: cache tenant settings, report export ~3x faster (#425)",
      "assertions": [
        "States the range, commit count (7), and date span at the top",
        "The removal of /v1/contacts is hoisted to the top of both artifacts with an 'Action required' line and names the !: marker that flagged it",
        "Customer notes contain no commit-speak — no ReportExporter.flush(), class names, ticket numbers, or author names",
        "The lodash bump and the ExportService refactor appear only in the internal changelog, never in customer notes, not even as 'various improvements'",
        "Every internal changelog line ends with its short hash, plus PR number where present",
        "The vague 'fix stuff' commit (8a9b0c1) goes in the Needs review list with its hash, not guessed into customer notes",
        "No marketing adjectives; the performance line states the measured change (~3x faster export) rather than 'blazing fast'",
        "Does not invent features or fixes not present in the log"
      ]
    },
    {
      "id": 2,
      "held_out": "2026-10-05",
      "prompt": "We're cutting 3.2.0 of Tidewater Booking tomorrow. The log since 3.1.0 is in commits.txt. I need two things: the \"what's new\" text for the customer email (goes to clinic owners and to the devs who integrate with our API and webhooks), and the entry for CHANGELOG.md in the repo. Marketing wants the headline to be SMS reminders and the Google Calendar sync, and wants to say exports are 5x faster. We're a hosted service, nobody runs our code themselves.",
      "assertions": [
        "Google Calendar sync is not announced as shipping in 3.2.0; the output says it was reverted (commit a4e8b60 / #225) and is not in this release",
        "The export speed-up is given as 41s to 14s for 50k bookings, or roughly 3x, and the requested '5x faster' claim is not used or is called out as unsupported by the log",
        "The webhook field rename (start/end to starts_at/ends_at in booking payloads) is called out as a breaking change that integrators must act on, even though its commit is labelled refactor",
        "The removal of X-Tidewater-Key header auth is called out as breaking with the instruction to switch to Bearer tokens, and breaking changes appear before the new features",
        "The 'wip' (3b7c5e2) and 'more fixes' (71f9d3b) commits are flagged as unclear and needing someone to check, not described as specific features or fixes",
        "The duplicate-calendar-events fix (#218) is not listed as a fix customers receive, because it was reverted along with the sync",
        "The customer email text does not mention the twilio or vite bumps, the BookingRepo rename, CI changes or the Node 20 server requirement",
        "Each line of the CHANGELOG entry carries a commit hash or PR number from the log"
      ]
    },
    {
      "id": 3,
      "held_out": "2026-10-05",
      "prompt": "need release notes for pressroom (our static site CLI, the people using it are our clients' developers), going out as 0.9.0 today. this is what's gone in since 0.8.2, half from git log --oneline and half from memory, sorry:\n\nd41c0aa add --watch flag\n77e2b19 fix crash on empty frontmatter\nbump deps\n3a9f6c1 config: `output_dir` now `outDir` (old key ignored)\nc08e512 misc cleanup\n9b1d7e4 faster builds\n\nalso Priya fixed something with image paths on windows, I don't have the commit",
      "assertions": [
        "The output_dir to outDir config change is flagged as breaking, with the action to rename the key in existing config files because the old key is now ignored",
        "'Faster builds' is reported without any invented figure or percentage, and the output says the size of the speed-up is not known",
        "The Windows image-path fix is included only as far as described and marked as unconfirmed or missing its commit; no specific bug behaviour is invented for it",
        "'misc cleanup' (c08e512) and 'bump deps' are not turned into user-facing claims",
        "The --watch flag and the empty-frontmatter crash fix both appear as user-facing items",
        "It does not invent dates, PR numbers, a commit count for commits it was not shown, or features not in the pasted list",
        "It delivers usable release notes in the reply rather than only asking questions, and lists the items that need confirming before sending"
      ]
    },
    {
      "id": 4,
      "held_out": "2026-10-06",
      "prompt": "Portside WMS 5.0.0 goes out to customers on Thursday. The log since 4.9.2 is in git-log.txt. Two things please: (1) upgrade notes for admins. Our customers run Portside on their own servers, so their IT people need to know what to do before they upgrade. (2) A short paragraph for the customer newsletter, three or four sentences. Sales want the newsletter to lead with batch scanning and to say the duplicate pick list problem is finally fixed. Also remind people that 4.x support ends on 31 December.",
      "assertions": [
        "The PostgreSQL 15+ requirement is called out as something admins must deal with before upgrading (the installer aborts on PostgreSQL 13 and 14), even though its commit is labelled chore",
        "The removal of /api/v1/stock is called out as a breaking change with /api/v2/stock-levels as the replacement, and the breaking items come before the new features in the admin notes",
        "Batch scanning is described as off by default (flag scan.batch.enabled, missed reads on older handhelds), not as a feature that works out of the box after upgrading",
        "The duplicate pick list problem is not described as fully fixed; the output says split shipments can still produce a duplicate line (#1893) or flags that the 'finally fixed' claim is not supported by the log",
        "The end of 4.x maintenance is given as 31 March 2027, or the conflict between the requested 31 December date and the log is flagged; 31 December is not repeated as fact",
        "The wms.mail.host to wms.smtp.host rename is described as a deprecation where the old key still works, not as a change that breaks existing configuration on upgrade",
        "The admin notes include the security fix for session tokens not being invalidated on password change (LSA-2026-04)",
        "The newsletter paragraph does not mention the spring-boot bump, the StockLedger refactor or the CI test shards"
      ]
    },
    {
      "id": 5,
      "held_out": "2026-10-06",
      "prompt": "need the what's new text for the app stores for Tallybird 1.6 (habit tracker, it's just me and Sam). short, people don't read these. stuff that's in: dark mode, home screen widgets (iOS only i think? Sam would know), fixed the bug where streaks reset if you logged right around midnight, a few crash fixes, Sam did something with sync, and we dropped iOS 15. iOS goes out tomorrow, android build is still stuck in review",
      "assertions": [
        "It delivers store text that could be pasted in as it stands, not only a list of questions",
        "Dark mode and the fix for streaks resetting around midnight both appear in the store text",
        "The end of iOS 15 support is mentioned in the iOS text or raised explicitly as something users on iOS 15 need to be told",
        "Widgets are not promised to Android users; the output flags that whether widgets are iOS-only needs confirming with Sam, or gives separate iOS and Android text with widgets only in the iOS one",
        "The sync change is not described as a specific improvement (such as faster or more reliable sync); it is left out or flagged as needing detail from Sam",
        "It does not invent crash counts, percentages, a release date for Android, or features that are not in the list",
        "The store text itself is short: about 100 words or fewer per store"
      ]
    }
  ]
}