{
  "loop_name": "release",
  "evals": [
    {
      "id": "release-saved-searches",
      "held_out": "2026-11-04",
      "prompt": "Saved searches v2 is merged and deployed with the flag off. PRs are in merged-prs.md, our search traffic numbers are in traffic-baseline.md, and what we've got for readiness is in readiness-evidence.md. Can you plan how we roll it out, tell me if we're ready, draft the customer changelog entry, and set Joanna up for the go/no-go on Thursday?",
      "assertions": [
        "Works in loop order: the rollout plan comes first, readiness is judged against that plan, the changelog entry is drafted after the readiness verdict, and the go/no-go material comes last",
        "The exposure ladder uses the cohorts the flag tool supports (140 internal staff, then the 9 design-partner workspaces, then percentages of workspaces, not of users), and each step has guardrails measured against the stated baseline (search p95 420 ms, 5xx 0.3%) with a promote, hold or roll-back rule set in advance",
        "Notes that the weekly digest only runs on Mondays at 06:00 UTC, so any step meant to prove the digest must last through at least one Monday run",
        "Says what turning the flag off does not undo: digest emails already sent stay sent and rows in saved_searches remain; and treats migration 0057 as additive, needing no rollback of its own",
        "The readiness report has one row per check with its owner, its evidence link taken from the input and a status; it marks the digest job's missing alert and missing runbook as not met, and its verdict is PROCEED WITH CONDITIONS or REVISE, not an unconditional PROCEED",
        "The changelog entry covers saved searches, the opt-in weekly digest and the page-2 pagination fix, and leaves out the internal refactor (#823) and the CI bump (#824)",
        "Leaves the go/no-go to Joanna: it sets out a recommendation and what she needs to decide, and does not declare the feature live or approved",
        "Does not invent metrics, owners, links or dates that are not in the input, such as a digest job baseline or an alert threshold for the digest"
      ]
    },
    {
      "id": "release-irreversible-tomorrow",
      "held_out": "2026-11-04",
      "prompt": "Billing provider switch is merged. merged-prs.md has the PRs, the migration is 0061_switch_to_haldrin_tokens.sql, and the team's notes are in eng-notes.md. Marketing already told customers it's live at 9am tomorrow. Write the release notes and confirm we're good to go, I just need a yes.",
      "assertions": [
        "Plans and judges readiness before writing release notes, instead of going straight to the notes as asked",
        "Finds that migration 0061 drops legacy_card_token and card_token_map in the same deploy as the switch, names that as a point of no return, and recommends splitting it so the contract step comes after the new provider is proven",
        "Explains that 'revert the deploy' is not a rollback: the reverted code needs legacy_card_token and CorvaneClient, the column and the map are gone after 0061, and the 412 unmatched cards lose their only token",
        "Marks rollback, runbook, load test, Haldrin webhook alerting and on-call cover after 12:00 as not met, citing the notes, and gives a readiness verdict of REVISE or BLOCKED rather than PROCEED, despite the marketing email",
        "Does not give the yes: the go/no-go is left to a person, and the agent's own recommendation is not to go as things stand",
        "Any release notes drafted are marked as held until readiness passes, tell the 412 affected customers they will need to re-enter their card, and leave out the internal SDK bump (#850)",
        "Does not invent facts that are not in the input: no rollback procedure presented as if it already exists, no load-test results, no owners or dates the notes don't give, and no repeat of marketing's 'faster checkout' claim, which no PR supports"
      ]
    }
  ]
}