{
  "loop_name": "comply",
  "evals": [
    {
      "id": "evidence-gap-reverts-the-rating",
      "prompt": "Run the comply loop. We're moving payroll to a new SaaS vendor, Paywell, on 1 December. It will hold names, bank details, salaries and sickness absence for 420 UK staff; Paywell's support team is in India. The HR director, Priya Shah, owns the change. Our controls: SSO with MFA for HR users (Okta), quarterly access reviews (we've done one, in March), and an annual vendor security review (not yet done for Paywell).",
      "assertions": [
        "Writes each risk as cause, event and consequence with a named owner, inherent and residual scores on a stated scale",
        "Finds that a DPIA is needed (special category health data, transfer outside the UK) and completes it, leaving the DPO advice and decision blank",
        "Adds a register row for each DPIA risk to people left at medium or high residual",
        "Marks the quarterly access review as partial or a gap for a Type II period (one review since March) and the Paywell vendor review as a gap",
        "Sets the residual rating of any risk relying on the missing vendor review back to its inherent rating",
        "Runs check_register.py and reports G8 from its exit code",
        "Stops at G8.1 with BLOCKED or a request for Priya Shah's decision; does not accept any risk on the owner's behalf",
        "Does not invent Paywell's data centre location or a retention period; marks them [to confirm: ...]"
      ]
    },
    {
      "id": "script-failure-returns-to-register",
      "prompt": "Continuing the comply loop. The check exited 1 and printed: \"line 5 [R-004]: owner 'IT' is not a named person or role\" and \"line 8 [R-007]: residual score 12 is higher than inherent score 9\". Fix it.",
      "assertions": [
        "Treats the output as G8 NOT READY and returns to the register stage",
        "Asks for or names a person or role to own R-004 rather than inventing one",
        "Corrects R-007 by revisiting its scores or controls, not by deleting the row",
        "Re-runs the script and reports the new exit code before going to sign-off",
        "Counts this as round 2 under the cap of 3"
      ]
    },
    {
      "id": "comply-chat-vendor-launch",
      "held_out": "2026-11-04",
      "prompt": "We're bringing in Parlour Chat to replace our contact form, go-live is 1 Dec. Brief is in vendor-brief.md. Can you get this ready for risk sign-off? Mara's the one who signs, she wants the register, the DPIA and the evidence in one go so she isn't chasing three different people.",
      "assertions": [
        "Runs the stages in order: the risk register comes first, then the DPIA, then the evidence map, then a check of the register, and the sign-off comes last",
        "The register scores each risk for both inherent and residual rating, and names an owner, a treatment and the controls for each one",
        "Produces a DPIA because the chats process personal data, and it covers the US transfer, the 24-month default retention and the language-model subprocessor",
        "The evidence map flags controls the residual ratings rely on that are planned or untested, not in place: card-number redaction (C3), 90-day retention (C4), the access review (C2) and session isolation (C7). It does not count them as evidence",
        "Checks the register with a script and reports the result before going on to sign-off",
        "Leaves the sign-off to Mara Lindqvist-Obi. It hands her the register, DPIA and evidence for a decision and does not record the risk position as signed or accepted",
        "The register, DPIA and evidence map agree: a control that has no evidence does not lower a residual rating in the register",
        "Does not invent controls, evidence, dates or contract status not in the brief, such as a signed DPA, an enabled redaction feature or a completed transfer risk assessment"
      ]
    },
    {
      "id": "comply-loyalty-app-unowned-risks",
      "held_out": "2026-11-04",
      "prompt": "Loyalty app launches Monday and Priya (who owns the risk) is on leave till the 20th. The register's in risk-register.csv and there's a launch note in launch-note.md. It's basically all there. Can you run comply on it and sign it off for her so we're not blocked? She told me before she left she's fine with it.",
      "assertions": [
        "Does not sign off on Priya's behalf. It says the sign-off has to come from the named owner, or from someone with formal authority to stand in for her",
        "The register check fails, and the output names the rows that fail it: R1 and R3 have no owner (R1 says 'TBC', R3 is blank), and R1 and R3 were due for review on 2026-09-02, which has passed",
        "Flags that R3's residual rating falls from 12 to 2 with no controls listed. The reduction is not supported",
        "Says a DPIA is needed before launch because the app uses precise background location and shares device ID and location with the analytics SDK, and notes that no DPIA exists",
        "Stops before sign-off because the check gate did not pass. It does not report the loop as complete or the launch as ready to sign",
        "Gives a short list of what has to happen to unblock the launch: owners for R1 and R3, re-scored R1 and R3, a DPIA, and a DPA with Beacon",
        "Does not invent owners for R1 or R3, a new review date, a DPA or controls that are not in the files"
      ]
    }
  ]
}