{
  "skill_name": "sql-review",
  "evals": [
    {
      "id": 1,
      "prompt": "Our Snowflake query for weekly active users per plan looks too high compared with the product dashboard. Can you review it?\n\nSELECT p.plan_name, DATE_TRUNC('week', e.event_ts) AS wk, COUNT(e.user_id) AS wau\nFROM events e\nJOIN subscriptions s ON s.user_id = e.user_id\nJOIN plans p ON p.plan_id = s.plan_id\nWHERE e.event_ts BETWEEN '2026-08-31' AND '2026-09-27'\n  AND e.user_id NOT IN (SELECT user_id FROM internal_users)\nGROUP BY 1, 2;\n\nsubscriptions keeps one row per plan change, so a user who upgraded has two rows. event_ts is TIMESTAMP_NTZ written in UTC. Acme reports weeks in UTC.",
      "assertions": [
        "Identifies that COUNT(e.user_id) counts events, not distinct users, so WAU is really weekly events",
        "Identifies that the subscriptions join fans out for users with plan changes and attributes their events to every plan they ever had",
        "Flags NOT IN against internal_users as returning no rows if that subquery contains a NULL, and proposes NOT EXISTS",
        "Flags BETWEEN on event_ts as dropping 27 September after midnight and proposes a half-open window",
        "Does not claim integer division is a problem, because there is no division and Snowflake's / returns a decimal anyway",
        "Provides a full corrected query in Snowflake syntax that picks one plan per user per week (for example the plan active at the event time)",
        "Provides verification queries, such as a uniqueness check on the subscription join and a reconciliation against a distinct-user count",
        "Does not invent row counts or magnitudes not given in the prompt"
      ]
    },
    {
      "id": 2,
      "held_out": "2026-10-18",
      "prompt": "Can you review monthly_donor_revenue.sql before it goes back on the trustees' dashboard? The schema is in schema.md and my notes are in context.md.",
      "assertions": [
        "Identifies the join to donor_addresses as repeating each donation once per address for donors with several addresses, the likely cause of the overstatement, and says SELECT DISTINCT does not fix it",
        "Does not accept the refund theory: refunds are negative rows in donations, so the query already includes them",
        "Flags that gift_aid_status <> 'rejected' also drops rows where the status is NULL, which the schema says is most donors, so it understates revenue as well",
        "Flags that months are cut in UTC while finance reports by UK local time, and that integer division truncates the pounds and the average",
        "Gives corrected PostgreSQL as a single statement (CTEs allowed) that uses one address per donor, such as the primary one, or aggregates before joining",
        "Does not state a corrected September total, since nothing can be run, and does not invent tables or columns that are not in the schema",
        "Keeps the review to under a page, plus the corrected query"
      ]
    },
    {
      "id": 3,
      "held_out": "2026-10-18",
      "prompt": "hi, not a techie, I'm a teacher. the head wants the number of pupils who missed more than 10 days this term. our attendance export is an SQLite file with one row per pupil per session, morning and afternoon. a colleague wrote this:\n\nSELECT COUNT(*) FROM attendance WHERE status = 'absent' GROUP BY pupil_id HAVING COUNT(*) > 10;\n\nis it right? just tell me yes or no and give me a fixed one, short please",
      "assertions": [
        "Says no, the query is not right",
        "Explains that it returns one row per pupil with a count, not a single number of pupils",
        "Points out that with two sessions a day, more than 10 days means more than 20 absent sessions",
        "Gives one corrected SQLite query that returns a single number",
        "Notes the query does not limit to this term, without inventing a date column name as if it were known (any name used is marked as a guess to check)",
        "Keeps the reply short, as asked"
      ]
    }
  ]
}