{
  "skill_name": "terraform-module",
  "evals": [
    {
      "id": 1,
      "prompt": "Write a Terraform module for Acme's Postgres database on AWS RDS. The payments team and the orders team will both call it. They need to pick instance class and storage size; everything else should be safe by default.",
      "assertions": [
        "Produces versions.tf, variables.tf, main.tf, outputs.tf, a basic usage example and a README with inputs and outputs tables",
        "versions.tf sets required_version and a provider version constraint with an upper bound on the major version",
        "Every variable has a type and a description, and inputs such as instance class or storage size have validation blocks with fixable error messages",
        "Defaults set storage_encrypted = true, publicly_accessible = false, deletion_protection = true and skip_final_snapshot = false",
        "The master password is not a literal or a variable default; it is a sensitive variable, a managed password, or a secret-manager reference",
        "A tags variable is merged with module defaults and applied to taggable resources",
        "The module contains no provider block, no backend block, and no hard-coded account ID, region or ARN",
        "Does not claim terraform validate or plan was run unless it was actually run"
      ]
    },
    {
      "id": 2,
      "prompt": "Turn this into a reusable module for Northwind's GCP projects: a GCS bucket for nightly exports, kept 90 days, readable by the analytics service account only.",
      "assertions": [
        "Uses labels with lowercase keys rather than AWS-style tags",
        "Sets public_access_prevention to enforced and uniform_bucket_level_access to true by default",
        "Grants access with a member-level IAM binding scoped to the named service account variable, not a project-wide or primitive role",
        "Adds a lifecycle rule deleting objects after a retention variable defaulting to 90",
        "Takes the project ID as a variable or data source rather than a literal",
        "Ends with the terraform fmt, validate and plan commands for the user to run"
      ]
    },
    {
      "id": 3,
      "held_out": "2026-10-18",
      "prompt": "Write the Terraform module for our archive bucket. The requirements are in requirements.md. Put it in modules/archive-bucket/.",
      "assertions": [
        "Writes the module files under modules/archive-bucket/",
        "The bucket is not publicly readable: public access is blocked and the website reads through the caller-supplied CloudFront distribution, and the reply explains how this meets both the website need and SEC-4",
        "Flags that deleting noncurrent versions after 30 days conflicts with keeping replaced master versions for 7 years, and does not silently apply the 30-day deletion to masters",
        "Flags that moving the JPEG derivatives the website shows to Glacier Deep Archive would stop them being served, rather than applying the 90-day transition to everything without comment",
        "The version constraints accept AWS provider 4.x and Terraform 1.5 (nothing requires provider 5.x)",
        "Encrypts with a customer-managed KMS key and applies the caller-supplied tags",
        "Does not add requirements that are not in requirements.md (for example cross-region replication or access logging) unless marked as optional or as assumptions"
      ]
    },
    {
      "id": 4,
      "held_out": "2026-10-18",
      "prompt": "hi, I'm the admin for a psychology research group. the university gave us an AWS account and I need somewhere to keep our survey data files (they have participants' answers in them). the IT person said 'write it as a terraform module'. I don't really know terraform. just the main file please and keep it short",
      "assertions": [
        "Delivers a main.tf without stopping to ask questions first",
        "The storage is private and encrypted by default, with public access blocked",
        "Respects 'just the main file': gives one short file, or briefly explains if anything else is strictly needed",
        "Notes that participants' answers are sensitive and suggests checking the university's data or ethics rules (for example on region or retention) rather than inventing them",
        "Does not invent account IDs, retention periods, regions or other requirements as if they had been given",
        "Includes a short plain-language explanation of how to use the file"
      ]
    }
  ]
}