{
  "skill_name": "sonar-onboard",
  "evals": [
    {
      "id": 1,
      "prompt": "Set this Python repo up for Sonar compliance — we use SonarCloud and run CI on GitHub Actions.",
      "assertions": [
        "sonar.projectKey, sonar.sources, sonar.tests and the coverage path reflect what is actually on disk, not generic example values",
        "The generated CI workflow blocks merge on a quality-gate failure rather than merely running a scan",
        "SonarCloud vs SonarQube server is chosen explicitly and the generated config matches that choice",
        "The coverage report path matches the language's real tooling output",
        "Setup steps that require a token or server-side action are named for the operator rather than silently assumed"
      ]
    },
    {
      "id": 2,
      "held_out": "2026-10-18",
      "prompt": "Set this repo up for Sonar. We run our own SonarQube server (host URL https://sonar.harrow-transit.test), not SonarCloud. The repo is in this folder. Add the scan to our pipeline and make merges fail if the quality gate fails, without breaking the jobs we already have. The token is kept as a masked CI variable called SONAR_TOKEN; don't put it anywhere in the repo. Also, the gate should require 90% coverage on new code. Add a short README section about the gate, under 15 lines.",
      "assertions": [
        "Writes a sonar-project.properties aimed at the self-hosted SonarQube server from the prompt, with no SonarCloud organisation setting",
        "Analysis covers both the Python package under src/wayline (tests in tests/) and the TypeScript frontend under web/src, with the *.test.ts files treated as tests rather than sources, and excludes the generated src/wayline/_generated and the vendored src/wayline/vendor code",
        "The Python coverage report path is build/coverage.xml, which the GitLab test job writes through its --cov-report flag (not the reports/coverage.xml default in pyproject.toml), and the TypeScript path is web/coverage/lcov.info",
        "The scan is added to .gitlab-ci.yml, the active CI, rather than the deprecated GitHub Actions workflow; the existing jobs are kept and the pipeline fails when the quality gate fails",
        "Says that the 90%-coverage-on-new-code condition is set on the quality gate in SonarQube itself (by someone with admin rights there), not in repository files, rather than implying a properties setting enforces it",
        "No token value, server address or organisation appears that is not in the input; the token is read only from the SONAR_TOKEN CI variable",
        "The README section is under 15 lines"
      ]
    },
    {
      "id": 3,
      "held_out": "2026-10-18",
      "prompt": "I look after the website for our community choir. A volunteer built it in plain HTML, CSS and a bit of JavaScript, there are no tests, and it lives in a public GitHub repo. The board now wants it 'Sonar checked', and someone said SonarCloud is free for public projects. Can you write whatever files I need to add to the repo (save them here) and tell me what I have to click? Keep the explanation short, a few lines. I'm not a developer.",
      "assertions": [
        "Writes the configuration files (a Sonar properties file and a GitHub Actions workflow, or equivalent) into the working folder rather than only describing them or asking questions",
        "Targets SonarCloud, as the user indicated, not a self-hosted server",
        "Values it cannot know, such as the SonarCloud organisation and project key, are clearly marked as placeholders the user must fill in",
        "Does not configure a test or coverage report path, since the site has no tests, and does not invent other project details",
        "Tells the user the steps they must do themselves, such as importing the project on SonarCloud and adding the token as a GitHub repository secret, and puts no token in any file",
        "The explanation is a few short lines in plain language, not a long tutorial"
      ]
    }
  ]
}