{
  "skill_name": "agents-md-generator",
  "evals": [
    {
      "id": 1,
      "prompt": "Write an AGENTS.md for this repo. It's a Python service: pyproject.toml declares pytest and ruff in dev dependencies, there's a Makefile with `run` and `migrate` targets, and .github/workflows/ci.yml runs `ruff check .` then `pytest -q`. No test script anywhere else. Migrations are in alembic/versions/.",
      "assertions": [
        "Commands are traced to their source — CI for ruff/pytest, Makefile for run/migrate — not inferred from the dependency list",
        "The test command is `pytest -q` as CI runs it, not a guessed `pytest` or `make test` that does not exist",
        "No command appears that cannot be traced to pyproject.toml, the Makefile, or the workflow",
        "Alembic migrations produce a convention or forbidden-action line (append-only / never edit a committed migration)",
        "A `Do not` section exists with concrete entries, not implied caution",
        "Output is at or under ~150 lines",
        "No virtue instructions appear — no 'clean', 'best practice', 'thorough', 'maintainable'",
        "Conventions included would each change agent behaviour if removed"
      ]
    },
    {
      "id": 2,
      "prompt": "Here's our AGENTS.md — audit it. It says: 'Run `npm test` to test. Write clean, maintainable code and follow best practices. Be thorough in reviews.' Our package.json has scripts for build and lint only, no test.",
      "assertions": [
        "Audit mode is used — a verdict and diff, not a silent rewrite",
        "`npm test` is flagged as unverifiable: package.json has no test script",
        "The three virtue instructions are named for removal, with the reason that they change no behaviour",
        "What the file is missing is stated — real commands (build, lint), file placement, forbidden actions",
        "The line-count position is reported rather than the file being failed for length",
        "A concrete replacement or diff is offered, not just criticism"
      ]
    },
    {
      "id": 3,
      "prompt": "Also wire it up for Claude Code and Copilot.",
      "assertions": [
        "Satellites are emitted as deltas only — each points at AGENTS.md for shared content",
        "No satellite repeats the command table, file placement, or conventions",
        "CLAUDE.md carries only Claude-Code-specific content; .github/copilot-instructions.md only Copilot-specific content",
        "If there is no genuinely tool-specific content, that is said rather than a file padded to justify itself"
      ]
    },
    {
      "id": 4,
      "prompt": "[Non-interactive run — no repo access] Generate an AGENTS.md for a typical Node microservice.",
      "assertions": [
        "No AGENTS.md is produced — the output is BLOCKED: need repo access",
        "No commands are invented from ecosystem convention",
        "The reason given is that every command must trace to a file in the tree"
      ]
    },
    {
      "id": 5,
      "held_out": "2026-10-18",
      "prompt": "We're starting to use coding agents on our sensor-ingest repo, which is the current working directory (README.md, CONTRIBUTING.md, pyproject.toml, .github/workflows/ci.yml, scripts/deploy.sh, data/raw/README.md, src/, tests/). Write an AGENTS.md at the repo root. The agents keep running `make test` and it fails, which is part of why we want this. Keep it under 60 lines and don't change any other file.",
      "assertions": [
        "Writes AGENTS.md at the repo root, under 60 lines, and modifies no other file",
        "Does not tell agents to run `make test` (there is no Makefile); the test command given is `uv run pytest -m \"not hardware\"` as in CI and CONTRIBUTING.md",
        "Gives ruff (`uv run ruff format` / `uv run ruff check`) as the formatter and linter, not black, resolving the README's stale instruction in favour of CI and CONTRIBUTING.md",
        "Tells agents not to run the tests marked hardware, because they need the live gateway on the plant network",
        "Forbids running scripts/deploy.sh and forbids editing, renaming or deleting anything in data/raw/",
        "Does not invent commands, tools or checks that appear nowhere in the repo (for example mypy, pre-commit, docker, or a Makefile target)",
        "Mentions that README.md is out of date (make test, black) as something for the humans to fix, without editing it"
      ]
    },
    {
      "id": 6,
      "held_out": "2026-10-18",
      "prompt": "my colleague says I should add an 'agents' file so the AI helper stops messing up my marking folder (it's the folder you're in). it keeps overwriting grades_final.csv and I lose my hand corrections. I'm a teacher, not a programmer. keep the file short, like 10 lines",
      "assertions": [
        "Writes an AGENTS.md in the folder of roughly 10 lines (no more than about 15)",
        "Notices that make_report.R itself writes grades_final.csv, so the file tells the agent not to run make_report.R (or to ask first), not only 'don't edit grades_final.csv'",
        "Tells the agent not to change raw_marks.csv or grades_final.csv",
        "Gives the run order from notes.txt (Rscript clean_marks.R, then Rscript make_report.R) without adding commands or packages the folder does not mention",
        "Explains the change to the teacher in plain language, in a few lines",
        "Delivers the file without stopping to ask questions first"
      ]
    }
  ]
}