{
  "skill_name": "codebase-tour",
  "evals": [
    {
      "id": 1,
      "prompt": "I'm new to this repo (it's the current directory) and my first ticket is to add a `carrier` filter to GET /shipments, so dispatchers can see one carrier's shipments at a time. Get me oriented before I start: how to run it, where the change goes, and anything that will bite me.",
      "assertions": [
        "Opens with a short list of the few things that matter most for this change, before any section, and does not open with a directory listing",
        "Every command given is cited to package.json or .github/workflows/ci.yml and marked as either run or only read; `npm run dev` is not given as a way to run the server",
        "Says where the README and the code disagree: there is no `dev` script (package.json has `start:watch`), there is no `src/routes/` (routes are registered in src/http/routes.ts), and config comes from env vars in src/config.ts, not config/default.json",
        "Says the change goes in `parseListQuery` (src/http/handlers/shipments.ts) and `buildShipmentFilter` (src/domain/shipments/filter.ts), and that openapi.yaml needs the new query parameter",
        "Names src/generated/api-types.ts as generated from openapi.yaml by `npm run gen:types`, says not to edit it by hand, and notes that CI fails if it drifts (.github/workflows/ci.yml)",
        "Names src/jobs/export-shipments.ts as a second copy of the shipment filtering that the change does not reach on its own, notes it has no tests, and raises whether it needs the carrier filter too",
        "Names tests/http/shipments.test.ts and tests/domain/filter.test.ts as the tests for the area, with a way to run just those through the repo's vitest script, and notes DATABASE_URL is required to start the server (src/config.ts)",
        "Does not invent commands, files, scripts or tools that are not in the repo"
      ]
    },
    {
      "id": 2,
      "prompt": "Never touched this codebase before. I need to add support for Northwind Credit Union's CSV export (header: Transaction Date, Merchant, Amount; dates like 2026-03-02). Where do I start, and how do I run the tests? The repo is the folder you're in.",
      "assertions": [
        "Says the README is wrong that a new parser module is picked up automatically: the PARSERS registry in src/ledgerline/parsers/__init__.py has to list it",
        "Names HEADER_SIGNATURES in src/ledgerline/detect.py as a second place the new bank must be added, or detection fails for the new header",
        "Gives the test command from noxfile.py or CI (`nox -s tests`, `nox -s lint tests`), says `python setup.py test` from the README cannot work because there is no setup.py, and notes the tests session skips tests marked `slow`",
        "Gives a way to run a single test file through the runner the repo already uses (nox posargs or pytest), labelled as derived from the repo's config rather than presented as a defined command",
        "Points to AcmeParser as the parser to copy and flags that HarborParser has no tests and does not use parse_amount or raise ParseError like the others",
        "Identifies src/ledgerline/_version.py as generated by setuptools-scm and not to be edited",
        "Each convention it states (ParseError naming the column, money.parse_amount for amounts, CSV samples in tests/fixtures/) is cited to a file in the repo",
        "Does not invent commands, files, scripts or tools that are not in the repo"
      ]
    },
    {
      "id": 3,
      "prompt": "I've got a ticket to switch webhook delivery retries in this repo to exponential backoff. I don't really know Go and I've never seen this code. Write me an orientation to docs/backoff-tour.md and don't change anything else.",
      "assertions": [
        "Writes docs/backoff-tour.md and modifies no other file in the repo",
        "Says the README is wrong about config.yaml: there is no such file, and retry settings are the env vars TIDEPOOL_MAX_ATTEMPTS and TIDEPOOL_RETRY_DELAY read in internal/config/config.go",
        "Says the README is wrong that `make test` includes the database tests: they sit behind the `integration` build tag and run with `make test-integration`, which needs Postgres and TIDEPOOL_DATABASE_URL",
        "Names `Deliver` in internal/dispatch/dispatcher.go as where the change goes, and the injectable `sleep` field used by internal/dispatch/dispatcher_test.go as the way to test the delays",
        "Names internal/replay/replay.go as a second retry loop with its own fixed delay that the change must also update or deliberately leave, and notes it has no tests",
        "Names internal/store/query.sql.go as generated by sqlc from query.sql, regenerated with `make generate`, and not to be edited by hand",
        "Every command is cited to the Makefile or .github/workflows/ci.yml and marked as run or only read; it does not claim to have run the integration tests",
        "Does not invent commands, files, scripts or tools that are not in the repo"
      ]
    },
    {
      "id": 4,
      "held_out": "2026-10-20",
      "prompt": "Hi! I joined the checkout team on Monday and I've been handed ticket SHIP-214: \"Raise the EU free-shipping threshold from EUR 50 to EUR 60. UK and US stay as they are.\" I'm new to this repo and haven't used FastAPI much. Before I start, can you get me oriented: how do I install, run and test this thing, and where exactly would I make the change?",
      "assertions": [
        "Gives install/run/test commands taken from the Makefile and .github/workflows/ci.yml (`make install` / `pip install -e \".[dev]\"`, `make run` / `uvicorn shipcalc.main:app`, `make test` / pytest) and explicitly says the README's `pip install -r requirements.txt`, `flask run` and `python -m unittest discover` are stale (there is no requirements.txt and the app is FastAPI, not Flask).",
        "Says the EU threshold's source of truth is data/rates.csv (line 2, `free_threshold` column), that src/shipcalc/generated/rates_table.py is generated (DO NOT EDIT header on line 1) and must not be hand-edited, and that the change is: edit the CSV, then run `make gen`; it notes that CI's `make check-generated` step fails if the generated file is out of sync with the CSV.",
        "Finds the second, hard-coded copy of the threshold in src/shipcalc/api/preview.py line 5 (`FREE_SHIPPING_MIN`, used by the /preview/banner route) and says it must also change to 60, or the cart banner and the actual quote will disagree.",
        "Traces the quote path with file:line citations: the /quote handler in src/shipcalc/api/routes.py (shipping_cost call at line 24) -> shipping_cost in src/shipcalc/shipping.py (line 13) -> the threshold comparison against RATES[region][\"free_threshold\"] at shipping.py line 15.",
        "Flags the feature flag SHIPCALC_FF_FREE_SHIPPING read in src/shipcalc/shipping.py line 10 (free shipping applies only when it is exactly \"on\", and it is not in .env.example or the Makefile run target), so a local manual check of the new threshold will show no change unless the flag is set; also notes SHIPCALC_API_KEY is required at startup (src/shipcalc/config.py line 12).",
        "Points out that shipping.py and preview.py have no tests while discounts and tax do (tests/unit/test_discounts.py, tests/unit/test_tax.py), and recommends adding boundary tests (e.g. 59.99 vs 60.00 for EU, plus UK/US unchanged) that follow the existing style: pytest functions, Decimal money values, and the flag set via monkeypatch or the environment.",
        "Ends with one concrete first step whose result can be checked (e.g. run `make test` and see it pass before changing anything, or edit the CSV, run `make gen` and confirm `git diff` touches only the EU line of rates_table.py), and keeps the tour focused on this ticket instead of walking through tax and discounts in detail.",
        "Invents no commands, files, functions or settings that are not in the repo (e.g. no requirements.txt, no `python -m shipcalc` or `flask run` as a working command, no existing test_shipping.py, no config value for the threshold outside rates.csv/preview.py)."
      ]
    },
    {
      "id": 5,
      "held_out": "2026-10-20",
      "prompt": "First week here, and my ticket is CAB-112: \"Let guests book stays of up to 21 nights (currently capped at 14).\" I'm new to this codebase. Can you walk me through the code for the booking flow, tell me where the change goes, and how I run and test it? I really don't want to break anything on my first PR.",
      "assertions": [
        "Gives commands from the root package.json and .github/workflows/ci.yml: `npm install`, `npm test` (which runs `node --test` in each workspace), `npm run gen` / `npm run gen:check`, and `npm run dev:api` (which reads ../../.env, so .env must be copied from .env.example first). It says the README's `yarn install`, `cd api && yarn start`, `yarn test` and the `api/config.js` reference are stale: the repo uses npm workspaces under packages/ and there is no api/ directory or config.js.",
        "Identifies packages/api/src/generated/schema.js as generated (DO NOT EDIT on line 1, written by scripts/gen-schema.mjs) and says to change the nights `maximum` in packages/api/openapi.json (line 24) and run `npm run gen`, noting that CI's `npm run gen:check` fails if schema.js is hand-edited or stale.",
        "Lists every copy of the 14-night limit that must move to 21: packages/api/openapi.json line 24 (via the regenerated schema.js), packages/api/src/services/bookings.js line 6 (`MAX_NIGHTS`), and packages/web/src/bookingForm.js line 2 (`MAX_NIGHTS`, the client-side check). It warns that changing only one of them gives a confusing 400 or 422, or a form that blocks valid stays. The README line also needs updating.",
        "Traces the POST /bookings request path with file:line citations: packages/api/src/server.js mounts the router -> packages/api/src/routes/bookings.js line 7 runs `validate('BookingRequest')` -> packages/api/src/middleware/validate.js line 15 enforces `maximum` from the generated schema (400) -> createBooking / maxNightsFor in packages/api/src/services/bookings.js enforces MAX_NIGHTS (422).",
        "Flags the `long-stays` feature flag (FEATURE_FLAGS env var parsed in packages/api/src/flags.js; checked at services/bookings.js line 10, giving members LONG_STAY_MAX_NIGHTS = 28). It notes the schema maximum is checked first, so members can never actually reach 28. It asks the ticket owner how the 21-night limit should interact with that flag instead of deciding alone.",
        "Notes that services/bookings.js and web/src/bookingForm.js have no tests while pricing and format do (packages/api/test/pricing.test.js, packages/web/test/format.test.js). It warns that a new test importing services/bookings.js will throw unless DATABASE_URL is set, because packages/api/src/db.js lines 2-4 check it at import time, and suggests setting it (plus FEATURE_FLAGS if needed) before a dynamic import, using node:test and node:assert/strict as the existing tests do.",
        "Ends with one concrete first step that can be checked (e.g. run `npm install && npm test` and see 5 passing tests, or change openapi.json, run `npm run gen` and confirm `npm run gen:check` passes and the diff shows only the nights maximum), scoped to this ticket.",
        "Invents no commands, files, functions or settings that are not in the repo (e.g. no yarn scripts, no api/config.js, no `npm run lint` or `npm run build`, no existing bookings test, no .env file claimed to exist)."
      ]
    }
  ]
}