{
  "skill_name": "dependency-upgrade-review",
  "evals": [
    {
      "id": 1,
      "prompt": "Dependabot opened PR #212 on harbor-notes: \"Bump quillmark from 3.4.1 to 4.1.0\". CI is green. The PR diff is in pr.diff, quillmark's changelog is quillmark-CHANGELOG.md, and I've attached src/render/, test/ and .github/workflows/ci.yml from the repo. Can I merge it?",
      "assertions": [
        "Gives exactly one verdict at the top, and it is DO NOT MERGE",
        "Reports that renderInline() was removed in 4.0.0 and that src/render/comment.js still imports and calls it, citing the file and line",
        "Flags that 4.0.0 changed the sanitize default to false and that src/render/note.js renders note bodies with quillmark.render(markdown) and no sanitize option, so raw HTML from users passes through",
        "Lists the lockfile changes the title does not mention: entity-codec 2.1.0 to 3.0.2 (a major, with its licence changing from MIT to MPL-2.0), html-escape-lite removed, and quill-native-highlight and prebuild-install added, with quill-native-highlight's new install script",
        "Says the green CI does not cover the change: the only test renders Markdown with no raw HTML, and no test imports src/render/comment.js",
        "Accounts for the skipped releases, not only 4.1.0: names the 3.5.0 link-title security fix and the setOptions() deprecation that src/render/note.js still uses",
        "Gives a rollback that pins quillmark back and regenerates package-lock.json (or reverts the PR), and names taking 3.5.x as the way to get the security fix without the breaking changes",
        "Opens with the verdict and the decisive reasons (the removed renderInline() call and the sanitize default) in plain words before any list, keeps the lockfile changes to a line or a short list rather than a table, and does not describe the session's network or tool access",
        "Does not invent versions, advisory IDs, APIs, files or tests that are not in the supplied input"
      ]
    },
    {
      "id": 2,
      "prompt": "Dependabot PR #88 on tidewater-tools: \"Bump the github-actions group with 2 updates\". Checks passed. Diff in pr.diff, the two workflow files after the change are in .github/workflows/, and I pasted the release notes from both action repos into release-notes.md. OK to merge?",
      "assertions": [
        "Gives MERGE AFTER CHECKS, not MERGE",
        "Treats the northwind-ci/checkout-lite SHA as unverified and gives the exact git ls-remote command for refs/tags/v3.2.0 and refs/tags/v3.2.0^{}, saying which line must equal 9b8e7a6d5c4b3a291807f6e5d4c3b2a1908f7e6d",
        "Says octo-tools/pypi-publish moving from @v2 to @v3 is a tag-only pin that can be moved after review, recommends pinning it to a commit SHA, and notes the step receives secrets.PYPI_TOKEN",
        "Says the passing checks never ran pypi-publish v3, because release.yml runs only on a v* tag push, and names a way to exercise it before the next real release",
        "Reports that release.yml still passes the deprecated password input (release.yml line 22) and that verify-metadata now defaults to true, so the next release can fail on an invalid long description",
        "Classes the jumps: checkout-lite v3.1.0 to v3.2.0 as minor, pypi-publish v2 to v3 as major",
        "Gives a rollback (revert the PR or restore the old uses: lines) and notes a revert cannot undo a release already published to PyPI with v3",
        "Does not report either SHA or the tag move as verified, and does not explain the gap by describing the session's network or tool access; it gives the command and the SHA it must equal instead",
        "Does not invent action versions, SHAs, inputs or workflow steps that are not in the supplied files"
      ]
    },
    {
      "id": 3,
      "prompt": "Renovate PR on our Go rate-limiter service: \"fix(deps): update module github.com/lanternhq/ratelimit to v0.9.0\". CI passed. Quick verdict please.\n\ngo.mod:\n-\tgithub.com/lanternhq/ratelimit v0.8.3\n+\tgithub.com/lanternhq/ratelimit v0.9.0\n+\tgithub.com/lanternhq/clock v1.2.0 // indirect\n\ngo.sum:\n-github.com/lanternhq/ratelimit v0.8.3 h1:AbCd=\n-github.com/lanternhq/ratelimit v0.8.3/go.mod h1:EfGh=\n+github.com/lanternhq/clock v1.2.0 h1:IjKl=\n+github.com/lanternhq/clock v1.2.0/go.mod h1:MnOp=\n+github.com/lanternhq/ratelimit v0.9.0 h1:QrSt=\n+github.com/lanternhq/ratelimit v0.9.0/go.mod h1:UvWx=\n\nRelease notes for v0.9.0:\n- Limiter.Wait(ctx) now returns ErrLimitExceeded immediately when burst is 0, instead of blocking.\n- NewLimiter(rate, burst) is deprecated; use New(Config{Rate: r, Burst: b}).\n- Uses github.com/lanternhq/clock for time, so tests can inject a fake clock.",
      "assertions": [
        "Gives MERGE AFTER CHECKS, not MERGE",
        "States that v0.8.3 to v0.9.0 is a 0.x minor bump, which semver treats as potentially breaking",
        "Names the Wait behaviour change and the NewLimiter deprecation, and gives a concrete search command for Wait( and NewLimiter( since no code was supplied, without claiming there are or are not hits",
        "Points out that only the v0.9.0 notes were supplied, so any releases between v0.8.3 and v0.9.0 are still unread",
        "Flags github.com/lanternhq/clock v1.2.0 as a new indirect module the PR title does not mention",
        "Says the passing CI only covers this if a test exercises the burst-0 Wait path, and lists that as a check rather than assuming it",
        "Gives a rollback command such as go get github.com/lanternhq/ratelimit@v0.8.3 followed by go mod tidy, or reverting the PR",
        "Stays short, as the user asked for a quick verdict: the verdict and its reasons come first, and there is no package table and no empty section",
        "Does not invent file names, call sites, tests or release versions that are not in the prompt"
      ]
    },
    {
      "id": 4,
      "held_out": "2026-10-20",
      "prompt": "Renovate opened PR #287 bumping quillstream from 2.4.1 to 3.1.0 and CI is green. The PR body is in PR.md, the diff is pr.diff, and I copied the upstream changelog into upstream/. Repo is checked out at main. Is this safe to merge? I'd like to get it in today.",
      "assertions": [
        "Gives the verdict DO NOT MERGE for #287 (not MERGE, and not MERGE AFTER CHECKS without code changes), and ties it to at least one breaking change that hits code in this repo.",
        "Points out that read_legacy_archive in src/ingest/reader.py imports quillstream.legacy.parse_frame inside the function, and that the 3.0.0 entry in the changelog (a release this PR skips) removes that module. Names quillstream.frames.decode(buf, strict=False) as the replacement and says the `header, payload = parse_frame(...)` unpacking has to change too, because decode returns a Frame object.",
        "Points out that fetch_with_retry in src/ingest/client.py retries only on `if result is None`. Under 3.0.0, Client.fetch raises quillstream.errors.QuillTimeout instead of returning None, so timeouts will escape as exceptions rather than being retried. Suggests catching QuillTimeout, or migrating first with 2.5.0's Client(raise_on_timeout=True).",
        "Flags that the Dockerfile builds FROM python:3.9-slim while 3.0.0 requires Python >=3.10 (and grainwire wheels only cover CPython 3.10–3.13), so the production image build or install will break. Notes that CI and pip-compile both ran on Python 3.11, so a green CI could not catch this.",
        "Lists the requirements.txt changes beyond the title, as shown in pr.diff: new transitive package grainwire 1.0.0, removed compat-six 1.16.2, and typing-shimx bumped 4.5.0 -> 4.9.1. Ties grainwire to the 3.1.0 changelog entry, including the sdist/Rust-toolchain caveat for platforms without wheels.",
        "Explains why the green check is weak evidence: tests/test_reader.py only tests read_stream, with a mocked Client whose fetch returns a payload. Nothing exercises read_legacy_archive (the lazy import means the missing module never fails at import or collection time) or the timeout/None path.",
        "Gives a concrete rollback: revert the PR, or restore quillstream==2.4.1 in requirements.in and re-run `pip-compile --generate-hashes`, then rebuild and redeploy the image. Optionally offers a staged path, e.g. moving to 2.5.0 first to surface the deprecation warnings.",
        "Does not invent versions, APIs or changelog entries that are not in the files (e.g. no made-up 3.0.1 fix, no Python-3.9 compatibility shim). Does not claim to have installed the package, run the tests, or checked the sha256 hashes against PyPI. If it recommends checking the hashes, it says that needs network access and gives the command rather than asserting they match."
      ]
    },
    {
      "id": 5,
      "held_out": "2026-10-20",
      "prompt": "Dependabot opened two PRs this morning and both are green: #412 bumps the bramblewick/setup-toolchain action pin (a teammate pushed an extra commit to it) and #413 bumps release-notary. Descriptions are PR-412.md / PR-413.md, diffs are pr-412.diff / pr-413.diff, upstream release notes are in upstream/. Are the action SHA pins right, and can I merge both before we push the v5.13.0 tag on Friday?",
      "assertions": [
        "Gives a separate verdict for each PR. #413 is DO NOT MERGE. #412 is not a plain MERGE: it is MERGE AFTER CHECKS or DO NOT MERGE until the pins are reconciled.",
        "Catches that pr-412.diff pins setup-toolchain to two different commits that are both labelled `# v3.3.0`: 8141d107d0837efc25f86e096a7eefd58e979515 in ci.yml (from Dependabot) and 7dd0aa52537f8f4f4002b505507186cdaec669c4 in release.yml (from mkowalczyk's hand-made commit). Says at most one of them can be the tag's commit, and that the release.yml one is the suspect because it was edited by hand and is the one that runs with production secrets.",
        "Says plainly that it cannot confirm offline which SHA (if either) the v3.3.0 tag points to. Gives a concrete command, e.g. `git ls-remote https://github.com/bramblewick/setup-toolchain 'refs/tags/v3.3.0*'` (using the peeled `^{}` line if the tag is annotated) or `gh api repos/bramblewick/setup-toolchain/git/ref/tags/v3.3.0`. Never states that either SHA is verified.",
        "For #412, notes that v3.3.0 switches the action to the node24 runtime, which needs self-hosted runners on v2.328.0 or newer. release.yml runs on [self-hosted, linux, release] and PR CI runs only on ubuntu-latest, so the first real test would be Friday's tag push. Also mentions the `cache` default flipping to true, which affects the release job.",
        "For #413, finds that scripts/release.mjs calls createNotary({ legacyMode: true, ... }). 1.9.0 removed legacyMode and now throws TypeError: Unknown option \"legacyMode\", so `npm run release` in release.yml will fail. Also notes that the switch to the v2 manifest layout breaks the downstream ledger-portal parser mentioned in the code comment.",
        "For #413, lists the package-lock changes beyond the title: a new transitive package notary-telemetry 0.2.1 with \"hasInstallScript\": true, resolved from https://npm.notary-cdn.io rather than registry.npmjs.org; ansi-tint 1.0.4 removed; semver-lite 2.1.0 -> 2.3.0; release-notary's engines raised to node >=20. Treats the install script plus the off-registry tarball as a supply-chain concern in its own right.",
        "Explains that the green checks never ran the changed code. ci.yml runs on pull_request with `npm ci --ignore-scripts` and never runs scripts/release.mjs. release.yml runs plain `npm ci`, so notary-telemetry's install script would first execute in the production environment, with RELEASE_SIGNING_KEY and NPM_TOKEN in the job and contents: write / id-token: write permissions. Mentions NOTARY_TELEMETRY=0 and/or keeping 1.8.x as mitigations.",
        "Gives a rollback for each PR: revert #412 to restore the c8aee2ae521bdb1ef1b97761673ebbe92c953b0f (v3.2.0) pins; revert #413, or pin release-notary back to 1.8.2 and regenerate package-lock.json. Does not invent versions, flags or APIs that are not in the files, and does not claim to have run npm install, the workflows, or the release script."
      ]
    },
    {
      "id": 6,
      "held_out": "2026-10-20",
      "prompt": "Dependabot opened PR #412 on our payments-gateway service bumping github.com/fernwire/httpkit from 1.8.1 to 1.8.4. It's only a patch bump and every check is green, so I'd like to merge it before the release cut this afternoon. Can I merge this dependency upgrade? The PR body is in PR.md, the diff is pr.diff, and I pasted the upstream changelog into vendor-notes/httpkit-CHANGELOG.md. The repo source is checked out here.",
      "assertions": [
        "Picks out the v1.8.3 entry in vendor-notes/httpkit-CHANGELOG.md (a version the Dependabot release notes in PR.md leave out): NewClient no longer applies a default 30-second Timeout, and the zero value now means no client-level timeout. It treats this as the key finding even though the bump looks like a patch.",
        "Points to internal/payments/client.go NewClient, which calls httpkit.NewClient with only WithBaseURL and WithRetries and whose comment says it relies on the 30s default (INC-2291), and to internal/payments/settle.go, which calls Charge with context.Background() and no deadline. So after the upgrade a hung gateway can block the settlement batch indefinitely. It also notes that internal/notify/client.go sets WithTimeout(5*time.Second) explicitly and is unaffected.",
        "Explains that the green CI does not test this change: client_test.go injects fakeDoer, so the real httpkit client is never exercised, and ci.yml runs `go test -race -short ./...` without `-tags integration`, so client_integration_test.go (TestChargeHungGatewayTimesOut) did not run on this PR. It runs only nightly or on manual dispatch via integration.yml.",
        "Gives a verdict of DO NOT MERGE as-is, or MERGE AFTER CHECKS on the specific condition that payments NewClient gains an explicit httpkit.WithTimeout(30*time.Second) (or a per-call context deadline in SettleBatch) and the integration test passes. It is never plain MERGE.",
        "Lists the go.mod/go.sum changes the PR title does not mention: indirect github.com/fernwire/backoff v0.3.0 -> v0.4.0 and a newly added indirect github.com/fernwire/otelshim v0.1.0. It ties them to the v1.8.2 (backoff, full-jitter) and v1.8.4 (WithTracer) changelog entries rather than calling them unexplained.",
        "Gives a concrete rollback: revert the merge commit, or `go get github.com/fernwire/httpkit@v1.8.1 && go mod tidy`, which also returns backoff to v0.3.0 and drops otelshim. It notes that adding the explicit WithTimeout is safe to keep on either version.",
        "Does not claim to have run tests or verified the go.sum hashes or upstream tags. It gives the network or execution steps as commands to run (for example `go mod verify`, `go test -tags integration -run TestChargeHungGatewayTimesOut ./internal/payments/`, or triggering integration.yml via workflow_dispatch).",
        "Invents no httpkit versions, options or behaviours beyond those in the files (for example no made-up v1.8.5, no claimed retry-on-POST change, no options other than WithBaseURL, WithRetries, WithTimeout and WithTracer)."
      ]
    },
    {
      "id": 7,
      "held_out": "2026-10-20",
      "prompt": "Renovate grouped this week's dev dependency updates into PR #88 for our invoice-widgets package: five npm packages in one go (details in PR.md, diff in pr.diff). After last quarter's incident with a grouped bump I'm nervous about these. Is this Renovate bump safe to merge, or should I split it up? Repo files are here.",
      "assertions": [
        "Ends with a verdict of MERGE. It does not downgrade to MERGE AFTER CHECKS or DO NOT MERGE over problems the files do not contain (for example an install script, a registry change, a maintainer change, or a breaking change that affects this repo), and it does not insist on splitting the PR without a concrete reason from the files.",
        "Checks bundlecrate 6.1.0's new Node >=18.18 minimum against the repo and finds it harmless: .nvmrc is 20.11.1, package.json engines is node >=20, and ci.yml uses node-version-file: .nvmrc.",
        "Checks the repo's uses of the changed APIs and finds nothing that needs action. vellum-test 2.8.0 deprecates vt.legacyTimers(), but tests/due-date.test.ts uses only vt.fakeTimers/vt.restoreTimers. lintwright 4.3.0's new no-floating-promise rule is only in lintwright:strict, while .lintwright.json extends lintwright:recommended.",
        "Lists the lockfile changes beyond the five headline bumps: @bundlecrate/resolver-core 1.0.0 added, bundlecrate-legacy-resolver 2.3.4 removed, and the transitive picolabel 0.9.1 -> 0.9.2. It ties each to a changelog entry (bundlecrate 6.1.0 resolver extraction, lintwright 4.3.0 picolabel update).",
        "States that no lockfile entry in pr.diff has \"hasInstallScript\" and that every resolved URL is on https://registry.npmjs.org/, which matches .npmrc. So there are no new install-time scripts and no registry switch.",
        "Says CI did exercise the changed code, mapping the packages to jobs in .github/workflows/ci.yml: lintwright -> lint, prettyquill -> format, tsforge-types -> typecheck, vellum-test -> test, bundlecrate -> build-and-smoke (which runs test:dist against the built bundle). It says the pass/fail status comes from PR.md and was not observed directly.",
        "Gives a rollback: revert the merge commit (restoring package.json and package-lock.json together) and run `npm ci`. It notes these are all devDependencies, so only the bundlecrate-built dist/ could reach users, and that build is covered by the smoke step.",
        "Does not claim to have verified integrity hashes, provenance or upstream tags offline. It invents no versions, rules or APIs not in the files, and frames anything network-dependent (for example `npm audit signatures`, `npm ci` locally) as an optional command, not a blocker."
      ]
    }
  ]
}