# Land the nightly bump's green PR; put a red one on the product dashboard. # # One bot keeps a product fresh: bump.yml, which moves the dev-kit, the ctl-* / # moq-* npm family, and the Norsk nightly pair and norsk-ctl pin in one run and # opens one PR on bump/nightly (ci-workflows docs/dashboard/ci-fragility.md, # proposal B2). It never pushes to main (proposal B): its own gate is lint, # typecheck and unit tests, and the tiers that break on an upstream change -- # smoke, integration and the guides that gate build-image's publish -- run on # the PR. This workflow is what lands it. Until bump.yml, three bots did that # job, each with its own branch (upgrade-latest/*, sync-dev-kit/*, # chore/sync-ctl-packages); those are still triaged, so their PRs still in # flight when a product moves to bump.yml land or are reported like any other. # Nothing used to act on bot PRs: green # ones sat unmerged (studio's ctl-sdk was twenty releases behind on 2026-10-03, # its PR open since 09-03) and red ones were invisible, because the bot that # opened them reports success. With a product count that only goes up, "someone # will notice" is not a process. # # So, for every open PR whose branch a bot owns: # - every check on its head has passed, every TIER that runs on a bot PR has a # green run on that head, and it merges cleanly -> squash-merge it. The # tiers are checks, smoke, integration and build-image (whose gate-guides # job runs the guides): each of those the head carries whose workflow runs # on pull_request. integration and build-image run on bot branches only, so # human PRs do not pay for them. A hand-owned integration.yml that runs on # push alone cannot gate, and still runs on main after the merge. "Nothing # failed yet" is not enough: a slow tier that has not started has no check # run at all, so bot-prs used to merge before integration had begun; # - it conflicts with main -> re-run bump.yml, ONCE per head, so it rebuilds # the night from today's main (see below). Asked FIRST, whatever its # checks say: the rebuild re-runs them all on the new base; # - a check failed, it still conflicts after that rebuild, or it is a draft # (the bot's own gate was red) -> list it, and report this repo's `bot-prs` # cell RED on the product dashboard, which is the Discord message people # actually read; # - checks or a tier still running -> leave it for the next trigger; a tier # that has not started an hour after checks finished never will (its # workflow is broken, or does not run on PRs), so that is red too; # - a tier integration.yml cancelled because the host was too contended to # judge it (its job carries a `ctl-contention-uncertain` annotation) -> not # yet judged, so neither red nor merged: re-run once a backoff has passed # (vars.BOT_PRS_CONTENDED_BACKOFF_MINUTES, default 30, doubling per # attempt), up to vars.BOT_PRS_CONTENDED_ATTEMPTS attempts (default 4); # after that it is red as "still contended after N attempts: needs a human # or a quieter runner", never as a failure. Any other cancelled check is # red, and a cancelled tier is re-run up to its third attempt, as before. # The cell goes green again once nothing is red. # # Why the conflict rebuild: when the three bots fired within minutes of each # other, each writing package.json and bun.lock from main as it stood, # whichever landed first left the others conflicting, and a bot rebuilt its # branch only on its next DAILY run (2026-10-09: commentary, probe and reuters # each had a sync-ctl-packages PR "needing a human" that a re-run fixed). One # bump ends that, but a human's merge to main can still make bump/nightly # conflict. bump.yml rebuilds from main when run: it force-pushes bump/nightly # from today's main and closes any older bump PR, and an old bot's PR for each # component it moved -- so it is the rebuild for those branches too, whose own # workflows the move to bump.yml deleted. The marker comment caps it at one # rebuild per head, so a rebuild that still conflicts goes to a human instead # of looping every hour. # # It triages the bot PRs that target one base: main, unless a dispatch names # another (`base`). A trial of a dev-kit change against a throwaway branch is # triaged, rebuilt and merged there, never on main, and reports nothing. # # Runs when any gating tier finishes (the prompt path), hourly as a sweep, # and on demand. No checkout: it only talks to the GitHub API, so it needs none # of the runner-workspace steps the building workflows carry. # # Single-sourced in @norskvideo/ctl-dev-kit (conventions/bot-prs.yml) and copied # verbatim into each product repo -- it reads its dashboard key from this repo's # bump.yml rather than carrying one -- so sync-drift creates it where # absent and check:drift fails a copy that diverges. Never hand-edit the copy. name: bot-prs on: workflow_run: workflows: [checks, smoke, integration, build-image] types: [completed] schedule: - cron: "23 * * * *" workflow_dispatch: inputs: base: description: "Triage the bot PRs that target this branch. main, unless trialling on a throwaway branch." default: main permissions: contents: read checks: read pull-requests: read # Reads which gating tiers have passed on a PR's head, and writes Actions: # re-running a cancelled tier and dispatching a bot's workflow to rebuild a # conflicting PR. Those went through the PAT, which has no Actions write, and # failed with HTTP 403 in every repo (2026-10-10). A workflow_dispatch made # with this token does start a run (exempt from the recursion guard). Merges, # branch deletes and comments stay on the PAT (WRITE_TOKEN). actions: write concurrency: # One at a time per repo, and never cancelled: a merge half-done by a run that # was cancelled is worse than a second run finding nothing left to do. group: bot-prs-${{ github.repository }} cancel-in-progress: false jobs: triage: runs-on: x64 steps: - name: Merge green bot PRs, report red ones id: triage env: # Reads go through the job's own token, which the permissions above # scope to exactly what is read; CI_DISPATCH_TOKEN (a PAT) has no # Checks access, and the check-runs call returned an error object. # Merges and branch deletes go through the PAT, because a push made # with GITHUB_TOKEN does not trigger main's own workflows. READ_TOKEN: ${{ github.token }} WRITE_TOKEN: ${{ secrets.CI_DISPATCH_TOKEN }} REPO: ${{ github.repository }} BASE: ${{ inputs.base || 'main' }} # A run integration.yml cancelled because the host was too contended # to judge it carries this annotation title on its job. Such a run is # not yet judged: re-run with backoff, up to the attempt budget. # Repo variables, so the verbatim copy needs no edit; unset or not a # number takes the default (4 attempts, 30 minutes doubling). CONTENDED_MARK: ctl-contention-uncertain CONTENDED_ATTEMPTS: ${{ vars.BOT_PRS_CONTENDED_ATTEMPTS }} CONTENDED_BACKOFF_MINUTES: ${{ vars.BOT_PRS_CONTENDED_BACKOFF_MINUTES }} run: | set -euo pipefail api="https://api.github.com/repos/$REPO" curl_as() { local tok="$1"; shift; curl -sS --retry 3 --retry-all-errors -H "Authorization: Bearer $tok" -H "Accept: application/vnd.github+json" "$@"; } gh_api() { curl_as "$READ_TOKEN" "$@"; } gh_write() { curl_as "$WRITE_TOKEN" "$@"; } # Actions writes (re-run a tier, dispatch a bot's workflow) use the # job token, which permissions grant actions: write; the PAT cannot. gh_actions() { curl_as "$READ_TOKEN" "$@"; } # An API error is a JSON object with a message; say what it was rather # than letting jq fail later on a missing field. expect() { local what="$1" filter="$2" body="$3" jq -e "$filter" >/dev/null 2>&1 <<<"$body" && return 0 echo "::error::$what: $(jq -r '.message // .' <<<"$body" 2>/dev/null || echo "$body")" exit 1 } # The bump owns bump/nightly (see bump.yml's PR step); the other three # are the retired bots' branches, still landed while in flight. pulls="$(gh_api "$api/pulls?state=open&per_page=100")" expect "listing open PRs" 'type == "array"' "$pulls" prs="$(jq -c --arg repo "$REPO" --arg base "$BASE" ' [.[] | select(.head.repo.full_name == $repo) | select(.base.ref == $base) | select(.head.ref | test("^(bump/nightly$|upgrade-latest/|sync-dev-kit/|chore/sync-ctl-packages$)")) | {number, ref: .head.ref, sha: .head.sha, draft, url: .html_url}]' <<<"$pulls")" # Does this workflow (on stdin) run on pull_request? Only its `on:` block # counts. The convention integration.yml and build-image.yml do, on bot # branches only; a hand-owned integration.yml that runs on push alone # cannot gate a PR. gates_bot_prs() { awk '/^on:/ { on = 1; next } on && /^[^ #]/ { on = 0 } on && /^ pull_request:/ { found = 1 } END { exit !found }' } # The tiers a bump must pass on its head before it reaches main: each # of these workflows the HEAD carries that runs on a bot PR. Read from # the head, not main, so a dev-kit sync that brings a tier gates on it. # Each file is read whole BEFORE it is judged: piped straight into # gates_bot_prs under pipefail, a file larger than the pipe buffer # failed the pipeline whenever the reader finished first, and # build-image.yml read as "does not run on bot PRs" (commentary #36). tiers_for() { local sha="$1" listing f text listing="$(gh_api "$api/contents/.github/workflows?ref=$sha")" expect "listing the workflows at $sha" 'type == "array"' "$listing" for f in checks.yml smoke.yml integration.yml build-image.yml; do jq -e --arg f "$f" 'any(.[]; .name == $f)' >/dev/null <<<"$listing" || continue # A tier it cannot read must not silently stop gating: fail instead. text="$(curl -fsS --retry 3 --retry-all-errors -H "Authorization: Bearer $READ_TOKEN" \ -H "Accept: application/vnd.github.raw" "$api/contents/.github/workflows/$f?ref=$sha")" || { echo "::error::could not read $f at $sha" >&2; return 1; } if gates_bot_prs <<<"$text"; then echo "$f" else echo "${f%.yml} does not run on bot PRs at ${sha:0:8}, so it does not gate the merge" >&2 fi done } # tier_verdict REQUIRED RUNS NOW -> "ok", "wait ", "rerun " # or "red ". REQUIRED: workflow files, one per line. RUNS: the # head's actions/runs listing. The newest pull_request run of each # counts, so a re-run after a red is honoured. A tier with no run yet is # waited for, unless checks finished over an hour ago: then it never # will start. A CANCELLED tier is re-run, up to its third attempt: the # guides queue on one concurrency group per repo and GitHub keeps one # pending run per group, cancelling the older, so three bot PRs opening # within the hour can cancel a guides run with nothing wrong in it. # CONTENDED (optional): the run ids contended_runs found. A cancelled # tier among them was never judged: it is re-run once the backoff # since its cancel has passed (CONTENDED_BACKOFF_MINUTES, doubling per # attempt), up to CONTENDED_ATTEMPTS attempts, then reported as still # contended. It never reads ok. tier_verdict() { local required="$1" runs="$2" now="$3" contended="${4:-}" f run id status conclusion checks_age local attempt max="${CONTENDED_ATTEMPTS:-}" base="${CONTENDED_BACKOFF_MINUTES:-}" backoff age local -a waiting=() failed=() rerun=() [[ "$max" =~ ^[1-9][0-9]*$ ]] || max=4 [[ "$base" =~ ^[0-9]+$ ]] || base=30 checks_age="$(jq -r --argjson now "$now" ' [.workflow_runs[] | select((.path | split("@")[0]) == ".github/workflows/checks.yml" and .event == "pull_request" and .status == "completed")] | max_by(.id) | if . == null then "" else ($now - (.updated_at | fromdateiso8601)) end' <<<"$runs")" while IFS= read -r f; do [ -n "$f" ] || continue run="$(jq -c --arg p ".github/workflows/$f" ' [.workflow_runs[] | select((.path | split("@")[0]) == $p and .event == "pull_request")] | max_by(.id) // empty' <<<"$runs")" if [ -z "$run" ]; then if [ -n "$checks_age" ] && [ "$checks_age" -gt 3600 ]; then failed+=("${f%.yml} never ran on this head (does its workflow run on bot PRs?)") else waiting+=("${f%.yml} has not started") fi continue fi status="$(jq -r .status <<<"$run")" conclusion="$(jq -r '.conclusion // ""' <<<"$run")" id="$(jq -r .id <<<"$run")" attempt="$(jq -r '.run_attempt // 1' <<<"$run")" if [ "$status" != "completed" ]; then waiting+=("${f%.yml} $status") elif [ "$conclusion" = "cancelled" ] && grep -qxF "$id" <<<"$contended"; then if [ "$attempt" -ge "$max" ]; then failed+=("${f%.yml} still contended after $attempt attempts: needs a human or a quieter runner") continue fi backoff=$((base * 60 * (1 << (attempt - 1)))) age=$((now - $(jq -r '.updated_at | fromdateiso8601' <<<"$run"))) if [ "$age" -ge "$backoff" ]; then rerun+=("$id") else waiting+=("${f%.yml} not yet judged, the host was contended (attempt $attempt of $max); re-running in $(((backoff - age + 59) / 60))m") fi elif [ "$conclusion" = "cancelled" ] && [ "$(jq -r '.run_attempt // 1' <<<"$run")" -lt 3 ]; then rerun+=("$(jq -r .id <<<"$run")") elif [ "$conclusion" != "success" ]; then failed+=("${f%.yml} $conclusion") fi done <<<"$required" if [ "${#failed[@]}" -gt 0 ]; then printf 'red %s\n' "$(printf '%s; ' "${failed[@]}" | sed 's/; $//')" elif [ "${#rerun[@]}" -gt 0 ]; then echo "rerun ${rerun[*]}" elif [ "${#waiting[@]}" -gt 0 ]; then printf 'wait %s\n' "$(printf '%s; ' "${waiting[@]}" | sed 's/; $//')" else echo ok fi } # contended_runs CHECK_RUNS -> the workflow run ids (one per line) that # integration.yml cancelled because the host was too contended to # judge them: a CANCELLED job of the run carries the CONTENDED_MARK # annotation. The marker on a job that failed (its cancel never # landed) is not one: that run is red. Annotations it cannot read # leave the run red, as every cancel was before. contended_runs() { local checks="$1" id notes while IFS= read -r id; do [ -n "$id" ] || continue notes="$(gh_api "$api/check-runs/$id/annotations?per_page=100")" if ! jq -e 'type == "array"' >/dev/null 2>&1 <<<"$notes"; then echo "::warning::could not read the annotations of check run $id; a cancel there counts as red" >&2 continue fi jq -e --arg m "$CONTENDED_MARK" 'any(.[]; .title == $m)' >/dev/null <<<"$notes" || continue jq -r --argjson id "$id" '.check_runs[] | select(.id == $id) | (.details_url // "") | (capture("/actions/runs/(?[0-9]+)").r // empty)' <<<"$checks" done < <(jq -r '.check_runs[] | select(.conclusion == "cancelled" and (.output.annotations_count // 0) > 0) | .id' <<<"$checks") | sort -u } # failed_checks CHECK_RUNS CONTENDED -> the names of the checks that # failed, comma separated. A cancelled job of a contended run (every # one of them: GitHub cancels the run's other jobs with it) is left to # tier_verdict, which re-runs it; any other cancel is red, as before. failed_checks() { jq -r --arg c "$2" '($c | split("\n") | map(select(. != ""))) as $contended | [.check_runs[] | select(.conclusion | IN("failure","cancelled","timed_out","action_required","startup_failure")) | select(.conclusion != "cancelled" or (((.details_url // "") | (capture("/actions/runs/(?[0-9]+)").r // "")) as $r | any($contended[]; . == $r) | not)) | .name] | unique | join(", ")' <<<"$1" } # The workflow that rebuilds each bot branch: bump.yml, for the retired # bots' branches too (their workflows went when bump.yml arrived, and # bump.yml supersedes their PRs for every component it moves). producer() { case "$1" in bump/nightly | upgrade-latest/* | sync-dev-kit/* | chore/sync-ctl-packages) echo bump.yml ;; *) echo "" ;; esac } red=() merged=() rebuilt=() while IFS= read -r pr; do [ -n "$pr" ] || continue n="$(jq -r .number <<<"$pr")"; ref="$(jq -r .ref <<<"$pr")"; sha="$(jq -r .sha <<<"$pr")" # Mergeability FIRST, before the checks are judged. A rebuild re-runs # every tier on today's base, so a conflicting PR is rebuilt whatever # its checks say: relay #11 (2026-10-10) carried a smoke failure from # before main moved, was reported red for it, and sat conflicting # for 90 minutes with its rebuild never asked for. Only a PR that # still conflicts after its rebuild, or merges cleanly with red # checks, is red. GitHub computes mergeability lazily; ask until it # has an answer. mergeable=null for _ in 1 2 3 4 5; do mergeable="$(gh_api "$api/pulls/$n" | jq -r .mergeable)" [ "$mergeable" != "null" ] && break sleep 3 done if [ "$mergeable" != "true" ]; then # One rebuild per head: the marker names the sha it was asked for. marker="bot-prs: rebuild requested for head $sha" wf="$(producer "$ref")" comments="$(gh_api "$api/issues/$n/comments?per_page=100")" expect "reading comments on #$n" 'type == "array"' "$comments" if [ -z "$wf" ]; then red+=("#$n $ref: does not merge cleanly with main, and no bot is known to rebuild it") elif jq -e --arg m "$marker" 'any(.[]; .body | contains($m))' >/dev/null <<<"$comments"; then red+=("#$n $ref: still does not merge cleanly with main after $wf rebuilt it") else code="$(gh_actions -o /dev/null -w '%{http_code}' -X POST "$api/actions/workflows/$wf/dispatches" -d "$(jq -nc --arg b "$BASE" '{ref:$b, inputs:{base:$b}}')")" if [ "$code" = "204" ]; then body="$marker -- it conflicted with $BASE, so $wf was re-run to rebuild the bump from today's $BASE. If the rebuilt branch conflicts too, this PR is reported for a human." gh_write -o /dev/null -X POST "$api/issues/$n/comments" -d "$(jq -nc --arg b "$body" '{body:$b}')" || true rebuilt+=("#$n") echo "#$n $ref: conflicts with main; re-ran $wf to rebuild it" else red+=("#$n $ref: does not merge cleanly with main, and re-running $wf failed (HTTP $code)") fi fi continue fi runs="$(gh_api "$api/commits/$sha/check-runs?per_page=100")" expect "reading checks for #$n" '.check_runs | type == "array"' "$runs" contended="$(contended_runs "$runs")" failed="$(failed_checks "$runs" "$contended")" pending="$(jq -r '[.check_runs[] | select(.status != "completed")] | length' <<<"$runs")" passed="$(jq -r '[.check_runs[] | select(.conclusion == "success")] | length' <<<"$runs")" if [ -n "$failed" ]; then red+=("#$n $ref: failing $failed"); continue; fi if [ "$(jq -r .draft <<<"$pr")" = "true" ]; then red+=("#$n $ref: draft (its bot's gate was red)"); continue; fi if [ "$pending" -gt 0 ] || [ "$passed" -eq 0 ]; then echo "#$n $ref: checks not finished; next time"; continue; fi # Every tier that runs on a bot PR must have PASSED on this head. required="$(tiers_for "$sha")" wf_runs="$(gh_api "$api/actions/runs?head_sha=$sha&per_page=100")" expect "reading workflow runs for #$n" '.workflow_runs | type == "array"' "$wf_runs" verdict="$(tier_verdict "$required" "$wf_runs" "$(date +%s)" "$contended")" case "$verdict" in ok) ;; red\ *) red+=("#$n $ref: ${verdict#red }"); continue ;; rerun\ *) for id in ${verdict#rerun }; do code="$(gh_actions -o /dev/null -w '%{http_code}' -X POST "$api/actions/runs/$id/rerun")" if grep -qxF "$id" <<<"$contended"; then echo "#$n $ref: a gating tier (run $id) was cancelled as uncertain, the host was contended; re-running it (HTTP $code)" else echo "#$n $ref: a gating tier (run $id) was cancelled; re-running it (HTTP $code)" fi done continue ;; *) echo "#$n $ref: ${verdict#wait }; next time"; continue ;; esac # sha pins the merge to what was checked: a bot force-pushing the # branch meanwhile makes this refuse rather than land unchecked code. res="$(gh_write -X PUT "$api/pulls/$n/merge" -d "$(jq -nc --arg sha "$sha" '{merge_method:"squash", sha:$sha}')")" if [ "$(jq -r '.merged // false' <<<"$res")" = "true" ]; then merged+=("#$n") echo "merged #$n ($ref)" # The bots recreate their branch on the next bump. gh_write -X DELETE "$api/git/refs/heads/$ref" -o /dev/null || true else red+=("#$n $ref: merge refused: $(jq -r '.message // "unknown"' <<<"$res")") fi done < <(jq -c '.[]' <<<"$prs") if [ "${#red[@]}" -gt 0 ]; then state=failure detail="$(printf '%s; ' "${red[@]}")" for r in "${red[@]}"; do echo "::warning::bot PR needs a human: $r"; done else state=success # Never empty: the dashboard step reads an empty detail as missing. merged_detail="${merged[*]:+merged ${merged[*]}}" rebuilt_detail="${rebuilt[*]:+rebuilding ${rebuilt[*]}}" detail="$(printf '%s' "$merged_detail${merged_detail:+${rebuilt_detail:+; }}$rebuilt_detail")" detail="${detail:-no bot PRs waiting}" fi echo "state=$state" >> "$GITHUB_OUTPUT" echo "detail=${detail%; }" >> "$GITHUB_OUTPUT" - name: Report to the product dashboard # Always: a triage that broke is exactly what nobody would otherwise see. # But only for main: a trial against another base must not paint the board. if: always() && (inputs.base || 'main') == 'main' env: TOKEN: ${{ secrets.CI_DISPATCH_TOKEN }} READ_TOKEN: ${{ github.token }} STATUS: ${{ steps.triage.outputs.state || 'failure' }} # By the step's outcome, not by an empty detail: a quiet run with nothing # merged once read "triage failed" beside a green status every hour. DETAIL: ${{ steps.triage.outcome == 'success' && steps.triage.outputs.detail || (steps.triage.outcome == 'success' && 'no bot PRs waiting' || 'the bot-PR triage itself failed; see the run') }} SHA: ${{ github.sha }} RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} REPO: ${{ github.repository }} run: | set -euo pipefail # The dashboard key is the one bump.yml reports under, read from it so # this file can stay byte-identical across products; upgrade-latest.yml # carried it before bump.yml did. -f: a 404 must not read as a key. product="" for wf in bump.yml upgrade-latest.yml; do product="$(curl -fsS -H "Authorization: Bearer $READ_TOKEN" -H "Accept: application/vnd.github.raw" \ "https://api.github.com/repos/$REPO/contents/.github/workflows/$wf" 2>/dev/null \ | sed -nE 's/^[[:space:]]*product:[[:space:]]*([^[:space:]]+).*/\1/p' | head -1 || true)" [ -z "$product" ] || break done if [ -z "$product" ] || [ "$product" = "__PRODUCT__" ]; then echo "::warning::no product key in bump.yml; not reporting to the dashboard" exit 0 fi ts="$(date -u +%Y-%m-%dT%H:%M:%SZ)" payload="$(jq -cn \ --arg product "$product" --arg pipeline bot-prs --arg conclusion "$STATUS" \ --arg sha "$SHA" --arg runUrl "$RUN_URL" --arg ts "$ts" --arg detail "$DETAIL" \ '{event_type: "ci-status", client_payload: ( {product: $product, pipeline: $pipeline, conclusion: $conclusion, sha: $sha, runUrl: $runUrl, ts: $ts} + (if $detail == "" then {} else {detail: $detail} end) )}')" code="$(curl -sS -o /dev/null -w '%{http_code}' -X POST \ -H "Authorization: Bearer $TOKEN" -H "Accept: application/vnd.github+json" \ https://api.github.com/repos/id3as/ci-workflows/dispatches -d "$payload")" [ "$code" = "204" ] || echo "::warning::dashboard dispatch returned HTTP $code"