/** * Giving an answered block a way back into the queue. * * A blocked or failed run leaves its state label on the issue, and eligibility * treats any state label as disqualifying (`routing.isEligible`). That is the * right interlock — it is what stops a second worker landing on live work * across a daemon restart — but until this verb existed it had no exit: the * orchestrator's brief tells it never to hand-edit a state label, so an issue * whose question it had just answered could only be left labelled, which means * never re-claimed and the answer inert. Nothing surfaces that either: the * issue does not fail, it simply stops existing as far as dispatch is * concerned. * * So the clearing is a daemon-owned act, through the same tracker port the * dispatcher writes labels with, and the brief's rule stays absolute. That * absoluteness is worth more than the exception it replaces: orphan detection * is only trustworthy while every state label on the tracker was written by * this package. The re-queue is the same port and the same principle: the * queue label goes back on by default (#184) so an answered block is * dispatchable again, and it stays off under `--no-requeue` or while any run * for the issue is still active. * * Nothing here writes to the run history, and that is a decision rather than * an omission. `RunState` describes what a worker process did; an answer is the * one event that happens outside every run, so no member fits it — folding it * into `merged` or `killed` would make `status` describe a run that never * reached either. The one store write is the label projection outbox (#201): * a decided label change is durable locally and projected with retry, so a 403 * defers it instead of throwing the verb, and the daemon's next tick applies it * even if this process dies before it does. Eligibility is read off the label * set an issue effectively carries — physical labels overlaid with that outbox — * and never off a run row, so the store has nothing else to say here. Leaving * history alone keeps both budgets honest: a block consumes an operational * continuation, while a real implementation failure consumes the separate * failed-attempt budget. */ import { hasContinuationBudget, hasFailedAttemptBudget } from "./admission.ts"; import { projectLabels } from "./label-projection.ts"; import { LIVE_STATES } from "./store.ts"; import { wakeDispatch } from "./wake.ts"; import type { Caps, ProjectConfig, RunRecord, Store, Tracker } from "./types.ts"; /** What one `unblock` did, and the state it found around it. */ export interface UnblockOutcome { /** State labels the tracker was asked to drop. */ cleared: string[]; /** Total run segments, retained for history and sequence numbering. */ attemptsUsed: number; failuresUsed: number; continuationsUsed: number; /** Newest attempt, when the store has one for this issue at all. */ latest?: RunRecord; /** Whether any run for the issue is still in an active state — the answer * to "will the dispatcher hold this issue as issue-active?" that #178 * found this verb guessing at. */ active: boolean; /** What the resident-daemon wake did, when this unblock restored the queue * label and therefore made work claimable now (#878). Absent when nothing * was requeued: there would be nothing for a pass to find. */ woken?: string; /** Set when a live worker (a claimed/running run) is on the issue. The * dispatcher holds a worker-backed run as issue-active unconditionally, * while worker-free pushed occupancy can bypass it (#175) — so this is * the signal the queue label and the in-flight wording follow. */ live?: true; /** Set when nothing was cleared because the newest attempt's work exists * only in its worktree. Carries the salvage failure verbatim. */ refused?: string; /** Set when `--force` recorded an operator's acceptance of that loss. */ forced?: true; /** Set when the queue label was re-added (the default); absent on * --no-requeue, on the refusal path, when a live worker (claimed or * running) is still on the issue, and when a spent budget withheld the * re-queue (requeueWithheld). A pushed-green/pushed-pending occupancy * is worker-free, so the label is still restored there (#175 bypasses it; * `isEligible` needs the label once the PR resolves). */ requeued?: true; /** Set when the re-queue was withheld because the issue's failed-attempt or * continuation budget is spent — the same admission gate the dispatcher * holds on (failed-attempts / continuations), so the queue label would * manufacture a candidate that can never be admitted (#348). The lifecycle * clears still happened; only the add was skipped. */ requeueWithheld?: true; /** Set when `--no-requeue` skipped the queue-label re-add. */ requeueSkipped?: true; /** Set when the tracker refused one or more of this verb's label ops, so * the label sync is owed rather than done: the intended label state is * durable in the store and the daemon retries it (#201). This says nothing * about claimability — `listReady` reads the tracker by label, so an issue * is dispatchable only once the queue label is physically present, which a * pending add has not delivered and `--no-requeue` never enqueues at all. * The number is what is still pending. */ labelSyncQueued?: number; } /** * Drop the state labels the issue could be carrying, as far as the store can * prove they are droppable. * * The two terminal labels come off unconditionally, because the tracker is the * only source of truth for which one is set and this process cannot read that * back through the Tracker port — inferring it from the newest run row would be * a guess that goes wrong exactly when a human has relabelled something by * hand. Removing a label an issue does not carry is a no-op: `gh issue edit * --remove-label` exits 0 on an absent label (verified against gh 2.97.0), and * the adapter swallows the 404 older paths return for one. * * `agent:in-progress` comes off only when the newest run row is terminal, and * that condition is the whole safety argument. The label means a worker process * exists, and clearing it from under a live run is how two workers end up on * one issue, so it used to be excluded outright. What that cost showed up on * 2026-08-09: veltro#331's newest row had already settled to `failed` (its PR * #332 was closed unmerged), `unblock 331` reported `cleared agent:blocked, * agent:failed` and `next tick eligible again`, and an immediate authoritative * `gh issue view 331` still showed `agent:in-progress` — the one label that * actually decides eligibility, with no verb anywhere able to remove it (#18). * A terminal newest row is proof the process is gone rather than an opinion * about it: it is written when the worker exits, by `reconcileOrphanedRuns` at * startup for a worker that died with its daemon, or by settlement once the PR * resolved. Once it is there, the interlock has nothing left to guard. * * No run row at all is deliberately *not* that proof, so that case still leaves * the label alone. A missing row means the store never saw the run, which is * indistinguishable from here to a claim that raced a store write — and being * wrong in that direction is the two-workers bug, where being wrong in the * other direction is a stuck issue an operator is already looking at. On the * paths the daemon does own it releases the label itself, next to the store * transition that proves the run ended (`releaseInProgress` in `daemon.ts`). */ export async function unblockIssue( project: ProjectConfig, tracker: Tracker, store: Store, issue: number, opts: { force?: boolean; requeue?: boolean } = {}, caps: Caps, ): Promise { const requeue = opts.requeue !== false; // Read before any label is touched: terminality is the whole of the argument // for clearing in-progress, so the row that carries it decides the set. const latest = store.latestRun(project.name, issue); const terminal = latest !== undefined && !LIVE_STATES.includes(latest.state); // Two different occupancies, and the label decisions each follow its own: // `active` (ACTIVE_STATES) is what *occupies the issue* — a pushed-green or // pushed-pending run has no worker process but its PR is live, so dispatch // must not start a fresh attempt behind it. `live` (LIVE_STATES) is what // *holds a worker process* (claimed/running). The queue label follows // `live`: the dispatcher only holds a claim as issue-active for a // worker-backed run (#175 bypasses worker-free pushed-green rows), so an // issue whose occupancy is pushed-only is genuinely re-queueable and // `isEligible` requires the label once that PR resolves closed-unmerged. const active = store.activeRuns(project.name).some((r) => r.issue === issue); const live = store.liveRuns(project.name).some((r) => r.issue === issue); const counts = { attemptsUsed: store.attemptsFor(project.name, issue), failuresUsed: store.failuresFor(project.name, issue), continuationsUsed: store.continuationsFor(project.name, issue), active, }; // The dispatcher's own admission gates, as daemon.ts computes them (#348): // every failed-implementation slot spent, or the continuation budget // exceeded, and the issue is held (`failed-attempts` / `continuations`) on // every subsequent tick forever. A re-queued issue in that state is exactly // the permanent admission hold this verb must not manufacture, so the // re-queue half below is decided on the same predicates that decide the hold. const budgetSpent = !hasFailedAttemptBudget(counts.failuresUsed, caps.maxAttemptsPerIssue) || !hasContinuationBudget(counts.continuationsUsed, caps.maxContinuationsPerIssue); // The one case where this verb refuses. Clearing the labels here re-queues an // issue whose next claim starts by force-removing the worktree that holds the // only copy of the last attempt's work (#118) — and unlike every other state // this function reasons about, that is unrecoverable. `--force` is the way // through, and it is deliberately a separate keystroke rather than a prompt // the operator can wave past: it records, on the row, that a human accepted // the loss or recovered the tree themselves. Narrowed to the row rather than // a boolean so the reason printed and the acknowledgement written are // provably about the same run. const held = latest !== undefined && latest.salvageError !== undefined && latest.salvageAckAt === undefined ? latest : undefined; if (held !== undefined && opts.force !== true) { return { cleared: [], ...counts, latest: held, refused: held.salvageError }; } if (held !== undefined) store.updateRun(held.id, { salvageAckAt: Date.now() }); const cleared: string[] = []; const ops: { issue: number; op: "add" | "remove"; label: string }[] = []; for (const label of new Set([ project.stateLabels.blocked, project.stateLabels.failed, ...(terminal ? [project.stateLabels.inProgress] : []), ])) { ops.push({ issue, op: "remove", label }); cleared.push(label); } // The re-queue half of the verb. Clearing the state labels makes the issue // *eligible*, but the dispatcher never sees an issue that does not carry the // queue label — without this the unblock ends in the same alive-but-inert // state it exists to end (#184). The label goes back on by default; // `--no-requeue` is the "about to close it" case and leaves dispatch alone. // Never under a *live worker*: the dispatcher holds a worker-backed run as // issue-active anyway, and restoring the label under one is exactly the // #178 inverted state. A pushed-green/pushed-pending run is worker-free — // its PR is live but no process writes to its branch — so a pushed // continuation still gets the label: the dispatcher bypasses worker-free // pushed-green rows (#175), and `isEligible` requires the queue label once // the PR resolves closed-unmerged. And never when the failed-attempt or // continuation budget is spent (#348): the dispatcher holds such an issue // forever, so the label would claim a candidacy the next admission cannot // deliver — the clears still happen, the add is withheld. let requeued: true | undefined; let requeueWithheld: true | undefined; let requeueSkipped: true | undefined; let labelSyncQueued: number | undefined; if (!live) { if (requeue) { if (budgetSpent) { requeueWithheld = true; } else { ops.push({ issue, op: "add", label: project.queueLabel }); requeued = true; } } else { requeueSkipped = true; } } // Apply through the same projector the daemon drains (#201): the clears and // the queue restore land in enqueue order, and a tracker that refuses (403, // rate limit) defers them with backoff instead of throwing the verb — which // is the concrete #184/#198 kill: `unblock` stops breaking on a 403. The // intended label state is durable; the daemon converges the tracker. The // issue only becomes claimable once the queue label itself lands — // `listReady` queries by label, so a pending queue-label add is invisible to // dispatch until projection succeeds. if (ops.length > 0) { store.enqueueLabelOps(project.name, ops); await projectLabels(store, tracker, project); // How much of THIS issue's sync is still owed (not the whole fleet's outbox): // a backlog on other issues is not this verb's lag to report. const stillPending = store.pendingLabelOpsFor(project.name, issue).length; if (stillPending > 0) labelSyncQueued = stillPending; } // An unblock that actually restored the queue label has made work claimable // now, so the resident daemon is poked instead of leaving the issue for the // next scheduled pass (#878). Only on a real restore: a withheld or skipped // requeue changed nothing to dispatch, and a wake following a no-op is a claim // of progress that did not happen. Best effort by construction — the label // mutation already committed, so an unreachable daemon is a reported line, // never an error here. const woken = requeued !== undefined ? await wakeDispatch(project.name) : undefined; return { cleared, ...counts, ...(woken === undefined ? {} : { woken }), ...(live ? { live: true as const } : {}), ...(latest === undefined ? {} : { latest }), ...(held === undefined ? {} : { forced: true as const }), ...(requeued === undefined ? {} : { requeued }), ...(requeueWithheld === undefined ? {} : { requeueWithheld }), ...(requeueSkipped === undefined ? {} : { requeueSkipped }), ...(labelSyncQueued === undefined ? {} : { labelSyncQueued }), }; } /** * What the operator reads back. It promises a re-claim only when one can * actually happen, because a promise made blindly sends someone away believing * work had resumed and they find out by waiting for it. Three things withhold * the re-claim promise: a live run still owns the issue through * `agent:in-progress`, a spent attempt budget makes the next tick escalate * rather than dispatch, and — since the in-progress label is only released for * a terminal run — an issue with no run row at all, where that label may still * be sitting there unread. A spent budget also withholds the queue label * itself, on the same predicate the dispatcher holds on (#348): re-adding it * would manufacture a candidate the next tick can never admit, and the report * says so instead of claiming a restore that did not happen. #18 was * filed against this function saying `next tick eligible again` in a case where * it was not, so the wording is a contract rather than prose. * * A refusal is the loudest thing this verb prints, and it prints instead of * everything else: the operator asked to re-queue an issue whose only copy of * real work is a directory, and the next line they read has to be the one that * stops them typing the same command again with `--force` on the end. */ export function formatUnblock( issue: number, o: UnblockOutcome, project: ProjectConfig, caps: Caps, ): string { const latest = o.latest; if (o.refused !== undefined) { return [ `#${issue}: REFUSED — nothing was cleared`, ` reason attempt ${latest?.attempt ?? "?"} could not commit its work: ${o.refused}`, ` only copy ${latest?.worktree === undefined || latest.worktree === "" ? "(worktree path not recorded)" : latest.worktree}`, " why re-claiming this issue removes that worktree, and the work is not on any ref", ` recover inspect the tree and commit or copy what matters, then re-run with --force`, ].join("\n"); } const lines = [`#${issue}: cleared ${o.cleared.join(", ")}`]; // When the tracker refused the sync, NO label is provably applied — the // "restored / eligible again" claims below would be a lie, so they branch to // the queued/retry wording instead (#201). const labelsPending = o.labelSyncQueued !== undefined; if (o.labelSyncQueued !== undefined) { lines.push( ` label sync queued (${o.labelSyncQueued} pending) — the daemon retries until the tracker matches`, ); } if (o.forced === true) { lines.push( ` forced the unsalvaged worktree was accepted as lost or already recovered by hand — ` + "the next claim removes it", ); } if (latest === undefined) { lines.push(" runs none recorded — the terminal labels were cleared anyway; eligibility is read off the tracker"); } else { lines.push(` runs ${o.attemptsUsed}, newest ${latest.state}`); lines.push(` failures ${o.failuresUsed} of ${caps.maxAttemptsPerIssue}`); lines.push(` continuations ${o.continuationsUsed} of ${caps.maxContinuationsPerIssue}`); } if (latest !== undefined && LIVE_STATES.includes(latest.state)) { // The newest run is live, so `active` is necessarily true and the queue // label was NOT re-added (unblockIssue only re-queues when no run for the // issue is active). Say that, like the sibling-active branch below does — // the label outcome is part of the contract (#184), and a live newest run // reaches this branch instead of `else if (o.active)`. lines.push( ` in flight attempt ${latest.attempt} is ${latest.state}, so the issue keeps ` + `"${project.stateLabels.inProgress}" until it ends — nothing is re-claimed before then`, ` queue "${project.queueLabel}" not restored — a run is still active; re-run unblock once it settles`, ); } else if (!hasFailedAttemptBudget(o.failuresUsed, caps.maxAttemptsPerIssue)) { lines.push( ` next tick not eligible: all ${caps.maxAttemptsPerIssue} failed attempts are spent. ` + "Rewrite the issue or raise maxAttemptsPerIssue.", ); if (o.requeueWithheld === true) { lines.push( ` queue "${project.queueLabel}" withheld — re-adding it would manufacture a candidate the ` + "dispatcher holds as failed-attempts forever; rewrite the issue or raise maxAttemptsPerIssue before re-queueing", ); } } else if (!hasContinuationBudget(o.continuationsUsed, caps.maxContinuationsPerIssue)) { lines.push( ` next tick not eligible: the ${caps.maxContinuationsPerIssue}-continuation budget was exceeded. ` + "Inspect progress or raise maxContinuationsPerIssue.", ); if (o.requeueWithheld === true) { lines.push( ` queue "${project.queueLabel}" withheld — re-adding it would manufacture a candidate the ` + "dispatcher holds as continuations forever; inspect progress or raise maxContinuationsPerIssue before re-queueing", ); } } else if (latest === undefined) { lines.push( ` next tick eligible once the issue carries "${project.queueLabel}" and no state label — with no ` + `run row to prove the worker is gone, "${project.stateLabels.inProgress}" was left in place, and ` + "on its own it keeps the issue ineligible", ); } else if (latest.state === "pushed-green" || latest.state === "pushed-pending") { // #178: an unblocked issue whose newest run pushed a green PR is not "free // for a fresh attempt" — the dispatcher continues the pushed run's branch, // and only once the PR resolves does the issue settle (#175). Say that // instead of promising a re-claim the next tick withholds. lines.push( latest.prUrl === undefined ? labelsPending ? ` next tick label sync queued (${o.labelSyncQueued} pending) — the daemon retries until the tracker matches` : o.requeued === true ? ` next tick eligible again — "${project.queueLabel}" restored (no-op if it was already present; ` + "the dispatcher still applies its open-PR check at claim time)" : ` next tick eligible again, as long as the issue still carries "${project.queueLabel}" ` + `(newest run is ${latest.state} with no recorded PR; the dispatcher still applies its open-PR check at claim time)` : ` next tick eligible as a continuation of ${latest.prUrl} — the pushed run stays active until ` + "that PR resolves; dispatch continues on its branch", ); } else if (o.active) { // A run other than the newest still occupies the issue. Two shapes hide // under `active`, and they demand opposite reports: a *live worker* // (claimed/running) holds the issue unconditionally — #178's misleading // case — while worker-free pushed runs (pushed-green/pushed-pending) are // the continuation shape, because #175 lets the dispatcher admit // pushed-green rows whose workers are gone. The queue label follows the // same split: off under a live worker, restored under pushed-only // occupancy. if (o.live === true) { lines.push( ` in flight a live worker is still on this issue, so the dispatcher holds it until that run settles — ` + `nothing is re-claimed before then ("${project.stateLabels.inProgress}" stays unless already released)`, ` queue "${project.queueLabel}" not restored — a live worker is still on this issue; re-run unblock once it settles`, ); } else { lines.push( ` next tick no live worker is on this issue — its active run(s) are worker-free pushes; ` + "dispatch continues their branch once its holds and open-PR check clear", ); if (labelsPending) { lines.push( ` queue label sync queued (${o.labelSyncQueued} pending) — the daemon retries until the tracker matches`, ); } else if (o.requeued === true) { lines.push(` queue "${project.queueLabel}" restored (no-op if it was already present)`); } } } else { lines.push( labelsPending ? ` next tick label sync queued (${o.labelSyncQueued} pending) — the daemon retries until the tracker matches` : o.requeued === true ? ` next tick eligible again — "${project.queueLabel}" restored (no-op if it was already present; ` + "the dispatcher still applies its open-PR check at claim time)" : ` next tick eligible again, as long as the issue still carries "${project.queueLabel}" ` + "(the dispatcher still applies its open-PR check at claim time)", ); } if (o.requeueSkipped === true) { lines.push(` queue "${project.queueLabel}" left untouched (--no-requeue)`); } // What happens next in time, not just in principle (#878): a restored label // that woke the daemon claims on an immediate pass, and one that could not // reach it waits for the scheduled pass. Both are honest; a report that says // "eligible again" without saying which invites the reader to assume the // faster one. if (o.woken !== undefined) lines.push(` dispatch ${o.woken}`); return lines.join("\n"); }