Skip to content

fix(work-unblock): leave outage-lost queued runs out of the re-fire room - #15478

Merged
kyle-sexton merged 4 commits into
mainfrom
fix/work-unblock-stale-queued-runs
Nov 12, 2026
Merged

kyle-sexton merged 4 commits into
mainfrom
fix/work-unblock-stale-queued-runs

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

No related issue: hub finding — outage-lost queued runs block the kill-switch re-fire pacer (sweep run 43123797947)

session: GitHub Support

Summary

The work-unblock sweep's kill-switch re-fire pacer counted every unfinished work-resolve-issue run. Twelve runs lost in GitHub's 2026-11-11 Actions outage have stayed queued since 20:30-20:43Z and will never start. Sweep run 43123797947 logged 12 work-resolve-issue run(s) unfinished, kill-switch re-fire room 0 of 6 and deferred all 13 paced re-fires, so the issue lane sat idle with agent-ready issues waiting.

Fix

  • New limits.stale-queued-minutes (default 60): a run still at status queued this many minutes after its created_at is left out of the count behind refire-queue-per-slot's room. The sweep logs one notice with how many it left out.
  • Why queued this long means lost: the admit job runs on a GitHub-hosted runner (work-resolve-issue.yml runs-on: ubuntu-24.04), and the slot wait is a job-level concurrency group on resolve-issue that runs after admit and pick-slot. A run waiting for a slot has therefore already started. A run that stays queued never got a runner for its first job.
  • Default 60 is a judgment call: a healthy run leaves queued within minutes on a hosted runner, and 60 leaves a wide margin.
  • runsInFlight now returns {title, status, created} records from the same five status listings, and both callers use them:
    • The kill-switch re-fire pacer leaves stale queued runs out of its count.
    • recurrenceRefusal (reopened fix issues) leaves stale queued runs out of its "runs that name no issue" count. Before, the lost runs refused every reopened fix issue with "cannot tell whether a lane run for it is in flight". Its title match (endsWith(" #N")) still counts every run, stale or not, because a run that names the issue may still be that issue's run, and ignoring it could relabel the issue twice.
  • An unreadable created_at parses to NaN and is never treated as stale, so that run still counts (fails safe).
  • The key goes in the usual places: limits.mjs, resolve.mjs limitsOf, resolve.test.mjs, the schema (1-1440), pr-pipeline.yaml with its reason, the README limits table, and a pr-pipeline changelog fragment (bump: minor).

Per R461: zero added GitHub calls. The filter reads created_at from the workflow_runs payload the five existing runs?status=… listings already return. The diff adds no actions(/github( call site.

Verification

node --test .github/scripts/work-unblock/label-unblocked.test.mjs .github/actions/resolve-config/resolve.test.mjs .github/actions/resolve-config/live-config.test.mjs: tests 621, pass 621, fail 0.

New cases:

  • 3 fresh unfinished runs plus 12 runs queued exactly 60 minutes: room 3 of 6, as with no lost runs, and the notice names 12.
  • Runs that still fill the room: queued 59:59, in_progress or pending for four hours, and queued with no readable creation time.
  • stale-queued-minutes: 120 moves the cutoff.
  • Reopened fix issue: 12 stale queued runs that name no issue no longer refuse the relabel. One fresh run that names no issue still refuses it. A stale queued run whose title ends #30 still counts as in flight.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01VQb9Q1HHi6JACp77x2ZbE7

kyle-sexton and others added 3 commits November 11, 2026 19:25
The kill-switch re-fire pacer counted every unfinished work-resolve-issue
run. Runs GitHub lost in an Actions outage stay `queued` and never start,
so they held the room at 0 and the issue lane idle with agent-ready
issues waiting (sweep run 43123797947: 12 unfinished, room 0 of 6).

A run still `queued` limits.stale-queued-minutes (default 60) after it
was created is no longer counted toward that room, and the sweep logs
how many it left out. The reopened fix issue's in-flight check still
counts every unfinished run. No new GitHub call: the filter reads
created_at from the run listings the sweep already makes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQb9Q1HHi6JACp77x2ZbE7
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 12, 2026 00:30
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T00:32:17.722765Z cd628bb Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

… check

recurrenceRefusal counted every unfinished run that names no issue, so
outage-lost queued runs refused every reopened fix issue's relabel with
"cannot tell whether a lane run for it is in flight". A run still queued
limits.stale-queued-minutes after it was created never starts, so it no
longer counts there. A run whose title names the issue still holds it,
stale or not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQb9Q1HHi6JACp77x2ZbE7
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed abf09a4fca3474243681103bf25c6701f3a26047: read 6 of 6 changed file(s); 0 finding(s), 0 as inline comments.

Code review (incremental, blocking only): PR #15478, cd628bb → abf09a4

Blocking findings: none. I read every hunk of all 6 files in the incremental scope. I also read the unchanged code those hunks call: runsInFlight, staleQueued, recurrenceRefusal, its one caller in restartLimited, and the test helpers reopened and laneRun. This lane already reviewed an earlier head, so this review reports blocking findings only.

The earlier finding is fixed. At cd628bb the description and the code disagreed (the prior summary comment). Now they agree:

  • recurrenceRefusal (L2152-L2155) still holds the issue for any run whose title ends #N, stale or not.
  • It now leaves stale queued runs that name no issue out of the untitled count.
  • now and limits reach it from restartLimited's own parameters (L2480, L2607). That is its only call site.
  • staleQueued gets the same {title, status, created} records the re-fire pacer uses.
  • An unreadable created_at gives NaN, so the run is not treated as stale and still counts.
  • The no-token path (runs === null) still returns before the .some and .filter calls.

Tests. The new test covers the three cases the description lists:

  • 12 stale untitled runs relabel the issue.
  • One fresh untitled run refuses the relabel.
  • A stale run named #30 still holds the issue.

The changelog fragment, the pr-pipeline.yaml comment, the README row and the schema description all match the new behavior.

I dropped one advisory point because this review is blocking only. The comment on an unchanged line, label-unblocked.test.mjs:4890, still says only the re-fire room ignores stale queued runs. The reopened-fix check now ignores them too when they name no issue. The test case it describes still passes, because that run names #30.

Verification. The author reports the tests pass. That is their claim and I have not checked it: this lane has no shell, so I ran nothing.

There are no paved-path, plugin-manifest or settled-thread items. The PR context contains no embedded instructions. The change costs nothing: it adds no GitHub calls and no extra lane runs.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed abf09a4fca3474243681103bf25c6701f3a26047: read 6 of 6 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #15478, incremental (cd628bb → abf09a4)

No security issues found. This review was blocking only: this lane already reviewed an earlier head, so it reports only CRITICAL and IMPORTANT findings. I read every hunk of all 6 files in the incremental scope.

What changed since the last review

recurrenceRefusal is the in-flight check that runs before a reopened fix issue is relabeled. It now leaves a stale queued run out of its "unfinished runs that name no issue" count, where a stale queued run is one still queued limits.stale-queued-minutes after it was created (label-unblocked.mjs#L2155). The title match (endsWith(" #N")) still counts every run, stale ones included. The code now does what the PR description says, which settles the description mismatch the last review noted. The rest of the change is a new test, the changelog fragment, and doc and schema wording.

Checks

  • Scope wiring. now and limits are already parameters of restartLimited (L2480), and the pacer at L2660 already uses them with the same staleQueued helper (L2099). A created_at that cannot be read gives NaN, so the run is never treated as stale and still counts. That fails safe.
  • Trust boundary. The check only applies to issues opened by ALERT_BOT that match FIX_ISSUE, which the unchanged guard at the top of recurrenceRefusal enforces. Its new inputs, status and created_at, come from GitHub's own workflow_runs payload. No pull request, issue or comment author controls them. The worst case of the narrower check is relabeling an issue whose run was actually lost, which could mean a duplicate lane run. That costs spend, not a leak, compromise or charge.
  • Trust controls. docs/conventions/pr-pipeline.yaml (trusted path) changes only a comment under limits.stale-queued-minutes. No lane slot, activity effect, token scope or merge setting changed, so the broker mints nothing wider. No merge-hold file, kill-switch action or trusted-trigger action is touched.
  • Embedded text. The PR context contains no instructions aimed at the lane.

Spend (not a hold, not counted as a finding)

Reopened fix issues that outage-lost runs used to hold can now be relabeled, which can start more work-resolve-issue lane runs. That spends prepaid credits or subscription usage only.

critical-findings: 0

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 7ca0046 Nov 12, 2026
56 checks passed
@kyle-sexton
kyle-sexton deleted the fix/work-unblock-stale-queued-runs branch November 12, 2026 01:12
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant