Skip to content

fix(pr-pipeline): resend pr-merge dispatches on a TLS handshake timeout and retry the rerun and alert POSTs - #10266

Merged
kyle-sexton merged 2 commits into
mainfrom
fix/9355-pr-merge-dispatch-retries
Oct 26, 2026
Merged

kyle-sexton merged 2 commits into
mainfrom
fix/9355-pr-merge-dispatch-retries

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Closes #9355

Summary

Four follow-ups to pr-merge.yml's dispatch retry and the rate-limit tests, deferred from review of #9350 and #9354.

  1. Hand-written gh rate-limit stderr in tests is replaced with the text run 39906176584 logged at 2026-10-23T05:30:57Z (dispatch-full-run, "Start the full CI run on main"). The test the issue named, test_rate_limited_is_a_429_or_a_403_that_names_a_rate_limit in test_babysit_gh.py, and gh_rate_limited itself were removed by #9456. The capture now feeds the tests that judge stderr alone: the full-run wait and the alert send in workflow.test.mjs, the re-run step in merge-job.test.mjs, test_babysit_postmerge.py's role-label read and test_babysit_verify_evidence.py's not-transient list. No capture of a secondary limit, a bare 429 or a GraphQL limit was found, so those strings stay hand-written.
  2. Both full-run dispatch loops also send again on Go's net/http: TLS handshake timeout. That error comes before the request is written, so a resend cannot start a second run. The rate-limit stops from #9920 run before the resend check: a 403 or 429 still stops the step red and is never sent again.
  3. The warning for an exhausted retry now says that a scheduled pr-merge run starts a run only for a lane merge, and that any other merge waits for the next push or for pr-require-checks' own schedule.
  4. The rerun-failed-jobs POST and the post-merge-demotion-alert dispatch POST now get the Q677 retry. fix(source-control): stop babysit GitHub calls once a rate-limit pool is exhausted #9920 already sent the re-run POST through api_budget_wrap_gh, but that wrapper only stops on a rate limit and never retries. The alert POST had neither a retry nor a rate-limit stop.

Fix

  • .github/workflows/pr-merge.yml:
    • The resend pattern is now dial tcp|tls: (handshake|first record)|net/http: TLS handshake timeout, in both dispatch loops, the re-run POST and the alert send.
    • The re-run POST gets at most three attempts, after a capped full-jitter wait (up to 10 s, then 20 s). The step's existing stopped check runs before any resend.
    • The alert send gets the same retry. A rate-limited alert send is sent once and stops red.
    • The budget comments for both jobs now count the new waits (up to 30 s each).
  • Tests:
    • workflow.test.mjs: a TLS handshake timeout is resent, in both jobs. A rate limit after the timeout still stops after two POSTs. The warning's new wording. The alert send's retry, its three-attempt cap, its rate-limit stop and no resend after a 502.
    • merge-job.test.mjs: the re-run POST's retry, its cap and no resend after a refusal. A rate limit after an unsent POST stops red.
  • Changelog fragments: pr-pipeline (patch) and source-control (none: test strings only).

Verification

  • Before the change, the 7 new tests fail when run against the old pr-merge.yml. After it, all pass.
  • node --test .github/scripts/pr-merge/*.test.mjs: 421 pass, 0 fail.
  • node --test .github/actions/log-api-budget/*.test.mjs: 102 pass, 0 fail.
  • babysit-prs Python suite (python3 -m unittest discover -s tests -t .): 3043 tests OK.
  • scripts/run-ruff.sh check and format --check on the touched Python: clean.
  • actionlint .github/workflows/pr-merge.yml: clean.
  • scripts/check-affected.sh --tests: 13 passed, 1 skipped.

Left out: a live run of the changed steps against GitHub (no TLS timeout or rate limit can be produced on demand); the stubbed-gh tests cover them.

Related

  • #9354 and #9350: the reviews that raised these items.
  • #9346: item 4 was first noted there.
  • #9920: the rate-limit stops this change keeps.
  • #9549: not covered here (the JS lane scripts' fetch clients). It shares only the Q677 retry rule.

🤖 Generated with Claude Code

kyle-sexton and others added 2 commits October 26, 2026 07:29
…ut and retry the rerun and alert POSTs

Both full-run dispatch loops also resend on Go's `net/http: TLS handshake
timeout`, which fails before the request is written; a rate limit still
stops the step red. The rerun-failed-jobs POST and the
post-merge-demotion-alert dispatch take the same Q677 retry (at most
three attempts, capped full-jitter backoff, only for a request that never
left); a rate-limited alert send stops red. The exhausted-retry warning
now says the scheduled pickup re-dispatches only lane merges. Tests quote
gh's rate-limit stderr as run 39906176584 logged it.

Closes #9355

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 26, 2026 12:40
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T12:42:31.973060Z 8774f5a Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed 8774f5aecf16689955c71645a71bb8c3d92a8a9c: 0 finding(s), 0 as inline comments.

No findings. I read the full diff, the PR context and the changed regions of .github/workflows/pr-merge.yml. I ran nothing: this lane has no shell, so the pass/fail counts in the PR description are author-claimed and unverified.

Closing issue #9355, each criterion checked against the diff:

  1. Real gh rate-limit stderr in tests: met. The captured run 39906176584 text is named in workflow.test.mjs, merge-job.test.mjs and the two babysit Python tests. The PR explains why test_babysit_gh.py is no longer the target: revert(pr-pipeline): stop falling back to the lanes App token on GITHUB_TOKEN exhaustion #9456 removed that test and gh_rate_limited.
  2. Resend on net/http: TLS handshake timeout: met. The pattern is added to both dispatch loops (pr-merge.yml:~316 and ~1903). The rate-limit stops from fix(source-control): stop babysit GitHub calls once a rate-limit pool is exhausted #9920 still run before the resend check. workflow.test.mjs covers the resend in both jobs, and a rate limit after the timeout still stops after two POSTs.
  3. Exhausted-retry warning reworded: met. Both jobs use the new wording, and a test asserts it for both.
  4. Q677 retry on the rerun-failed-jobs POST and the alert POST: met.
    • Both retry at most three times, with a capped full-jitter wait of up to 10 s, then 20 s, only when gh's error shows the request never left.
    • In the re-run step, stopped runs before the resend check, so a rate limit exits red first.
    • In the alert send, the rate-limit grep runs before the resend check.
    • Tests cover the retry, the three-attempt cap, a refusal or 502 that is not resent, and a rate limit after an unsent send.

Shell logic. cmd && break and cmd && exit 0 do not trip set -e. After the loops, $try holds the attempt number, so the "after N attempt(s)" messages are accurate. The loop's wait is reassigned by the later for wait in … read-back loop, which is harmless.

Changelog fragments. Patch for pr-pipeline and none for the test-only source-control change both fit the diff. No plugin manifests changed, so no manifest <field> lines apply.

Spend. The change adds up to 30 s of retry waits per step inside existing job timeouts and WAIT_CAP budgets. That is runner time and API calls only, so there is no hold.

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 8774f5aecf16689955c71645a71bb8c3d92a8a9c: 0 finding(s), 0 as inline comments.

No security issues found in this change.

critical-findings: 0

What I read: the whole diff (7 files) and PR context. For .github/workflows/pr-merge.yml, a trusted path, I also read the head file's re-run step in full (pr-merge.yml lines 363-448).

Trust controls: none of the control classes is touched.

  • The change does not edit merge-holds.mjs, enable-auto-merge.mjs, superseded.mjs, clear-escalations.mjs, babysit_risk.py, babysit_verify_evidence.py or docs/conventions/pr-pipeline.yaml.
  • The pr-merge.yml hunks are confined to the dispatch-full-run and post-merge-dispatch jobs' dispatch, re-run and alert-send steps. They do not touch any step that runs a merge-hold script or sets such a script's environment.
  • No permissions:, token scope, kill-switch or trusted-actor logic changes.

Logic checked in the new retry loops:

  • Resend condition: a request is sent again only when gh's stderr matches dial tcp, tls: (handshake|first record) or net/http: TLS handshake timeout. Those errors occur before any request byte is written, so a resend cannot start a duplicate run or a second re-run.
  • Rate limit precedence:
    • In the re-run step, stopped runs before the resend check, so a 403/429 rate limit still exits red.
    • In the alert send, the rate-limit grep runs before the resend check and exits 1.
    • In both loops a rate limit is never sent again.
  • Loop bounds: both loops are capped at three attempts, with a jittered wait of at most 10 s, then 20 s. set -e does not break the cmd && break and cmd && exit 0 forms, because they sit in an && list.
  • Alert send: it still fails the job when the send is not delivered, as before. A success still exits 0, and a non-failure conclusion exits 0 without a send, as before.
  • Shell injection: no new ${{ }} expressions appear inside run: blocks. The variables used are the existing env-bound RUN_ID (regex-validated in the re-run step), REPO and head_sha.
  • Logging: stderr from gh is echoed to the job log, as the existing code already did. It carries no credential.

Tests and changelog: the other files are test-only (stubbed gh) and changelog fragments. They carry a captured GitHub rate-limit message with a request ID and timestamp, which is not sensitive.

Spend: the extra retry waits (up to 30 s per job) spend only Actions time already within the existing budget, which the diff's comments re-count. They use no paid plan or billing setting, so there is no hold.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 26, 2026
Merged via the queue into main with commit 6596979 Oct 26, 2026
72 checks passed
@kyle-sexton
kyle-sexton deleted the fix/9355-pr-merge-dispatch-retries branch October 26, 2026 14:50
kyle-sexton added a commit that referenced this pull request Oct 26, 2026
Brings in #10266 (pr-merge dispatch and POST retries); merged cleanly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore(pr-pipeline): pr-merge dispatch and read-token review follow-ups

1 participant