Skip to content

fix(pr-pipeline): round a Retry-After stop up so a short one is waited out - #17560

Open
kyle-sexton wants to merge 2 commits into
mainfrom
fix/rate-guard-retry-after-ceil
Open

kyle-sexton wants to merge 2 commits into
mainfrom
fix/rate-guard-retry-after-ceil

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

No related issue: main-red run 44631001257

Summary

Main went red in run 44631001257: "a read that rides out a short rate limit leaves the process green" in check-trusted-trigger.test.mjs got a 403 with retry-after 1, and instead of waiting the read failed the step (no call to it before 2026-11-19T22:51:44Z (retry-after); this step fails, logged at 22:51:44.010).

The cause is a clock race in the rate guard, not test environment. The failing step's env listing has no API_BUDGET_* variable, and waitBudgetMs (rate-guard.mjs:98-101) can only raise the 60 s budget.

  • classify (rate-guard.mjs:510-512) stamped a Retry-After stop as floor(now / 1000) + retryAfter. An answer at x.99 s got a stop only a few ms ahead.
  • observe captures now for classify, awaits the body reads, then decides the resend with waits(url) (rate-guard.mjs:845) on a fresh now(). Once the clock has crossed the second, until() returns null, waits is false, and a read the client would have resent after a second fails the step.
  • check-trusted-trigger.mjs:271 and the guard loop (rate-guard.mjs:911) make the same waits decision and miss the same way.

Fix

The Retry-After candidate now rounds the answer's time up (Math.ceil(now / 1000) + retryAfter), so the stop is always at least retry-after seconds out, as GitHub asks. The x-ratelimit-reset and fallback paths are unchanged. rate-guard.vectors.json uses whole-second clocks, so no vector changes and the other ports keep parity.

A regression test in check-trusted-trigger.test.mjs drives createGitHub on a latch clock that ticks 5 ms per read, starting 970 ms into a second. latchClock gains start and tick options for it. It sits beside the ride-out test because adding it to rate-guard.test.mjs crossed the 1000-line size check.

Verification

  • The new test fails on the old code with the same 403 GitHubError main saw, and passes with the fix.
  • A sweep of the clock start from 900 to 1000 ms in 5 ms steps: on the old code, starts 965, 970 and 975 fail; with the fix, every start passes.
  • node --test on rate-guard.test.mjs (167/167), check-trusted-trigger.test.mjs (486/486), and log-api-budget.test.mjs with pr-merge/merge-job.test.mjs all pass.
  • No other ...process.env spawn in these test files shares the cause: the race was in the guard, so any real-clock test that hits a short retry-after is covered by the code fix.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01KeBVgbERyakZ33T4AetfRj

kyle-sexton and others added 2 commits November 19, 2026 18:15
…d out

classify stamped a Retry-After stop as floor(now) + retry-after. An
answer late in a second got a stop that ended a few ms later, before
observe re-read the clock to decide the resend, so waits() saw no stop
and a read the client would have resent after one second failed the
step instead. Main went red on this in run 44631001257.

The Retry-After candidate now rounds the answer's time up, so the stop is
always at least retry-after seconds out. Vectors use whole-second clocks,
so no vector changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KeBVgbERyakZ33T4AetfRj
…e ride-out test

Move the regression test out of rate-guard.test.mjs, which it pushed
over the 1000-line size check, into check-trusted-trigger.test.mjs,
where it drives createGitHub on a latch clock that ticks 5 ms per read
from 970 ms into a second: the old floor stamping fails it with the
same 403 main saw, and the rounded-up stop passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KeBVgbERyakZ33T4AetfRj
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T23:20:58.031422Z e78593a PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e78593acf5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

const nowSeconds = Math.floor(now / 1000);
const candidates = [
retryAfter !== null ? [nowSeconds + retryAfter, "retry-after"] : null,
retryAfter !== null ? [Math.ceil(now / 1000) + retryAfter, "retry-after"] : null,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Account for rounding when checking the retry wait budget

When a response carries Retry-After: 60 and arrives between whole-second boundaries, rounding up makes the computed wait exceed the default 60,000 ms budget. For example, with now = 1700000000500, the new reset is 60,500 ms away, so createRateLimitLatch.waits() returns false and observe() fails the step instead of allowing retryServerErrors() to resend the read; the previous calculation allowed that retry. Preserve the conservative reset rounding, but account for its padding in the budget comparison so a delay equal to the configured budget remains retryable.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant