Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
b1bb0ad
fix(pr-pipeline): read the queued pass record again across a 5xx outage
kyle-sexton Oct 30, 2026
7a81238
fix(pr-pipeline): size the verdict listing and cold-read cost from th…
kyle-sexton Oct 30, 2026
e70f8dd
Merge remote-tracking branch 'origin/main' into fix/verify-evidence-r…
kyle-sexton Oct 30, 2026
20948b5
test(pr-pipeline): write the sleep stub with a quoted heredoc
kyle-sexton Oct 30, 2026
06ec402
Merge remote-tracking branch 'origin/main' into fix/verify-evidence-r…
kyle-sexton Oct 30, 2026
05e0b5b
fix(pr-pipeline): count the cold path's own listing in the headroom need
kyle-sexton Oct 30, 2026
9c9cc6b
fix(pr-pipeline): size the verdict listing to the limit its caller ki…
kyle-sexton Oct 30, 2026
e212a9d
Merge remote-tracking branch 'origin/main' into fix/verify-evidence-r…
kyle-sexton Oct 30, 2026
ba3e4e5
fix(pr-pipeline): skip the round when the verdict listing outruns ver…
kyle-sexton Oct 30, 2026
bf43ccd
Merge remote-tracking branch 'origin/main' into fix/verify-evidence-r…
kyle-sexton Oct 30, 2026
c5338ef
fix(pr-pipeline): quiet the expected verify:fail warning and log the …
kyle-sexton Oct 30, 2026
43e8571
Merge remote-tracking branch 'origin/main' into fix/verify-evidence-r…
kyle-sexton Oct 30, 2026
c1e1a53
fix(pr-pipeline): annotate a round skipped because the verdict listin…
kyle-sexton Oct 30, 2026
5a83017
Merge remote-tracking branch 'origin/main' into fix/verify-evidence-r…
kyle-sexton Oct 30, 2026
d8bf9a9
Merge remote-tracking branch 'origin/main' into fix/verify-evidence-r…
kyle-sexton Oct 30, 2026
8c56902
refactor(pr-merge): keep pythonVerify doc block attached to pythonVerify
kyle-sexton Oct 30, 2026
80c2176
Merge remote-tracking branch 'origin/main' into fix/verify-evidence-r…
kyle-sexton Oct 30, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
bump: patch
---

### Fixed

- In the merge queue, `ci-status` reads the counted pass's `verdict.json` again after 5, 15 and 30 s
when GitHub answers a 5xx or the request gets no response; any other failure still fails at once.
Three 503s over 26 s failed merge-group run
[41063238235](https://github.com/melodic-software/claude-code-plugins/actions/runs/41063238235)
under the old 5 and 10 s waits.
- The cold-read headroom check works out a cold read's cost from the live artifact listing, its
`total_count` and the FAILs it lists, plus that listing's own pages, in place of a fixed 1,546 sized for 77 pages and 793 FAIL
reads. A listing that does not read holds the pull request.
- `enable-auto-merge.mjs` (verifier reads and `capped-close`) and `verify-cap.mjs` tell the verdict
reader the limit they kill it at (240 s, 30 s for a kept-answer lookup, 120 s), so a listing that
cannot finish in that time refuses with the pages, the budget and the caller named rather than
ending as a bare `verify:unreadable`.
- `verify-cap.mjs` skips the address-feedback or fix-ci round, for that run only, when the
reader's listing does not fit in its 120 s, as it does when a sibling FAIL run cannot be read:
either may hide the FAIL cap. Any other refusal still lets the round run, now with a warning
naming the refusal and its detail instead of none, except `verify:fail` on an address-feedback
round sent for that FAIL, which expects it.
- `verify-cap.mjs` logs one line per read summarizing the reader's stderr, its
`babysit-listing-miss` lines first, so a listing that missed GitHub's live count shows in the
dispatch job's log.
- The address-feedback and fix-ci dispatchers annotate a skip for a listing that does not fit in
`verify-cap.mjs`'s limit as `::warning title=verify-deadline-skip::`, with the reader's detail,
since that skip recurs every run while pr-merge's carried listing is missing or stale.
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
bump: patch
---

### Fixed

- `babysit_verify_evidence.py` reads GitHub's live artifact `total_count` from one page before its
whole listings and sizes their shared timeout from it: each whole listing and its 7-page top-up at
1 s a page, at most 300 s (`LISTING_TIMEOUT_SECONDS`), or less when the caller names the limit it
kills the reader at in `BABYSIT_VERIFY_DEADLINE_SECONDS` (and itself in `BABYSIT_VERIFY_CALLER`),
less 15 s for the reads after the listing. A whole listing that cannot fit refuses
`verify:unreadable` before paging, with the pages, the seconds they need and the budget (and the
caller) in the detail, instead of being killed with no reason. That detail starts with
`DEADLINE_UNREADABLE` when the caller's limit, not the 300 s cap, is what it does not fit in. The
caller's limit bounds the count read too, and the whole listings' timeout is at least gh's 60 s
per-call default unless the cap or that limit leaves less.
10 changes: 8 additions & 2 deletions .github/scripts/pr-merge/enable-auto-merge.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -1104,6 +1104,11 @@ export function cacheLine(number, cache) {
// merge.verify-cache as the run applies it: `serve` only when it says so.
export const cacheMode = (env) => (env.VERIFY_CACHE === "serve" ? "serve" : "shadow");

// The reader's env with the limit it is killed at, so it sizes its listing to
// that limit and names this caller when a listing cannot fit
// (babysit_verify_evidence.py DEADLINE_ENV and CALLER_ENV).
const readerEnv = (env, timeoutMs) => ({ ...env, BABYSIT_VERIFY_DEADLINE_SECONDS: String(timeoutMs / 1000), BABYSIT_VERIFY_CALLER: "enable-auto-merge.mjs" });

// The gate's own reader, one process per pull request; the run shares its
// artifact listing, head-ref map and run cache through the BABYSIT_* files.
// `--shared-listing`: the first read's listing serves the whole run, an allow
Expand All @@ -1115,6 +1120,7 @@ export const cacheMode = (env) => (env.VERIFY_CACHE === "serve" ? "serve" : "sha
// kept refusal is served (`serve`) only to a pull request not armed, which is
// always read in full, and never on `fresh`; `cacheOnly` reads only a kept
// one, returning no `refusal` on a miss or when there is no file to read.

export function pythonVerify(env, { spawn = spawnSync, clock = Date.now } = {}) {
// `rung`: the lane room's effective rung, which the gate scores at.
const verify = async ({ number, headRefOid: head, baseRefOid: base, autoMergeRequest }, rung, { fresh = false, cacheOnly = false } = {}) => {
Expand Down Expand Up @@ -1143,7 +1149,7 @@ export function pythonVerify(env, { spawn = spawnSync, clock = Date.now } = {})
}
const started = clock();
const timeout = args.includes("--answers-only") ? CACHE_ONLY_TIMEOUT_MS : VERIFY_TIMEOUT_MS;
const done = spawn("python3", [...args, String(number), head], { encoding: "utf8", env, timeout });
const done = spawn("python3", [...args, String(number), head], { encoding: "utf8", env: readerEnv(env, timeout), timeout });
const cost = { number, calls: ghCalls(done.stderr), ms: clock() - started };
let evidence;
if (done.status === 0) {
Expand Down Expand Up @@ -1237,7 +1243,7 @@ export function pythonClose(env, spawn = spawnSync) {
for (const label of holds) {
args.push("--hold-label", label);
}
const done = spawn("python3", [...args, String(number), head], { encoding: "utf8", env, timeout: VERIFY_TIMEOUT_MS });
const done = spawn("python3", [...args, String(number), head], { encoding: "utf8", env: readerEnv(env, VERIFY_TIMEOUT_MS), timeout: VERIFY_TIMEOUT_MS });
let answer = null;
try {
answer = JSON.parse(String(done.stdout).trim().split("\n").at(-1));
Expand Down
11 changes: 9 additions & 2 deletions .github/scripts/pr-merge/enable-auto-merge.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -3482,7 +3482,8 @@ test("serve asks for a kept answer only for an unarmed pull request, and never o
const reader = shadowReader({ 1: MISSING, 2: MISSING });
const { spawned } = reader;
const timeouts = [];
const spawn = (cmd, args, options) => (timeouts.push(options.timeout), reader.spawn(cmd, args));
const deadlines = [];
const spawn = (cmd, args, options) => (timeouts.push(options.timeout), deadlines.push([options.env.BABYSIT_VERIFY_DEADLINE_SECONDS, options.env.BABYSIT_VERIFY_CALLER]), reader.spawn(cmd, args));
const verify = pythonVerify(env, { spawn, clock: () => 0 });
const log = console.log;
console.log = () => {};
Expand All @@ -3498,6 +3499,10 @@ test("serve asks for a kept answer only for an unarmed pull request, and never o
assert.deepEqual(spawned.map(mode), ["serve", "shadow", "shadow", "serve"]);
assert.deepEqual(spawned.map((args) => args.includes("--answers-only")), [false, false, false, true]);
assert.deepEqual(timeouts, [VERIFY_TIMEOUT_MS, VERIFY_TIMEOUT_MS, VERIFY_TIMEOUT_MS, CACHE_ONLY_TIMEOUT_MS]);
// babysit_verify_evidence.py DEADLINE_ENV and CALLER_ENV: each read is told
// the limit it is killed at (240 s, or 30 s for a kept-answer lookup).
const caller = "enable-auto-merge.mjs";
assert.deepEqual(deadlines, [["240", caller], ["240", caller], ["240", caller], ["30", caller]]);
assert.deepEqual(spawned[0].slice(spawned[0].indexOf("--answers-out"), spawned[0].indexOf("--answers-out") + 6), ["--answers-out", "/tmp/out.json", "--answers-mode", "serve", "--answers-in", "/tmp/in.json"]);
});

Expand Down Expand Up @@ -3655,7 +3660,7 @@ test("an approved hold-path escalation keeps its label while anything else stand
test("pythonClose runs the reader's capped-close at the base with the hold labels, and throws on a failed exit", () => {
const seen = [];
const spawn = (cmd, args, options) => {
seen.push({ cmd, args, token: options.env.GH_TOKEN });
seen.push({ cmd, args, token: options.env.GH_TOKEN, deadline: options.env.BABYSIT_VERIFY_DEADLINE_SECONDS, caller: options.env.BABYSIT_VERIFY_CALLER, timeout: options.timeout });
return { status: 0, stdout: '{"closed": true, "restarted": [7], "parked": []}\n' };
};
const base = "b".repeat(40);
Expand All @@ -3666,6 +3671,8 @@ test("pythonClose runs the reader's capped-close at the base with the hold label
assert.deepEqual(seen[0].args.slice(1), ["capped-close", "--repo", "o/r", "--base-sha", base, "--hold-label", "do-not-merge", "--hold-label", "needs-human", "5", HEAD]);
// Its writes keep the lanes App token.
assert.equal(seen[0].token, "app");
// Its reader is told the 240 s limit it is killed at (DEADLINE_ENV).
assert.deepEqual([seen[0].timeout, seen[0].deadline, seen[0].caller], [VERIFY_TIMEOUT_MS, "240", "enable-auto-merge.mjs"]);
const failing = () => ({ status: 1, stdout: '{"closed": false, "error": "close: HTTP 403"}\n', stderr: "" });
assert.throws(() => pythonClose(env, failing)({ number: 5, headRefOid: HEAD, baseRefOid: base }, []), /capped-close exited 1: close: HTTP 403/);
assert.throws(() => pythonClose(env, spawn)({ number: 5, headRefOid: HEAD, baseRefOid: "main" }, []), /not a commit SHA/);
Expand Down
11 changes: 9 additions & 2 deletions .github/scripts/pr-unblock/feedback-dispatch.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ import { commentItems, gateSource, listLabelEvents, prHold, reviewReader } from
import { emptyRecord, isHandled, markHandled, readRecord, writeRecord } from "./dispatch-record.mjs";
import { anyHandles, KILL_SWITCH_STOP, LANE_STOP, laneStop, limitResetsAt, permanentlyHandles } from "./limit-stop.mjs";
import { budgetedFetch } from "./request-budget.mjs";
import { final as capFinal, verifyCapReader } from "./verify-cap.mjs";
import { FAIL as VERIFY_FAIL, final as capFinal, deadlineWarning, verifyCapReader } from "./verify-cap.mjs";

export const DEFAULTS = Object.freeze({
base: "main",
Expand Down Expand Up @@ -505,13 +505,20 @@ export async function dispatchFeedback({
// later pass clears no cap, and a push past it could never merge, so
// pr-merge closes the pull request for a fresh attempt. It closes only the
// lanes App's: an adopted owner's is never skipped.
const capped = owner ? null : await verifyCapped({ number, head, base: pull.base?.sha });
// A round sent for pr-verify's FAIL at this head (notesSince) expects the
// reader's FAIL refusal there.
const expected = !passOnly && notesSince !== null ? [VERIFY_FAIL] : [];
const capped = owner ? null : await verifyCapped({ number, head, base: pull.base?.sha, expected });
if (capped) {
// The cap never clears for the head: later runs skip it before any
// read. A sibling read that failed is read again next run.
if (capFinal(capped)) {
markHandled(record, settings.checkName, head);
}
const warning = deadlineWarning(number, capped);
if (warning) {
log(warning);
}
return skip(`#${number}: pr-verify's FAILs are at the cap or cannot all be read (${capped}); pr-merge closes it for a fresh attempt, so no ${settings.activity} round`);
}
const { workflowRunId } = await api.dispatchWorkflow({
Expand Down
42 changes: 41 additions & 1 deletion .github/scripts/pr-unblock/feedback-dispatch.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ test("a PR whose pr-verify FAILs are at the cap gets no address-feedback round",
const { result, lines } = await decide(api, resolvedLane(), { verifyCapped });
assert.equal(result, null);
assert.deepEqual(api.calls.dispatch, []);
assert.deepEqual(asked, [{ number: 42, head: HEAD, base: undefined }]);
assert.deepEqual(asked, [{ number: 42, head: HEAD, base: undefined, expected: [] }]);
assert.match(lines.join("\n"), /^skip: #42: .*verify:repeated-fail/m);
// The cap never clears for the head: a later run at it reads nothing.
const record = emptyRecord();
Expand All @@ -232,6 +232,23 @@ test("a PR whose sibling FAIL run cannot be read gets no round this run and is r
assert.equal(reads, 2);
});

// A listing that does not fit in verify-cap's limit (babysit_verify_evidence.py
// DEADLINE_UNREADABLE) recurs every run while pr-merge's carried listing is
// missing or stale, so its skip is a warning annotation, not a plain line.
test("a PR skipped because the listing did not fit warns with the reader's detail; a sibling skip does not", async () => {
const deadline = "verify:unreadable: the artifact listing does not fit in the caller's limit: repository artifacts: 10450 artifacts in 105 pages need 112 s";
const api = fakeApi();
const { result, lines } = await decide(api, resolvedLane(), { verifyCapped: () => deadline });
assert.equal(result, null);
assert.deepEqual(api.calls.dispatch, []);
assert.deepEqual(
lines.filter((line) => line.startsWith("::warning title=verify-deadline-skip::")),
[`::warning title=verify-deadline-skip::#42: ${deadline}`],
);
const sibling = await decide(fakeApi(), resolvedLane(), { verifyCapped: () => "verify:unreadable: a FAIL run of a pull request for the same issue" });
assert.ok(!sibling.lines.some((line) => line.includes("verify-deadline-skip")), sibling.lines.join("\n"));
});

// A verifier PASS at a new head clears no cap, so the pass-only dispatch
// starts no round either.
test("a pass-only dispatch of a PR whose pr-verify FAILs are at the cap gets no round", async () => {
Expand All @@ -242,6 +259,29 @@ test("a pass-only dispatch of a PR whose pr-verify FAILs are at the cap gets no
assert.match(lines.join("\n"), /^skip: #42: .*verify:repeated-fail/m);
});

// A round sent for pr-verify's FAIL at the head (notesSince set) finds that
// FAIL (babysit_verify_evidence.py FAIL, "verify:fail") there by design, so
// the reader is told to expect it; a review run's round and a pass expect none.
test("only a round sent for pr-verify's FAIL at the head expects the reader's FAIL refusal", async () => {
for (const [extra, expected] of [
[{ notesSince: Date.parse("2026-10-17T19:00:00Z") }, ["verify:fail"]],
[{}, []],
[{ passOnly: true }, []],
]) {
const asked = [];
const verifyCapped = (request) => {
asked.push(request);
return null;
};
await decide(fakeApi(), resolvedLane(), { ...extra, verifyCapped });
assert.deepEqual(
asked.map((request) => request.expected),
[expected],
JSON.stringify(extra),
);
}
});

test("a PR the reader does not find at the cap dispatches as before", async () => {
const api = fakeApi();
const { result } = await decide(api, resolvedLane(), { verifyCapped: () => null });
Expand Down
6 changes: 5 additions & 1 deletion .github/scripts/pr-unblock/fix-ci-dispatch.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ import { failingChecks } from "./fix-ci-context.mjs";
import { prHold, reviewReader } from "./gate-source.mjs";
import { KILL_SWITCH_STOP, killSwitchStop, laneStop, releasesHead } from "./limit-stop.mjs";
import { serverFailure } from "./request-budget.mjs";
import { final as capFinal } from "./verify-cap.mjs";
import { final as capFinal, deadlineWarning } from "./verify-cap.mjs";

export const DEFAULTS = Object.freeze({
base: "main",
Expand Down Expand Up @@ -668,6 +668,10 @@ export async function dispatchCiFixes({
// It closes only the lanes App's: an adopted owner's is never skipped.
const verifyCap = node.author?.databaseId === appSenderId ? await verifyCapped({ number: node.number, head: node.headRefOid, base: node.baseRefOid }) : null;
if (verifyCap) {
const warning = deadlineWarning(node.number, verifyCap);
if (warning) {
log(warning);
}
log(`skip fix-ci #${node.number}: pr-verify's FAILs are at the cap or cannot all be read (${verifyCap}); pr-merge closes it for a fresh attempt, so no fix-ci round`);
// The cap never clears for the head: later runs skip it before any
// read. A sibling read that failed is read again next run.
Expand Down
16 changes: 16 additions & 0 deletions .github/scripts/pr-unblock/fix-ci-dispatch.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,22 @@ test("a head whose sibling FAIL run cannot be read is skipped but not recorded",
assert.deepEqual(api.calls.dispatch, []);
});

// A listing that does not fit in verify-cap's limit (babysit_verify_evidence.py
// DEADLINE_UNREADABLE) recurs every run while pr-merge's carried listing is
// missing or stale, so its skip is a warning annotation, not a plain line.
test("a head skipped because the listing did not fit warns with the reader's detail; a sibling skip does not", async () => {
const deadline = "verify:unreadable: the artifact listing does not fit in the caller's limit: repository artifacts: 10450 artifacts in 105 pages need 112 s";
const { api, lines } = await run({ pages: [[pr(5)]] }, { verifyCapped: () => deadline });
assert.deepEqual(api.calls.dispatch, []);
assert.deepEqual(
lines.filter((line) => line.startsWith("::warning title=verify-deadline-skip::")),
[`::warning title=verify-deadline-skip::#5: ${deadline}`],
);
const sibling = await run({ pages: [[pr(5)]] }, { verifyCapped: () => "verify:unreadable: a FAIL run of a pull request for the same issue" });
assert.deepEqual(sibling.api.calls.dispatch, []);
assert.ok(!sibling.lines.some((line) => line.includes("verify-deadline-skip")), sibling.lines.join("\n"));
});

// A fork PR that pushed the same commit lists its runs under the same head SHA.
// Its newer run, still going, would read as a re-run in flight and hold fix-ci
// off (reconcilerOwns); readRequiredRun keeps it out of the tested run.
Expand Down
Loading
Loading