Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Changed

- **pr-verify PASS and PASS-NOTES artifacts now expire after 7 days instead of 30, unless the pull request's body or issue links changed since its head commit.** Such a pass is read only at an open pull request's current head, so outliving that head is dead weight in the repository-wide artifact listing; a pass that expires on a held head fails closed as `verify:missing` and the next push, edit or label has pr-verify judge it again. A pass judged after a body edit or link change may supersede a FAIL at its head, which the merge gate still reads at a later merge-only head, so it keeps 90 days like the FAIL. FAIL verdicts keep 90 days for the 80-day look-back. The `pr-review-main-*` row records why its 90 days stay for now.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Changed

- **The verify rules sync suite pins verdict lifetimes against the gate.** A new case runs `write-verdict.mjs` `retentionDays` and fails when a FAIL would expire before `babysit_verify_evidence.TOO_OLD_DAYS`, or when a pass lasts anything but its recorded 7 days.
3 changes: 2 additions & 1 deletion .github/artifact-retention.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"scripts/credential-sweep.sh --since defaults to 90 days ago, GitHub's default artifact retention"
],
"max-reader": "plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py",
"reason": "A FAIL verdict must outlive the 80-day FAIL look-back (babysit_verify_evidence.py TOO_OLD_DAYS = 80, ten days under 90): a shorter retention drops earlier FAILs, and the gate then stops refusing verify:earlier-fail and the FAIL cap. scripts/credential-sweep.sh also assumes 90-day artifacts. write-verdict.mjs retentionDays keeps FAIL at max and PASS or PASS-NOTES 30 days (an expired PASS fails closed as verify:missing); the row records the longest.",
"reason": "A FAIL verdict must outlive the 80-day FAIL look-back (babysit_verify_evidence.py TOO_OLD_DAYS = 80, ten days under 90): a shorter retention drops earlier FAILs, and the gate then stops refusing verify:earlier-fail and the FAIL cap. scripts/credential-sweep.sh also assumes 90-day artifacts. write-verdict.mjs retentionDays keeps FAIL at max, a PASS or PASS-NOTES judged after a body edit or issue link change since the head commit at max too (it may supersede a FAIL there, which the gate reads at later heads of the same change), and any other PASS or PASS-NOTES 7 days: it is read only at the open pull request's current head, and an expired one fails closed as verify:missing until the next pr-require-checks run (push, edit or label) has pr-verify judge the head again. The row records the longest.",
"computed-by": ".github/scripts/pr-verify/write-verdict.mjs",
"uploads": [
".github/workflows/pr-verify.yml#check-intent",
Expand All @@ -28,6 +28,7 @@
],
"max-reader": "plugins/source-control/skills/babysit-prs/scripts/babysit_review_evidence.py",
"reason": "A verdict that reported findings must outlive a later clean review of the same tree, or its expiry would lower the count the gate reads; the gate counts the most findings any unexpired verdict of the tree reported.",
"pending-cut": "About 238 a day, 21,400 at 90 days. CRITICAL counts already carry on the record job's check runs (babysit_merge.py carried_critical), but the below-CRITICAL total security_review_clean reads does not; a short window needs that total read from the check runs first.",
"uploads": [
".github/workflows/pr-review-main.yml#review",
".github/workflows/pr-review-main.yml#review-subscription",
Expand Down
39 changes: 29 additions & 10 deletions .github/scripts/pr-verify/resolve-pr.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@
// trusted people edited its body or connected or disconnected an issue, at
// least once (relinkedByPeople): reason `relinked`. The gate weighs the pass that
// judgment writes by the same rule (babysit_verify_evidence `_superseded`).
// A judgment also reports relink-since-head: whether the PR's body or links
// changed since the head commit (relinkedSince), which sets how long its pass
// lasts (write-verdict.mjs retentionDays).

import { appendFileSync, writeFileSync } from "node:fs";
import process from "node:process";
Expand Down Expand Up @@ -131,28 +134,42 @@ const trustedHuman = (actor, humans) => actor?.__typename === "User" && humans.h
// see. A failed read, a list cut short, or an act with no readable time is false.
export async function relinkedByPeople(github, repository, number, humans, since) {
const start = Date.parse(since ?? "");
if (Number.isNaN(start)) {
return false;
}
const acts = Number.isNaN(start) ? null : await relinkActs(github, repository, number);
const inside = (acts ?? []).filter(([at]) => Date.parse(at) >= start);
return inside.length > 0 && inside.every(([, actor]) => trustedHuman(actor, humans));
}

// Every body edit and issue connect or disconnect on the pull request, each
// [time, actor], read whole (RELINK_QUERY); null for a failed read, a list
// cut short, or an act with no readable time.
async function relinkActs(github, repository, number) {
const [o, r] = repository.split("/");
let pull;
try {
pull = (await github("POST", "/graphql", { query: RELINK_QUERY, variables: { o, r, n: number } }))?.data?.repository?.pullRequest;
} catch {
return false;
return null;
}
const edits = pull?.userContentEdits;
// totalCount counts every timeline item; filteredCount the kinds asked.
const events = pull?.timelineItems;
if (!Array.isArray(edits?.nodes) || edits.totalCount !== edits.nodes.length || !Array.isArray(events?.nodes) || events.filteredCount !== events.nodes.length) {
return false;
return null;
}
const acts = [...edits.nodes.map((node) => [node?.editedAt, node?.editor]), ...events.nodes.map((node) => [node?.createdAt, node?.actor])];
if (acts.some(([at]) => Number.isNaN(Date.parse(at ?? "")))) {
return false;
}
const inside = acts.filter(([at]) => Date.parse(at) >= start);
return inside.length > 0 && inside.every(([, actor]) => trustedHuman(actor, humans));
return acts.some(([at]) => Number.isNaN(Date.parse(at ?? ""))) ? null : acts;
}

// Whether anyone edited the pull request's body or connected or disconnected
// an issue from `since`, the head commit's time, on. Only then can a pass
// judged at the head supersede a FAIL there (babysit_verify_evidence
// `_superseded` needs such an act after the FAIL, which comes after the
// commit), so write-verdict.mjs keeps such a pass as long as a FAIL. A time
// or read that fails answers true.
export async function relinkedSince(github, repository, number, since) {
const start = Date.parse(since ?? "");
const acts = Number.isNaN(start) ? null : await relinkActs(github, repository, number);
return acts === null || acts.some(([at]) => Date.parse(at) >= start);
}

// babysit_merge_policy.names_nothing: whether a GraphQL answer says only that
Expand Down Expand Up @@ -651,6 +668,7 @@ export async function resolvePull({ github, readRefs = github, readLog, reposito
return {
proceed: true,
reason: ownFail !== undefined ? "relinked" : passes.length > 0 ? "spec-changed" : "ok",
relinkSinceHead: ownFail !== undefined || (await relinkedSince(github, repository, pull.number, run.head_commit?.timestamp)),
prNumber: pull.number,
headSha: head,
baseSha: pull.base.sha,
Expand Down Expand Up @@ -739,6 +757,7 @@ export async function main(env = process.env, { github, fetchImpl } = {}) {
`head-sha=${result.headSha ?? ""}`,
`base-sha=${result.baseSha ?? ""}`,
`head-ref=${result.headRef ?? ""}`,
`relink-since-head=${result.relinkSinceHead ?? ""}`,
];
appendFileSync(env.GITHUB_OUTPUT, `${lines.join("\n")}\n`);
const summary = {
Expand Down
39 changes: 38 additions & 1 deletion .github/scripts/pr-verify/resolve-pr.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,8 @@ test("a passing same-repository run resolves its one PR at the run's head", asyn
linkedIssues: [LINKED],
specKind: "closes",
specText: [{ number: 7, spec_kind: "closes", body: "the spec", comments: [] }],
// The fixture's run names no head commit time, so a relink cannot be ruled out.
relinkSinceHead: true,
});
});

Expand Down Expand Up @@ -870,6 +872,41 @@ test("a FAIL named without issues stands while the PR links no issue a FAIL name
}
});

// The head commit, after ORIGINAL and before FAILED_AT.
const COMMITTED_AT = "2026-10-19T12:00:00Z";
const committed = (overrides = {}) => ({ run: upstream({ head_commit: { id: HEAD, timestamp: COMMITTED_AT } }), ...overrides });

test("a judgment after an edit or link change since the head commit reports a relink, so its pass keeps a FAIL's retention", async () => {
// This PR's FAIL at the head, named with an issue it no longer links, so
// the head is judged again with reason ok, and the gate may find its pass
// superseding the FAIL (babysit_verify_evidence `_superseded`).
const fail = { artifacts: [artifact(`pr-verify-41-${HEAD}-fail-i5`, 77)], runs: { 77: verifyRun(77, { created_at: FAILED_AT }) } };
const linked = await resolve(committed({ ...fail, relink: relinkAnswer({ events: [{ createdAt: "2026-10-20T08:00:00Z", actor: PERSON }] }) }));
assert.equal(linked.reason, "ok");
assert.equal(linked.relinkSinceHead, true);
for (const [label, setup] of [
["a failed read", committed(fail)],
["no head commit time", { ...fail, relink: relinkAnswer() }],
["edits cut short", committed({ ...fail, relink: relinkAnswer({ editsTotal: 101 }) })],
]) {
assert.equal((await resolve(setup)).relinkSinceHead, true, label);
}
});

test("a judgment with no edit or link change since the head commit reports none", async () => {
// Only the original text, stamped before the head commit.
const resolved = await resolve(committed({ relink: relinkAnswer() }));
assert.equal(resolved.proceed, true);
assert.equal(resolved.relinkSinceHead, false);
});

test("a relinked judgment reports a relink", async () => {
const relink = relinkAnswer({ events: [{ createdAt: "2026-10-20T08:00:00Z", actor: PERSON }] });
const resolved = await resolve(relinked(relink));
assert.equal(resolved.reason, "relinked");
assert.equal(resolved.relinkSinceHead, true);
});

test("after a trusted relink, another PR's FAIL or one named with the new issues still stops the judgment", async () => {
const relink = relinkAnswer({ events: [{ createdAt: "2026-10-20T08:00:00Z", actor: PERSON }] });
for (const name of [`pr-verify-99-${HEAD}-fail`, `pr-verify-41-${HEAD}-fail-i7`]) {
Expand Down Expand Up @@ -1283,7 +1320,7 @@ test("main writes the outputs, the linked issues and the spec text the workflow
);
assert.equal(
readFileSync(output, "utf8"),
`proceed=true\nreason=ok\npr-number=41\nhead-sha=${HEAD}\nbase-sha=${BASE}\nhead-ref=feat/x\n`,
`proceed=true\nreason=ok\npr-number=41\nhead-sha=${HEAD}\nbase-sha=${BASE}\nhead-ref=feat/x\nrelink-since-head=true\n`,
);
assert.deepEqual(JSON.parse(readFileSync(spec, "utf8")), [LINKED]);
assert.deepEqual(JSON.parse(readFileSync(specText, "utf8")), [{ number: 7, spec_kind: "closes", body: "the spec", comments: [] }]);
Expand Down
2 changes: 2 additions & 0 deletions .github/scripts/pr-verify/workflow.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,8 @@ test("the upload is each job's only artifact step and follows the written verdic
assert.match(uploads[0].block, /if: steps\.verdict\.outcome == 'success'/, job);
// The verdict step names the retention (write-verdict.mjs retentionDays).
assert.equal(input(uploads[0].block, "retention-days"), "${{ steps.verdict.outputs.retention-days }}", job);
// A pass's retention follows this job's resolver (resolve-pr.mjs relinkedSince).
assert.match(step("Write the verdict", job), /^ {10}RELINK_SINCE_HEAD: \$\{\{ steps\.pr\.outputs\.relink-since-head \}\}$/m, job);
// A verdict needs a judge step that succeeded, which run-claude-cli never
// reports with a lane-class, so a credit stop writes none.
assert.match(step("Write the verdict", job), /^ {8}if: steps\.judge\.outcome == 'success'$/m, job);
Expand Down
25 changes: 16 additions & 9 deletions .github/scripts/pr-verify/write-verdict.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -231,14 +231,21 @@ export function artifactName(verdict) {
return name;
}

// How many days the verdict's artifact lasts. A PASS or PASS-NOTES counts
// only at the head it names, so one that expires refuses as verify:missing
// until pr-verify judges the head again. A FAIL, or anything else, keeps 90
// days, the maximum: the merge gate counts an earlier FAIL until the pull
// request is TOO_OLD_DAYS (80) old (babysit_verify_evidence.py), and
// carry-run-cache.mjs RETENTION_DAYS matches it.
export function retentionDays(verdict) {
return ["pass", "pass-notes"].includes(verdict) ? 30 : 90;
// How many days the verdict's artifact lasts. A PASS or PASS-NOTES judged
// after no body edit or issue connect or disconnect since the head commit
// (resolve-pr.mjs relinkedSince, RELINK_SINCE_HEAD `false`) is read only at
// the open pull request's current head, so one that expires refuses as
// verify:missing until the next pr-require-checks run (a push, edit or label)
// has pr-verify judge the head again; 7 days keeps a week of a held head.
// Any other pass may supersede a FAIL at its head after a relink
// (babysit_verify_evidence `_superseded`), which the merge gate still reads
// at a later head of the same change, so it keeps a FAIL's 90 days. A FAIL,
// or anything else, keeps 90 days, the maximum: the merge gate counts an
// earlier FAIL until the pull request is TOO_OLD_DAYS (80) old
// (babysit_verify_evidence.py), and carry-run-cache.mjs RETENTION_DAYS
// matches it.
export function retentionDays(verdict, mayRelink = true) {
return ["pass", "pass-notes"].includes(verdict) && mayRelink === false ? 7 : 90;
}

export function main(env = process.env) {
Expand Down Expand Up @@ -299,7 +306,7 @@ export function main(env = process.env) {
mkdirSync(env.VERDICT_DIR, { recursive: true });
writeFileSync(path.join(env.VERDICT_DIR, "verdict.json"), `${JSON.stringify(verdict, null, 2)}\n`);
const name = artifactName(verdict);
appendFileSync(env.GITHUB_OUTPUT, `name=${name}\nverdict=${verdict.verdict}\nretention-days=${retentionDays(verdict.verdict)}\n`);
appendFileSync(env.GITHUB_OUTPUT, `name=${name}\nverdict=${verdict.verdict}\nretention-days=${retentionDays(verdict.verdict, env.RELINK_SINCE_HEAD !== "false")}\n`);
console.log(`write-verdict: ${name}`);
}

Expand Down
21 changes: 14 additions & 7 deletions .github/scripts/pr-verify/write-verdict.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -53,11 +53,16 @@ test("each verdict names its artifact with the PR, the head and the verdict", ()
}
});

test("a PASS or PASS-NOTES lasts 30 days and a FAIL, or anything else, 90", () => {
assert.equal(retentionDays("pass"), 30);
assert.equal(retentionDays("pass-notes"), 30);
test("a PASS or PASS-NOTES with no relink since the head commit lasts 7 days, and any other verdict 90", () => {
assert.equal(retentionDays("pass", false), 7);
assert.equal(retentionDays("pass-notes", false), 7);
// A pass after a relink may supersede a FAIL, which later heads still read.
for (const verdict of ["pass", "pass-notes"]) {
assert.equal(retentionDays(verdict, true), 90, verdict);
assert.equal(retentionDays(verdict), 90, `${verdict}, relink unknown`);
}
for (const verdict of ["fail", "", undefined, "PASS", "pass-notes-x"]) {
assert.equal(retentionDays(verdict), 90, String(verdict));
assert.equal(retentionDays(verdict, false), 90, String(verdict));
}
});

Expand Down Expand Up @@ -311,10 +316,12 @@ test("main writes the verdict file and the artifact name, and nothing on bad out
assert.equal(readFileSync(env.GITHUB_OUTPUT, "utf8"), `name=pr-verify-41-${HEAD}-fail-i7\nverdict=fail\nretention-days=90\n`);

const okFile = execution("ok", JSON.parse(ok));
const passed = { ...env, VERDICT_DIR: path.join(dir, "verdict-ok"), GITHUB_OUTPUT: path.join(dir, "out-ok"), EXECUTION_FILE: okFile };
main(passed);
const okVerdict = JSON.parse(ok).verdict;
assert.equal(readFileSync(passed.GITHUB_OUTPUT, "utf8"), `name=pr-verify-41-${HEAD}-${okVerdict}\nverdict=${okVerdict}\nretention-days=30\n`);
for (const [relink, days] of [["false", 7], ["true", 90], [undefined, 90]]) {
const passed = { ...env, VERDICT_DIR: path.join(dir, `verdict-ok-${relink}`), GITHUB_OUTPUT: path.join(dir, `out-ok-${relink}`), EXECUTION_FILE: okFile, RELINK_SINCE_HEAD: relink };
main(passed);
assert.equal(readFileSync(passed.GITHUB_OUTPUT, "utf8"), `name=pr-verify-41-${HEAD}-${okVerdict}\nverdict=${okVerdict}\nretention-days=${days}\n`, String(relink));
}
const unparsable = path.join(dir, "unparsable.json");
writeFileSync(unparsable, "{not json");
for (const [label, extra] of [
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/pr-verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -382,6 +382,7 @@ jobs:
HEAD_SHA: ${{ steps.pr.outputs.head-sha }}
BASE_SHA: ${{ steps.pr.outputs.base-sha }}
HEAD_REF: ${{ steps.pr.outputs.head-ref }}
RELINK_SINCE_HEAD: ${{ steps.pr.outputs.relink-since-head }}
SPEC_FILE: ${{ github.workspace }}/pr-verify-data/spec.json
SPEC_TEXT_FILE: ${{ github.workspace }}/pr-verify-data/spec-text.json
VERDICT_DIR: ${{ runner.temp }}/pr-verify-verdict
Expand Down Expand Up @@ -754,6 +755,7 @@ jobs:
HEAD_SHA: ${{ steps.pr.outputs.head-sha }}
BASE_SHA: ${{ steps.pr.outputs.base-sha }}
HEAD_REF: ${{ steps.pr.outputs.head-ref }}
RELINK_SINCE_HEAD: ${{ steps.pr.outputs.relink-since-head }}
SPEC_FILE: ${{ github.workspace }}/pr-verify-data/spec.json
SPEC_TEXT_FILE: ${{ github.workspace }}/pr-verify-data/spec-text.json
VERDICT_DIR: ${{ runner.temp }}/pr-verify-verdict
Expand Down Expand Up @@ -1103,6 +1105,7 @@ jobs:
HEAD_SHA: ${{ steps.pr.outputs.head-sha }}
BASE_SHA: ${{ steps.pr.outputs.base-sha }}
HEAD_REF: ${{ steps.pr.outputs.head-ref }}
RELINK_SINCE_HEAD: ${{ steps.pr.outputs.relink-since-head }}
SPEC_FILE: ${{ github.workspace }}/pr-verify-data/spec.json
SPEC_TEXT_FILE: ${{ github.workspace }}/pr-verify-data/spec-text.json
VERDICT_DIR: ${{ runner.temp }}/pr-verify-verdict
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# test-scope: .github/scripts/pr-verify/resolve-pr.mjs .github/workflows/pr-require-checks.yml .github/scripts/pr-merge/merge-holds.mjs
# test-scope: .github/scripts/pr-verify/resolve-pr.mjs .github/scripts/pr-verify/write-verdict.mjs .github/workflows/pr-require-checks.yml .github/scripts/pr-merge/merge-holds.mjs
"""The verifier's two shared rules agree across languages: the gate's
(`babysit_verify_evidence`, `babysit_classify`) and the resolver's
(`.github/scripts/pr-verify/resolve-pr.mjs`, run here under node).
Expand Down Expand Up @@ -211,6 +211,32 @@ def test_the_head_ref_digest_is_the_same(self) -> None:
# The first 16 hex digits of `printf %s feat/x | sha256sum`.
self.assertEqual(answer["digests"][0], "79b4cc554b80dca6")

def test_a_fail_outlives_the_look_back_and_a_pass_its_held_head(self) -> None:
# A FAIL the gate still counts must not expire before the pull
# request turns too old (TOO_OLD_DAYS), or verify:earlier-fail and the
# FAIL cap stop refusing. So must a pass judged after a relink: it may
# supersede a FAIL at its head (`_superseded`), which a later head of
# the same change still reads. Any other pass is read only at the open
# head, and 7 days is the window .github/artifact-retention.json records.
writer = RESOLVER.parent / "write-verdict.mjs"
script = (
f"import {{ retentionDays }} from {json.dumps(writer.as_uri())};"
" console.log(JSON.stringify(["
' retentionDays("fail", false), retentionDays("pass", true),'
' retentionDays("pass-notes", true), retentionDays("pass", false),'
' retentionDays("pass-notes", false)]));'
)
proc = subprocess.run(
["node", "--input-type=module", "-e", script],
capture_output=True,
text=True,
check=True,
)
fail, relinked, relinked_notes, passed, notes = json.loads(proc.stdout)
for days in (fail, relinked, relinked_notes):
self.assertGreater(days, ve.TOO_OLD_DAYS)
self.assertEqual((passed, notes), (7, 7))

# Each FAIL-shaped name, and the issues its name carries read off by hand,
# or None for a name no reader takes. One verdict has one name: a PASS
# carries no issues, and issues are ascending and listed once.
Expand Down
Loading