Skip to content

perf(reconcile): read pass artifacts only for heads whose spec moved - #10527

Merged
kyle-sexton merged 5 commits into
mainfrom
perf/reconcile-lost-runs-prefilter
Oct 28, 2026
Merged

kyle-sexton merged 5 commits into
mainfrom
perf/reconcile-lost-runs-prefilter

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

No related issue: a GITHUB_TOKEN spend cut from the 2026-10-27 spend ranking, where maintenance-reconcile-lost-runs is #3 at about 8.1k calls a day.

Summary

The lost-run reconciler now reads a passed PR's pr-verify pass artifacts only when that PR's spec moved after the head's earliest checks run. The re-runs it plans are unchanged. ci-status now also writes the exclusive api-calls notice, so the hub can rank its spend.

Calls per run.

  • Before: mean 173 per run, measured from the job's api-calls notice over 32 runs on 2026-10-27 (spend ranking, row 3).
  • After: about 101-105, derived as 173 minus the measured cut below.

Measured cut. I ran the plan step (reads only) of the old and new reconcile.mjs back to back against live state, on a user token. A counting fetch wrapper refused any write.

sample plan-step calls, old plan-step calls, new pass artifact reads pass run reads GraphQL same plan
first 148 76 72 → 12 16 → 3 4 → 5 yes, 0 items
second, 20:32Z 144 76 68 → 12 14 → 3 4 → 5 yes, the same 2 flake items
  • Per run: 68-72 fewer calls in the plan step, about 47-49%.
  • Per day: at the cron's ≈47 runs a day, about 3,200-3,400 fewer GITHUB_TOKEN calls. The daily figure is derived from these two samples alone.
  • Pre-filter keep rate: in both samples, the first pre-filter kept 3 of the 17-18 passed PRs.

Fix

  • Pre-filter first (reconcile.mjs). The existing prefilterSpec now runs first over every passed PR, bounded by the earliest created_at among the head's pr-require-checks runs. passStarts and the existing pre-filter, with the real pass starts, run only for the PRs it keeps. judgeSpec is unchanged.
  • Why it can drop only what the old path dropped:
    • pr-verify starts only on workflow_run of a completed pr-require-checks run at that head (pr-verify.yml on:).
    • So every pass at a head started after some checks run at that head was created. The bound is therefore at most the newest pass start.
    • Any spec time after the newest pass is then also after the bound, so the first stage keeps a superset of what the old pre-filter kept.
    • The rules that keep a PR for reasons other than time apply the same in both stages: an unreadable time, a comment page that is not complete, or a GraphQL failure.
  • Assumptions behind the bound:
    • The checks run that started a pass must still appear in the head's run listing. It can be missing only if someone deleted it by hand.
    • If a pass's run cannot be read (a null start), the old path always went on to the full read; the first stage can now drop that PR. The outcome is the same, because the pre-filter reads the same times specChanged weighs and none of them moved after the bound.
  • Why the bound is not the newest checks run. The ranking proposed the newest checks run's run_started_at, but that is not safe. A label or body edit after a pass starts a newer checks run, and pr-verify then skips that head as already judged (pr-verify.yml concurrency comment). A spec edit between the pass and that newer run would be missed. A new test pins this case, and it fails under that bound.
  • Behavior differences:
    • The "its spec could not be checked" summary lines now come after the per-PR lines.
    • Such a line no longer appears for a PR that the first stage drops.
    • A head that has no pass but whose spec moved now costs one GraphQL query more than before.
  • ci-status (pr-require-checks.yml). The lanes step now prints the same api-calls github-token=N not-modified=M notice the log-api-budget action prints, from the same reading file. The lanes-hold reader records into that file (API_BUDGET_TOKEN: github-token). The aggregate-results action's calls are not recorded, so N is a lower bound, like every other job's.
  • Declined: listing pr-verify once instead of 9 status listings. There were 511 pr-verify runs in the last 24 h. That is 6 pages, over LANE_RUN_PAGES = 5, so a single listing would fail the run today. With the cap raised it saves about 3 calls a run, about 140 a day. At about 900 runs a day it would cost more than the 9 listings.

Verification

  • reconcile tests. node --test in .github/scripts/maintenance-reconcile-lost-runs: 117 pass. The new tests are:
    • A spec edited after a pass, followed by a newer checks run, still reaches the full read and re-runs the newer run. This fails under the newest-checks bound.
    • Five passed PRs with untouched specs make 0 pass artifact reads and 1 GraphQL query, 17 calls in all. On origin/main's reconcile.mjs the same test makes 20 artifact reads.
    • The existing spec-case table now pins the pre-filter reads and pass reads for each case.
  • log-api-budget tests. node --test on log-api-budget.test.mjs: 37 pass. A new test runs ci-status's notice block and compares its output with the action's.
  • Other checks:
    • scripts/ci-ordered-skips.test.sh: 13 pass.
    • actionlint is clean.
    • runner-policy: "Runner policy passed."
    • scripts/check-affected.sh --tests: 10 passed and 6 skipped. It ran no test suites, because the 61 selected suites are left to their CI lanes.
    • Of the 38 selected Node suites, 2,105 tests pass. 10 fail only in .github/actions/resolve-config, on Cannot find package 'ajv', because that package is not installed in this worktree.

Related

  • pr-require-checks.yml is on ci-status's hold list, so this change goes through hold review.
  • The spend ranking's cut 1 (the artifact listing in babysit_verify_evidence.py) is separate and is not in this PR.

🤖 Generated with Claude Code

kyle-sexton and others added 3 commits October 27, 2026 16:19
The lost-run reconciler read four pass artifacts and each pass's run for
every passed PR on every tick, before the one GraphQL pre-filter dropped
almost all of them. Run the pre-filter first against the head's earliest
checks run, which every pass at the head started after, so it drops only
heads the old path also dropped; the pass reads and the existing pre-filter
then run for the heads it keeps.

ci-status now also writes the api-calls notice log-api-budget writes, so
the hub can rank its spend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T21:15:46.755377Z 63f1a40 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 0b9a8c6903f42055602bc518764432f2dd9f91fc: 0 finding(s), 0 as inline comments.

No security findings.

critical-findings: 0

Scope read. The diff has 3 files: reconcile.mjs, reconcile.test.mjs and log-api-budget.test.mjs. None of them matches a trusted path. The PR description also says pr-require-checks.yml changed (a ci-status api-calls notice). That file is not in .claude-lane/pr.diff, so I did not review that change. If it lands in this PR, it is a trusted-path file and needs its own pass. No plugin manifests changed, so no manifest <field> lines apply.

What I checked. The change moves the existing time-based prefilterSpec ahead of the pass-artifact reads. The bound is the earliest pr-require-checks run's created_at at the head. The reconciler's job is to re-run pr-verify when a spec changes after a pass, so a wrong skip would leave a stale pass standing.

  • The bound is sound.
    • pr-verify.yml triggers only on workflow_run of pr-require-checks (lines 43-47). So every pass at a head started after some checks run at that head was created.
    • Spec times after the newest pass are therefore also after the earliest checks run. The first stage keeps a superset of what the old pre-filter kept.
    • own is filtered to this repository's heads (reconcile.mjs:502), so a fork run at the same SHA cannot supply the earliest time.
  • Attacker input. An outside author cannot back-date a GitHub-server-set updatedAt or lastEditedAt. Every fail-safe in prefilterSpec still keeps the PR: an unreadable time, an incomplete comment page, a GraphQL failure, or a null start. A missing or non-numeric created_at becomes null, which prefilterSpec keeps (reconcile.mjs:423).
  • Regression test. The newer-checks-run case is pinned, so the bound cannot silently move to the newest run.
  • Decision authority. judgeSpec is unchanged and still makes the skip or re-run decision.
  • Token and write surface. The change adds no new writes, tokens or permissions, and it widens no trust control. It adds one batched GraphQL read and removes REST reads.

Spend. The change cuts GITHUB_TOKEN API calls only. It adds no billed resource and no hold.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 0b9a8c6903f42055602bc518764432f2dd9f91fc: 0 finding(s), 0 as inline comments.

No findings. I read the diff, .github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs around plan, passStarts and prefilterSpec, the pr-verify.yml triggers, and the ci-status lanes step in pr-require-checks.yml. I did not run any tests. The PR's test counts and call-count measurements are author-claimed and unverified by me.

What I checked

  • The bound is sound. pr-verify.yml triggers only on workflow_run of a completed, successful pr-require-checks run at the same head (lines 45-47 and 73-75). Every pass artifact at a head therefore starts after that head's earliest checks run was created. A spec time after the newest pass is also after that earliest creation time. The first prefilterSpec stage keeps every head the old single stage kept. A null or NaN bound keeps the head, because prefilterSpec keeps any candidate with an unparseable start.
  • No crash on an empty list. Math.min(...[]) returns Infinity, and new Date(Infinity).toISOString() would throw. That path is unreachable, because isPassedChecks(checks) requires a run whose path is in RECORDED_UPSTREAMS in the same own list.
  • Control flow is intact. Rate-limit errors are still rethrown from the new loop. Other passStarts failures are still logged and re-run nothing. The second prefilterSpec and the judgeSpec stage run unchanged on the survivors.
  • Tests match the risk. One test shows a spec edit followed by a newer checks run still re-runs. Another shows five untouched heads make 0 artifact reads and 1 GraphQL query. The case table now pins the read counts for each case. The new log-api-budget test extracts the actual ci-status notice line from pr-require-checks.yml (line 5280 at head) and compares its output with the action's.

Notes, not counted as findings

  • The change spends fewer GITHUB_TOKEN calls per run, with one extra GraphQL query when a head has no pass but its spec moved. It involves no billed resource, so I set no hold.
  • The diff has 3 files and does not include pr-require-checks.yml. The PR description's api-calls notice in ci-status is already present at head, so I reviewed it only through the new test.
  • The bound assumes the earliest checks run still appears in the head's run listing. The PR acknowledges this (it could be missing only if someone deleted the run by hand), and I agree it is acceptable.
  • The PR has no closing issue, so there were no acceptance criteria to check.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 27, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 27, 2026
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

Unattended merge: risk HIGH at head 63f1a404db9cf97ece703d7ef4e8f2319b2ded99. Only LOW, or MED on class, size and reported review-finding reasons alone, merges without a human. A floor path holds it: a human merges it in person, never an agent or a lane.

HIGH high-path:.github/workflows/pr-require-checks.yml: .github/workflows/pr-require-checks.yml
MED evidence-unbound:claude-review: not bound: lane-source-changed
MED evidence-unbound:claude-security-review: not bound: lane-source-changed

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

Unattended merge: risk HIGH at head 63f1a404db9cf97ece703d7ef4e8f2319b2ded99. Only LOW, or MED on class, size and reported review-finding reasons alone, merges without a human. A floor path holds it: a human merges it in person, never an agent or a lane.

HIGH high-path:.github/workflows/pr-require-checks.yml: .github/workflows/pr-require-checks.yml
MED evidence-unbound:claude-review: not bound: lane-source-changed
MED evidence-unbound:claude-security-review: not bound: lane-source-changed

This refusal has stood unchanged at this head, so a person is needed: the pr-merge lane added the human-gated label.

@melodic-automation-lanes melodic-automation-lanes Bot added the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 28, 2026
@kyle-sexton kyle-sexton removed the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 28, 2026
@kyle-sexton
kyle-sexton enabled auto-merge October 28, 2026 01:25
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 28, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 28, 2026
@melodic-automation-lanes melodic-automation-lanes Bot added the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 28, 2026
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

The pr-unblock lane stopped on this pull request and needs a person: a conflicted path is a workflow, submodule config or a settings file, which the lane does not resolve.

  • Reason: protected-path-conflict
  • Resume key: 63f1a404db9cf97ece703d7ef4e8f2319b2ded99 (the head the lane stopped on)

To resume, deal with the reason above, remove the needs-human label (and do-not-merge, if set), then dispatch the pr-unblock workflow with pr-number=10527 and activity=resolve-conflicts:

gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=10527 -f activity=resolve-conflicts

kyle-sexton and others added 2 commits October 27, 2026 21:41
…uns-prefilter

Conflict in pr-require-checks.yml ci-status: both sides added the
api-calls notice. Keep main's api_budget_emit_calls helper line over the
branch's inline awk (same output), and point the branch's executing test
at that one-line block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

The pr-unblock lane escalated this pull request as protected-path-conflict, and that no longer holds at 0b9a8c6903f42055602bc518764432f2dd9f91fc: GitHub reports no conflict with main. pr-merge removed needs-human, so the lanes run on it again. To keep a person on it, add needs-human back: a label a person adds is never removed.

@melodic-automation-lanes melodic-automation-lanes Bot removed the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 28, 2026
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit 5d612cf Oct 28, 2026
57 of 58 checks passed
@kyle-sexton
kyle-sexton deleted the perf/reconcile-lost-runs-prefilter branch October 28, 2026 03:32
@github-actions

Copy link
Copy Markdown
Contributor

Post-merge mutation testing

  • Range: f16ac44..5d612cf
  • Merged pull request: #10527
  • Outcome: no result. The run's summary says why.

The run's summary has the full report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant