Skip to content

perf(ci): skip Check Tests at the job level when the PR has a diff base - #17553

Open
kyle-sexton wants to merge 2 commits into
mainfrom
perf/check-tests-skip
Open

kyle-sexton wants to merge 2 commits into
mainfrom
perf/check-tests-skip

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

No related issue: GitHub Optimization loop cut (Check Tests runs a runner only when it has work).

Summary

Check Tests started a runner on every PR, checked out the tree, and then exited at once whenever a diff base existed, because check-plugins already scans the changed test files. That was a 9-18 s checkout with 0 s of work (runs 44614871783, 44622815071), about 369 jobs and 132 runner-minutes a day. The branch lived in the shell because a skipped job used to turn CI Status red. The ordered-skip check now handles that.

Fix

  • Select Tests publishes run_check_tests: ${{ steps.base.outputs.ref == '' }}. An unset base renders true, so the scan runs.
  • Check Tests' job if: adds && needs.select-tests.outputs.run_check_tests == 'true'. The shell branch and DIFF_BASE are gone.
  • ci-status's ordered-skip step (Check each skipped lane against scope) adds check-tests to its lane pattern and maps the lane to its row with gsub("-"; "_"), so check-tests reads run_check_tests. It turns skipped into success only when select-tests succeeded and that row is exactly "false". Any other skip stays skipped, and treat-skipped-as: fail fails it.
  • scripts/check-docs-only-gate.sh lists the new row and allows this job condition, since it rejects any output or job if: it does not list.

Verification

  • bash scripts/ci-ordered-skips.test.sh: 21 pass, 0 fail. It runs the workflow's own step body. Cases include: check-tests skipped beside a diff base passes; skipped with no diff base stays skipped; an empty row stays skipped; a run that failed keeps its failure; other lanes do not borrow the row.
  • bash scripts/check-docs-only-gate.test.sh: 91 pass, 0 fail, including a case that rejects a != '' row.
  • bash scripts/check-cant-fail-scan.test.sh: 24 pass, 0 fail.
  • actionlint .github/workflows/pr-require-checks.yml: clean.

Expected after merge: Check Tests shows as skipped on PRs with a diff base, CI Status stays green, and about 369 jobs/day disappear. Merge latency is unchanged, since the job was never on the critical path.

Same condition as the removed shell branch

The new skip reads exactly the value the old step read, so it narrows nothing:

  • The removed step ran the scan when DIFF_BASE was empty, with DIFF_BASE: ${{ needs.select-tests.outputs.lane_base }}.
  • lane_base is ${{ steps.base.outputs.ref }} (pr-require-checks.yml:855), and the new row is ${{ steps.base.outputs.ref == '' }}, the same output.
  • A manifest change does not empty the base. "A pull request diffs against its base branch, and a merge group against the queue's base commit, a release group included, so a manifest change selects only the suites that declare they read it (test-scope)" (pr-require-checks.yml:961-965, scripts/resolve-diff-base.sh). The check-tests header's older wording, "resolves to the whole tree (no usable base, or a manifest change)", predates that rule.
  • This PR's own run ran every test shard because scripts/plan-test-lanes.sh plans every lane whole for a pr-require-checks.yml change (whole=1) while the diff base stays set. The old step also returned early there (its DIFF_BASE was non-empty), and check-plugins scanned the changed test files.

Related

  • Hub baseline sent before this change, per the agreement to send baselines before any pr-require-checks.yml change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KeBVgdbwavfuWZGmKnMhWX

kyle-sexton and others added 2 commits November 19, 2026 17:17
Check Tests started a runner on almost every pull request, checked out
(9-18 s), then exited in shell because a diff base existed: about 369
jobs a day of checkout-and-exit. With a base, check-plugins already scans
the changed test files.

Select Tests now publishes run_check_tests, true only when the diff base
is empty (an unset base also renders true, so the scan runs). Check Tests
carries it as an ordered skip in its job condition, and the shell branch
is gone.

The skip passes CI Status through the existing ordered-skip step (id
lanes): it turns a skipped check-tests into success only when Select
Tests succeeded and run_check_tests is exactly 'false'; any other skip
stays skipped and the aggregate (treat-skipped-as: fail) reds it. The
docs-only gate pins the new row and the job condition, and
ci-ordered-skips.test.sh covers both directions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KeBVgdbwavfuWZGmKnMhWX
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 19, 2026 22:37
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T22:39:38.930611Z c328753 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed c328753c93634aa3cb0de546d78e396258fe69d8: read 6 of 6 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #17553, skip Check Tests at the job level when a diff base exists

No security issues found.

This PR changes a merge hold: the ci-status ordered-skip step in .github/workflows/pr-require-checks.yml. I read every hunk of all 6 changed files and checked whether the hold now lets anything through that the base version stopped. It doesn't.

  • Same signal as before. The new row is run_check_tests: ${{ steps.base.outputs.ref == '' }} (L847). It reads the same value as lane_base: ${{ steps.base.outputs.ref }} (L855), which the removed shell branch read through DIFF_BASE.
  • Same result when there is a diff base. In the base version the job exited 0 without scanning, so it reported success. Now it skips, and the aggregate counts that skip as success. ci-status gets the same result in both versions.
  • Fails closed. The aggregate turns skipped into success only when select-tests succeeded and the row is exactly "false" (L7031-L7033). If select-tests fails, the job is skipped through its needs edge (its if: has no status function) and stays skipped, so the aggregate fails it. An empty or unset base renders the row true, so the full scan runs.
  • The lane-name mapping can't be borrowed. sub("^(test|lint)-"; "") | gsub("-"; "_") maps check-tests to run_check_tests. Lane names matching test-[a-z]+ contain no further dash, so the mapping for existing lanes is unchanged. The regex is anchored, so check-plugins cannot match. The tests cover that case, and check-docs-only-gate.sh allows exactly the one check-tests / run_check_tests pairing.
  • Nothing else widens. No token scope, trusted actor, permissions: block, or untrusted text path changes. The rest of the diff is docs, comments and tests.

Spend: this reduces CI runner usage (about 369 fewer jobs a day, by the PR's estimate). It is not a cost hold.

critical-findings: 0

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed c328753c93634aa3cb0de546d78e396258fe69d8: read 6 of 6 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #17553, perf(ci): skip Check Tests at the job level when the PR has a diff base

I found nothing to block or flag. I read all 6 changed files in full.

What I checked:

  • The new row has the same value as the old shell branch. run_check_tests: ${{ steps.base.outputs.ref == '' }} reads the same step output that lane_base publishes (pr-require-checks.yml#L855), and lane_base is what the removed DIFF_BASE read. So the job skips in exactly the cases where it used to exit 0. If Resolve the diff base fails, the row reads true, but select-tests has failed, so check-tests skips behind its needs edge. The ordered-skip step then keeps that result as skipped because $s.result != "success", and the aggregate fails it. This still fails closed.
  • The ordered-skip mapping is correct for every lane. With sub("^(test|lint)-"; "") | gsub("-"; "_"):
    • test-python maps to run_python.
    • lint-shell maps to run_shell.
    • check-tests maps to run_check_tests.
    • Other check-* jobs fail the lane regex, so they cannot borrow the row. The new cases in scripts/ci-ordered-skips.test.sh cover this.
  • The structural gate matches the workflow. In scripts/check-docs-only-gate.sh, the expected-row table entry matches the workflow line exactly. The job-condition allowance maps only check-tests to the check_tests row. Check 12 still requires the aggregate to read steps.lanes.outputs.results. The new tests cover acceptance, another check lane borrowing the row, and an inverted != '' row.
  • The extracted step body still tests the whole-tree path. scripts/check-cant-fail-scan.test.sh checks that the scan fails on a hollow test and passes on a clean tree.
  • Nothing downstream breaks when the job is skipped. post-merge-demotion-alert/red-main.mjs and pr-verify/probe/replay.mjs name the job, but neither treats a skipped Check Tests as a failure.

Loosened check: not a loosening. The ci-status lane pattern and the gate's allowed job conditions each gain an entry. Even so, the set of changes that pass is unchanged:

  • With a diff base, the job used to run and exit 0. It now skips and passes only against run_check_tests == 'false'.
  • With no base, the whole-tree scan still runs.
  • Any other skip stays red.

No input that failed before now passes.

Loosened-check verdict: fail (default: the prompt has no loosened_check_verdict: or loosened_check_patterns: line); patterns: 0. The prompt has no loosened_check_decision_record: line, so I did not check the citation requirement. The PR has no closing issue, so there were no issue criteria to check.

Unverified: the pass counts in the PR body (21, 91 and 24 tests, and a clean actionlint run) are the author's claims. I ran nothing to confirm them.

Spend: none. The change only removes runner jobs: about 369 a day of Check Tests on PRs with a diff base.

Verifier dropped 0 of 0 candidates.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

The pr-verify verifier failed head c328753c93634aa3cb0de546d78e396258fe69d8. What it wrote, as data:

What it wrote (33 lines)
Summary: The spec is PR #17553's own trusted title and body (no linked issue). It asks for `Check
  Tests` to skip at the job level whenever the old shell branch would have returned early, with
  `ci-status` treating that skip as an ordered skip. Most of the wiring is correct. The jq
  lane-to-row mapping (`gsub("-"; "_")` gives `run_check_tests`) works and leaves the existing
  `test-*` and `lint-shell` lanes unchanged. The `check-docs-only-gate.sh` allowlist and its tests
  are consistent, the new `ci-ordered-skips` cases are real and can fail, and the rewritten
  `check-cant-fail-scan` step test still checks both a failing and a passing whole-tree scan. In run
  44630416526 `CI Status` passed and `Check Tests` was skipped
  (https://github.com/melodic-software/claude-code-plugins/actions/runs/44630416526/job/135953446234).
  The blocking problem is that the new skip condition uses a different value from the one the
  removed shell branch used. The old step ran the whole-tree hollow-test scan whenever
  `needs.select-tests.outputs.lane_base` was empty. Per the job's own comment, that happens both
  when there is no usable base and when `select-tests` resolves a pull request or merge group to the
  whole tree because of a manifest change. The new `run_check_tests` row is `steps.base.outputs.ref
  == ''`, which only checks whether a base exists. So a whole-tree resolution that still has a base
  now skips the scan, and `ci-status` turns that skip into success. That quietly narrows a check and
  goes beyond the stated aim of only cutting runs that did nothing. The PR's own run hints at this:
  every Python, Bash and Node shard ran, which suggests a whole-tree selection, yet `Check Tests`
  was skipped. No binary files are in the diff, and no text in the diff or PR body tries to steer
  the reviewer.
- `.github/workflows/pr-require-checks.yml:847`: `run_check_tests: ${{ steps.base.outputs.ref == ''
  }}` only checks whether a base exists, but the shell branch it replaces checked `lane_base`, which
  is also empty when `select-tests` resolves to the whole tree on a manifest change. Build the row
  from the same step output that sets `lane_base` (true when `lane_base` would be empty). Otherwise
  whole-tree runs that have a base skip the hollow-test scan and `ci-status` counts the skip as
  success.
- `scripts/check-docs-only-gate.sh:173`: the pinned `run_check_tests` row must match the corrected
  expression in `pr-require-checks.yml`. Update it, and the copies in
  `scripts/check-docs-only-gate.test.sh` and the grep in `scripts/ci-ordered-skips.test.sh`, at the
  same time.
- `scripts/ci-ordered-skips.test.sh:66`: add a case or assertion showing that the `run_check_tests`
  row is true whenever `lane_base` is empty (a whole-tree resolution with a base, such as a manifest
  change), so the skip cannot hide a whole-tree scan.

This pull request leaves the merge queue, and ci-status holds it out while it carries pr-verify-failed. A pr-verify pass removes the label, at a new commit or at this same head: once a trusted person edits the body, and only trusted people have edited it since the fail, this head is judged again, so a fail on the body or the issues it links needs only the body fixed, no new commit.

@github-actions github-actions Bot added the pr-verify-failed pr-verify failed this head; a pass at a new commit, or after a trusted body edit, removes it. label Nov 19, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-verify-failed pr-verify failed this head; a pass at a new commit, or after a trusted body edit, removes it.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant