Repository navigation
feat(pr-pipeline): write the lane record into check-run annotations and lane.json #7766
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
b43fb60
feat(pr-pipeline): restart limit-stopped lanes and show lane failures…
kyle-sexton 30982b0
fix(harness-ops): match lane classes without a GNU-looking \b
kyle-sexton b861403
fix(pr-pipeline): harden limit-stop restarts against forged records a…
kyle-sexton c29c723
fix(pr-pipeline): address review findings on lane limit restarts
kyle-sexton 7b99dcd
Merge remote-tracking branch 'origin/main' into feat/lane-limit-restarts
kyle-sexton cbec5a8
feat(pr-pipeline): write the lane record into check-run annotations a…
kyle-sexton 1fdc2ce
Merge commit '7b99dcd58c5d9eef06830fca637140cfa1a2cd30' into feat/lan…
kyle-sexton 6f0905b
docs(pr-pipeline): state who can forge the lane record
kyle-sexton 14f6e19
Merge remote-tracking branch 'origin/main' into feat/lane-record-prod…
kyle-sexton ce7a790
ci(pr-pipeline): pin report-lane-outcome to ci-workflows v0.43.0
kyle-sexton 5045248
fix(pr-pipeline): let a lane record restart only a run that reported …
kyle-sexton File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| // The lane record annotation for a posted check run: the run's failure class, | ||
| // limit reset and turn headroom, which ci-workflows' report-lane-outcome | ||
| // computed in the run job from the execution file. Every field is a run job | ||
| // output and is checked against a fixed shape here; nothing the model wrote | ||
| // reaches the message. A rate-limit message keeps the exact shape | ||
| // .github/scripts/pr-unblock/limit-stop.mjs parses. | ||
|
|
||
| const CLASSES = new Set([ | ||
| "auth", | ||
| "rate-limit", | ||
| "overloaded", | ||
| "max-turns", | ||
| "timeout", | ||
| "other", | ||
| "skipped-validation", | ||
| "incomplete", | ||
| ]); | ||
| const RESETS = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(?::\d{2}(?:\.\d+)?)?Z$/; | ||
| const COUNT = /^(0|[1-9][0-9]{0,5})$/; | ||
| const PATH = ".github/workflows/pr-run-activity-write.yml"; | ||
|
|
||
| // One check-run annotation, or null when the run job left no record: the | ||
| // skill did not run, or the class is not one report-lane-outcome emits. | ||
| export function laneRecordAnnotation({ | ||
| lane, | ||
| failureClass = "", | ||
| resets = "", | ||
| turns = "", | ||
| maxTurns = "", | ||
| }) { | ||
| if (failureClass !== "" && !CLASSES.has(failureClass)) { | ||
| return null; | ||
| } | ||
| const tokens = [ | ||
| failureClass === "rate-limit" && RESETS.test(resets) && `resets=${resets}`, | ||
| COUNT.test(turns) && `turns=${turns}`, | ||
| COUNT.test(maxTurns) && `max-turns=${maxTurns}`, | ||
| ].filter(Boolean); | ||
| if (failureClass === "" && tokens.length === 0) { | ||
| return null; | ||
| } | ||
| const record = tokens.map((token) => ` ${token}`).join(""); | ||
| const at = { path: PATH, start_line: 1, end_line: 1 }; | ||
| if (failureClass === "") { | ||
| return { ...at, annotation_level: "notice", message: `${lane} completed:${record}.` }; | ||
| } | ||
| if (failureClass === "skipped-validation") { | ||
| return { | ||
| ...at, | ||
| annotation_level: "warning", | ||
| message: `${lane} concluded: success class=skipped-validation${record} with no execution evidence.`, | ||
| }; | ||
| } | ||
| const skipped = /review/iu.test(lane) ? "AI review" : lane; | ||
| return { | ||
| ...at, | ||
| annotation_level: "failure", | ||
| message: | ||
| failureClass === "rate-limit" | ||
| ? `${skipped} skipped: usage limit reached; re-run after it resets ` + | ||
| `(${lane} exited with: failure class=rate-limit${record}). Last SDK result: in the run log.` | ||
| : `${lane} exited with: failure class=${failureClass}${record}.`, | ||
| }; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| import assert from "node:assert/strict"; | ||
| import { test } from "node:test"; | ||
|
|
||
| import { limitResetsAt } from "../../scripts/pr-unblock/limit-stop.mjs"; | ||
| import { laneRecordAnnotation } from "./lane-record.mjs"; | ||
|
|
||
| const ENDED = "2026-10-17T12:00:00Z"; | ||
| const RESETS = "2026-10-17T15:30:00Z"; | ||
| const run = { completed_at: ENDED }; | ||
|
|
||
| test("a rate-limit record is the limit stop limit-stop.mjs reads, with its reset", () => { | ||
| const annotation = laneRecordAnnotation({ | ||
| lane: "pr-unblock", | ||
| failureClass: "rate-limit", | ||
| resets: RESETS, | ||
| turns: "12", | ||
| maxTurns: "30", | ||
| }); | ||
| assert.equal(annotation.annotation_level, "failure"); | ||
| assert.equal(annotation.title, undefined); | ||
| assert.equal( | ||
| annotation.message, | ||
| "pr-unblock skipped: usage limit reached; re-run after it resets (pr-unblock exited with: failure " + | ||
| `class=rate-limit resets=${RESETS} turns=12 max-turns=30). Last SDK result: in the run log.`, | ||
| ); | ||
| assert.equal(limitResetsAt(run, [annotation]), Date.parse(RESETS)); | ||
| }); | ||
|
|
||
| test("a rate-limit record with no reset reads as a limit stop on the fallback reset", () => { | ||
| const annotation = laneRecordAnnotation({ lane: "pr-unblock", failureClass: "rate-limit" }); | ||
| assert.equal(limitResetsAt(run, [annotation]), Date.parse(ENDED) + 5 * 3_600_000); | ||
| }); | ||
|
|
||
| test("a review lane's limit stop opens with AI review, as limit-stop.mjs expects", () => { | ||
| const annotation = laneRecordAnnotation({ lane: "pr-review", failureClass: "rate-limit", resets: RESETS }); | ||
| assert.match(annotation.message, /^AI review skipped: /); | ||
| assert.equal(limitResetsAt(run, [annotation]), Date.parse(RESETS)); | ||
| }); | ||
|
|
||
| test("a malformed reset is dropped, never passed into the message", () => { | ||
| const annotation = laneRecordAnnotation({ | ||
| lane: "pr-unblock", | ||
| failureClass: "rate-limit", | ||
| resets: "2099-01-01T00:00:00Z) x=(", | ||
| }); | ||
| assert.doesNotMatch(annotation.message, /resets=/); | ||
| }); | ||
|
|
||
| test("another class is a failure record that limit-stop.mjs does not read as a limit stop", () => { | ||
| const annotation = laneRecordAnnotation({ lane: "pr-unblock", failureClass: "max-turns", turns: "31", maxTurns: "30" }); | ||
| assert.equal(annotation.annotation_level, "failure"); | ||
| assert.equal(annotation.message, "pr-unblock exited with: failure class=max-turns turns=31 max-turns=30."); | ||
| assert.equal(limitResetsAt(run, [annotation]), null); | ||
| }); | ||
|
|
||
| test("a success records turn headroom as a notice", () => { | ||
| const annotation = laneRecordAnnotation({ lane: "pr-unblock", turns: "4", maxTurns: "30" }); | ||
| assert.equal(annotation.annotation_level, "notice"); | ||
| assert.equal(annotation.message, "pr-unblock completed: turns=4 max-turns=30."); | ||
| }); | ||
|
|
||
| test("a validation skip is a warning that names its success outcome", () => { | ||
| const annotation = laneRecordAnnotation({ lane: "pr-unblock", failureClass: "skipped-validation" }); | ||
| assert.equal(annotation.annotation_level, "warning"); | ||
| assert.match(annotation.message, /concluded: success class=skipped-validation/); | ||
| }); | ||
|
|
||
| test("no record, an unknown class, or non-numeric turns add nothing", () => { | ||
| assert.equal(laneRecordAnnotation({ lane: "pr-unblock" }), null); | ||
| assert.equal(laneRecordAnnotation({ lane: "pr-unblock", failureClass: "forged class" }), null); | ||
| assert.equal(laneRecordAnnotation({ lane: "pr-unblock", turns: "4 x=1", maxTurns: "" }), null); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.