Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 15 additions & 9 deletions .github/actions/report-check-run/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ conclusion is `success`, `failure`, or `neutral` with one skip reason.
| `run-result` | required | `needs.run.result`; in the read file, the `broker-check` job's result when that is not `success` |
| `pr-head-sha` | `''` | The PR's current head SHA, read from the API by the report job at report time |
| `act-outcome` | `''` | The run job's `act-outcome` output (`steps.act.outcome`) |
| `run-failure` | `''` | The run job's `run-failure` output: the reason a step before the act step failed with |
| `not-run-reason` | `''` | A skip reason the caller's own workflow gives for an activity job that never ran |
| `gate-reason` | `''` | The run job's gate stop reason output; empty when every gate proceeded |
| `head-sha` | `''` | The run job's `head-sha` output; the check is posted on it |
| `can-commit` | `''` | The run job's `can-commit` output; only `false` skips the signed-commit requirement |
Expand All @@ -26,7 +28,7 @@ conclusion is `success`, `failure`, or `neutral` with one skip reason.

The report job trusts only its own inputs and the run job's runner-computed outputs: lane and
activity from its `workflow_call` inputs, `needs.run.result`, and the run job's `act-outcome`,
`can-commit`, gate reason and `head-sha` outputs. The gate reason and head SHA come from the gate
`run-failure`, `can-commit`, gate reason and `head-sha` outputs. The gate reason and head SHA come from the gate
steps, which run before any head-influenced step. The verdict file is written later in that job, so
it is untrusted: it is checked against its contract, and its `lane`, `activity`, gate stop reason
and `head-sha` are cross-checks only. Any disagreement with the inputs is a failure. The verdict's
Expand All @@ -41,20 +43,24 @@ and `head-sha` are cross-checks only. Any disagreement with the inputs is a fail
1. `run-result` `cancelled`: neutral `superseded-sha` when `pr-head-sha` is a 40-hex SHA other
than the posted head SHA; failure otherwise. A job timeout or a manual cancel also reads
`cancelled`, so only a newer head shows the run was superseded.
2. No verdict, a malformed verdict, or one whose lane, activity, gate stop reason or `head-sha`
2. `run-result` `skipped` with a `not-run-reason`: neutral with that reason, and no verdict is
read. A caller posts this for an activity job its own workflow did not run, such as
`publish-feedback` after an `address-feedback` run with no feedback round.
3. No verdict, a malformed verdict, or one whose lane, activity, gate stop reason or `head-sha`
differs from the inputs: failure.
3. `config: invalid`: failure.
4. A `gate-reason`: neutral with the reason mapped through
4. `config: invalid`: failure.
5. A `gate-reason`: neutral with the reason mapped through
[`gate-skip-reasons.json`](gate-skip-reasons.json); an unmapped reason fails. A gate stop
carries no `head-sha`, so the check goes on `head-sha-fallback`.
5. An unverified or missing signed-commit result when `can-commit` is not `false`,
6. An unverified or missing signed-commit result when `can-commit` is not `false`,
`act-outcome` `failure`, or `dirty-tree: true`: failure. No skip reason overrides these.
6. A verdict `skip-reason`: neutral with that reason when `run-result` is `success` and
7. A verdict `skip-reason`: neutral with that reason when `run-result` is `success` and
`act-outcome` is `success` (a script that exited 0 with a reason) or `skipped` (the act step
never ran, so no head code wrote the verdict); failure otherwise.
7. `act-outcome` other than `success`: failure, since nothing explains the skip.
8. `run-result` other than `success`: failure.
9. Otherwise success.
8. `run-result` other than `success`: failure, with `run-failure` appended when set. A run job
that failed before the act step leaves `act-outcome` `skipped`, so this comes first.
9. `act-outcome` other than `success`: failure, since nothing explains the skip.
10. Otherwise success.
- Every neutral reason must be in `$defs/skip-reason` of `<base-path>/docs/conventions/pr-pipeline/pr-pipeline.schema.json`.
A reason outside it, or an unreadable schema, turns the neutral into a failure.
- `details_url` is this workflow run's page, built from the runner's own environment.
Expand Down
16 changes: 16 additions & 0 deletions .github/actions/report-check-run/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,20 @@ inputs:
description: The run job's act-outcome output (`steps.act.outcome`).
required: false
default: ""
run-failure:
description: >
The run job's run-failure output: the reason a step before the act step
failed with, such as request-lane-token's. Appended to a failed run
job's summary; empty when no such step failed.
required: false
default: ""
not-run-reason:
description: >
A skip reason from the caller's own workflow for an activity job that
did not run, with run-result `skipped`. Posts neutral with it and needs
no verdict. Empty for every run job's report.
required: false
default: ""
gate-reason:
description: >
The run job's gate stop reason output, from its gate steps; empty when
Expand Down Expand Up @@ -105,6 +119,8 @@ runs:
ACTIVITY: ${{ inputs.activity }}
RUN_RESULT: ${{ inputs.run-result }}
ACT_OUTCOME: ${{ inputs.act-outcome }}
RUN_FAILURE: ${{ inputs.run-failure }}
NOT_RUN_REASON: ${{ inputs.not-run-reason }}
GATE_REASON: ${{ inputs.gate-reason }}
HEAD_SHA: ${{ inputs.head-sha }}
PR_HEAD_SHA: ${{ inputs.pr-head-sha }}
Expand Down
24 changes: 18 additions & 6 deletions .github/actions/report-check-run/decide.mjs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
// Pure check-run decision for a pipeline activity. Lane, activity, run result,
// act outcome, gate reason, head SHA and the signed-commit result are the
// report job's own inputs or the run job's runner-computed outputs. The verdict
// act outcome, gate reason, run failure, not-run reason, head SHA and the
// signed-commit result are the report job's own inputs or the run job's
// runner-computed outputs. The verdict
// was written after head code ran: it is validated, its lane, activity, gate
// reason and head SHA must match those inputs, and it is never the source of
// the name, the head SHA, the outcome or any text in the check.
Expand Down Expand Up @@ -43,7 +44,7 @@ function isVerdict(verdict) {
export function decide(verdict, inputs) {
const { lane, activity, runResult, actOutcome, signedCommits } = inputs;
const { gateReason, headSha, headShaFallback, prHeadSha } = inputs;
const { gateSkipReasons, skipReasons } = inputs;
const { gateSkipReasons, skipReasons, runFailure, notRunReason } = inputs;
if (
![lane, activity].every(
(part) => NAME.test(part ?? "") && part !== RESERVED_NAME,
Expand Down Expand Up @@ -81,6 +82,11 @@ export function decide(verdict, inputs) {
? neutral("superseded-sha")
: failure("The run was cancelled and no newer head superseded it.");
}
// A caller whose activity job never ran (its precondition was not met)
// names the skip itself; there is no run job and so no verdict.
if (runResult === "skipped" && notRunReason) {
return neutral(notRunReason);
}
if (verdict == null) {
return failure("The run left no verdict.");
}
Expand Down Expand Up @@ -134,13 +140,19 @@ export function decide(verdict, inputs) {
? neutral(verdict["skip-reason"])
: failure("A skip reason came with an act step that did not finish.");
}
// A run job that failed before the act step leaves the act outcome skipped,
// so the run result is checked first and names the failed step's reason.
if (runResult !== "success") {
return failure(
runFailure
? `The run job did not succeed: ${runFailure}`
: "The run job did not succeed.",
);
}
if (actOutcome !== "success") {
return failure(
"The activity step did not run and no skip reason explains it.",
);
}
if (runResult !== "success") {
return failure("The run job did not succeed.");
}
return post("success", "Passed", "The activity ran and passed.");
}
69 changes: 69 additions & 0 deletions .github/actions/report-check-run/decide.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -385,6 +385,75 @@ test("a run job that did not succeed posts failure even when the act step succee
assert.equal(report.conclusion, "failure");
});

// A run job that fails before its act step, for example on a broker denial at
// the lane-token step, leaves the act outcome skipped.
const NOT_RUN = verdict({ config: "not-read", outcome: "not-run" });

test("a run job that failed before the act step names the run-job failure, not the act step", () => {
const report = posted(
decide(NOT_RUN, inputs({ runResult: "failure", actOutcome: "skipped" })),
);
assert.equal(report.conclusion, "failure");
assert.equal(report.output.summary, "The run job did not succeed.");
});

test("a run job failure carries the failed step's reason into the summary", () => {
const runFailure =
"lane-token-denied: the broker answered 403 with reason workflow-modified";
const report = posted(
decide(
NOT_RUN,
inputs({ runResult: "failure", actOutcome: "skipped", runFailure }),
),
);
assert.equal(report.conclusion, "failure");
assert.equal(
report.output.summary,
`The run job did not succeed: ${runFailure}`,
);
});

// A job that never ran its activity (publish-feedback after an
// address-feedback run with no feedback round) reports with no verdict.
test("a not-run reason on a skipped run posts neutral with that reason and needs no verdict", () => {
const report = posted(
decide(
null,
inputs({
runResult: "skipped",
actOutcome: "",
signedCommits: null,
notRunReason: "not-applicable",
}),
),
);
assert.equal(report.conclusion, "neutral");
assert.equal(report.head_sha, HEAD);
assert.equal(report.output.summary, "Skip reason: not-applicable");
});

test("a not-run reason outside the skip-reason enum posts failure", () => {
const report = posted(
decide(
null,
inputs({ runResult: "skipped", notRunReason: "looks-fine" }),
),
);
assert.equal(report.conclusion, "failure");
});

test("a not-run reason never neutralizes a run job that ran", () => {
for (const runResult of ["success", "failure"]) {
const report = posted(
decide(
verdict(),
inputs({ runResult, actOutcome: "failure", notRunReason: "not-applicable" }),
),
);
assert.equal(report.conclusion, "failure", runResult);
}
});

test("every key of the shipped gate-skip-reasons.json posts neutral with its mapped reason", () => {
const shipped = JSON.parse(
readFileSync(
Expand Down
2 changes: 2 additions & 0 deletions .github/actions/report-check-run/report-check-run.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,8 @@ export async function main({
runResult: env.RUN_RESULT,
actOutcome: env.ACT_OUTCOME,
gateReason: env.GATE_REASON ?? "",
runFailure: env.RUN_FAILURE ?? "",
notRunReason: env.NOT_RUN_REASON ?? "",
headSha: env.HEAD_SHA ?? "",
signedCommits: signedCommitsFrom(env),
headShaFallback: env.HEAD_SHA_FALLBACK,
Expand Down
13 changes: 13 additions & 0 deletions .github/actions/report-check-run/report-check-run.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,19 @@ test("a verdict claiming a no-pr gate while the env reason is empty POSTs failur
assert.equal(calls[0].body.head_sha, FALLBACK);
});

test("a run job that failed before the act step POSTs failure carrying RUN_FAILURE", async () => {
const { calls } = await run({
RUN_RESULT: "failure",
ACT_OUTCOME: "skipped",
RUN_FAILURE: "lane-token-denied: the broker answered 403 with reason workflow-modified",
});
assert.equal(calls[0].body.conclusion, "failure");
assert.equal(
calls[0].body.output.summary,
"The run job did not succeed: lane-token-denied: the broker answered 403 with reason workflow-modified",
);
});

test("a malformed env head SHA exits non-zero with no API call", async () => {
const { code, calls } = await run({ HEAD_SHA: "abc" });
assert.equal(code, 1);
Expand Down
1 change: 1 addition & 0 deletions .github/actions/request-lane-token/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ run, and mints nothing ([Check mode](#check-mode)).
| `token` | The lane token, masked. Set only when every check passed |
| `expires-at` | The expiry the broker returned |
| `mint-id` | `<check_run_id>-<run_attempt>`, the broker's audit key for this mint |
| `failure-reason` | The one-line reason the step failed with (`lane-token-denied: ...`); empty on a success |

## What it does

Expand Down
6 changes: 6 additions & 0 deletions .github/actions/request-lane-token/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,11 @@ outputs:
mint-id:
description: The broker's `<check_run_id>-<run_attempt>` for this mint.
value: ${{ steps.request.outputs.mint-id }}
failure-reason:
description: >
The one-line reason the step failed with, such as `lane-token-denied: ...`;
empty on a success.
value: ${{ steps.request.outputs.failure-reason }}

runs:
using: composite
Expand Down Expand Up @@ -118,6 +123,7 @@ runs:
fail() {
echo "::error::$1"
printf -- '- %s\n' "$1" >>"$GITHUB_STEP_SUMMARY"
printf 'failure-reason=%s\n' "$1" >>"$GITHUB_OUTPUT"
exit 1
}
# One printable line, at most 300 characters: broker text is data.
Expand Down
22 changes: 20 additions & 2 deletions .github/actions/request-lane-token/request-lane-token.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -150,14 +150,17 @@ test("with superseded-ok, any other denial reason still fails red", () => {
const run = request({ ...MERGE, SUPERSEDED_OK: "true" }, { status: "403", body: { reason: "already-minted" } });
assert.equal(run.status, 1);
assert.match(run.stdout, /::error::lane-token-denied: the broker answered 403 with reason already-minted/);
assert.equal(run.output, "");
assert.equal(run.output, "failure-reason=lane-token-denied: the broker answered 403 with reason already-minted\n");
});

test("without superseded-ok, a 403 workflow-modified still fails red", () => {
const run = request(MERGE, SUPERSEDED);
assert.equal(run.status, 1);
assert.match(run.stdout, /::error::lane-token-denied: the broker answered 403 with reason workflow-modified: \.github\/workflows\/pr-merge\.yml differs/);
assert.equal(run.output, "");
assert.equal(
run.output,
"failure-reason=lane-token-denied: the broker answered 403 with reason workflow-modified: .github/workflows/pr-merge.yml differs from refs/heads/main\n",
);
});

test("a superseded-ok other than true or false fails before the broker request", () => {
Expand Down Expand Up @@ -201,6 +204,21 @@ test("an expected upstream head without a dispatcher run id, or not a SHA, fails
}
});

// The run job maps this output, and report-check-run appends it to a failed
// run job's check summary.
test("a broker denial fails red and writes its reason as the failure-reason output", () => {
const run = request(
{},
{ status: "403", body: { reason: "workflow-modified", detail: "the workflow file changed" } },
);
assert.equal(run.status, 1);
assert.match(
run.output,
/^failure-reason=lane-token-denied: the broker answered 403 with reason workflow-modified: the workflow file changed$/m,
);
assert.doesNotMatch(run.output, /^token=/m);
});

test("an empty dispatcher run id leaves the body as it was", () => {
const run = request({ LANE: "pr-refine", ACTIVITY: "fix-docs" });
assert.equal(run.status, 0, run.stdout);
Expand Down
55 changes: 55 additions & 0 deletions .github/actions/resolve-config/callers.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,18 @@
// (the caller workflow itself stays editable in a same-repo PR); a failed resolution
// leaves the review on the reusable's defaults; intake never runs without
// resolved arguments; and each sparse checkout holds what this action loads.
// Also: pr-unblock.yml leaves a publish-feedback check on the head whether or
// not its reply job runs.
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import { readFileSync } from "node:fs";
import { test } from "node:test";
import { fileURLToPath } from "node:url";

import { parse } from "yaml";

import { main as reportCheckRun } from "../report-check-run/report-check-run.mjs";

const read = (relative) => readFileSync(new URL(relative, import.meta.url), "utf8");
const workflow = (name) => parse(read(`../../workflows/${name}`));
const RESOLVE = "./.base/.github/actions/resolve-config";
Expand Down Expand Up @@ -147,3 +152,53 @@ test("intake-triage.yml: intake runs only with claude-args resolved from triage-
assert.equal(runs("false", ""), false);
assert.equal(intake.with["claude-args"], "${{ needs.born-triaged.outputs.claude-args }}");
});

// address-feedback's head-sha output is empty after a gate stop.
test("pr-unblock.yml: publish-feedback is reported neutral when address-feedback ran no feedback round", async () => {
const head = "c".repeat(40);
const { jobs } = workflow("pr-unblock.yml");
const publish = jobs["publish-feedback"];
const notRun = jobs["publish-feedback-not-run"];
const context = (round, headSha, activity = "address-feedback") => ({
"inputs.activity": activity,
"needs.address-feedback.outputs.feedback-round": round,
"needs.address-feedback.outputs.head-sha": headSha,
});
assert.equal(evaluate(publish.if, context("true", head)), true);
assert.equal(evaluate(notRun.if, context("true", head)), false);
assert.equal(evaluate(publish.if, context("false", head)), false);
assert.equal(evaluate(notRun.if, context("false", head)), true);
assert.equal(evaluate(notRun.if, context("false", "")), false, "a gate stop gated no head to post on");
assert.equal(evaluate(notRun.if, context("", "", "resolve-conflicts")), false);

// The step's inputs reach report-check-run.mjs through the action's env map.
const action = parse(read("../report-check-run/action.yml"));
const step = notRun.steps.find((s) => s.uses === "./.base/.github/actions/report-check-run");
const values = { ...context("false", head), "github.token": "test-token" };
const env = {};
for (const [name, value] of Object.entries(action.runs.steps[0].env)) {
const input = value.match(/^\$\{\{ inputs\.([\w-]+) \}\}$/)[1];
const given = step.with[input] ?? action.inputs[input].default ?? "";
env[name] = /^\$\{\{/.test(given) ? evaluate(given, values) : String(given);
}
const calls = [];
const code = await reportCheckRun({
env: {
...env,
BASE_PATH: fileURLToPath(new URL("../../../", import.meta.url)),
GITHUB_REPOSITORY: "melodic-software/claude-code-plugins",
GITHUB_API_URL: "https://api.example.test",
},
fetchImpl: async (url, init) => {
calls.push(JSON.parse(init.body));
return { ok: true, status: 201, json: async () => ({ id: 1 }) };
},
log: () => {},
});
assert.equal(code, 0);
assert.equal(calls.length, 1);
assert.equal(calls[0].name, "pr-unblock / publish-feedback");
assert.equal(calls[0].head_sha, head);
assert.equal(calls[0].conclusion, "neutral");
assert.equal(calls[0].output.summary, "Skip reason: not-applicable");
});
Loading
Loading