Skip to content

fix(evals): fail the validity gate when a case's scaffold_script never ran - #10919

Closed
melodic-automation-lanes[bot] wants to merge 9 commits into
mainfrom
work-resolve-issue/issue-10504-40838028768-1
Closed

melodic-automation-lanes[bot] wants to merge 9 commits into
mainfrom
work-resolve-issue/issue-10504-40838028768-1

Conversation

@melodic-automation-lanes

@melodic-automation-lanes melodic-automation-lanes Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Closes #10504

Summary

Opened by the work-resolve-issue lane. The run's own summary:

run-validity.py has a new scaffold check. It reads context.scaffold_script from each case's <suite.root>/<dir>/case.yaml and fails a pass that sets one unless the operator passes the new --scaffolded flag, which records that the eval ran with --scaffold. The verdict reason names the case. An unreadable case.yaml, a missing case directory, or no suite root or case dir to find it by reports UNCHECKED, so the run is INVALID. The plugin-evals docs list no result field that records --scaffold, so the gate cannot read it from the result and takes the flag (the re-triage's fallback). The SKILL.md Run step already adds --scaffold when a case sets a scaffold_script (#10438), so this change covers only the validator half.

Not verified

  • run-validity.test.sh and test_run_validity.py ran locally at 36dc479: pytest 64 passed, run-validity.test.sh OK.
  • scripts/check-changelog-fragments.sh --check-required origin/main ran as written and passed.
  • The re-triage's first route, comparing a scaffolded and an unscaffolded result and trace and keying on the field that differs, was not possible: no claude plugin eval run is possible here. The documented result fields record no --scaffold, so the check uses the --scaffolded operator flag (the re-tri
  • Known parser limits the review flagged, left as is because every case.yaml in this repo uses the plain block form: a value on the next line, a quoted nested key, a multi-line flow mapping and a UTF-8 BOM read as no script; a scaffold_script nested deeper under context reads as one
  • tidy: passes 1-3 ran (no edits); pass 4 (ai-slop:audit fix) was skipped to keep the run's turn reserve for the checks, review and outcome

Fix

  • plugins/evals/skills/plugin-eval/scripts/run-validity.py: new scaffold check (yaml_scalar, sets_scaffold, check_scaffold), the --scaffolded flag, and docstring entries for both
  • plugins/evals/skills/plugin-eval/scripts/test_run_validity.py: new tests for a scaffold case with and without --scaffolded (block, commented and flow YAML), a case with no scaffold_script, an unreadable case.yaml and a result with no suite root; existing VALID or exact-verdict tests that have no cas
  • plugins/evals/skills/plugin-eval/scripts/fixtures/run-validity/r2-clean-runs.json: each case carries a dir, as cases[].dir in a real result does
  • plugins/evals/skills/plugin-eval/SKILL.md: Reading the delta step 1 names the scaffold check and when to pass --scaffolded; new Sources row for the JSON result section
  • plugins/evals/skills/plugin-eval/reference/reading-results.md: Read order step 1 adds --scaffolded
  • .changes/evals/work-resolve-issue-issue-10504-5c1d9a3e.md: patch fragment for the evals plugin

Verification

The runner ran each check from the base copy against b559753:

  • scripts/check-changelog-fragments.sh --check: passed

The run reports:

  • read https: //code.claude.com/docs/en/plugin-evals: context.scaffold_script runs only when --scaffold is passed; the JSON result section lists partial, aggregates, cases[] name/aggregates/error/aborted/skippedPaidGraders, costUsd, durationSeconds and claudeVersion, and no field records --scaffold
  • .lane/checks/bin/check-changelog-fragments (run after the review fixes): all 3 changelog fragments under .changes/ are valid
  • .lane/checks/bin/check-affected (run after the review fixes): 12 passed, 6 skipped over 6 changed files
  • fresh-context review subagent over the whole diff: traced every old and new test against the new code and found none failing. It found a fail-open (a missing case directory or suite root read as 'no script') and a '# comment' right after the key read as a value. Both are fixed, and the two root-usin
  • The lane run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40838028768

Left out: the items under Not verified.

Related

#10504

Lane-Activity: work-resolve-issue/resolve-issue
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed e1398f21bcc969a7cb3fb9c551c85b66084e4a47: 0 finding(s), 0 as inline comments.

No security issues found in this pull request. I read the whole diff and the PR context, and ran no code.

The change adds a scaffold check and a --scaffolded flag to the eval validity gate (plugins/evals/skills/plugin-eval/scripts/run-validity.py), plus tests, a fixture edit, docs and a changelog fragment.

  • Trusted paths and manifests: No changed file matches a TRUSTED PATHS glob, no trust control is touched, and no plugin.json or marketplace.json changed.
  • Path handling: The validator takes suite.root and cases[].dir from the operator-supplied result JSON and opens only case.yaml there, as UTF-8, parsed with regexes. Nothing from the file is executed or written. A crafted dir could point the read at another case.yaml, which only changes whether the check passes, so I count it as no security finding.
  • Fail-closed behaviour: An unreadable case.yaml, a missing case directory, or a missing suite root or case dir reports UNCHECKED, which makes the run INVALID. A case that sets a scaffold script without --scaffolded fails the check. This is the stricter direction, and the gate protects score validity rather than a security boundary.
  • --scaffolded flag: It is an operator assertion about the operator's own eval run, so a wrong flag only affects their own verdict.
  • Instruction surface: The SKILL.md edit to the PowerShell-fallback sentence is a rewording. It still says to run the same command, unchanged, through the PowerShell tool, so no standing rule is removed or softened.
  • PR context: It contains no imperative text aimed at the lane.
  • Spend: The change adds no spend.

critical-findings: 0

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed e1398f21bcc969a7cb3fb9c551c85b66084e4a47: 0 finding(s), 0 as inline comments.

No blocking findings.

What I checked (read only; I ran no tests or scripts):

  • check_scaffold and sets_scaffold in plugins/evals/skills/plugin-eval/scripts/run-validity.py, against each closing-issue (fix(evals): run scaffold scripts in plugin-eval and fail validity when a case was never scaffolded #10504) acceptance criterion.
  • I traced the new test inputs through the parser by reading it. They cover block, commented, flow, next-line and BOM forms, plus the ~ and empty-value no-script forms. The parser returns the verdict each test expects.
  • The remaining callers of run-validity.py: run-validity.test.sh, the SKILL.md and reading-results.md commands, and the plugin's own evals. None of them passes --scaffolded, and none needs to. They ask for the command or the verdict rule; they don't run the script. I found no other consumer.

Closing-issue criteria (#10504):

  • Scaffold check names the case and is non-VALID without the signal: met. test_a_case_with_a_scaffold_script_is_invalid_unless_scaffolded asserts the INVALID verdict, the case name and the PASS path with --scaffolded.
  • A fixture fails without the signal and passes with it: met by the same test.
  • An unreadable case.yaml reports UNCHECKED, not PASS: met. The test is test_an_unreadable_case_yaml_leaves_the_scaffold_check_unchecked, and a missing case dir or suite root is also UNCHECKED (the no-suite-root test).
  • A case with no context.scaffold_script is unaffected: met by test_a_case_with_no_scaffold_script_is_unaffected.
  • SKILL.md names the new check: met, in Reading the delta step 1 and the Sources row.
  • scripts/check-changelog-fragments.sh --check-required origin/main exits 0: author-claimed. I did not run it. The new fragment has bump: patch and a ### Fixed entry, which fits a validator bug fix.

Design note, not a finding: --scaffolded is an operator assertion the gate can't verify, because no documented result field records --scaffold. The PR and issue decision 2 both name this as the fallback, and SKILL.md's Sources row says to read the field instead if the docs add one.

Minor, not counted: the SKILL.md diff also rewrites the unrelated "bare word eval" sentence ("run the same command, unchanged, through the PowerShell tool instead") and re-wraps the step 1 paragraph. Neither changes meaning.

Other lines:

  • Manifests: no plugin.json or marketplace.json change, so no manifest lines apply.
  • Paved paths: none checked.
  • Spend: none. The change adds a local validation check and no new eval runs.
  • No imperative text in the PR body or issue asks the lane to approve, skip a check or post elsewhere.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 15f7f7d4be7b67f81a4e5cd6ea3c5b12a1f83465; the summary is in this lane's pull request comment.

claude-lane: claude-review

Lane-Activity: pr-unblock/address-feedback
Lane-Thread: PRRT_kwDOTCGFQM6xfzvW
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

pr-unblock / address-feedback finished round 1 of max 4 on 13dab33.

@melodic-automation-lanes

melodic-automation-lanes Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Unattended merge: native auto-merge leaves this pull request unarmed at head e1398f21bcc969a7cb3fb9c551c85b66084e4a47. The lanes clear this reason on their own; the next scheduled run judges it again.

pr-merge's gate refuses its verifier verdict (verify:repeated-fail)

Lane-Activity: pr-unblock/address-feedback
Lane-Thread: PRRT_kwDOTCGFQM6xgfwP
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

pr-unblock / address-feedback finished round 2 of max 4 on edc3718.

melodic-automation-lanes Bot and others added 3 commits October 29, 2026 10:50
…t-line scaffold_script

SKILL.md's validity-gate step now names the scaffold check where it says
what run-validity.py judges: it fails a case whose case.yaml sets
context.scaffold_script unless --scaffolded is passed, and a case
directory it cannot find or a case.yaml it cannot read leaves the check
unchecked and the run INVALID. The paragraph is wrapped so SKILL.md stays
at 499 lines, under the skill check's 500-line cap.

sets_scaffold reads case.yaml as utf-8-sig, so a byte order mark no
longer hides the context key, and a scaffold_script key with no inline
value counts as set when a more indented line follows it; a bare key
followed by a sibling, or ~, still counts as no script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t --scaffolded

run-validity.py's scaffold check returns PASS under --scaffolded before it
reads any case.yaml, so SKILL.md now says a missing case directory or an
unreadable case.yaml leaves the check unchecked only without the flag, and
the trace-check sentence reads as it did before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lane-Activity: pr-unblock/resolve-conflicts
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 30, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 30, 2026
@kyle-sexton

Copy link
Copy Markdown
Member

Merging as a session call. Basis:

  • pr-verify verdict at the current head e1398f2 is pass-notes: artifact pr-verify-10919-e1398f21bcc969a7cb3fb9c551c85b66084e4a47-pass-notes from run https://github.com/melodic-software/claude-code-plugins/actions/runs/40988925752 (pr-verify, workflow_run on main, conclusion success).
  • The three earlier fail verdicts (heads 15f7f7d, 8be3dcc, 5a45528) were addressed by later pushes; the verifier passed 36dc479 and e1398f2 after them.
  • ci-status, ci-lanes, claude-review-status and security-review pass; the only non-green status is pr-merge/unarmed, which reports the verify:repeated-fail refusal itself.
  • 0 unresolved review threads.
  • verify:repeated-fail is not one of the owner's three stop reasons (intent change, a finding that changes intent or primary design, a real secret, account or billing risk). The lane's permanent hold on it is being removed as a defect in a separate pull request.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 30, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 30, 2026
@kyle-sexton

Copy link
Copy Markdown
Member

Closing for a fresh attempt at #10504: this PR is at the verify cap (verify:repeated-fail, 3 failing heads). Under #10998 the pr-merge lane closes a capped lanes PR for a fresh attempt, but its runs have not reached this PR (the enable-auto-merge step logged 'the lane's room could not be read').

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(evals): run scaffold scripts in plugin-eval and fail validity when a case was never scaffolded

1 participant