Repository navigation
fix(evals): fail the validity gate when a case's scaffold_script never ran - #10919
melodic-automation-lanes[bot] wants to merge 9 commits into
Conversation
Lane-Activity: work-resolve-issue/resolve-issue
|
claude-security-review reviewed No security issues found in this pull request. I read the whole diff and the PR context, and ran no code. The change adds a
critical-findings: 0 |
|
claude-review reviewed No blocking findings. What I checked (read only; I ran no tests or scripts):
Closing-issue criteria (#10504):
Design note, not a finding: Minor, not counted: the SKILL.md diff also rewrites the unrelated "bare word Other lines:
|
Lane-Activity: pr-unblock/fix-ci
Lane-Activity: pr-unblock/address-feedback Lane-Thread: PRRT_kwDOTCGFQM6xfzvW
|
pr-unblock / address-feedback finished round 1 of max 4 on 13dab33. |
Lane-Activity: pr-unblock/fix-ci
|
Unattended merge: native auto-merge leaves this pull request unarmed at head |
Lane-Activity: pr-unblock/address-feedback Lane-Thread: PRRT_kwDOTCGFQM6xgfwP
|
pr-unblock / address-feedback finished round 2 of max 4 on edc3718. |
Lane-Activity: pr-unblock/fix-ci
…t-line scaffold_script SKILL.md's validity-gate step now names the scaffold check where it says what run-validity.py judges: it fails a case whose case.yaml sets context.scaffold_script unless --scaffolded is passed, and a case directory it cannot find or a case.yaml it cannot read leaves the check unchecked and the run INVALID. The paragraph is wrapped so SKILL.md stays at 499 lines, under the skill check's 500-line cap. sets_scaffold reads case.yaml as utf-8-sig, so a byte order mark no longer hides the context key, and a scaffold_script key with no inline value counts as set when a more indented line follows it; a bare key followed by a sibling, or ~, still counts as no script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t --scaffolded run-validity.py's scaffold check returns PASS under --scaffolded before it reads any case.yaml, so SKILL.md now says a missing case directory or an unreadable case.yaml leaves the check unchecked only without the flag, and the trace-check sentence reads as it did before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lane-Activity: pr-unblock/resolve-conflicts
|
Merging as a session call. Basis:
|
Closes #10504
Summary
Opened by the work-resolve-issue lane. The run's own summary:
Not verified
Fix
Verification
The runner ran each check from the base copy against b559753:
The run reports:
Left out: the items under Not verified.
Related
#10504