Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changes/evals/work-resolve-issue-issue-10504-5c1d9a3e.md
Comment thread
melodic-automation-lanes[bot] marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Fixed

- **plugin-eval:** the validity gate's new `scaffold` check marks a run INVALID when a case's `case.yaml` sets `context.scaffold_script` and the gate was not given `--scaffolded`. Without `--scaffold` the eval skips such a case's script, both arms run in an empty workspace, and the gate had still printed `verdict: VALID`. Pass `--scaffolded` when the eval ran with `--scaffold`, since no documented result field records it. A `case.yaml` or case directory the gate cannot read, or a result with no suite root or case directory to find it by, leaves the check unchecked. A `case.yaml` that starts with a UTF-8 byte order mark, or puts the `scaffold_script` value on the next line, still counts as setting a script. ([#10504](https://github.com/melodic-software/claude-code-plugins/issues/10504))
24 changes: 12 additions & 12 deletions plugins/evals/skills/plugin-eval/SKILL.md
Comment thread
melodic-automation-lanes[bot] marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
Expand Up @@ -263,26 +263,26 @@ a committed `mocks/.replay/` so agent mocks replay without a model call.
the JSON is the only record of what happened.

Where a harness guard other than the worktree refusal above blocks a Bash command containing the
bare word `eval`, run the same command through the PowerShell tool instead. The command text is
unchanged; only the tool differs.
bare word `eval`, run the same command, unchanged, through the PowerShell tool instead.

## Reading the delta

Read in this order. Stopping early at any step is the finding.

1. Run the validity gate before reading any number:
1. Run the validity gate ([read order](reference/reading-results.md#read-order)) before reading any number:

```bash
python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/run-validity.py" results.json --runs <the run count the eval used>
python3 "${CLAUDE_PLUGIN_ROOT}/skills/plugin-eval/scripts/run-validity.py" results.json --runs <the run count the eval used> [--scaffolded]
```

The run count is the run's `--runs`, else the cases' `runs`, else 3. Without the traces
`--keep-temp` kept it reports the trace checks unchecked and the run INVALID. Report a score,
delta, or interval only from `verdict: VALID` (exit 0), naming any warnings it printed. On
`verdict: INVALID` (exit 1), report INVALID with every reason on that line and no number, then
fix the cause and rerun. Tell the user to post that INVALID line and its reasons in place of
the number; "post nothing" is not the instruction. Exit 2 means the file could not be read or
an argument was wrong: say which. The steps below still apply to a VALID run.
The run count is the run's `--runs`, else the cases' `runs`, else 3. Without the traces `--keep-temp` kept it reports
the trace checks unchecked and the run INVALID. Pass `--scaffolded` when the eval ran with `--scaffold`, since no result
field records it; without it the `scaffold` check fails a case whose `case.yaml` sets `context.scaffold_script`, and a
case directory it cannot find or a `case.yaml` it cannot read leaves that check unchecked and the run INVALID. Report a
score, delta, or interval only from `verdict: VALID` (exit 0), naming any warnings it printed. On `verdict: INVALID`
(exit 1), report INVALID with every reason on that line and no number, then fix the cause and rerun. Tell the user to
post that INVALID line and its reasons in place of the number; "post nothing" is not the instruction. Exit 2 means the
file could not be read or an argument was wrong: say which. The steps below still apply to a VALID run.

A with-arm denial aimed at or under the plugin's directory, at a directory above it (which covers
its files), or with no absolute path or no known plugin directory, is a FAIL: the agent reached
Expand Down Expand Up @@ -489,7 +489,7 @@ record a drift outcome in this plugin's CHANGELOG.
| Worktree Bash guard refusal of the user's `!` command | Observed on Claude Code 2.1.285 ([records.md](reference/records.md#worktrees)) | 2026-10-01 | A release note touches worktree isolation or `plugin eval` |
| CLI linked-worktree refusal | Observed on Claude Code 2.1.289 ([records.md](reference/records.md#worktrees)) | 2026-10-04 | A release note changes the limit or the CLI's repository screening |
| Model and judge resolution | <https://code.claude.com/docs/en/plugin-evals#command-options> | 2026-10-01 | The table changes its `--model` or `--judge-model` row |
| `--scaffold` for a case with a `scaffold_script`, and the zero delta without it | <https://code.claude.com/docs/en/plugin-evals#add-setup-or-history-with-case-yaml>; the `--scaffold` row of <https://code.claude.com/docs/en/plugin-evals#command-options>; two discarded passes on Claude Code 2.1.309 and 2.1.310 (issue #9414) | 2026-10-27 | That section changes when a `scaffold_script` runs, or a release note touches `--scaffold` |
| `--scaffold` for a case with a `scaffold_script`, the zero delta without it, and the gate's `--scaffolded` because no documented result field records `--scaffold` | <https://code.claude.com/docs/en/plugin-evals#add-setup-or-history-with-case-yaml>; the `--scaffold` row of <https://code.claude.com/docs/en/plugin-evals#command-options>; two discarded passes on Claude Code 2.1.309 and 2.1.310 (issue #9414); <https://code.claude.com/docs/en/plugin-evals#json-result> | 2026-10-27 (JSON result: 2026-10-29) | That section changes when a `scaffold_script` runs, a release note touches `--scaffold`, or the JSON result section lists a field recording `--scaffold` (then read it in place of the flag) |
| Flag order, trust refusal, flag defaults | `claude plugin eval --help`; <https://code.claude.com/docs/en/plugin-evals#command-options> | 2026-09-12 | `--help` no longer matches, or a release note touches the flag set |
| Cost anchors (list-price estimates, this plugin's read-only suite, Claude Code 2.1.287 and 2.1.296) | [records.md, Cost anchors](reference/records.md#cost-anchors) | 2026-10-11 | A release note touches `plugin eval`, the served model changes, or a pass averages more than 0.2 USD per run |
| Spoke reach | <https://code.claude.com/docs/en/plugin-evals#grant-tools>; kept traces in [records.md](reference/records.md#spoke-reach) | 2026-10-04 | A release note touches `plugin eval` or sandbox permissions, or a kept trace on 2.1.289 or later shows a spoke `Read` denied |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,12 @@ statement of record: with `--json <file>` the terminal summary table is suppress

## Read order

1. `scripts/run-validity.py <json> --runs <n>`. Only `verdict: VALID` lets a number be reported;
1. `scripts/run-validity.py <json> --runs <n>`, plus `--scaffolded` when the eval ran with
`--scaffold`, since no documented result field records it. Without it the `scaffold` check
fails every case whose `case.yaml` sets `context.scaffold_script`, because both arms of that
case ran in an empty workspace; a `case.yaml` or case directory it cannot read, or a result with
no `suite.root` or case `dir` to find it by, leaves the check unchecked and the run INVALID.
Only `verdict: VALID` lets a number be reported;
on `verdict: INVALID`, report INVALID and its reasons. The command and its exits are in
[SKILL.md, Reading the delta](../SKILL.md#reading-the-delta).
2. `partial`. When `true`, `partialReason` is `cost_ceiling`, `interrupted`, or `auth_failed`. The
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"cases": [
{
"name": "control-no-trigger",
"dir": "evals/control-no-trigger",
"runsPerCase": 3,
"graders": [
{"name": "names-conftest", "type": "regex"},
Expand All @@ -22,6 +23,7 @@
},
{
"name": "measurable-criterion",
"dir": "evals/measurable-criterion",
"runsPerCase": 3,
"graders": [
{"name": "four-properties", "type": "llm"},
Expand All @@ -38,6 +40,7 @@
},
{
"name": "noise-before-gain",
"dir": "evals/noise-before-gain",
"runsPerCase": 3,
"graders": [
{"name": "noise-verdict", "type": "llm"},
Expand Down
108 changes: 104 additions & 4 deletions plugins/evals/skills/plugin-eval/scripts/run-validity.py
Comment thread
melodic-automation-lanes[bot] marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
@@ -1,17 +1,25 @@
#!/usr/bin/env python3
"""run-validity - decide whether a `claude plugin eval` run's score may count.

run-validity <aggregate-result.json> --runs N
run-validity <aggregate-result.json> --runs N [--scaffolded]

--runs is the run count the eval was invoked with (its --runs, else the case
default). Reads the result file and, where each run's tracePath resolves, the
kept trace (`--keep-temp`). A relative tracePath or suite root is read relative
to the result file. Prints one line per check, detail lines indented under it:
default). --scaffolded says the eval ran with --scaffold, which no documented
result field records. Reads the result file and, where each run's tracePath
resolves, the kept trace (`--keep-temp`). A relative tracePath or suite root
is read relative to the result file. Prints one line per check, detail lines
indented under it:

complete partial is false
paid graders no run skipped its paid graders
run errors no run errored, aborted, or ended its trace in an error
row count every case holds N rows per arm; runsPerCase is ignored
scaffold no case's case.yaml sets context.scaffold_script, unless
--scaffolded says the eval ran it; without --scaffold both
arms of such a case ran in an empty workspace. A case.yaml
that cannot be read, a missing case directory, or no suite
root or case dir to find it by, leaves the case unchecked;
a case directory with no case.yaml has no script
traces how many tracePaths resolve, and what that leaves unchecked
session started no run ended at 0 turns with no run error, no failed Bash
result reports the sandbox failing to start, and no run of 2
Expand Down Expand Up @@ -76,6 +84,8 @@
CASE_NAME = "name"
CASE_DIR = "dir" # cases[].dir, relative to suite.root
CASE_PROMPT = "prompt.md" # <suite.root>/<dir>/prompt.md, its frontmatter
CASE_YAML = "case.yaml" # <suite.root>/<dir>/case.yaml, its context block
YAML_CONTEXT, YAML_SCAFFOLD = "context", "scaffold_script" # context.scaffold_script
CASE_RUNS_STATED = "runsPerCase" # the case default, not the run count
CASE_ARMS = "arms"
CASE_GRADER_DEFS = "graders" # cases[].graders[]: name, type, config
Expand Down Expand Up @@ -320,6 +330,90 @@ def check_complete(result, runs):
return PASS, "partial is false", [], None


def yaml_scalar(value):
"""A one-line YAML value without its comment or quotes; "" for null."""
value = re.sub(r"(^|\s+)#.*$", "", value).strip().strip("'\"")
return "" if value.lower() in ("~", "null") else value


def sets_scaffold(case, root):
"""(whether the case's case.yaml sets context.scaffold_script, None) or
(None, why it could not be read). A case directory with no case.yaml has
no script; a case directory that does not exist is unread."""
case_dir = case.get(CASE_DIR)
if not root or not isinstance(case_dir, str):
return None, "no suite root or case dir to find its case.yaml by"
if not os.path.isdir(os.path.join(root, case_dir)):
return None, "no case directory at %s" % os.path.join(root, case_dir)
path = os.path.join(root, case_dir, CASE_YAML)
try:
with open(path, encoding="utf-8-sig") as handle:
text = handle.read()
except FileNotFoundError:
return False, None
except (OSError, ValueError) as error:
return None, "cannot read %s (%s)" % (path, error)
in_context = False
lines = [
line
for line in text.splitlines()
if line.strip() and not line.lstrip().startswith("#")
]
for index, line in enumerate(lines):
top = re.match(r"([^\s#][^:]*):\s*(.*)", line)
if top:
in_context = top.group(1).strip("'\"") == YAML_CONTEXT
if in_context and top.group(2).startswith("{"):
pair = re.search(r"\b%s\s*:\s*([^,}]*)" % YAML_SCAFFOLD, top.group(2))
return bool(pair and yaml_scalar(pair.group(1))), None
continue
pair = re.match(r"(\s+)%s:\s*(.*)" % YAML_SCAFFOLD, line)
if in_context and pair:
if not re.sub(r"(^|\s+)#.*$", "", pair.group(2)).strip():
# No inline value: a more indented next line holds it.
after = lines[index + 1] if index + 1 < len(lines) else ""
indent = len(after) - len(after.lstrip())
return indent > len(pair.group(1)), None
return bool(yaml_scalar(pair.group(2))), None
return False, None


def check_scaffold(cases, root, scaffolded):
"""A case that sets context.scaffold_script runs it only under --scaffold;
without it both arms ran in an empty workspace and the case measured nothing."""
if scaffolded:
return PASS, "--scaffolded: the eval ran every case's scaffold_script", [], None
declared, unread = [], []
for case in cases:
sets, why = sets_scaffold(case, root)
if sets is None:
unread.append("case %s: %s" % (case.get(CASE_NAME), why))
elif sets:
declared.append(str(case.get(CASE_NAME)))
if declared:
return (
FAIL,
"%s set context.scaffold_script and the eval is not marked --scaffolded"
% plural(len(declared), "case"),
[
"case %s: both arms ran unscaffolded; rerun with --scaffold and pass"
" --scaffolded here" % name
for name in declared
]
+ unread,
"case %s set%s a scaffold_script the eval did not run"
% (", ".join(declared), "" if len(declared) > 1 else "s"),
)
if unread:
return (
UNCHECKED,
"%s with no readable case.yaml" % plural(len(unread), "case"),
unread,
"scaffold unchecked in %s" % plural(len(unread), "case"),
)
return PASS, "no case sets context.scaffold_script", [], None


def check_paid(runs):
hits = [r.where for r in runs if r.row.get(RUN_SKIPPED_PAID) is True]
if hits:
Expand Down Expand Up @@ -804,6 +898,11 @@ def main(argv=None):
required=True,
help="the run count the eval was invoked with",
)
parser.add_argument(
"--scaffolded",
action="store_true",
help="the eval ran with --scaffold, so every case's scaffold_script ran",
)
args = parser.parse_args(argv)
if args.runs < 1:
sys.stderr.write("error: --runs must be 1 or more\n")
Expand Down Expand Up @@ -845,6 +944,7 @@ def main(argv=None):
("paid graders", check_paid(runs)),
("run errors", check_errors(runs)),
("row count", check_rows(cases, args.runs, arms)),
("scaffold", check_scaffold(cases, root, args.scaffolded)),
]
lines = []
for name, (status, summary, details, _) in checks:
Expand Down
Loading
Loading