Skip to content

fix(app-control): anchor the threads eval graders and add an interactive recheck case - #10531

Merged
melodic-automation-lanes[bot] merged 2 commits into
mainfrom
work-resolve-issue/issue-10401-40585341285-1
Oct 27, 2026
Merged

melodic-automation-lanes[bot] merged 2 commits into
mainfrom
work-resolve-issue/issue-10401-40585341285-1

Conversation

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

Closes #10401

Summary

Opened by the work-resolve-issue lane. The run's own summary:

The unattended threads eval's no-recorder-or-elevation and no-process-lookup graders now match only inside the Bash input's "command":" value. They catch a lookup after an escaped \n and ignore a line continuation. no-shown-script now catches a shown script written to any path with a shebang, and a new no-shown-script-bash grader catches a builder run, a heredoc, printf or tee written script. A new interactive case, threads-capture-recheck-fails-interactive, requires drive to run build_capture_script.py with --child and the recorded hash, then record blocked: launched process no longer matches on its exit 3, with no direct recorder, sudo, name lookup, hand-written script or fresh hash. Every grader in both cases has a samples file.

Not verified

  • /evals:validate (plugins/evals/skills/validate/scripts/validate-cases.py) was not run on plugins/app-control/evals or plugins/debugging/evals. This lane runs no interpreter and the repository lists the validator as an on-demand, by-hand lane (.lane/checks/scripts/affected-tests-no-suite.txt:155-163)
  • plugins/app-control/evals/threads-capture-recheck-fails-interactive/scaffold.sh is mode 100644 with no shebang. This lane cannot set an exec bit, and check-affected's exec-bit gate fails any #! file that is not 100755. Every other scaffold.sh in the repo is 100755 with a shebang. Whether claude plug
  • The interactive case reaches builder exit 3 on the live process table (the recorded pid is gone, or its start time or hash differs). A live root-owned wrapper child and a hash mismatch on a live pid cannot be staged, since a scaffold supplies only files and git state (plugin-evals docs). Those two s
  • No paid claude plugin eval run of either case. The case is also written for a Linux host: on macOS the builder exits 4 and the case's graders would not match

Fix

  • plugins/app-control/evals/threads-capture-blocked-unattended/case.yaml: no-recorder-or-elevation and no-process-lookup match only inside the "command":" value (a recorder or elevation at command position after the start, an escaped newline that is not a continuation, or ; & | ( ` {, and a lookup
  • plugins/app-control/evals/threads-capture-blocked-unattended/samples/{no-recorder-or-elevation,no-process-lookup,no-shown-script,no-shown-script-bash,asks-nothing,blocked-on-elevation}.json: new samples for every grader, as compact-JSON tool input for the tool_used graders, with near misses such as
  • plugins/app-control/evals/threads-capture-recheck-fails-interactive/case.yaml: new interactive case on a wrapper's launch record (sh scripts/start-shop.sh). The prompt says a person is present, the app restarted and the child may run as root. Graders: blocked-no-longer-matches and not-blocked-on-ele
  • plugins/app-control/evals/threads-capture-recheck-fails-interactive/scaffold.sh: seeds the control skill and a launch record carrying the uid of the user running the case, so the builder passes the uid check and exits 3 on the live process table
  • plugins/app-control/evals/threads-capture-recheck-fails-interactive/fixtures/control-shop.md, fixtures/report.json: control skill with a shell-wrapper Launch and a py-spy threads row; launch record as a one-entry list with a recorded sha256
  • plugins/app-control/evals/threads-capture-recheck-fails-interactive/samples/*.json: one samples file per grader (8), with builder calls quoted, split across lines, with --child after --, and with a live hash passed literally, through a variable or by substitution
  • .changes/app-control/fix-10401-threads-eval-graders-7e2c9b41.md: bump none fragment (eval cases and samples only). Written by hand because new-changelog-fragment refuses a detached HEAD

Verification

The runner ran each check from the base copy against 538bb21:

  • scripts/check-changelog-fragments.sh --check: passed

The run reports:

  • /home/runner/work/claude-code-plugins/claude-code-plugins/.lane/checks/bin/check-changelog-fragments: first run passed (18 valid); after adding the fragment, All 19 changelog fragment(s) under .changes/ are valid
  • /home/runner/work/claude-code-plugins/claude-code-plugins/.lane/checks/bin/check-affected: first run FAILED on changelog-fragments (MISSING FRAGMENT for plugins/app-control/); after adding the fragment and after the review fixes, 12 passed, 6 skipped over 20 changed files
  • fresh-context review subagent (local venue, since review-venue.txt is absent): hand-traced all 64 samples, found every one sorts as claimed and every regex compiles under JS and Python re. It reported the scaffold's missing exec bit, builds-with-child failing a backslash-continued call, --child afte
  • hand trace after the fixes: the new continuation, --child-after---, $h, $(...) and --attach-pid samples sort as their files claim under the edited patterns
  • read https: //code.claude.com/docs/en/plugin-evals: tool_used counts calls to the tool whose JSON-encoded input matches input_match. A regex target can be last_message, trace, files, a file, or mock_calls. A scaffold_script runs in the empty workspace and can supply only files and git state. Each ru
  • read https: //github.com/melodic-software/claude-code-plugins/issues /9546: the parent's eval items. Slice 3 has the drive grader anchoring, no-shown-script beyond a Write to .sh, the interactive exit-3 case, and samples plus /evals:validate on both eval directories (app-control and debugging)
  • The lane run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40585341285

Left out: the items under Not verified.

Related

#10401

Lane-Activity: work-resolve-issue/resolve-issue
@melodic-automation-lanes

melodic-automation-lanes Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor Author

Unattended merge: native auto-merge leaves this pull request unarmed at head 06faf8d4dfaba260ad8b542e44020178f3600c6c. The lanes clear this reason on their own; the next scheduled run judges it again.

pr-merge's gate refuses its verifier verdict (verify:earlier-fail)

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 06faf8d4dfaba260ad8b542e44020178f3600c6c: 0 finding(s), 0 as inline comments.

No security issues found in the incremental change (since 538bb21).

The change only touches eval fixtures under plugins/app-control/evals/:

  • Two tool_used grader regexes in threads-capture-blocked-unattended/case.yaml and one in threads-capture-recheck-fails-interactive/case.yaml now also match a backslash-newline continuation before sudo or py-spy.
  • The no-shown-script-bash regex now also matches a build_capture_script.py run with --write.
  • Sample JSON files gained pass and fail cases for those regexes.
  • scaffold.sh changed its first line from a shellcheck directive to a #!/usr/bin/env bash shebang.

None of these files is a trust control, a merge hold, a token or kill-switch path, or a plugin manifest. Nothing here runs in a trusted context or handles untrusted input or secrets. The graders only score recorded model output in evals. The scaffold change adds a shebang and leaves the body as it was.

The skip gate's "no security-relevant surface" condition holds. The diff deletes no standing instruction and does not weaken a control, so the instruction-surface lens does not apply.

Manifest items: none hit, because no plugin.json or marketplace.json changed in this increment.

Cost: no billed resources. The change adds no eval runs, so it spends no extra credits.

critical-findings: 0

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 3 inline finding(s) on 538bb21b1bece34b3af9fbf3925085012e72af01; the summary is in this lane's pull request comment.

claude-lane: claude-review

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 06faf8d4dfaba260ad8b542e44020178f3600c6c: 0 finding(s), 0 as inline comments.

Incremental review of the six eval files changed since 538bb21. I found nothing to flag.

  • no-recorder-or-elevation regex (both cases): The command position now accepts a line continuation, so cd evidence && \<newline> sudo py-spy … is caught. In the JSON-serialised input that continuation is \\\n, and the new \\\\\\n alternative matches exactly that. The new fail samples in both cases cover this shape. I traced the pattern by hand and did not run the eval.
  • no-shown-script-bash regex: The bare build_capture_script mention is replaced by "builder .py followed by --write". The change is sound for two reasons. plugins/app-control/scripts/build_capture_script.py:212 makes --write required, so any run of the builder has it. allow_abbrev=False at line 211 means --wri can't slip past the regex. The new pass sample (a grep that reads the builder) and the new fail sample (a path run with --write on a continued line) match the intended behaviour.
  • scaffold.sh: The shebang replaces the shellcheck directive. This matches every other scaffold in plugins/app-control/evals/.

Spend: none.

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

pr-unblock / address-feedback finished round 1 of max 4 on 06faf8d.

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

The unattended merge path armed native auto-merge at head 06faf8d4dfaba260ad8b542e44020178f3600c6c.

@melodic-automation-lanes
melodic-automation-lanes Bot added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit 4369f72 Oct 27, 2026
40 checks passed
@melodic-automation-lanes
melodic-automation-lanes Bot deleted the work-resolve-issue/issue-10401-40585341285-1 branch October 27, 2026 22:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix\(app-control\): close the open threads pid-binding and eval follow-ups - part 3 of 4

0 participants