Skip to content

fix: restore skill text and eval files a prompt audit changed without eval evidence - #16379

Merged
kyle-sexton merged 5 commits into
mainfrom
fix/restore-uneval-skill-cuts
Nov 15, 2026
Merged

kyle-sexton merged 5 commits into
mainfrom
fix/restore-uneval-skill-cuts

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

No related issue: restores skill text and eval files a prompt audit changed without eval evidence

Summary

Ten prompt-audit pull requests cut shipped skill and agent text with no before-and-after eval evidence, and edited eval cases and graders without re-calibration. docs/review/skills-and-agents.md and plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md ("Cutting text") let a cut ship only with eval evidence of no loss. This puts the earlier text back where the audit's edit was a cut.

Fix

  • Listed cuts (54): 47 restored to their earlier text. 1 already carried it (plugins/playwright/skills/playwright/SKILL.md). 6 keep main's text because review showed the earlier text was false at head, so the audit's edit was a correction: the audit-automation-gaps checklist's inline ladder, both Last updated: headers in the permission criteria files, the machine-health C:-only abort, the promotion-evidence plan's line numbers, and the be-concise quotes from the ai-slop catalog.
  • 174 further hunks from the ten diffs under plugins/*/skills/ and plugins/*/agents/ that removed or shortened instruction text, or dropped a directive, number, condition, example, reason, emphasis label or dated record, restored to the pre-audit text.
  • 5 eval files restored (the three mcp-tools case.yaml criteria and two user-interface graders). Eval edits that fixed a broken or contradictory case keep main's text: the price-before-running grader, the testing e2e-mcp-server-no-ui case (its Skill grader needs Skill allowed), the multi-agent expected outcome, and the firecrawl and x descriptions whose earlier line numbers no longer match SKILL.md.
  • Left as main has them: case-only edits, stale paths, names, counts and facts corrected, restated upstream facts swapped for a pointer, as-of dates added, and added text. Review threads on this PR record each correction that was first restored and then put back.
  • Changelog fragments: a patch fragment for each of the 39 plugins whose shipped files change, and a none fragment for mcp-tools and user-interface, whose only change is an eval file.

Verification

  • scripts/check-changelog-fragments.sh --check: all fragments valid.
  • scripts/sync-shared-copies.sh --check: all registered copies match.
  • scripts/check-changed-skills.sh origin/main: 0 failed.
  • scripts/check-changed-links-online.sh origin/main: every added or changed external link resolves; offline lychee over the changed markdown: 0 errors.
  • Local suites over restored files passed: planning interview-defenses, standards-binding, plan-size-keys and plan-stress-test; ai-slop detect; harness-config finding-ids and instruction-scan; lane-launcher; observability probes; docpage, pr-verify, pr-unblock and work-resolve node tests.
  • The pre-push check-affected hook passed on each pushed head.

Related

Restores text changed by #16328, #16330, #16331, #16332, #16333, #16334, #16336, #16337, #16339 and #16340.

Intentional-removal: the audit's replacement wording is removed on purpose; its earlier text returns until eval evidence shows no loss.

🤖 Generated with Claude Code

kyle-sexton and others added 2 commits November 15, 2026 11:11
… eval evidence

Revert the instruction cuts that #16330, #16331, #16333, #16334, #16336,
#16337, #16339 and #16340 made to shipped skill and agent text with no
before-and-after eval evidence: the 54 listed cuts plus 192 further hunks
that removed or shortened instruction text, numbers, conditions, examples,
reasons or dated records. Restore the eval case, prompt and grader files
those pull requests changed to their earlier text. Case-only edits, stale
facts corrected, upstream facts swapped for pointers and added text stay.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T16:19:59.612366Z 2ba492b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2ba492bba3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/x/skills/read/SKILL.md Resolved
The price-before-running grader's FAIL clause contradicted its own PASS
clause and prompt (24 runs, 4.2 to 5 USD); x:read's GET-response detail
belongs behind its existing pointer; monitor.md's dispatched-subagent
exception matches the file's opening contract. Each was a correction or
pointer swap, not a cut, so main's text stays.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The restored x-thread-unroll description carried the bare template
https://threadreaderapp.com/thread/<id>.html, which the online link
check extracts as https://threadreaderapp.com/thread/ and fails on a
404. Main's code span around it stays; the words are unchanged.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 575a91a9684d5673a21f548da0c7547a03a81b49: read 7 of 7 changed file(s); 0 finding(s), 0 as inline comments.

Security review: incremental, b783a90 → 575a91a (PR #16379)

No security issues found in this push. Because this lane had already reviewed an earlier head, this was a blocking-only review: only CRITICAL and IMPORTANT findings would have been reported.

critical-findings: 0

Scope: I read every hunk of all 7 files in .claude-lane/incremental.diff. None of them matches a TRUSTED PATHS glob, and none is a plugin manifest, hook, script, workflow or settings file. All 7 are prose in skill and reference markdown. This push keeps the audit's edits wherever the restored text was stale or contradictory, which is what the PR description says it does.

What I checked:

  • code-tidying/skills/batch-simplify/SKILL.md: the issue-title rule now points at reference/issue-house-style.md and no longer names Conventional Commits. Two controls are unchanged:
    • Filing still needs an explicit request from the user.
    • The scope that keeps agent and enforcement config out of the autonomous simplifier's reach is untouched.
  • harness-config permission criteria (both files): each drops a Last updated: header. These are metadata lines, and no permission rule changes.
  • kindle-dedrm/skills/manage/reference/workflow.md: only a comment on the placeholder tag changed. The fail-closed guard is still in place: the script exits when PINNED_TAG_FROM_VERSIONS_MD has not been substituted.
  • knowledge/skills/video-digest/context/watch-pipeline.md: the context-cost fan-out trigger is reworded to say no script counts the reads, so the agent decides. This is routing for context cost, not a security control.
  • babysit-loop/.../promotion-evidence-implementation-plan.md: stale line-number citations were removed. The fail-closed behaviour it describes is the same.
  • babysit-prs/reference/safety.md: one wording fix about who may have armed an auto-merge: enable-auto-merge.mjs replaces "before this rule". The file is documentation and changes no merge-hold code.
  • Removed or weakened instructions: none of these hunks deletes, narrows or softens a rule that a protected class depends on. I did not read the instruction-exception register for this incremental pass; I judged each change by what it would allow instead.
  • Embedded text: the PR description contains nothing that tries to steer this lane.

Spend: this push changes nothing that spends API credits or could create a charge.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 21 inline finding(s) on b783a9077849731e87981d59745e387ed66a89ef; the summary is in this lane's pull request comment.

claude-lane: claude-review

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 575a91a9684d5673a21f548da0c7547a03a81b49: read 7 of 7 changed file(s); 0 finding(s), 0 as inline comments.

Code review: incremental, b783a90 → 575a91a (blocking only)

I reviewed only the push since b783a90, and only for blocking findings, because this lane already reviewed an earlier head. The push changes 7 files, and I read every hunk in all 7.

Earlier findings: the push resolves them. Each changed hunk puts back main's text where my earlier review showed the audit had made a correction or a pointer swap, not a cut:

  • code-tidying/skills/batch-simplify/SKILL.md:269: issue titles follow the house style again (reference/issue-house-style.md, read before the write). The refactor(<area>): <what> rule is gone.
  • harness-config/.../audit-permission-grants/reference/criteria.md and audit-permission-state/reference/criteria.md: the out-of-date Last updated: lines are removed.
  • kindle-dedrm/skills/manage/reference/workflow.md:42: the fallback comment points at reference/versions.md again. It no longer names the rollback tag v10.0.20.
  • knowledge/skills/video-digest/context/watch-pipeline.md:283-288: the text again says no script counts the reads or sets a threshold. The made-up "read-count" signal is gone.
  • source-control/skills/babysit-loop/reference/promotion-evidence-implementation-plan.md:25-33: the out-of-date line-number citations are replaced with name references.
  • source-control/skills/babysit-prs/reference/safety.md:946: the text again names enable-auto-merge.mjs as one of the things that arm auto-merge, which matches the paragraph above it.

The other files I flagged earlier are no longer in the pull request; it now has 200 changed files, down from 229. They include write-for-agents, audit-automation-gaps, firecrawl case.yaml, map-dependencies, attribution gotchas.md, course-digest storage-schema.md, docpage-digest, machine-health, multi-agent, boris, repo-hygiene, the x evals, be-concise and rename-references. Their changelog fragments (machine-health, multi-agent, repo-hygiene, writing, x) are gone too, so no fragment is left for a plugin with nothing shipped.

Not counted (advisory, dropped because this review is blocking only): the PR description still says 42 plugins get patch fragments, and four none fragments including multi-agent and x. The file list no longer matches either count.

No new blocking findings in this push.

Cost: the push only edits documentation text. It spends nothing and starts no billed resource.

Verifier dropped 0 of 0 candidates (no candidates; no verifier subagent ran).

The review round showed these restored passages were false or
contradictory at head, so the audit's edits were corrections or
pointer swaps, not cuts: stale line numbers and dates, a rollback tag
given as the pin, size and gate counts, a read-count signal no script
emits, a C:-only abort the trigger and code contradict, examples that
break the rule beside them, house-style and C4-key facts, plugin-repo
counts worded as the user's repository, and eval descriptions and
graders that cite moved lines or contradict their own case. Main's text
stays for each; fragments for plugins left with no change are dropped.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit 17619e8 Nov 15, 2026
207 of 209 checks passed
@kyle-sexton
kyle-sexton deleted the fix/restore-uneval-skill-cuts branch November 15, 2026 17:32
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant