Repository navigation
refactor(adhd,evals): drop one self-check line and add outcome cases - #13541
Conversation
Remove the instructions that tell the model to re-read or re-check its own output with no new evidence: adhd shape's first-line/last-line check, ai-slop's second read after the self-audit, tidy-diff's closing read and second read, evals design's confirm-before-finishing, and audit-instructions' attestation that the run added no interactive gates. Opus 5 verifies its own work without being told to. Task specifications, run steps, independent verifiers, gates and evidence rules stay. ai-briefing, education and architecture had no line to remove. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eck-lines-batch-2
Add design-grading-hygiene-recorded, which checks that the criteria doc names a grader model other than the one under test and states the re-run cost, and audit-fix-self-audit-vocabulary, which checks that the self-audit removes vocabulary the detector does not flag. Keep the rules the removed sentences carried as drafting rules: the self-audit is one pass, and the target-model fail-loud stop is not an interactive gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The before/after run of design-grading-hygiene-recorded scored 1.00 with the line and 0.67 without it: two of three runs without it did not state the re-run cost. That is a drop of a third, so the cut does not ship. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…elf-check cuts A cut to a shipped skill needs eval evidence of no loss, and these skills' cases grant Bash, which plugin eval cannot run on this host. Restore rewrite-guide.md, tidy-diff and generated-code-residue.md, and audit-instructions to their main text. The new ai-slop case stays for a later run; its fragment becomes a none fragment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eck-lines-batch-2
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3e191bf674
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
claude-security-review reviewed Security review: PR #13541, incremental (fc511be → 8b1c91d)No security issues found. I read the whole change in the one file in incremental scope. This was a blocking-only review because the lane already reviewed an earlier head, so it would have reported only CRITICAL or IMPORTANT findings. There were none. What changed:
This is release-bookkeeping data. Nothing in it runs as code. It touches no plugin manifest, workflow, trust control, merge hold, token scope, or trusted path. It also does not delete, narrow or soften any standing instruction. Other checks:
critical-findings: 0 |
|
claude-review reviewed Code review: PR #13541 (incremental,
|
…n's cut grader-model-differs now accepts a judge named directly, such as "graded by Claude Opus", and rejects one naming the generator. audit-fix-self-audit-vocabulary gives each fact its own grader, fact-streaming among them, with labeled samples for every grader. The re-run of design-grading-hygiene-recorded scored 0.83 with design's confirm-before-finishing line and 1.00 without it, so the cut is re-applied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eck-lines-batch-2
Three eval passes did not show the cut is free of loss: the re-run-cost grader passed 7 of 9 runs with the line and 6 of 9 without it, and one pass dropped from 1.00 to 0.67. A cut ships only with eval evidence of no loss, so the line stays and only the new eval case ships for this plugin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eck-lines-batch-2
The fragment front matter takes only bump; the reason moves to the body, matching the ai-slop fragment in this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Update: the evals:design cut is reverted at 8b1c91d. Across three passes the re-run-cost grader passed 7/9 with the line and 6/9 without it, and one pass fell from 1.00 to 0.67, so no-loss is not shown and the line stays (docs/review/skills-and-agents.md:6-8). This PR now ships the adhd:shape cut, the grader fixes and the two new eval cases; the evals fragment is bump: none. |
|
The pr-verify verifier failed head What it wrote (25 lines)This pull request leaves the merge queue, and ci-status holds it out while it carries |
No related issue: owner decision in the poteto session, 2026-11-05 (no new issues while issues are driven to zero)
Summary
Batch 2 of removing self-check instructions, covering eight plugins: ai-briefing, harness-config, education, ai-slop, adhd, evals, code-tidying and architecture. One cut ships, adhd:shape, with a before/after eval pair that shows no loss. The evals:design cut was tried and reverted, because its eval pair did not show no loss (see Verification), so the line at
plugins/evals/skills/design/SKILL.md:215stays. Three cuts are restored until they can be evaluated on a host where plugin eval can grant Bash. Two new outcome cases are added.Owner decision (2026-11-05, poteto session), verbatim: "Go w/ recomendations" on this option: "Remove the lines that ask the model to re-read or double-check its own answer, from every skill and agent. Keep the steps that run real tests, builds or type-checks. Put the verifier-subagent steps behind the new verification setting."
Basis: Prompting Claude Opus 5: "Claude Opus 5 verifies its own work without being told to. If your prompt contains explicit verification instructions ("include a final verification step for any non-trivial task," "use a subagent to verify"), remove them: instructions like these cause over-verification on Claude Opus 5, and removing them reduces wasted tokens with no loss in quality."
Repo rule:
docs/review/skills-and-agents.md:6-8says a cut to an existing skill ships only with eval evidence of no loss.Fix
Line numbers are from origin/main at
2f59a2505c80e8f8c2d8259e50f3bcc3281857b5. None of the eight plugins changed upstream throughb489e435c7cc15ea6db127b2d826b970c1039dd7.Removed (1)
plugins/adhd/skills/shape/SKILL.md:205-206:52), 3 "End with one concrete next action" (:75) and 5 "Restate state every turn" (:92)Restored (1)
plugins/evals/skills/design/SKILL.md:215keeps its main text, "Before finishing, confirm and". The cut was tried and reverted at8b1c91d89d(see Verification).Pending eval (host without Docker links)
These cuts are identified as SELF but restored to their main text exactly, because their cases grant
Bashand this host cannot run them (see Verification).plugins/ai-slop/skills/audit/reference/rewrite-guide.md:261-262audit-fix-self-audit-vocabulary(new, in this PR)plugins/code-tidying/skills/tidy-diff/SKILL.md:77-78tidy-diff-behavior-changing-cleanup-proposed(existing)plugins/code-tidying/skills/tidy-diff/reference/generated-code-residue.md:17-19,:28tidy-diff-behavior-changing-cleanup-proposedplugins/harness-config/skills/audit-instructions/SKILL.md:406-408ai-skills-scope-report-only(existing)For each plugin, build a before copy at
2f59a2505cwith the same frozen cases and an after copy with the cut applied, both outside any git repository. Read the case'sscaffold.sh, then run both at the same time:Then run
run-validity.py <json> --runs 3on each result, andnoise-report.py <after.json> --threshold 0.8 --baseline <before.json> --margin 0.05.Kept borderline lines
plugins/adhd/skills/clarify/SKILL.md:25plugins/adhd/skills/shape/SKILL.md:197-203plugins/ai-briefing/skills/generate/SKILL.md:224-230plugins/ai-briefing/skills/generate/reference/build-pipeline.md:117briefing.mdcan be edited and re-run between the two emits (:120), andslide-generation.md:160-164defines its own slide-time filterplugins/ai-briefing/skills/generate/reference/build-pipeline.md:199plugins/ai-slop/skills/audit/SKILL.md:297-299rewrite-guide.md:133)plugins/ai-slop/skills/audit/reference/rewrite-guide.md:105-107plugins/ai-slop/skills/audit/reference/rewrite-guide.md:258-260plugins/ai-slop/skills/audit/context/persist-findings.md:63plugins/architecture/skills/record-decision/SKILL.md:88plugins/architecture/skills/record-decision/SKILL.md:90plugins/architecture/skills/improve/actions/deepening.md:66plugins/architecture/skills/improve/actions/conventions.md:54plugins/code-tidying/skills/batch-simplify/context/reference.md:79plugins/education/skills/explain/SKILL.md:41plugins/education/skills/explain/SKILL.md:46plugins/education/skills/explain/SKILL.md:106plugins/education/skills/quiz-me/SKILL.md:87plugins/education/skills/teach/context/glossary.md:63plugins/evals/skills/plugin-eval/reference/case-authoring.md:3plugins/harness-config/skills/audit-automation-gaps/SKILL.md:89plugins/harness-config/skills/audit-automation-gaps/SKILL.md:205plugins/harness-config/skills/audit-automation-gaps/SKILL.md:408plugins/harness-config/skills/audit-instructions/SKILL.md:375plugins/harness-config/skills/audit-instructions/reference/conflict-criteria.md:525plugins/harness-config/skills/audit/SKILL.md:377No line with
fresh-contextdispatch wording or afresh-eyes-exemptmarker was edited.New eval cases
plugins/evals/evals/design-grading-hygiene-recorded: the criteria doc names a grader model other than the one under test and states the re-run cost. Regex graders, with labeled samples.grader-model-differsalso accepts a judge named directly ("graded by Claude Opus") and rejects one naming the generator ("graded by Claude Sonnet").plugins/ai-slop/evals/audit-fix-self-audit-vocabulary: a fix removes model vocabulary the detector does not flag ('pivotal', 'delves', 'leverages', 'showcases') and the chat closers. It keeps each fact, and each fact has its own grader (fact-streaming,fact-2gb,fact-dry-run), the wayaudit-fix-plain-wordsdoes it. Regex graders on the file, each with labeled samples.Fragments: adhd (patch), evals (
none), ai-slop (none, for the new case).Verification
plugins/skill-quality/scripts/check-skill.shover theskills/root of all eight plugins, on the final tree: 44 passed, 0 failed.validate-cases.py(/evals:validate) on each plugin'sevals/, run one plugin at a time: exit 0 and 0 FAIL for all eight, the two new cases included.scripts/run-plugin-tests.shover the 97 suites in the eight plugins, before the restores: 4 harness-config suites failed (audit-permission-grants/scripts/permission-rule-check.test.sh,audit/scripts/fix-plugin-drift.test.sh,audit-permission-state/scripts/no-writes.test.sh,audit-permission-state/scripts/permission-state-primary-dir.test.sh). This PR changes nothing in their directories, andno-writes.test.shalso fails in the main checkout. Everything else passed. No ai-slop test was rerun.docs/review/skills-and-agents.md:6-8.Eval evidence
Each pair ran at the same time with the same flags, on plugin copies outside any repository: before =
2f59a2505cwith the same frozen cases, after = the branch with that cut applied. Flags:--trust-plugin --keep-temp --judge-model opus --threshold 0.8 --max-cost-usd 5 --no-publish --ablation none, runs 3,--case <case>. The comparison is before against after, so only the with-plugin arm ran.sh-multistep-action-first(existing; regex graders on the first and last line)/adhd:shapedirectly)design-grading-hygiene-recorded(new)grader-model-differs)Pass 1 was INVALID (a
skill-firedgrader cannot pass on a slash-invoked prompt, so it was removed). On its other graders that pass scored 2 of 3 runs passing on both versions. Pass 3's after copy is the before copy with only line 215 changed.Read these numbers with care.
grader-model-differspassed in every run of every pass, so the grader fix does not explain pass 2's loss. Every difference between versions comes fromrerun-cost-stated. Across the three passes, that grader passed 7 of 9 runs before the cut and 6 of 9 after it. That is within the run-to-run noise these 3-run passes show: before scored 1.00 in pass 2 and 0.83 in pass 3. One case cannot establish non-inferiority, as each compare verdict says ("n too small to call").Cost: adhd 0.25 USD per version; evals 0.39 to 0.45 USD per version per pass (list-price estimates).
ai-slop, code-tidying and harness-config were not run. Their cases grant
Bash, and plugin eval refuses aBash-granting run while~/.dockerholds symlinks (contextsandfeatures.jsonpoint under/mnt/c/). The owner decided against a link swap, so those cuts wait for a host without the links.Related
🤖 Generated with Claude Code