Skip to content

chore: apply prompt-audit fixes to discipline through guardrails plugins - #16337

Merged
kyle-sexton merged 5 commits into
mainfrom
chore/prompt-audit-discipline-to-guardrails
Nov 15, 2026
Merged

kyle-sexton merged 5 commits into
mainfrom
chore/prompt-audit-discipline-to-guardrails

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

No related issue: prompt-audit findings from an owner-requested full-read audit

Summary

Group 4 of 10 from a full-read prompt audit (Anthropic's prompt-audit guide) of instruction text, covering 17 plugins from discipline to guardrails. A second agent verified every edit before it reached this branch. The edits lower capital-letter emphasis (ADR 0098), correct stale paths, counts and section names, replace restated upstream facts with pointers, and settle contradictions between files toward the newer rule.

This is instruction text, with these code-carried exceptions:

  • plugins/guardrails/hooks/block-root-delete-target.sh: two block messages now name the real cause (a cd the guard cannot follow, not only CDPATH) and say where a literal path goes for robocopy and .NET delete calls. Blocking logic is unchanged.
  • plugins/disk-hygiene/skills/clean/scripts/hygiene.py, plugins/disk-hygiene/skills/setup/scripts/python3_alias_probe.py: capitals lowered in help and verdict strings.
  • plugins/guardrails/.claude-plugin/plugin.json: description counts 18 guards (the manifest has 18 guard switches); two option descriptions drop a release version and issue number.
  • plugins/evals/evals/price-before-running/graders/estimate.md: the FAIL clause used stale bounds (36 runs, 3 to 4.5 USD) that failed the case's own expected answer; it now matches the PASS clause (24 runs, 4.2 to 5 USD).
  • plugins/firecrawl/evals/fc-map-before-crawl/case.yaml: judge criteria cite SKILL.md rules by name instead of drifted line numbers.

Reverted after review: the research-sweep.js move of the fetch stage to the retrieval role (a behavior change with no test pinning it), and two non-pointer skill cuts without before/after eval evidence (discovery research practitioner-test.md, the education lessons.md "Workspace HTML" bullet). Those files carry main's text.

Fix

  • Applied group-04.patch (made against 7045703ffdd) to current main with git apply --3way: all 129 files applied cleanly, none dropped.
  • Regenerated plugins/guardrails/README.md's options block (scripts/sync-plugin-options-docs.py) and docs/catalog.md (scripts/generate-catalog.mjs) from the manifest change.
  • Removed two double blank lines left by deleted paragraphs, and the scan-flags.md row from scripts/docs-anchor-baseline.txt, which the validator reported stale once that file's link gained an anchor.
  • extract-ssot verify Gate 1 now excludes generator-owned regions and the candidate SSOT's own file from reproduction counts, matching identify and Lesson 14.
  • Added a patch changelog fragment for each of the 17 plugins.

Verification

  • scripts/check-affected.sh --uncommitted: 16 passed, 1 skipped (changed-skills runs in CI), after the two fixes above.
  • scripts/validate-plugins.sh --only "<17 plugins>": "All plugin manifests and the catalog validated."
  • scripts/check-changelog-fragments.sh --check-required origin/main: every changed fragment-mode plugin (17) has a fragment.
  • scripts/sync-shared-copies.sh --check: all 904 registered copies match.
  • Pre-push check-affected on the review-fix head: passed after merging current main.
  • Left out: full .github suite and the guardrails hook suites (CI runs them); bash -n passed on the changed hook.

Related

N/A

🤖 Generated with Claude Code

Apply the verified prompt-audit edits for 17 plugins (discipline to
guardrails): capital-letter emphasis lowered per ADR 0098, stale paths
and counts corrected, restated upstream facts replaced by pointers, and
contradictions between files settled toward the newer rule.

Code-carried text: research-sweep.js runs its fetch stage on the
retrieval role and drops a fixed summary length; two root-delete guard
block messages name their real causes; hygiene.py and the alias probe
lose capitals in message strings; the guardrails manifest counts 18
guards and drops history from two option descriptions; one grader and
one judge criterion lose stale numbers.

Regenerated the guardrails options README block and docs/catalog.md,
removed a stale docs-anchor baseline row, and added patch changelog
fragments for every touched plugin.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 15, 2026 13:59
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T14:02:41.021104Z 448d499 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 448d4999c5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/fix-docs
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 43722088079

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/apply-patch
Model rung 0
Leg base
Run job success
Activity step success
Commits pushed c638c2e
Run 43722088079

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 0d8e0063cff893b261d2fbe5e4cb1c982e61bc8e: read 8 of 8 changed file(s); 0 finding(s), 0 as inline comments.

Security review: incremental pass (dcde2d1 → 0d8e006), PR #16337

No security issues found.

This lane already reviewed an earlier head (blocking only: yes), so this pass reports only CRITICAL and IMPORTANT findings. It found neither.

critical-findings: 0

Scope. I read every hunk of all 8 files listed in .claude-lane/incremental-files.txt from .claude-lane/incremental.diff. The push restores main's wording to text the earlier head had cut. None of these files matches a TRUSTED PATHS: glob. None is a manifest, workflow, hook, script or trust control, so the manifest and trust-control checks don't apply.

  • Three .changes/** fragments (discovery, disk-hygiene, gaming): the descriptions now match the restored text. Changelog prose only.
  • plugins/discipline/skills/sweep-all/SKILL.md: restores the measured token-cost example (~170K tokens per fork, ~1.4M per pass) and keeps "Budget the sweep as a deliberate spend". This is cost guidance the model reads, not a cap or gate.
  • plugins/discovery/skills/research/SKILL.md: restores the "individual Bash calls, one command per call" wording for the git branch --show-current fallback, and the "2-3 sentence" summary length. The rule still runs the git read only when no topic argument is given, so nothing is loosened.
  • plugins/disk-hygiene/skills/clean/SKILL.md: adds a sentence saying that enforcing "workers return evidence only" through a subagent hook is parked. It adds no permission and removes no gate, and the "Do not treat a worker PowerShell recycle or delete as belt-denied" rule after it is unchanged.
  • plugins/disk-hygiene/skills/clean/reference/safety-model.md: restores the .pulumi-write-test-* / disk-hygiene: resolve the managed-state exclusion contradicting the plugin's own baseline hint #3860 rationale. "Do not re-add a baseline hint for managed state without that lane" is unchanged.
  • plugins/docs-hygiene/skills/extract-ssot/context/lessons.md: restores a provenance sentence about where lessons 1-11 came from.

Instruction-surface lens. No standing rule is deleted, narrowed or softened. Every edit restores or keeps the earlier constraint text, so there was no protected class to check against the instruction exception register.

Settled threads. All three settled threads are marked lines changed since resolution: no, and none of their files is in this push. Nothing is re-raised.

Embedded text. The PR context and settled-threads files contain no directives aimed at this lane.

Spend. No change to model, effort, turn limits or schedules. The sweep-all text is advisory cost guidance only.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 2 inline finding(s) on c638c2e67ea47980c88eeb84382e419ff324ac51; the summary is in this lane's pull request comment.

claude-lane: claude-review

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 0d8e0063cff893b261d2fbe5e4cb1c982e61bc8e: read 8 of 8 changed file(s); 0 finding(s), 0 as inline comments.

Code review, incremental (dcde2d1..0d8e006, 8 files)

No findings. I read every hunk of the 8 files in this incremental scope. This lane already reviewed earlier heads (blocking only: yes), so this review covers blocking findings only. I found none.

This push puts back instruction text that an earlier commit had cut. It also updates the changelog fragments so they stop describing the cuts it undoes.

What changed, and what I checked

  • plugins/discovery/skills/research/SKILL.md
    • The "Summary" item says "2-3 sentence answer" again.
    • The branch-read paragraph goes back to its earlier wording: individual Bash calls, one command per call.
    • The discovery fragment no longer says the PR drops "fixed step or sentence counts", so the fragment matches the restored text.
  • plugins/discipline/skills/sweep-all/SKILL.md
    • The figure from a real run is back: about 170K tokens per fork, about 1.4M for an 8-corrector pass.
    • The discipline fragment never claimed this figure was removed, so it is still accurate.
  • plugins/disk-hygiene/skills/clean/SKILL.md: the sentence saying hook enforcement for subagents is parked is back. No gate or approval rule changes.
  • plugins/disk-hygiene/skills/clean/reference/safety-model.md
  • .changes/gaming/...: the fragment no longer claims an incident history paragraph was dropped. I checked the full diff: the remaining gaming changes are the dated Dagherbou verification row in fork-comparison.md, the refetch wording in upstream-watch.md, and the casing changes in setup/SKILL.md. The fragment now matches them.
  • plugins/docs-hygiene/skills/extract-ssot/context/lessons.md: the sentence saying where Lessons 1-11 came from is back. Lesson numbering and the append-only rule are unchanged.

All three changed fragments stay at bump: patch, which fits changes to instruction text only.

Not covered or not run

  • Closing issue: none, so there are no acceptance criteria to check.
  • Settled threads and prior findings: none of them cover these lines.
  • Embedded instructions: no text in the PR context, settled threads or prior findings tried to give this lane instructions.
  • Verification: I ran no tests. Any test results the PR reports are the author's claim and I have not checked them.

Verifier dropped 0 of 0 candidates.

kyle-sexton and others added 2 commits November 15, 2026 10:25
- Restore research-sweep.js to main: moving the fetch stage to the
  retrieval role is a behavior change with no test pinning it.
- Restore practitioner-test.md and the education lessons.md "Workspace
  HTML" bullet to main: both were non-pointer cuts to shipped skill text
  without the before/after eval evidence skill-criteria.md requires.
- verify Gate 1 now excludes generator-owned regions and the candidate
  SSOT's own file from reproduction counts, matching identify and the
  Lesson 14 mitigation that names both actions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 15, 2026
@kyle-sexton kyle-sexton added the do-not-merge Hard merge gate: do not merge while applied. label Nov 15, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 15, 2026
A cut to an existing skill ships only with eval evidence of no loss
(docs/review/skills-and-agents.md), so these audit edits go back to
main's text:

- disk-hygiene clean SKILL.md: the parked "workers return evidence
  only" hook note.
- disk-hygiene safety-model.md: why the .pulumi-write-test-* hint was
  removed (#3860) rather than exempted.
- docs-hygiene compress target-types.md: the fired 2026-08-15 recheck
  trigger history.
- docs-hygiene extract-ssot lessons.md: where lessons 1-11 were seeded.
- gaming dlss5 reversal-matrix.md: the case that prompted drift
  handling.
- discipline sweep-all SKILL.md: the measured per-fork token figures.
- discovery research SKILL.md: the one-command-per-call repository
  context wording and the 2-3 sentence summary bound.

The discovery, disk-hygiene and gaming changelog fragments drop the
claims these restores made untrue.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton kyle-sexton removed the do-not-merge Hard merge gate: do not merge while applied. label Nov 15, 2026
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit 83af7e4 Nov 15, 2026
77 checks passed
@kyle-sexton
kyle-sexton deleted the chore/prompt-audit-discipline-to-guardrails branch November 15, 2026 16:00
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Post-merge mutation testing

Killed 0, survived 4, timeout 0, no coverage 0, invalid 0, not run 2.
Covered-code score 0%; mutation score 0%.

File Killed Survived Timeout No coverage Invalid Not run
plugins/disk-hygiene/skills/clean/scripts/hygiene.py 0 3 0 0 0 0
plugins/disk-hygiene/skills/setup/scripts/python3_alias_probe.py 0 1 0 0 0 0
plugins/guardrails/hooks/block-root-delete-target.sh 0 0 0 0 0 2

Surviving mutants

Location Operator Disposition Cause
plugins/disk-hygiene/skills/clean/scripts/hygiene.py:4782 other productive no-assertion
plugins/disk-hygiene/skills/clean/scripts/hygiene.py:4787 other productive no-assertion
plugins/disk-hygiene/skills/clean/scripts/hygiene.py:4788 other productive no-assertion
plugins/disk-hygiene/skills/setup/scripts/python3_alias_probe.py:122 other productive no-assertion

The run's summary has the full report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant