Repository navigation
fix(plugin-profiles): prove the per-hook setup question is asked and carry review fixes - #16507
Conversation
… is asked The case prompt stated the always-on or per-profile answer up front, so a pass showed the answer recorded, not the question asked. The prompt now answers the work interview and the recording offer only, and the graders check that the final reply asks about notes-hooks and that no profile is saved with --yes before the answer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp
…in check check stopped with exit 2 on a malformed user or repository profile, though those levels are reported, never failed. It now reports the file and its error and checks the other levels; a malformed shipped file still stops the check with the file named. scripts/check-plugin-profiles.sh now runs the check with an empty home and config folder and the checked tree as the project, so the caller's installed plugins and own profiles never reach the CI result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp
…ction The operative-phrasing check matched "is|if|when enabled" only, so a "## Next" gated on plugins that "are enabled" or covering one that "is not enabled" passed as a plain mention. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp
… remedy The drift delta sweep and /verification:confirm Stage 1 gated on a plugin being enabled but took the fallback when it was "absent". Per the seam-phrasing contract the fallback covers a plugin that is not enabled, and the report names claude plugin enable, preceded by claude plugin install when the plugin is not installed. The plugin-options cluster comment in the cross-plugin source registry said every other copy sits at a path of its own; the harness-ops and plugin-profiles copies sit at the canonical path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp
…ommand Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Lane run report
Failed GitHub API calls
|
Lane-Activity: pr-refine/fix-docs
Lane run report
Failed GitHub API calls
|
|
claude-security-review reviewed Security review: PR #16507, head ba5b547 (incremental since effa5ae)No security issues found. This was an incremental review, limited to blocking findings only (the lane already reviewed an earlier head). I read every hunk of both files in scope:
Neither file matches a trusted-path glob. No plugin manifest changed and no standing instruction was weakened. Both files are eval fixtures and reach no credentialed lane. Spend: no billing impact. The eval's run budget is unchanged. critical-findings: 0 |
|
claude-review reviewed Code review: PR #16507 (incremental, effa5ae..ba5b547, blocking only)I read every hunk of both files in the incremental scope:
This lane has reviewed an earlier head, so this pass reports blocking findings only. What this push changes: it applies the second fix offered in this lane's earlier finding (discussion_r4582274119).
The Settled: that thread is resolved. The owner's reply states the fix, and the job reports no line changes since resolution. I am not raising it again. Not verified: I ran nothing. The Closing issue: the PR only refers to #16434 ( Manifests: none changed. Spend: removing a tool from an eval case only affects prepaid model usage when the case runs. It is not a billing change. No new findings. Verifier dropped 0 of 0 candidates. |
…in the reply Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp
Post-merge mutation testingKilled 5, survived 2, timeout 0, no coverage 0, invalid 0, not run 0.
Surviving mutants
The run's summary has the full report. |
Refs: #16434
Summary
#16434 stayed open after #16483 because the per-hook setup case answered its own question in the prompt, so the headless session never had to ask it live. This PR rewrites that case so one headless
claude -psession can show the question being asked. It also carries seven review suggestions left from the last pr-review-main verdicts on #16450 and #16483.Fix
Per-hook setup case (
plugins/plugin-profiles/evals/setup-asks-per-hook-plugin-and-saves-user-profile/): the prompt answers the work interview and the recording offer, but leaves notes-hooks' always-on or per-profile choice open. The new graders read:per-hook-question-asked: the final reply names notes-hooks and asks always on or per profile.no-profile-saved-before-answer: fails anyprofiles.mjs save … --yes.The three save graders and their samples are dropped. Saving at user level was proven by the three headless sessions on 03e149a (feat(plugin-profiles)!: switch plugin sets by profile and remove the core bundle #16483).
profiles.mjs check: a malformed user or repository profile is reported by its path, and the check still exits 0; a malformed shipped profile still exits 2.scripts/check-plugin-profiles.sh: the gate runs with an empty temp home and config folder, so the caller's own profiles can no longer change its result.check-skill.shcheck 27: a## Nextgated on plugins that "are enabled" or "is not enabled" now warns, as "is enabled" already did.Seam fallbacks:
plugins/autonomy/reference/routines/drift-delta-sweep.mdandplugins/verification/skills/confirm/SKILL.mdsay "not enabled" instead of "absent", and name the enable remedy. In the autonomy file the remedy is named by surface class, since its reference contracts ban vendor names.scripts/cross-plugin-source-registry.txt: the comment now names where each copy sits.Not changed:
skills/profile/SKILL.md## Next: not a defect..claude/rules/skill-bodies-state-current-rules.mdsays a Next section carries no availability gate.plugins/testing/evals/run-e2e-route-install-suggestion/case.yaml: its tightened grader matchesrun-e2e/SKILL.mdand its samples. Its LLM grader has not been run since that change.Verification
check-skill.test.sh: all assertions passed.profiles.test.mjswithlib/check.test.mjs: 77/77.check-plugin-profiles.test.sh: 4/4.validate-cases.pyonplugins/plugin-profiles/evals: only the existing allowed_tools warnings. An up-front-answer reply and asave --yescall, offered as must-pass samples in a scratch copy, were both rejected.scripts/check-plugin-profiles.sh: passes against the catalog. Pre-pushcheck-affected: passes.Remaining before
Closes #16434: one headlessclaude -psession for this case, run in a throwaway config after the owner signs in to it. It is graded by the case's graders, and the config is logged out afterwards.Related
🤖 Generated with Claude Code
https://claude.ai/code/session_015XWwmfnVygLgeDpAXkfoJp