Repository navigation
fix(plugin-profiles): prove the per-hook setup question is asked and carry review fixes #16507
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
fe6a722
fix(plugin-profiles): make the per-hook setup case prove the question…
kyle-sexton 2e4e40b
fix(plugin-profiles): report unreadable user and repository profiles …
kyle-sexton cbeeba6
fix(skill-quality): flag "are enabled" and negated gates in a Next se…
kyle-sexton 4d16809
docs(verification): word plugin fallbacks as not enabled and name the…
kyle-sexton 5aad6d5
fix(plugin-profiles): guard the profile gate's temp-home delete
kyle-sexton 84c1dde
test(plugin-profiles): use a dictionary word for the unknown-key fixture
kyle-sexton 2c7df0c
docs(autonomy): name the enable remedy by surface class, not vendor c…
kyle-sexton effa5ae
Apply fix-docs on fix/plugin-profiles-followups
melodic-automation-lanes[bot] ba5b547
fix(plugin-profiles): have both per-hook graders expect the question …
kyle-sexton File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| bump: patch | ||
| --- | ||
|
|
||
| ### Changed | ||
|
|
||
| - The drift delta sweep records a lane as not run when its plugin is not enabled, and its report names enabling the plugin as the remedy, after installing it when it is not installed. |
7 changes: 7 additions & 0 deletions
7
.changes/plugin-profiles/fix-plugin-profiles-followups-3f9a1c27.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| bump: patch | ||
| --- | ||
|
|
||
| ### Fixed | ||
|
|
||
| - `check` reports a user or repository profile file it cannot read, naming the file, and checks the other levels instead of stopping; a shipped profile it cannot read still stops the check with the file named. |
7 changes: 7 additions & 0 deletions
7
.changes/skill-quality/fix-plugin-profiles-followups-5d4f92b3.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| bump: patch | ||
| --- | ||
|
|
||
| ### Fixed | ||
|
|
||
| - The `## Next` operative-phrasing check also flags `are enabled` and a negated gate (`is not enabled`, `when not enabled`). |
7 changes: 7 additions & 0 deletions
7
.changes/verification/fix-plugin-profiles-followups-c71a05e9.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| bump: patch | ||
| --- | ||
|
|
||
| ### Changed | ||
|
|
||
| - `/verification:confirm` runs the project's own build, test and lint commands when the `toolchain` plugin is not enabled, and its report names `claude plugin enable toolchain`, preceded by `claude plugin install toolchain` when it is not installed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
8 changes: 4 additions & 4 deletions
8
...ugin-profiles/evals/setup-asks-per-hook-plugin-and-saves-user-profile/prompt.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,13 +1,13 @@ | ||
| --- | ||
| description: "AC15, AC25. The scaffold adds the local marketplace fixture-market to the run's temporary home and installs notes-hooks (ships a SessionStart hook, skill jot) and draw (skill sketch). Setup should run the CLI's inventory and usage commands, settle per-profile for notes-hooks, preview the save of profile docs with --dry-run, then save it with --level user --yes on the stated yes. The slash prompt loads the skill without a Skill call, so no skill-fired grader; the CLI graders are with-only (the CLI ships only with the plugin) and match the script path however it is spelled; no-hand-written-profile fails a profile file written with the Write tool. This case cannot run in the claude plugin eval sandbox today: the run's configuration directory is write-denied there, and the nested claude that inventory and install call is not reachable. It is kept for when the harness allows nesting; the behavior is proven by headless claude -p sessions and, for install, by scripts/install-real-claude.test.mjs instead. Needs --scaffold --allow-tools Bash Skill Write." | ||
| description: "AC25. The scaffold adds the local marketplace fixture-market to the run's temporary home and installs notes-hooks (ships a SessionStart hook, skill jot) and draw (skill sketch). The prompt answers the work interview and declines the recording offer but leaves the per-hook choice open, so one headless session shows setup running inventory and usage, then ending its turn on the question whether notes-hooks is always on or per profile, with no profile saved. per-hook-question-asked reads the final reply; no-profile-saved-before-answer fails any save run with --yes; no-hand-written-profile fails a profile file written with the Write tool. The slash prompt loads the skill without a Skill call, so no skill-fired grader; the CLI graders are with-only (the CLI ships only with the plugin) and match the script path however it is spelled. This case cannot run in the claude plugin eval sandbox today: the run's configuration directory is write-denied there, and the nested claude that inventory calls is not reachable. It is kept for when the harness allows nesting; the behavior is proven by a headless claude -p session instead. Needs --scaffold --allow-tools Bash Skill Write." | ||
| tags: [plugin-profiles, profile, setup, outcome] | ||
| runs: 1 | ||
| max_turns: 30 | ||
| timeout_seconds: 600 | ||
| allowed_tools: [Skill, Bash, Read, Write, AskUserQuestion] | ||
| expected_outcome: "Setup lists notes-hooks and draw with their skills, records notes-hooks as per profile, previews profile docs, then saves it at user level through the CLI." | ||
| allowed_tools: [Skill, Bash, Read, Write] | ||
| expected_outcome: "Setup lists notes-hooks and draw with their skills, then asks whether notes-hooks is always on or per profile and stops there, saving nothing." | ||
| --- | ||
|
|
||
| /plugin-profiles:profile setup | ||
|
|
||
| I write documentation and sometimes sketch diagrams. My answers up front, so you need not ask: notes-hooks is per profile, not always on. Make one profile named docs at user level holding notes-hooks and draw, described as "Write docs and sketch diagrams." Yes, save it once you have shown me the file. | ||
| I write documentation and sometimes sketch diagrams; building docs is my only phase, with no exploring, planning, reviewing, interface design or harness upkeep. No session recording for now. I want one profile named docs at user level, described as "Write docs and sketch diagrams." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
36 changes: 36 additions & 0 deletions
36
...ls/setup-asks-per-hook-plugin-and-saves-user-profile/samples/per-hook-question-asked.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| { | ||
| "pass": [ | ||
| { | ||
| "answer": "Installed: notes-hooks (jot) and draw (sketch).\n\nnotes-hooks ships a SessionStart hook. Should notes-hooks be always on, in every session, or per profile, loaded only by the profiles you add it to?" | ||
| }, | ||
| { | ||
| "answer": "**notes-hooks** runs a hook at session start. Per profile or always on for it?", | ||
| "why": "the two choices in the other order" | ||
| }, | ||
| { | ||
| "answer": "Phase 4 of 7. notes-hooks ships hooks: do you want it **always-on** or **per-profile**?", | ||
| "why": "hyphenated choices, markdown bold" | ||
| } | ||
| ], | ||
| "fail": [ | ||
| { | ||
| "answer": "" | ||
| }, | ||
| { | ||
| "answer": "Saved docs at ~/.claude/plugin-profiles/docs.yaml. As you said, notes-hooks is per profile, not always on. Want me to apply docs to this folder?", | ||
| "why": "the per-hook answer was given up front and recorded, the question was never asked" | ||
| }, | ||
| { | ||
| "answer": "Recorded notes-hooks as per profile rather than always on, since you only build docs. Shall I preview the docs profile?", | ||
| "why": "the answer guessed instead of asked" | ||
| }, | ||
| { | ||
| "answer": "Which phases do you work in: exploring, planning, building or reviewing?", | ||
| "why": "a different question" | ||
| }, | ||
| { | ||
| "answer": "Should draw be always on or per profile?", | ||
| "why": "asks about a plugin that ships no hooks" | ||
| } | ||
| ] | ||
| } |
58 changes: 0 additions & 58 deletions
58
...up-asks-per-hook-plugin-and-saves-user-profile/samples/save-previewed-before-written.json
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.