Skip to content

feat(playbooks): add reference-independence and config-before-generator skill rules - #16612

Merged
kyle-sexton merged 2 commits into
mainfrom
docs/skill-authoring-reference-generator-rules
Nov 16, 2026
Merged

kyle-sexton merged 2 commits into
mainfrom
docs/skill-authoring-reference-generator-rules

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

No related issue: two authoring rules decided for the skill-authoring criteria, with no tracking issue.

Summary

playbooks:skill-authoring's criteria (reference/skill-criteria.md) had no rule that a reference file must work outside the skill's step sequence, and no rule for when a shipped skill may generate a repository-specific skill instead of reading repository config.

Fix

Two short rules, each with its reason, in the sections that already hold the neighbouring rules:

  • Reference files: "A reference works without the steps." Another skill or a person loads a reference through the skill's invocation or a documented action (never by path, which keeps the public-surface contract /docs-hygiene:audit-encapsulation checks) and acts on it without running the steps. So a reference names its own inputs and never leans on a value an earlier step set. No existing statement covered this; the nearest bullet ("Each pointer says what the file holds and when to read it") is about pointers, so the rule sits directly after it.
  • Body: "Config before a generated skill." A shipped skill generates a repository-specific skill (as /app-control:map generates a project's control skill) only where repository configuration cannot express the difference; when config can carry it, keep one shipped skill and read the config. Placed after "Layer on a skill, never restate it", the other rule against copies.

Two eval cases, one per rule, and a minor changelog fragment for playbooks.

Verification

Both new cases ran through claude plugin eval (2 runs per arm, with-plugin and no-plugin baseline, haiku judge). The plugin was copied outside the repository first, because the CLI refuses a plugin whose repository has many linked worktrees.

Case With Without Δ
skill-authoring-reference-works-without-steps 1.00 0.50 +0.50
skill-authoring-config-before-generated-skill 1.00 0.25 +0.75
  • reference-works-without-steps: the behavior grader passed in both arms. The base model already removes an obvious step dependency, so the Δ comes from the criteria-read grader. The case guards the rule against regression; it does not show the rule changes behavior.

  • config-before-generated-skill: the first run's grader required an explicit config entry, and it failed the with-plugin runs, which read the linter from the repository's own linter config on every run. That also satisfies the rule, so the grader now accepts either form. On the re-run, the behavior grader passed 2/2 with the plugin and 1/2 without it.

  • validate-cases.py plugins/playbooks/evals: no FAIL; the new cases add only the standard "judge samples need calibration" WARN every llm-graded case here carries.

  • check-skill.sh plugins/playbooks/skills: skill-authoring PASS, 0 errors, 3 warnings, all pre-existing (SKILL.md is unchanged).

  • plugins/playbooks/skills/skill-authoring/scripts/update.test.sh: 32/32 passed.

Related

None beyond the rules above: no other issue or PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC

kyle-sexton and others added 2 commits November 16, 2026 14:15
…or skill rules

skill-authoring's criteria gain two rules:

- Reference files: a reference works without the skill's steps. Another
  skill or a person loads it through the skill's invocation or a
  documented action, never by path, and acts on it without running the
  step sequence, so it names its own inputs.
- Body: a shipped skill generates a repository-specific skill (as
  /app-control:map does) only where repository configuration cannot
  express the difference; when config can carry it, keep one shipped
  skill and read the config.

Each rule has an eval case under plugins/playbooks/evals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
…grader

The config-before-generated-skill grader required an explicit config
entry and failed rewrites that read the linter from the repository's
own linter config on every run, which the rule also allows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 19:19
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T19:20:36.843530Z 9859cb9 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed 9859cb9713c50adfa3cc45b4d112857a8887cf1b: read 14 of 14 changed file(s); 0 finding(s), 0 as inline comments.

Code review: no findings

This PR adds two rules to plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:

  • Config before a generated skill, in the Body section, after "Layer on a skill, never restate it".
  • A reference works without the steps, in the Reference files section, after the pointer bullet.

It also adds one eval case per rule and a minor changelog fragment for playbooks.

What I checked:

  • Rules. Both rules carry their reason, which matches the file's own "Why:" convention. The paths they cite exist at head: docs/contracts/config-cascade/README.md and /app-control:map (plugins/app-control/skills/map/SKILL.md). The config rule agrees with the existing "Stack-specific tools" section, whose step 4 writes the chosen setup into project config rather than into the skill.
  • Eval cases. Each case.yaml follows the same shape and execution block as the neighbouring skill-authoring-* cases, for example skill-authoring-layers-on-the-owning-skill. Each LLM grader states pass and fail conditions that match its case's expected_outcome. The calibration samples cover the fail modes each grader names: still generates a copy, keeps the copy as an option, still points at a step, drops the naming rule.
  • Duplicate files. The two reads-criteria graders and their samples are byte-identical across the two cases (recorded in identical.txt). Each sits at an eval grader or sample path, where it does the same thing, so I read it once.
  • Changelog. A minor bump fits added criteria. No plugin manifest changes, so there are no manifest lines.
  • Closing issues. None; the PR says it has no tracking issue.

The eval scores and validator results in the PR description are the author's claims; I did not run them.

Spend: the two new eval cases add plugin-eval runs on prepaid API credit (20 turns and 600 s max per run). This is not a hold.

Verifier dropped 0 of 0 candidates.

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 9859cb9713c50adfa3cc45b4d112857a8887cf1b: read 14 of 14 changed file(s); 0 finding(s), 0 as inline comments.

Security review: no security issues found

I read the whole diff: all 12 files on the read list. The two copies listed in identical.txt are byte-identical to their first paths, and their paths do nothing different.

What changed

  • plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md gains two rules. Both are additions; no existing rule is deleted, narrowed or softened:
    • "Config before a generated skill"
    • "A reference works without the steps"
  • Two new eval cases: skill-authoring-config-before-generated-skill and skill-authoring-reference-works-without-steps.
    • Each case.yaml sets allowed_tools: [Read, Skill], max_turns: 20 and timeout_seconds: 600. That is a read-only tool set with no Bash, Write or network tools.
    • Each adds prompts, LLM and tool_used graders, and calibration samples.
  • A minor changelog fragment for playbooks.

Lenses checked

  • Instruction surface: the change only adds rules. Neither new rule removes or weakens a guardrail in a protected class. The "never by its path" wording reinforces the existing public-surface contract.
  • Trust controls and trusted paths: no changed file matches a trusted-path glob. Nothing touches the trusted-trigger check, the lane token, the kill switch, the broker or the merge holds.
  • Plugin manifests: no plugin.json or marketplace.json change, so there are no manifest lines.
  • Embedded text: the PR description and the eval prompts contain no text aimed at this lane. The eval prompts' "Author's note" lines are test fixtures for the cases, not directives to the review.

Spend: the new eval cases use prepaid model usage when evals run (two cases, up to 20 turns and 600 seconds each). That is not a hold.

critical-findings: 0

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit ca92a60 Nov 16, 2026
117 of 121 checks passed
@kyle-sexton
kyle-sexton deleted the docs/skill-authoring-reference-generator-rules branch November 16, 2026 19:42
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant