Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1373,6 +1373,10 @@ jobs:
run: scripts/check-skill-precompute-compose.sh --all
- name: Test the shared listing-budget reporter
run: bash plugins/skill-quality/scripts/check-listing-budget.test.sh
- name: Test the invocation-probe harness
run: bash plugins/skill-quality/scripts/measure-invocation.test.sh
- name: Validate invocation probes
run: bash plugins/skill-quality/scripts/measure-invocation.sh validate plugins/skill-quality/probes
# Report-only (always exit 0 — see the script's own header): pools every
# plugin's skills root into ONE shared aggregate, the shape a consumer who
# installs the whole marketplace actually experiences.
Expand Down
2 changes: 1 addition & 1 deletion plugins/skill-quality/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
"name": "skill-quality",
"version": "0.24.10",
"version": "0.24.11",
"description": "Skill-authoring QA tooling: a static contract checker that runs twenty-six deterministic checks over a Claude Code skill (frontmatter, explicit invocation mode, description/verb-contract polarity, per-skill listing-entry cap, trigger-keyword preservation, line caps, broken internal refs, markdownlint, gotchas surface, evals presence, precompute opportunity, completion-criteria signal, injection shell-declaration, fresh-eyes declaration conformance), a shared skill-listing budget reporter across a set of skills, and a bundled evals.json schema plus a deterministic eval-quality lint (duplicate case identities, missing fixtures, empty or vague grading criteria, set-coverage warnings). Runs against any repo's skills directory via the convention-resolution ladder, with no baked layout.",
"author": {
"name": "Melodic Software",
Expand Down
17 changes: 17 additions & 0 deletions plugins/skill-quality/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,23 @@
All notable changes to the `skill-quality` plugin are documented here. Format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning.

## [0.24.11] - 2026-09-28

### Added

- **`measure-invocation`: a repeatable description-invocation probe harness (#3526).** Scores
whether a skill's listing text would win the requests it should, with a train/validation split
and a false-trigger rate. Default method is a deterministic lexical listing-overlap floor that
runs without a model. CI validates the probe schema and runs the harness tests; `score` and
`compare` are run by hand. `emit-plugin-eval` writes `claude plugin eval` cases for an
on-demand live run; `compare` prints the delta versus a baseline so a rewrite cannot hide a
validation drop behind a train gain. Seed probes (20 queries each) cover `skill-quality:check`
and `mcp-tools:audit`. `score` fails on a competitor it cannot load, `compare` fails unless
both reports cover the same skills, and `emit-plugin-eval` refuses a non-empty output
directory. Fleet-wide description rewrites stay attended. Command:
`bash plugins/skill-quality/scripts/measure-invocation.sh validate|score|compare|emit-plugin-eval`.
Contract: `reference/invocation-probes.md`.

## [0.24.10] - 2026-09-28

### Added
Expand Down
22 changes: 19 additions & 3 deletions plugins/skill-quality/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@

A Claude Code plugin for **skill-authoring QA**: it runs a static, deterministic contract gate over a
skill directory, reports the shared listing-budget estimate across a set of skills, and validates a
skill's `evals.json` against a bundled schema plus a deterministic eval-quality lint. No model
invocation anywhere. The same checks run identically in a session, a pre-commit hook, or CI.
skill's `evals.json` against a bundled schema plus a deterministic eval-quality lint, and scores
description-driven auto-invocation probes (`measure-invocation`). Default paths invoke no model.
The same checks run identically in a session, a pre-commit hook, or CI.

The drift static analysis catches best is a rewrite silently dropping a `description` trigger
phrase, which can degrade a skill's auto-invocation. Check 3 compares the trigger phrases against
Expand All @@ -13,7 +14,7 @@ the reviewer to confirm the description still names that intent, or to restore t

| Skill | What it does |
|---|---|
| `/skill-quality:check` | Runs the contract gate (`check`), reports the shared listing budget (`listing-budget`), or schema-validates and quality-lints evals (`validate-evals`) for one skill, a set of roots, or every skill. |
| `/skill-quality:check` | Runs the contract gate (`check`), reports the shared listing budget (`listing-budget`), schema-validates and quality-lints evals (`validate-evals`), or scores description auto-invocation probes (`measure-invocation`). |
| `/skill-quality:setup` | Check-only: resolves and verifies the skills directory and prints the guidance for routing a non-default `skills_root` change through Claude Code. |

## Checks
Expand Down Expand Up @@ -123,6 +124,7 @@ because the budget it reports is the shared one.
/skill-quality:check validate-evals my-skill # schema-check + quality-lint evals.json
/skill-quality:check listing-budget # report the shared budget over the resolved root
/skill-quality:check listing-budget plugins/*/skills # pool every plugin's root into one aggregate
/skill-quality:check measure-invocation # score description auto-invocation probes
```

## Skills directory is never baked in
Expand Down Expand Up @@ -169,6 +171,20 @@ stands alone.
- `npx` (Node) is optional; without it the markdownlint check downgrades to a warning and the other
twenty-five still gate.

## Description-invocation probes

`measure-invocation` scores whether a skill's listing text would win the requests it should
(and stay quiet on the ones it should not). Default method is a deterministic lexical
listing-overlap floor; `emit-plugin-eval` writes `claude plugin eval` cases for a live run.
Contract: [`reference/invocation-probes.md`](reference/invocation-probes.md).

```shell
bash plugins/skill-quality/scripts/measure-invocation.sh validate plugins/skill-quality/probes
bash plugins/skill-quality/scripts/measure-invocation.sh score plugins/skill-quality/probes
bash plugins/skill-quality/scripts/measure-invocation.sh compare \
plugins/skill-quality/probes/baselines/listing-overlap.json /tmp/score.json
```

## Configuration

<!-- BEGIN GENERATED: plugin options. Edit plugin.json, then run scripts/sync-plugin-options-docs.py -->
Expand Down
Loading