Skip to content

ci(skills): ratchet stack-specific code fences in skill bodies and amend ADR 0101 - #16649

Merged
kyle-sexton merged 4 commits into
mainfrom
feat/no-per-stack-ratchet
Nov 16, 2026
Merged

kyle-sexton merged 4 commits into
mainfrom
feat/no-per-stack-ratchet

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Refs: #16634
No related issue: owner decision recorded in docs/plugin-philosophy.md (no per-stack examples in shipped skills)

Summary

Shipped skills carry no ecosystem-specific examples: no code sample written for one stack, and no link to one tool's documentation. This PR makes two follow-ups to that rule. ADR 0101's Source citation row still kept any documentation link, so it is amended. A count ratchet stops new stack-specific code fences from landing in skill and agent bodies while the existing ones are cut.

Fix

  • docs/adr/0101-...md: the status line, the Source citation row and a dated ## Amendment (2026-11-16) section. The amendment states the rule and its reason, points at the plugin philosophy's "Hardcoded consumer specifics", and names the ratchet. This amends the record in place, following the repository's own convention: ADRs 0018, 0028 and 0075 each carry an accepted; amended <date> status line with a dated amendment section. That convention wins over /architecture:record-decision's default of a new record.
  • scripts/skill-stack-fences.py: a per-file counter. It prints <path> fences=<n> for each tracked plugins/*/skills/**/*.md and plugins/*/agents/**/*.md file holding a counted fence, and exits 2 when git fails or no file is in scope, so a broken scope can't read as 0.
  • scripts/test_skill_stack_fences.py and scripts/skill-stack-fences.test.sh (the python_probe wrapper) are shaped like the js-complexity-share counter's. Fifteen cases cover counted and excluded tags, tags with attributes, list-indented and tilde fences, fences behind blockquote and list-marker prefixes (and the next block a missed list-item opener would hide), closers held to the opener's blockquote depth, a blockquote fence ending with its blockquote (CommonMark), an unclosed blockquote fence that must not swallow a later fence, fences nested inside another fence, closing-fence length, scope, sorted output, and exit 2 both with no file in scope and outside a repository.
  • .performance/ratchets.json: a new per-file skill-stack-fences row with 39 file ceilings and 174 fences in total, recorded by ratchet.py add --per-file --break-glass off. The existing Check performance counter ceilings step in check-plugins runs ratchet.py check --file .performance/ratchets.json, so no workflow edit is needed.

No plugin is touched, so no plugin needs a version bump or changelog fragment.

The counting rule

A fence counts when it opens a code block in a skill or agent body and the first word of its info string (lowercased, {, . and } stripped) is not in the excluded set. Every other tag counts, so a new language (rust, kotlin) is counted without an edit. Untagged fences do not count. A fence nested inside another fence is content and does not count.

Excluded group Tags Basis
Shells the agent runs commands through bash sh shell zsh console powershell pwsh ps1 batch bat cmd The skill runs these through its own shell tool. For example, plugins/machine-health/skills/audit/reference/windows/check-catalog.md holds 25 powershell checks the audit runs.
Data, prose and diagram formats a skill reads or emits text txt plaintext markdown md json jsonc jsonl yaml yml toml ini csv diff mermaid gitconfig Output templates and contracts, not stack samples.
Skill pre-compute block ! Commands the skill itself runs at load time, e.g. plugins/knowledge/skills/video-digest/SKILL.md.
Queries and patterns skills run against their own data sql regex plugins/debugging/skills/analyze-profile/reference/formats/cpuprofile.md:40-42 is a query the skill runs with --param. plugins/docs-hygiene/skills/rename-references/context/patterns.md:43-45 is the pattern the sweep uses.

Out of scope by directory: evals, tests, test, __tests__, fixtures, testdata, vendor. These hold eval inputs, test data, and upstream copies a sync replaces, such as plugins/playwright/skills/playwright/vendor/, which would otherwise fail CI on a sync PR. This scope follows scripts/js-complexity-share.py.

Basis for per-file over one total: /toolchain:ratchet "Per-file counters". A count that files own is held per file, so cutting a fence in one file cannot pay for a new one elsewhere.

Known imprecision: some counted js fences are code a skill executes, such as the course-digest adapters, the playwright references and the harness-ops extraction. The tag cannot tell those apart, so they count. A declared-ecosystem surface or executed adapter that needs a new fence raises its file's ceiling in the PR that adds it, with the reason in the body.

Today's count is 174 fences in 39 files at f552369. #16634 reported 179 in 51 files. The figures differ because of scope: #16634's count also took in plugins/*/reference/**, while this one covers skills and agents only, drops the eval, test, fixture and vendor directories, and excludes sql and regex.

Verification

  • python3 scripts/test_skill_stack_fences.py: written first and failed (7 of 9 cases, counter absent), then 9 of 9 pass. The review fix for container prefixes added three cases, which failed first (3 of 12) and now pass: 12 of 12. The second review fix, which holds closers to the opener's blockquote depth, added two cases. Both failed first (2 of 14) and now pass. A third fix makes a blockquote fence end when its blockquote ends, per CommonMark. Its updated mirror case and one new case both failed first (2 of 15), and all 15 now pass. bash scripts/skill-stack-fences.test.sh: 15 pass. The count is unchanged after that fix: 174 fences in 39 files.
  • ratchet.py check on a file holding only the new row: OK skill-stack-fences: every file at or below its ceiling (fences=174 total). With a python fence appended to an uncounted skill file it reported ABOVE ... fences=1 > ceiling 0 and exited 1. With the file restored it exited 0. The full ratchets.json was not run locally because the hook-census rows need strace, so CI runs it.
  • scripts/check-affected.sh --no-fetch: 16 passed, 3 skipped over 5 changed files.
  • The following passed: scripts/run-ruff.sh check and format --check; shellcheck; check-fixture-git-isolation.sh --check; check-test-tmp-cleanup.sh --check; check-silent-skips.sh; check-purged-em-dashes.sh; check-vocabulary.py; check-adr-numbers.sh --check; check-discriminating-test-skips.sh; check-line-citations.py; check-lane-coverage.sh --check; check-cant-fail-scan.sh origin/main; and the commit-time comment lint.

Related

Not in this PR

  • Plugin-level plugins/*/reference/** files. They are model-read under ADR 0101 and docs(philosophy): ship no ecosystem-specific examples in skills #16634 proposed counting them; this ratchet covers skill and agent bodies only.
  • Matching links to tool documentation. That would need a per-stack domain list, the same maintenance the rule exists to avoid.
  • A scripts/suite-seconds.txt row for the new suite. The rows are measured from CI runs, and until one is added the lane planner sizes an unlisted suite at the mean.

🤖 Generated with Claude Code

…end ADR 0101

Shipped skills carry no ecosystem-specific examples. ADR 0101's Source
citation row still kept any documentation link, so it is amended in
place: a link to one ecosystem's tool documentation, or an example
written for one stack, is no longer a sanctioned citation.

scripts/skill-stack-fences.py counts, per file, the code fences in
tracked plugins/*/skills/**/*.md and plugins/*/agents/**/*.md whose tag
is not a shell, data or prose format, diagram, pre-compute block, or the
sql and regex skills run against their own data. Eval, test, fixture
and vendor directories are out of scope. A per-file skill-stack-fences
row in .performance/ratchets.json freezes today's 174 fences in 39
files; the existing ratchet step in check-plugins checks it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 21:52
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T21:54:21.052042Z 3f3b2fc Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f3b2fcd75

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/skill-stack-fences.py Resolved
@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/fix-docs
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 43964540026

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 538c13a1cbd017ae3f153a323549e7083e6ac194: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

claude-review: blocking-only incremental review of 538c13a1. This lane already reviewed earlier heads of this pull request, so this pass reports only blocking findings. It covers the change since b77d4ffa. I read every hunk of both files in .claude-lane/incremental.diff (scripts/skill-stack-fences.py and scripts/test_skill_stack_fences.py), plus count_fences in full at head.

No blocking findings.

Earlier finding resolved. At b77d4ffa, the summary reported that a blockquote fence the blockquote ends without a closer swallowed the rest of the file. This head fixes that. In count_fences (L98-L110), a line whose leading > count is below quote_depth now ends the fence and falls through to the opener match. That matches CommonMark: laziness does not apply to fenced code. I traced both new and updated tests by hand against the new logic:

  • test_blockquote_fence_ends_with_its_blockquote counts 1. CommonMark also gives 1: the bare ``` opens an untagged top-level fence, and that fence holds `> ```` and ```go as content.
  • test_unclosed_blockquote_fence_does_not_swallow_a_later_fence counts 2, as CommonMark does.

I also traced the case the settled thread on old line 63 raised (> ```` inside a top-level fence). It still counts as content: depth 0 can never fall below quote_depth, and the closer still needs exactly quote_depth` markers. So this push does not reopen that settled point.

The module docstring now describes the new rule.

Not verified by me: the PR body's claims (15 of 15 tests pass, 174 fences in 39 files, the ratchet.py check results) are the author's. I have no shell and ran nothing.

Closing issue: none (Refs: #16634 only), so there are no acceptance criteria to check.

Embedded text: none found in the PR context, settled-threads or prior-findings files.

Spend: this push only adds one more unittest case to the CI run. That uses prepaid CI time and is not a hold.

Verifier dropped 0 of 0 candidates.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 538c13a1cbd017ae3f153a323549e7083e6ac194: read 2 of 2 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #16649 (incremental since b77d4ff): no security issues found

Scope. This review covers only what changed since b77d4ffa. I read every hunk of the two files in .claude-lane/incremental.diff: scripts/skill-stack-fences.py and scripts/test_skill_stack_fences.py. Neither path is on the trusted-path list. This lane already reviewed an earlier head (blocking only: yes), so only CRITICAL and IMPORTANT findings would be reported. There are none.

What changed

  • scripts/skill-stack-fences.py
    • The counter now ends an open fence when a line carries fewer > markers than the opener did, which matches CommonMark. That line is then read as a normal line that may open a new fence. Before this, only an exact-depth closer ended the fence.
    • The new QUOTE_MARKERS regex is [\s>]*. It always matches, possibly with zero length, so .group(0) cannot raise. It has a single quantifier and no nesting, so a crafted line cannot cause catastrophic backtracking.
    • Nothing else changed. The input is still the repository's tracked skill and agent .md files. The counter holds no credentials, writes nothing and passes nothing to a shell. Its only output is the per-file fence count that feeds the skill-stack-fences ratchet, which is a content-policy check, not a security control.
  • scripts/test_skill_stack_fences.py
    • The mirror test is renamed, and its expected count changes from 2 to 1 under the new rule.
    • A new test checks that an unclosed blockquote fence does not swallow a later fence. Both tests run in the existing isolated temporary repositories.

Checks

  • Trust controls: none touched. Nothing changes the trusted-trigger check, token scope, kill switch, token broker, merge holds or any .github/actions/ action.
  • Plugin manifests: none changed.
  • Instruction-surface deletions: none.
  • Embedded instructions: the PR context file has none aimed at this lane.
  • Spend: nothing beyond the existing CI run of the counter.

Earlier posts: the lane's earlier summary comment (#16649 (comment)) reported 0 findings, so nothing carries forward.

critical-findings: 0

…fixes

The opening-fence pattern allowed only whitespace before the delimiter,
so a fence inside a blockquote (`> ```python`) or right after a list
marker (`- ```python`, `1. ```python`) was missed, and the missed
opener's closer then read as an untagged opener that hid the next real
block. Any mix of `>` markers, list markers and their indentation now
comes off before an opener or a closer is matched.

Three regression cases cover the blockquote form, the list-marker form
and the hidden next block. Today's count is unchanged: 174 fences in 39
files, so the ceilings stand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on fc3ee2f983879e95aa5d39010c5e6f1c862aba6e; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment thread scripts/skill-stack-fences.py Resolved
kyle-sexton and others added 2 commits November 16, 2026 17:56
…depth

The container-prefix fix let a closer strip prefixes its opener never
had, so a `> ```` content line inside an open top-level fence closed it
and the real block after it went uncounted. A fence now records its
opener's blockquote depth, and a closer may carry only indentation and
exactly that many `>` markers; any other line, a list marker included,
is content of the open fence.

Two regression cases cover a blockquote closer inside a top-level fence
and its mirror, a blockquote fence that closes only on a `>` closer.
Today's count is unchanged: 174 fences in 39 files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under CommonMark a fence opened inside a blockquote ends when the
blockquote does, and lazy continuation cannot apply inside a fenced
block. The counter kept such a fence open until a `>` closer, so an
unclosed blockquote fence swallowed later real fences and undercounted.
A line with fewer `>` markers than the opener now ends the fence and is
then read as a normal line that may open one.

The mirror case now expects the CommonMark result, and a new case checks
that an unclosed blockquote fence followed by a top-level python fence
counts both. Today's count is unchanged: 174 fences in 39 files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 833c1ac Nov 16, 2026
63 checks passed
@kyle-sexton
kyle-sexton deleted the feat/no-per-stack-ratchet branch November 16, 2026 23:12
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant