Skip to content

refactor(planning): one owner for the authoring-formats resolution ladder - #11168

Merged
kyle-sexton merged 5 commits into
mainfrom
host-run/10119-ac-format-ladder-owner
Oct 30, 2026
Merged

kyle-sexton merged 5 commits into
mainfrom
host-run/10119-ac-format-ladder-owner

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Closes #10119
Refs: #6917

Summary

/planning:interview (in context/ears.md, since #11074) and /planning:prd each restated the acceptance_criteria_format resolution ladder, and /planning:design restated the same ladder for diagram_dialect. The ladder now lives once, in plugins/planning/context/authoring-formats.md. Each skill points at that file and keeps only what is specific to its key: the default, what a degrade emits, and design's unset (no C4 view emitted) provenance label.

Fix

  • New plugins/planning/context/authoring-formats.md: the ladder steps, the resolver's exit codes, the untrusted-input rule, and why the ladder sits in the plugin rather than in a citation of the publisher's docs. It arrives through Read unsubstituted, so it writes <plugin-root> and <project-root>, and each reading skill's body supplies both values.
  • interview's context/ears.md, prd and design: each ladder block is replaced with a pointer plus its key-specific lines.
  • docs/contracts/authoring-formats (1.0.4, clarification): a consuming plugin restates the ladder once, in a file it bundles, instead of once per skill. The config-cascade row now says the same.
  • The context/ears.md is unchanged pin is repinned with tests/repin.sh. Only the ladder moved out of ears.md; the EARS tag vocabulary and bullet shapes are unchanged.
  • Two new outcome cases cover the convention paths the move touches. They were written before any after-version run and are the same in both versions:
    • interview-ears-convention-tags-criteria: ears produces tagged AC<n>: [<pattern>] criteria in the Brief.
    • design-dbml-data-dialect: dbml produces an entity-relationships.md labeled dialect: dbml with a dbml fence.

Verification

  • git grep -n -i 'acceptance_criteria_format' -- plugins/: the steps appear only in context/authoring-formats.md. The skills, README and CHANGELOG mention the key without restating them.
  • bash plugins/planning/tests/interview-defenses.test.sh: PASS after the repin.
  • scripts/check-affected.sh --tests on the committed head: 14 passed, 2 skipped.
  • validate-cases.py plugins/planning/evals: no FAIL.

Before and after evals

Setup:

  • Claude Code 2.1.314, subscription login, agent model claude-opus-5-5 in every trace, judge opus.
  • Before is origin/main; after is this branch. Both copies sit outside the repository and run the same case set.
  • Before and after ran concurrently with the same flags: claude plugin eval <copy>/plugins/planning --ablation none --scaffold --trust-plugin --keep-temp --allow-tools Bash Write Edit --judge-model opus --threshold 0.8 --runs 3 --no-publish.
  • --ablation none runs the with-plugin arm only, so each comparison is before versus after, not with versus without.

Mean case score over 3 runs:

Skill Case Before After
interview interview-ears-convention-tags-criteria (new) 1.00 1.00
interview interview-consumer-standard-override 1.00 1.00
interview interview-lock-restates-problem 1.00 1.00
interview interview-prd-seeds-brief 1.00 1.00
prd prd-handoff-scope-clear 1.00 1.00
prd prd-next-unsure-midflow 0.67 1.00
design design-dbml-data-dialect (new) 1.00 1.00
design design-brief-nfr-not-reasked 0.93 1.00
design design-fact-placed-once 1.00 1.00
design design-one-way-door-adr-prompt 0.67 0.67
design design-prototype-reversibility-deferred 1.00 1.00
  • Validity. run-validity.py --runs 3 prints verdict: VALID on all four result files: two passes, before and after each. The only warning is skill fired, which is expected because each prompt starts with the slash command.
  • Noise report, 7-case pass. noise-report.py after --baseline before --margin 0.05: change +0.01, 95% interval -0.01 to +0.03, non-inferior at margin 0.05.
  • Noise report, 4-case pass. Change +0.08, interval -0.08 to +0.25, not shown non-inferior at margin 0.05. With 4 cases the interval is wider than the margin. No case's score went down.
  • New cases. In both versions, the new cases call the bundled resolve-convention-home.sh in 3 of 3 runs and pass 3 of 3. Moving the ladder into the context file did not lose the ears or dbml path.
  • Failing grader. design-one-way-door-adr-prompt fails its one-way-with-adr grader in every run of both versions, so this change did not cause it.
  • Final head. The before and after passes compared origin/main with this branch as first written, before refactor(planning): move the interview EARS ladder and coverage-prompt exemption into one context file each #11074 moved interview's ladder into context/ears.md. After rebasing onto refactor(planning): move the interview EARS ladder and coverage-prompt exemption into one context file each #11074 and switching the shared file to placeholders, the two convention cases were rerun on the final head: 3 of 3 each, every run read context/authoring-formats.md and called the resolver, run-validity.py verdict: VALID. The other cases' text did not change in the rebase.
  • Cost. The CLI's list-price estimate is 91.3 USD in total. 37.4 USD of that went to a first pass that stopped at its 15 USD cost ceiling; that pass was discarded and rerun.

Related

🤖 Generated with Claude Code

kyle-sexton and others added 2 commits October 30, 2026 01:08
…dder

The acceptance_criteria_format ladder was restated in interview's
context/ears.md and in prd, and the same ladder for diagram_dialect in
design. It now lives once in context/authoring-formats.md, written with
<plugin-root> and <project-root> placeholders that each reading skill's
body fills; each skill keeps only its key's default, degrade target and
provenance label.

Adds eval cases for the ears and dbml convention paths, and repins the
ears.md digest:
context/ears.md is unchanged: d44ad1a0d4b3ceba53b4f962afd8afee2d939dc5b68c305a0318dd18a31ecc54 -> 903332451df15c686d46259080f54f82824287b1efe4199a42716e5e95609c6f
Only the ladder moved out; the EARS tag vocabulary and bullet shapes are
unchanged.

Refs: #10119

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 30, 2026 07:10
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T07:14:42.024932Z 6903a0f Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 467cef38e86324ccf287a333becf1585475987cf: 0 finding(s), 0 as inline comments.

No security findings in the incremental change (since 6903a0f).

All 12 changed files are eval fixtures, sample data and one changelog line under plugins/planning/evals/** and .changes/**. None of them is a trusted path, a trust control, a plugin manifest or an instruction surface.

  • prd-ears-convention-tags-criteria/scaffold.sh (new): it runs git init and writes fixed heredocs inside the eval's working directory. It reads no environment variables, secrets or network input. It takes no input from the PR or the agent, so there is nothing to inject.
  • Grader regexes (dbml, interview, prd): these run over the agent-written artifact inside the eval harness. The patterns use lazy line-by-line quantifiers, so the worst case is polynomial matching on a small document. There is no credentialed path or trusted context behind them.
  • Eval case.yaml and prompt.md: the prompt's allowed_tools are Read, Glob, Grep, Skill, Bash, Write and Edit, scoped to the throwaway scaffold repo. This matches the existing eval cases and widens no token or tool grant.
  • .changes entry: it is a prose edit only.
  • Spend: the new prd-ears-convention-tags-criteria case runs three times at up to 30 turns each, which uses prepaid API credit. That is not a hold.

manifest: no manifest touched.
critical-findings: 0

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 467cef38e86324ccf287a333becf1585475987cf: 0 finding(s), 0 as inline comments.

No findings in the incremental review (changes since 6903a0f). I hand-traced each regex and the sample files, and ran none of them, so these checks are by reading only.

  • design-dbml-data-dialect grader: the two lookaheads are bounded by (?!-{3}), so scope: data and dialect: dbml must both sit inside the frontmatter. I traced the three new fail samples (scope: system, body-only scope, body-only dialect) and each is rejected. The existing pass samples still match.
  • interview-ears-convention-tags-criteria grader:
    • The negative lookahead that rejects any untagged AC<n>: bullet in the section cannot be sidestepped by backtracking [^\n]*. The next token must be a literal \n, so only the end-of-heading position can proceed.
    • The double space after the colon is handled: backtracking [ \t]* hits the (?![ \t]|tag) guard.
    • The mixed fail samples, including the one with no trailing newline, are rejected. The pass sample with a later ### Captured assumptions section stays inside the lookahead's (?!#) boundary.
    • The ### Acceptance criteria heading it now requires matches the Brief template in interview/context/loop.md and context/ears.md.
  • prd-ears-convention-tags-criteria (new case):
    • The regex grader checks that at least one criterion is tagged. The LLM grader every-criterion-tagged checks that all four are, and its mixed-criteria sample is a fail.
    • The prompt uses the synthesize action, as the PRD skill's unattended path requires.
    • The tag names match the skill's table.
    • The scaffold follows the existing interview-ears-convention-tags pattern.
    • Neither PRD sample file's pass entries are rejected by the regex.
  • Changelog fragment: the wording fix is accurate and a patch bump fits.

Cost: the new prd-ears-convention-tags-criteria eval case adds 3 model runs, which spend prepaid API credit. No hold.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6903a0f2ca

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/planning/skills/prd/SKILL.md Resolved
@kyle-sexton
kyle-sexton enabled auto-merge October 30, 2026 07:23
@kyle-sexton kyle-sexton added the do-not-merge Hard merge gate: do not merge while applied. label Oct 30, 2026
kyle-sexton and others added 2 commits October 30, 2026 03:32
…e in the convention cases

criteria-tagged now fails when any AC bullet in the Brief's acceptance
criteria section is untagged, not only when none is tagged; a mixed
tagged/untagged sample is added as a fail. dbml-artifact now requires
scope: data and dialect: dbml inside the frontmatter, with wrong-scope
and body-only samples as fails.

Refs: #10119

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
prd-ears-convention-tags-criteria runs /planning:prd in a repository
whose convention home selects acceptance_criteria_format: ears and checks
that every acceptance criterion the PRD records carries an exact EARS tag,
so prd's pointer to the shared resolution ladder is covered by an outcome.

Refs: #10119

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 30, 2026
@kyle-sexton kyle-sexton removed the do-not-merge Hard merge gate: do not merge while applied. label Oct 30, 2026
Merged via the queue into main with commit a1b88a6 Oct 30, 2026
70 of 71 checks passed
@kyle-sexton
kyle-sexton deleted the host-run/10119-ac-format-ladder-owner branch October 30, 2026 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor(planning): one owner for the acceptance_criteria_format resolution ladder

1 participant