Repository navigation
feat(review): split merge-risk rating out of explain-change - #12954
Conversation
Explaining a change and rating its merge risk serve different consumers and change for different reasons, so /review:merge-risk now owns the rubric, the blind checker, the result contract and the demote rule, and /review:explain-change consumes its rows as the risk map. The rubric absorbs warpdotdev/common-skills merge-risk's five areas, grading alignment against the closing issue, rating a one-way door HIGH on its own instead of averaging, and rating trust-widening changes HIGH. The door definition moves to lib/one-way-doors.md, synced into planning and review. pull-request view now returns the PR body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
- Impact's trust floor fires only on a change that widens authority or trust, not on ordinary skill or instruction text. - Under --dir the caller owns freshness, so the head re-read is skipped instead of restarting on the caller's stale reads. - The closing issue is read by qualified github: ID, never a bare number another tracker could resolve. - The blind checker gets the door definition's path. - Three claude plugin eval cases replace the legacy evals.json. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
…gged fixture typo Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 41b443f184
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
claude-security-review reviewed Security review (incremental,
|
|
claude-review reviewed Code review, incremental (
|
…copy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
- explain-change keeps the checker's level on a disputed risk row. - Under --dir, a saved issue.md is the closing-issue read, so the eval cases test the skill rather than the prompt. - The alignment floor also covers an issue this run could not read. - review bumps major: --risk-only is removed. - The planning fragment says the door definition widened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
The before/after eval found explain-change skipping /review:merge-risk in 2 of 3 runs when the reader limited its reads, and merge-risk searching the plugin for the door definition. explain-change now always runs merge-risk, which reads only the saved PR files plus its rubric and door definition at fixed paths. Adds three claude plugin eval cases for explain-change's risk map. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
…ation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
An offer reply restated the step 3 clause about merge-risk's reads (eval pass 4: 2 of 5 after offers, both failures). merge-risk owns its reads, so explain-change only says to always run it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
…grader Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
|
Eval evidence on the final skill text (claude plugin eval, claude-opus-5-5, opus judge, before = origin/main, both run concurrently with identical flags):
Caveats: the cases were written in the authoring session, the llm graders are uncalibrated, and the judge model equals the tested model. |
… not drifting lines Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
No related issue: design agreed with the owner in session; follows #6943 and #7024.
Summary
Explaining a change and rating its merge risk serve different readers and change for different
reasons, yet the rating shipped as
--risk-only, a flag that swapped out explain-change's wholeprocedure. This splits it out:
/review:merge-riskowns the rubric, the blind checker, the resultcontract and the demote rule, and
/review:explain-changeconsumes its rows as the risk map.The rubric absorbs the useful parts of warpdotdev/common-skills
merge-riskand rejects its weighted average, which lets an irreversible change score medium. Each idea's
disposition is in
docs/upstream/warpdotdev-common-skills-agents-skills-merge-risk.md.Fix
plugins/review/skills/merge-risk/: new skill.reference/rubric.mdis the one rubric the raterand the checker both read: five required areas (alignment with the closing issue, impact,
verification, reversibility, error detection),
LOWtoCRITICAL, floors not scores. A one-waydoor rates
HIGHalone; a change that widens an automated actor's authority or a gate's trustrates
HIGH(the class the feat(review): add a risk-only action to explain-change that can only demote an unattended merge #7024 replay missed on feat(ci): add trusted-actor gate, kill-switch, text selection and signed-commit actions for CI lanes #6205); alignment grades against the closingissue, never the PR body. Under
--dirthe caller owns freshness, so the head re-read runs onlyon the skill's own reads. The closing issue is read by qualified
github:ID.plugins/review/skills/explain-change/: step 3 runs/review:merge-risk <url> --dir <dir>;--risk-onlyandreference/risk-only.mdare gone.lib/one-way-doors.md: the door definition, single-sourced and synced into planning and review(ADR 0019 shared copies);
/planning:design's reversibility reference now points at it.plugins/source-control/skills/pull-request/scripts/read-pr.sh:viewreturnsbody, somerge-risk reads the closing-issue line without a second forge read.
claude plugin evalcases underplugins/review/evals/merge-risk-*, offline through--dirfixtures; the moved legacy case is retired.mattpocock-skills.mdrowprnow names/review:merge-risk.Verification
plugins/source-control/skills/pull-request/scripts/read-pr.test.sh: pass, includingview keeps the bodyanda null body reads as empty.plugins/review/tests/explain-change.test.sh: 138 pass, 0 fail.plugins/review/tests/merge-risk.test.sh: 5 pass, 0 fail.check-skill.sh: merge-risk and explain-change PASS, 0 warnings; planning design PASS.validate-cases.py plugins/review/evals: no findings for the merge-risk cases.scripts/sync-shared-copies.sh --checkand--check-bump origin/main: pass.scripts/check-upstream-drift.sh --linkson both records: all links resolve.Pre-push
check-affected: pass after regeneratingdocs/catalog.mdanddocs/skill-cheat-sheet.md.Before/after
claude plugin eval(opus-5-5, 3 runs per case, before = origin/main, run concurrently): explain-change 1.00 before and after on 3 cases,non-inferior at margin 0.05; merge-risk with plugin 1.00 vs 0.14 without (+0.86, 95% interval +0.66 to +1.00). Both passes VALID. A first pass caught explain-change skipping the merge-risk call under a read limit; fixed before the rerun.Left out: the eval cases were written in this session and the llm graders are uncalibrated. The new rubric has not been replayed against past unattended merges; that replay is #12984, the precondition for owner question Q57 (wiring
demoteinto a merge gate). No gate readsdemoteyet.Related
demote.🤖 Generated with Claude Code
https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5