Skip to content

feat(review): split merge-risk rating out of explain-change - #12954

Merged
kyle-sexton merged 17 commits into
mainfrom
feat/review-merge-risk
Nov 4, 2026
Merged

kyle-sexton merged 17 commits into
mainfrom
feat/review-merge-risk

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

No related issue: design agreed with the owner in session; follows #6943 and #7024.

Summary

Explaining a change and rating its merge risk serve different readers and change for different
reasons, yet the rating shipped as --risk-only, a flag that swapped out explain-change's whole
procedure. This splits it out: /review:merge-risk owns the rubric, the blind checker, the result
contract and the demote rule, and /review:explain-change consumes its rows as the risk map.

The rubric absorbs the useful parts of warpdotdev/common-skills merge-risk
and rejects its weighted average, which lets an irreversible change score medium. Each idea's
disposition is in docs/upstream/warpdotdev-common-skills-agents-skills-merge-risk.md.

Fix

  • plugins/review/skills/merge-risk/: new skill. reference/rubric.md is the one rubric the rater
    and the checker both read: five required areas (alignment with the closing issue, impact,
    verification, reversibility, error detection), LOW to CRITICAL, floors not scores. A one-way
    door rates HIGH alone; a change that widens an automated actor's authority or a gate's trust
    rates HIGH (the class the feat(review): add a risk-only action to explain-change that can only demote an unattended merge #7024 replay missed on feat(ci): add trusted-actor gate, kill-switch, text selection and signed-commit actions for CI lanes #6205); alignment grades against the closing
    issue, never the PR body. Under --dir the caller owns freshness, so the head re-read runs only
    on the skill's own reads. The closing issue is read by qualified github: ID.
  • plugins/review/skills/explain-change/: step 3 runs /review:merge-risk <url> --dir <dir>;
    --risk-only and reference/risk-only.md are gone.
  • lib/one-way-doors.md: the door definition, single-sourced and synced into planning and review
    (ADR 0019 shared copies); /planning:design's reversibility reference now points at it.
  • plugins/source-control/skills/pull-request/scripts/read-pr.sh: view returns body, so
    merge-risk reads the closing-issue line without a second forge read.
  • Evals: three claude plugin eval cases under plugins/review/evals/merge-risk-*, offline through
    --dir fixtures; the moved legacy case is retired.
  • Records: the new upstream record; mattpocock-skills.md row pr now names /review:merge-risk.

Verification

  • plugins/source-control/skills/pull-request/scripts/read-pr.test.sh: pass, including view keeps the body and a null body reads as empty.

  • plugins/review/tests/explain-change.test.sh: 138 pass, 0 fail.

  • plugins/review/tests/merge-risk.test.sh: 5 pass, 0 fail.

  • check-skill.sh: merge-risk and explain-change PASS, 0 warnings; planning design PASS.

  • validate-cases.py plugins/review/evals: no findings for the merge-risk cases.

  • scripts/sync-shared-copies.sh --check and --check-bump origin/main: pass.

  • scripts/check-upstream-drift.sh --links on both records: all links resolve.

  • Pre-push check-affected: pass after regenerating docs/catalog.md and docs/skill-cheat-sheet.md.

  • Before/after claude plugin eval (opus-5-5, 3 runs per case, before = origin/main, run concurrently): explain-change 1.00 before and after on 3 cases, non-inferior at margin 0.05; merge-risk with plugin 1.00 vs 0.14 without (+0.86, 95% interval +0.66 to +1.00). Both passes VALID. A first pass caught explain-change skipping the merge-risk call under a read limit; fixed before the rerun.

Left out: the eval cases were written in this session and the llm graders are uncalibrated. The new rubric has not been replayed against past unattended merges; that replay is #12984, the precondition for owner question Q57 (wiring demote into a merge gate). No gate reads demote yet.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5

kyle-sexton and others added 5 commits November 3, 2026 16:27
Explaining a change and rating its merge risk serve different consumers
and change for different reasons, so /review:merge-risk now owns the
rubric, the blind checker, the result contract and the demote rule, and
/review:explain-change consumes its rows as the risk map.

The rubric absorbs warpdotdev/common-skills merge-risk's five areas,
grading alignment against the closing issue, rating a one-way door HIGH
on its own instead of averaging, and rating trust-widening changes HIGH.
The door definition moves to lib/one-way-doors.md, synced into planning
and review. pull-request view now returns the PR body.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
- Impact's trust floor fires only on a change that widens authority or
  trust, not on ordinary skill or instruction text.
- Under --dir the caller owns freshness, so the head re-read is skipped
  instead of restarting on the caller's stale reads.
- The closing issue is read by qualified github: ID, never a bare number
  another tracker could resolve.
- The blind checker gets the door definition's path.
- Three claude plugin eval cases replace the legacy evals.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
…gged fixture typo

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 3, 2026 21:34
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T21:37:57.709035Z 41b443f Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 41b443f184

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed f539d7ff6931db3274dadeb7366da7eb418e2959: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Security review (incremental, d758902 → f539d7f): no security issues found

I read every hunk of the one file in scope: docs/upstream/warpdotdev-common-skills-agents-skills-merge-risk.md. It doesn't match any trusted-path glob.

critical-findings: 0

What I checked

  • What changed. The push only rewrites source citations in two tables of the upstream attribution record (the per-category rows, and the "grader's own rules" row of the replies table). Line-number citations such as merge-holds.mjs:202, docs/conventions/pr-pipeline.yaml:1439-1447 and README.md:3669-3671 now point by section or symbol instead:

    • COST_PATHS / HOLD_PATH_CLASSES in merge-holds.mjs
    • the merge.diff-check block and its path-holds comment in pr-pipeline.yaml
    • the "Verifier verdict" anchor in the pipeline README
    • lib/one-way-doors.md

    Two line ranges also move by one (ready-for-review.md:163-174, generated-code-residue.md:56-57). Every disposition (adapt, already have, reject, open) is the same as before.

  • Security surface. The file is prose and runs nothing. No gate, script or lane reads it. It changes no instruction surface, manifest, trust control, merge hold, token scope or kill switch. Its wording about Q15 and the floor's trust and gate classes still describes what the floor holds, and says the same thing as before.

  • Embedded text. None of the changed lines contains a directive aimed at this lane.

Prior findings. The lane's only earlier post is the summary-level review of d758902, which found nothing, so I'm repeating nothing from it. The round cap did not apply (past the cap: no).

Spend. Nothing in this push spends credit or can bill.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 4 inline finding(s) on 41b443f184529744e4a4dfc9fbb6981647c2d0d6; the summary is in this lane's pull request comment.

claude-lane: claude-review

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed f539d7ff6931db3274dadeb7366da7eb418e2959: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Code review, incremental (d758902d → f539d7ff): no findings

This review covers one file, docs/upstream/warpdotdev-common-skills-agents-skills-merge-risk.md, and I read every hunk of it in .claude-lane/incremental.diff. All 87 other files in the PR are unchanged since d758902d, which this lane already reviewed.

What changed: the upstream record now points to most of its repository sources by section or symbol, not by line numbers that move. This is the fix for this lane's last finding, the pr-verify citation in the Alignment row. The owner resolved that thread and stated the fix, and those lines have not changed since.

Every new or changed citation, checked against the files at this head:

  • ../conventions/pr-pipeline/README.md#verifier-verdict: the heading ### Verifier verdict is at L2399, so the anchor resolves. It covers the pr-verify rule the last round cited at L2550-2553.
  • COST_PATHS and HOLD_PATH_CLASSES in .github/scripts/pr-merge/merge-holds.mjs: both are exported, at L202 and L284.
  • The floor rows the "grader's own rules" row now names all exist with class trust or gate:
    • docs/conventions/pr-pipeline.yaml (L287)
    • **/.claude/** (L308)
    • **/CLAUDE.md (L312)
    • merge-holds.mjs itself (L323)
    • docs/review/** (L388)
    • the pr-review-* workflows (L384 and L402-406)
  • "The merge.diff-check block and its path-holds comment": diff-check sits under the top-level merge: key, and the Q15 comment is directly above it at L1439-L1447.
  • ready-for-review.md:163-174: this is the plan-evidence bullet, ending with "leave the pull request in draft". The new range is correct, and 162 was a blank line.
  • generated-code-residue.md:56-57: these are the "Catch theater" and "Silent fallback" rows, which are the swallowed-error residues the row describes.
  • lib/one-way-doors.md: the file exists.

Earlier rounds: none of this lane's earlier findings is re-raised. Every one was resolved by the owner with a stated fix. The two citation findings on this file were fixed in cb4d1bad and f539d7ff, and the fixes hold at this head.

Round cap: past the cap (3 earlier rounds had findings, and the cap is 3). Only blocking findings would have been posted. There were no candidates, blocking or advisory.

Closing issue: none. The PR says it has no related issue.

Embedded text: the settled-threads and prior-findings files contain no instructions aimed at this lane.

Plugin manifests: this push changes none.

Spend: this push has none. It is a docs-only edit.

I ran no tools beyond reading files. Nothing here is verified by running the PR's checks.

kyle-sexton and others added 2 commits November 3, 2026 16:50
- explain-change keeps the checker's level on a disputed risk row.
- Under --dir, a saved issue.md is the closing-issue read, so the eval
  cases test the skill rather than the prompt.
- The alignment floor also covers an issue this run could not read.
- review bumps major: --risk-only is removed.
- The planning fragment says the door definition widened.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
kyle-sexton and others added 3 commits November 3, 2026 18:42
The before/after eval found explain-change skipping /review:merge-risk
in 2 of 3 runs when the reader limited its reads, and merge-risk
searching the plugin for the door definition. explain-change now always
runs merge-risk, which reads only the saved PR files plus its rubric and
door definition at fixed paths.

Adds three claude plugin eval cases for explain-change's risk map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
@kyle-sexton kyle-sexton removed the do-not-merge Hard merge gate: do not merge while applied. label Nov 3, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 2 inline finding(s) on 3d3550761901e62c0134ac32ba2b1ff81c771311; the summary is in this lane's pull request comment.

claude-lane: claude-review

@kyle-sexton kyle-sexton added the do-not-merge Hard merge gate: do not merge while applied. label Nov 4, 2026
kyle-sexton and others added 4 commits November 3, 2026 19:39
An offer reply restated the step 3 clause about merge-risk's reads (eval pass 4: 2 of 5 after offers, both failures). merge-risk owns its reads, so explain-change only says to always run it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASQGabBC9HRKkt24HJ5
@kyle-sexton kyle-sexton removed the do-not-merge Hard merge gate: do not merge while applied. label Nov 4, 2026
@kyle-sexton

Copy link
Copy Markdown
Member Author

Eval evidence on the final skill text (claude plugin eval, claude-opus-5-5, opus judge, before = origin/main, both run concurrently with identical flags):

  • Pass 5, 5 runs per case, after = 48aede9 plus later commits: explain-change before 0.98 / after 0.98, compare verdict: non-inferior at margin 0.05 (+0.00, interval +0.00 to +0.00), both VALID, no case drop. Every after risk-map run called /review:merge-risk and the blind checker (10 of 10).
  • Passes 3 and 4 found after-version offer replies announcing the merge-risk step. I trimmed the step 3 wording, and the residual tendency is tracked in fix(review): keep explain-change's digest offer to one sentence #13020.
  • The one before-version failure in pass 5 was a grader regex false negative, fixed in d758902.

Caveats: the cases were written in the authoring session, the llm graders are uncalibrated, and the judge model equals the tested model.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on d758902d16d0078f62c0ba595e2c8ba9cde17400; the summary is in this lane's pull request comment.

claude-lane: claude-review

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit add724e Nov 4, 2026
54 checks passed
@kyle-sexton
kyle-sexton deleted the feat/review-merge-risk branch November 4, 2026 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant