Skip to content

feat(architecture): report scan defects on their own and route review-class problems to their skills - #16194

Merged
kyle-sexton merged 7 commits into
mainfrom
feat/architecture-scan-defect-reports
Nov 14, 2026
Merged

kyle-sexton merged 7 commits into
mainfrom
feat/architecture-scan-defect-reports

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

No related issue: two owner decisions on what a design scan does with problems that are not about module shape, small enough to land directly.

Summary

architecture:improve reproduces any live-bug claim a scan makes before it reaches the report, but said nothing about where the defect goes next. It stayed inside its refactor candidate, so it surfaced only if that candidate was picked, and nothing kept a security defect's details out of a public tracker. The scan also had no rule for code-review problems it notices along the way.

Fix

  • The deepening action's Phase 2 now names each reproduced live defect in the reply before the candidate question, security defects first.
  • A defect that is not a security one goes to /bugs:write --file when that plugin is installed, so the report is saved to a file and the user is offered the choice to file it; nothing is filed without the user's answer. Otherwise the reply carries the report (title, reproduction steps, expected against actual, fix location).
  • A security defect is stated to the user and kept out of any tracker item, pull request, commit message or other text that may become public; the user is told to report it privately, and the skill files nothing. A pointer names GitHub's page on private reporting.
  • A dead-code claim is not a defect, and an unverified claim is named as unverified, not reported.
  • Review-class problems (types that opt out of checking, discarded errors, a hard-coded value) stay as evidence on their candidate. The reply names the owning skill, /code-metrics:audit type-debt or /review:quality-gate code, each when installed; the scan adds no finding or badge for them.

Verification

  • Three outcome cases under plugins/architecture/evals/, each 3 runs per arm on Claude Code 2.1.325: deepening-defect-reported-on-its-own, deepening-security-defect-kept-private and deepening-review-class-names-owner passed 3 of 3 with the plugin and 0 of 3 without it. These load only the architecture plugin, so the defect case exercises the fallback report.
  • The installed path, deepening-defect-goes-to-bugs-write (906c717): a stand-in bugs plugin inside the case directory puts /bugs:write in the skill list, because the eval CLI loads a second plugin only from inside the case directory. Run twice, 3 runs per arm: the with-plugin arm passed 6 of 6 (the scan called /bugs:write with --file, named the defect before the candidate question, and ran no tracker write or commit). The no-plugin arm loads neither plugin, so its failure is built in; 3 of its 6 runs failed the graders and 3 did not start, on a sandbox error about the machine's Docker credential store. That leaves the run-validity verdict at INVALID for both runs, on the baseline arm only.
  • What the stand-in does not measure: what the real /bugs:write does with --file. Checked once outside the eval harness: a headless session with this branch's plugins/architecture and plugins/bugs both loaded, on the same scaffolded repository, in auto permission mode. The scan reproduced the defect, called /bugs:write --file, the skill wrote a 52-line report with type: bug-report frontmatter to the bugs plugin's data directory, the reply gave its path and the command that would file it, and nothing was filed or committed. Two earlier attempts in acceptEdits mode with no approval surface had that write denied (the data directory is outside the working directory); both replies then carried the report inline and said the file was not saved.
  • The two existing deepening cases still pass with the plugin.
  • check-affected passed in the pre-push hook on each pushed head.
  • Left out: a case for an unverified bug claim being named as unverified instead of reported (the existing light-level case covers the labeling only); the LLM graders were not run through judge calibration.

Related

N/A

🤖 Generated with Claude Code

kyle-sexton and others added 3 commits November 14, 2026 11:25
…s own

A design scan can reproduce a live bug while looking for shallow
modules. Until now the defect stayed inside its refactor candidate and
surfaced only if that candidate was picked.

The deepening action now names each reproduced live defect in the reply,
security defects first. A defect that is not a security one goes to
/bugs:write when that plugin is installed, else a report in the reply.
A security defect is stated to the user and kept out of any text that
may become public, with a pointer to private reporting.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…wning skills

A design scan also notices problems that are not about module shape,
such as types that opt out of checking and discarded errors. The
deepening action now keeps them as evidence on their candidate and
names the owning skill in the reply, without findings of its own.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 14, 2026 17:34
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-14T17:36:09.754600Z 337d6e9 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 337d6e9ba5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 906c717f5e1d49993ddb743427c568a81cb23c6f: read 11 of 11 changed file(s); 2 finding(s), 2 as inline comments.

Security review (incremental, blocking only): 2 manifest flags, no exploitable issue

This lane already reviewed an earlier head, so this review reports only CRITICAL and IMPORTANT findings. Manifest flag lines are reported at either severity.

What I read. The incremental scope has 11 files, all in the new eval case plugins/architecture/evals/deepening-defect-goes-to-bugs-write/. I read every hunk of each one in .claude-lane/incremental.diff. None of them matches a trusted-path glob.

The manifest (bugs-stub/.claude-plugin/plugin.json). This is a new plugin manifest, so the plugin-manifest rules apply. Its fields:

  • manifest name: flag, added in a new eval-only stand-in manifest; runs nothing. Inline comment. IMPORTANT.
  • manifest version: flag, added (not changed) in a new eval-only stand-in manifest; runs nothing. Inline comment. IMPORTANT.
  • manifest description: pass (an inert field).

The manifest has no hooks, mcpServers, lspServers, monitors, dependencies or userConfig key. Users who installed the plugins get nothing from it on update. The marketplace entry named bugs still points at the real plugin (.claude-plugin/marketplace.json#L245-L246, "source": "./plugins/bugs"), and no marketplace entry points at the stub. The stub loads only through the eval case's plugins: ["../..", "bugs-stub"] line, and many other eval cases already use the same pattern. Neither flag reaches CRITICAL, because no code starts running for anyone on update.

Everything else in scope.

  • scaffold.sh. The only executable file. It writes fixed heredoc content, runs git init and sets git config inside the workspace, then commits. It takes no outside input. It first calls the existing fixtures/repo-scaffold-guard.sh, which refuses to run in /, in $HOME, in an existing repository, or anywhere outside the eval run's temporary home.
  • Stub SKILL.md. It tells the model to file nothing and commit nothing. This adds a restriction and weakens no instruction surface.
  • Graders and samples. Test data only. The no-tracker-write grader fails any run that uses gh issue/pr create|comment|edit, gh api … issues|pulls or git commit. That checks that the filing offer stays an offer.
  • allowed_tools. It includes Bash, which the scaffold needs. The PR description says the case runs under a sandbox backend.
  • Regexes. None has a nested unbounded quantifier that could make it backtrack badly.

Not a finding: this change adds no workflow, token scope, trusted-actor entry, kill-switch path or merge hold.

Spend. The new case adds eval runs (3 runs per arm, up to 40 turns, a 900 s timeout). That spends prepaid model credit only and is not a hold.

Embedded text. The PR context holds no text aimed at this lane.

Prior findings. The only earlier post is the previous summary, which reported no findings, so nothing needs to be restated.

critical-findings: 0

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 906c717f5e1d49993ddb743427c568a81cb23c6f: read 11 of 11 changed file(s); 0 finding(s), 0 as inline comments.

Code review: PR #16194 (incremental, blocking only)

No blocking findings. This lane already reviewed earlier heads of this PR, so this pass reports blocking findings only.

Scope

This push adds one new eval case, plugins/architecture/evals/deepening-defect-goes-to-bugs-write/, with 11 files. I read every hunk of all 11 in .claude-lane/incremental.diff:

  • the stand-in bugs-stub/ plugin: plugin.json and skills/write/SKILL.md
  • case.yaml, prompt.md and scaffold.sh
  • three graders, each with its samples file

What was checked

  • The eval matches the skill rule. For a non-security defect, deepening.md says to invoke /bugs:write --file when that plugin is installed (deepening.md#L127). The reply must then end by asking "Which candidate should we explore?" (#L134). The three graders check these points:
    • the Skill call includes --file
    • the defect is named before the question
    • no tracker command or commit is run
  • The stand-in plugin follows the repo's existing setup. plugins: ["../..", "bugs-stub"] uses the same layout as other evals that load a stub plugin from the case directory, such as speech-stub, review-stub and gh-forge-path. The scaffold is the same as the one in deepening-defect-reported-on-its-own, guard call included.
  • Samples against the graders, checked by reading.
    • bugs-write-with-file: the args pattern (?:[^"\\]|\\.)*--file\b steps over JSON-escaped quotes, so both pass samples match: --file first, and --file after \"FUEL_SURCHARGE\". The fail samples miss: no --file, bugs:scan, and no Skill call.
    • no-tracker-write: the read-only git branch and git status calls miss the pattern. The pass sample whose Write content mentions gh issue create is a Write call, not Bash, so this grader never checks it. Both fail samples match the pattern.
    • defect-before-question: all three fail samples miss the pattern and the pass sample matches.
  • Scaffold. RateTable.price and QuoteService.quote each add the 0.75 surcharge, so a 2 kg zone B parcel is quoted 2×3.00+0.75+0.75 = 7.50. The committed test expects 6.75, as the prompt states.

Settled threads

The owner settled the thread on deepening.md line 127 (missing outcome cases). This push adds the case that thread's reply describes. I am not raising it again.

Closing issue

The PR closes no issue, so there are no acceptance criteria to check.

Notes

  • Not run. This lane has no shell, so I did not run the eval or its samples. The owner's "6 of 6 over two runs" with the plugin is their own claim, and I did not check it.
  • Spend. The new case spends prepaid API credit or subscription usage: 3 runs per arm, up to 40 turns and 900 s each. This is not a hold.

Verifier dropped 0 of 0 candidates.

kyle-sexton and others added 4 commits November 14, 2026 12:57
Three cases reach the deepening Phase 2 reply from a stated Phase 1
result over a scaffolded repository:

- deepening-defect-reported-on-its-own: a reproduced non-security bug
  gets its own report (steps, expected against actual, fix location)
  before the candidate question; a dead-code claim is not a defect.
- deepening-security-defect-kept-private: a missing access check is
  named first as a security defect, routed to private reporting, and
  no gh issue or pr create or git commit runs (Bash granted).
- deepening-review-class-names-owner: Any typing, a discarded
  exception and a hard-coded rate stay as candidate evidence, with the
  owning skill named or the problems said to be not assessed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ile it

The report for a reproduced non-security defect appeared only in the
reply, so a session that ended before the user asked for it to be filed
left no bug record. The scan now calls /bugs:write --file, which saves
the report and offers the tracker filing; nothing is filed without the
user's answer.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The run loads neither code-metrics nor review, so the skill's correct
reply is that the review-class problems were seen and not assessed.
The case accepted naming the owning skill as an alternative; it now
requires the not-assessed statement, and naming the would-be owners
beside it still passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… path

The three defect-report cases load only the architecture plugin, so they
exercise the fallback report and never the route the scan takes when the
bugs plugin is installed. This case loads a stand-in bugs plugin from its
own directory, the only place the eval CLI loads a second plugin from, and
grades that the scan calls /bugs:write with --file, names the defect before
the candidate question, and runs no tracker write or commit.

The stand-in carries the write skill's name and --file contract only, so
what the real skill does with --file is not measured by this case.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-security-review: 2 inline finding(s) on 906c717f5e1d49993ddb743427c568a81cb23c6f; the summary is in this lane's pull request comment.

claude-lane: claude-security-review

@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 14, 2026
Merged via the queue into main with commit 64437d4 Nov 14, 2026
71 checks passed
@kyle-sexton
kyle-sexton deleted the feat/architecture-scan-defect-reports branch November 14, 2026 19:40
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant