Repository navigation
feat(mutation-testing): add seed-faults, hand-seeded defects graded as caught over planted - #16632
Conversation
…s caught over planted New skill /mutation-testing:seed-faults plants a few hand-written, realistic defects one at a time in a throwaway worktree outside the repository, runs the selected tests against each, and reports seeds caught over seeds planted with each surviving seed's file:line and change, as an estimate of suite thoroughness, never a defect count. Nothing planted is committed; the worktree is removed and the user's checkout is checked against a snapshot before any result is reported. principles explains seeded faults against generated mutants and names the skill in Next. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 69fcb397be
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The skill keeps its decision (agent-driven, tool-agnostic) and drops the per-tool documentation links; the PR body carries them as the basis. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
A seed that keeps a file's size and lands in the same second as the previous run could run the cached build of the original code and read as survived. seed-faults now finds the repository's compiled and cached artifacts from its toolchain and clears them, or forces a rebuild, in the throwaway worktree before each seed. The happy-path eval case gains a grader for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
One eval run reported caught over planted with no estimate wording. The report section now requires the word beside the ratio, with its reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o grader A correct reply said "none of the 3 bugs" and failed a grader that took digits only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lane run report
Failed GitHub API calls
|
|
claude-security-review reviewed Security review: PR #16632, incremental (3b3ab89 → 9d10548)No security issues found. This review covered only the changes since What changed
Manifest: no plugin or marketplace manifest changed in this push. Trust controls: none touched. The file matches no trusted-path glob. The push changes no trusted-actor list, token broker, kill switch, merge hold or Spend (not a finding): none. critical-findings: 0 |
|
claude-review reviewed Code review (incremental, blocking only):
|
The DOI link resolves to a publisher page that refuses automated clients, so the online link check fails on it. The open copies found answer non-browser clients with 202 or 403 too, so the skill gives the full reference with the DOI as plain text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No related issue: an owner decision from a design interview (2026-11-16).
Summary
mutation-testingmeasured a suite only with operator-generated mutants. Nothing let a suite (or areviewer) be graded on realistic, hand-written defects, the fault-seeding use Pezzè and Young
describe ("Fault seeding can be used to evaluate the thoroughness of a test suite", Software
Testing and Analysis: Process, Principles, and Techniques, Wiley 2008, ch. 16 "Fault-Based
Testing").
Fix
/mutation-testing:seed-faults. Plants a few realistic defects (5 by default,--count) one at a time in a detached worktree created in a temp directory outside therepository, runs the selected tests on each after a green baseline in that worktree, and
reports seeds caught over seeds planted with each surviving seed's
file:lineand change.Invalid (non-building) seeds are counted apart.
carry no marker; the ledger stays outside the worktree.
--reviewergrades a fresh-contextreviewer given whole files and a neutral brief.
the user's checkout is compared with a Phase 0 snapshot (
git status --porcelain,HEAD)before any result is reported.
ICSE 2005: hand-seeded faults "seem to be harder to
detect than real faults", so the ratio is an estimate of thoroughness, never a defect count.
links. Basis, read 2026-11-16: the StrykerJS
Plugins page lists test runners,
reporters, checkers and ignorers, and no mutator plugin; the
mutmut docs describe no custom
mutation operators. The test command is resolved as
auditdoes and never guessed.## Next.seed-faults(the literature's term). Basis: three blind naming generators each rankedseed-faultsorseed-defectsfirst;seed-faultsmatches both cited sources.Verification
Two eval cases, 3 runs per arm, with the plugin and with no plugin (
claude plugin eval, run froma copy outside git with the documented Docker link swap):
sf-no-test-command-asks: asks for the test command, runs nothing, plants nothingsf-reports-caught-over-planted: worktree, no commit, tests untouched, source intact,x of 3, survivorfile:line, "estimate".gitread-only, sogit worktree addfails there and the runs fellback to a copy outside the repository; the grader checks the
git worktree addattempt, notits removal.
\bdid notmatch a
git worktree addafter a newline in a multi-line command. The pattern was fixed and thesix traces re-graded offline with it: 3/3 with plugin, 0/3 without. The no-plugin arm reports a
ratio too but no survivor location and no estimate caveat.
validate-cases.pyon the plugin's evals: exit 0.check-changed-skills.sh origin/main: 2 skillschecked, 0 failed.
check-affected.sh: passes after shortening the manifest description;generate-catalog --checkandgenerate-cheatsheet --checkin sync.Related
--reviewer) has no eval case.🤖 Generated with Claude Code
https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC