Skip to content

feat(mutation-testing): add seed-faults, hand-seeded defects graded as caught over planted - #16632

Merged
kyle-sexton merged 6 commits into
mainfrom
feat/mutation-testing-seed-faults
Nov 16, 2026
Merged

kyle-sexton merged 6 commits into
mainfrom
feat/mutation-testing-seed-faults

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

No related issue: an owner decision from a design interview (2026-11-16).

Summary

mutation-testing measured a suite only with operator-generated mutants. Nothing let a suite (or a
reviewer) be graded on realistic, hand-written defects, the fault-seeding use Pezzè and Young
describe ("Fault seeding can be used to evaluate the thoroughness of a test suite", Software
Testing and Analysis: Process, Principles, and Techniques
, Wiley 2008, ch. 16 "Fault-Based
Testing").

Fix

  • New skill /mutation-testing:seed-faults. Plants a few realistic defects (5 by default,
    --count) one at a time in a detached worktree created in a temp directory outside the
    repository, runs the selected tests on each after a green baseline in that worktree, and
    reports seeds caught over seeds planted with each surviving seed's file:line and change.
    Invalid (non-building) seeds are counted apart.
  • What is graded never sees the seeds. The planter may read tests and never edits them; seeds
    carry no marker; the ledger stays outside the worktree. --reviewer grades a fresh-context
    reviewer given whole files and a neutral brief.
  • Nothing planted is kept. No commit anywhere, the worktree is removed on every exit path, and
    the user's checkout is compared with a Phase 0 snapshot (git status --porcelain, HEAD)
    before any result is reported.
  • Reported as an estimate. Andrews, Briand and Labiche,
    ICSE 2005: hand-seeded faults "seem to be harder to
    detect than real faults", so the ratio is an estimate of thoroughness, never a defect count.
  • Agent-driven and tool-agnostic. The skill states that decision and carries no tool-doc
    links. Basis, read 2026-11-16: the StrykerJS
    Plugins page lists test runners,
    reporters, checkers and ignorers, and no mutator plugin; the
    mutmut docs describe no custom
    mutation operators. The test command is resolved as audit does and never guessed.
  • principles gains a "mutants or hand-seeded faults?" entry and names the skill in ## Next.
  • Name: seed-faults (the literature's term). Basis: three blind naming generators each ranked
    seed-faults or seed-defects first; seed-faults matches both cited sources.

Verification

Two eval cases, 3 runs per arm, with the plugin and with no plugin (claude plugin eval, run from
a copy outside git with the documented Docker link swap):

Case With plugin No plugin
sf-no-test-command-asks: asks for the test command, runs nothing, plants nothing 3/3 0/3
sf-reports-caught-over-planted: worktree, no commit, tests untouched, source intact, x of 3, survivor file:line, "estimate" 3/3 (after re-grade, below) 0/3
  • The eval sandbox mounts .git read-only, so git worktree add fails there and the runs fell
    back to a copy outside the repository; the grader checks the git worktree add attempt, not
    its removal.
  • In the second run one with-plugin run scored a miss because the grader's leading \b did not
    match a git worktree add after a newline in a multi-line command. The pattern was fixed and the
    six traces re-graded offline with it: 3/3 with plugin, 0/3 without. The no-plugin arm reports a
    ratio too but no survivor location and no estimate caveat.
  • validate-cases.py on the plugin's evals: exit 0. check-changed-skills.sh origin/main: 2 skills
    checked, 0 failed. check-affected.sh: passes after shortening the manifest description;
    generate-catalog --check and generate-cheatsheet --check in sync.

Related

  • Known gap: the reviewer mode (--reviewer) has no eval case.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC

…s caught over planted

New skill /mutation-testing:seed-faults plants a few hand-written,
realistic defects one at a time in a throwaway worktree outside the
repository, runs the selected tests against each, and reports seeds
caught over seeds planted with each surviving seed's file:line and
change, as an estimate of suite thoroughness, never a defect count.
Nothing planted is committed; the worktree is removed and the user's
checkout is checked against a snapshot before any result is reported.

principles explains seeded faults against generated mutants and names
the skill in Next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
@kyle-sexton
kyle-sexton marked this pull request as ready for review November 16, 2026 20:16
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T20:18:46.475136Z 69fcb39 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 69fcb397be

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

kyle-sexton and others added 4 commits November 16, 2026 15:19
The skill keeps its decision (agent-driven, tool-agnostic) and drops the
per-tool documentation links; the PR body carries them as the basis.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
A seed that keeps a file's size and lands in the same second as the
previous run could run the cached build of the original code and read as
survived. seed-faults now finds the repository's compiled and cached
artifacts from its toolchain and clears them, or forces a rebuild, in
the throwaway worktree before each seed. The happy-path eval case gains
a grader for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWmASPFjTu4anSp6BowqVC
One eval run reported caught over planted with no estimate wording. The
report section now requires the word beside the ratio, with its reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o grader

A correct reply said "none of the 3 bugs" and failed a grader that took
digits only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/fix-docs
Model rung 0
Leg handoff
Run job success
Activity step skipped
Commits pushed none (read lane)
Run 43952749230

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 9d1054862527b0df6191e5548981a2c12de231cb: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Security review: PR #16632, incremental (3b3ab89 → 9d10548)

No security issues found.

This review covered only the changes since 3b3ab892. I read the whole hunk of the one file on the incremental list in .claude-lane/incremental.diff. This lane already reviewed an earlier head (blocking only: yes), so this pass reports only CRITICAL and IMPORTANT findings.

What changed

  • plugins/mutation-testing/skills/seed-faults/SKILL.md: the ICSE 2005 citation (Andrews, Briand and Labiche) is now plain text with the DOI written out. Before, it was a markdown hyperlink to doi.org. This is a citation in a skill body. It removes, narrows or softens no standing instruction. The guidance around it is unchanged: report the ratio as an estimate and never as a count of defects. The instruction-surface lens therefore finds nothing.

Manifest: no plugin or marketplace manifest changed in this push.

Trust controls: none touched. The file matches no trusted-path glob. The push changes no trusted-actor list, token broker, kill switch, merge hold or .github/actions/ code.

Spend (not a finding): none.

critical-findings: 0

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 9d1054862527b0df6191e5548981a2c12de231cb: read 1 of 1 changed file(s); 0 finding(s), 0 as inline comments.

Code review (incremental, blocking only): 3b3ab892..9d105486

This round covers the one file the push changed, plugins/mutation-testing/skills/seed-faults/SKILL.md. I read its whole hunk in .claude-lane/incremental.diff. This lane has already reviewed an earlier head (blocking only: yes), so this round reports blocking findings only.

What changed
The Andrews, Briand and Labiche citation is now plain text. It used to be a Markdown link to https://doi.org/10.1145/1062455.1062530. The new text gives the title in quotes, the venue (Proceedings of the 27th International Conference on Software Engineering, ICSE 2005) and the DOI as plain text. The quoted finding ("seem to be harder to detect than real faults") and the guidance after it read the same as before.

Assessment

  • The edit only changes how the source is cited. The authors, title, venue and DOI all match the old link, and the skill's instructions say the same thing as before.
  • The rewrapped line 128 is longer than the lines around it. That is a formatting point a linter would catch, not a blocking issue.

No blocking findings.

Settled threads. The one settled thread, the cache-clearing point on seed-faults/SKILL.md around original line 80, is marked lines changed since resolution: yes. This push changes only the citation near line 125, so it does not touch the code that thread was about, and I raise nothing new on it.

Verifier dropped 0 of 0 candidates.

The DOI link resolves to a publisher page that refuses automated
clients, so the online link check fails on it. The open copies found
answer non-browser clients with 202 or 403 too, so the skill gives the
full reference with the DOI as plain text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 974db1f Nov 16, 2026
48 checks passed
@kyle-sexton
kyle-sexton deleted the feat/mutation-testing-seed-faults branch November 16, 2026 21:09
@github-actions github-actions Bot added the merge-path: owner-agent Post-merge accounting record (README.md, Merge paths) label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-path: owner-agent Post-merge accounting record (README.md, Merge paths)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant