You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(code-tidying): give the tidy-diff comment-triage eval case the comment tooling #12924
#12703 got tree-sitter and pygments into the eval sandbox for the remove-comments and audit cases through plugins/code-tidying/evals/fixtures/comment-tooling-seed.sh, called from seed.sh. The tidy-diff cases never get it, so no run can exercise one of tidy-diff's applied, proven comment deletions (change-shape.sh exits 3 without the parser).
Correction (lane 2, 2026-11-04; agent choice, not an owner answer): the cases that seed through td-seed.sh are /code-tidying:tidy cases. tidy's comment tidying (#15 in skills/tidy/reference/tidyings.md) is a judgment delete with no parser proof, and its allowed-tools grants no change-shape.sh, so td-seed.sh needs no tooling. The tidy-diff cases are the five tidy-diff-* cases, each with its own inline scaffold. Of those, only tidy-diff-triage-exempt-and-class-b-kept lists code lines holding comments for the comment pass to prove. The other four list no code comment (behavior-changing-cleanup-proposed, wrapper-inlined-only-in-rows), list markdown only (markdown-skips-code-passes), or stop before any pass (no-lines-file-derives-no-scope).
Done when
tidy-diff-triage-exempt-and-class-b-kept/scaffold.sh installs the tooling through comment-tooling-seed.sh, and the case grades the proven deletion of its narrating comment.
A before/after pair on that case (before: the tooling call removed; after: as committed), each arm VALID per run-validity.py, shows what the tooling does to its score, with a negative control as code-tidying: make the remove-comments body rules measurable #12703 used. The after traces show the tooling reached the runs (a change-shape.sh exit other than 3).
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood: td-seed.sh, and any other code-tidying seed whose skill path needs the comment tooling, should install it through comment-tooling-seed.sh, as seed.sh does after #12703. Then a before/after eval pair on the tidy-diff cases should show what the change does to their scores. Each arm needs to be VALID per run-validity.py, with a negative control. The seed edit can be checked in the lane. The eval pair needs a host run, which the host-run label already covers. No duplicates found among the open issues.
Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
changed the title [-]test(code-tidying): give the tidy-diff eval cases the comment tooling[/-][+]test(code-tidying): convert the tidy-diff eval cases and give them the comment tooling[/+]on Nov 4, 2026
Body corrected (agent choice, not an owner answer): the td-* cases are /code-tidying:tidy's, whose comment tidying needs no parser proof, so td-seed.sh stays as it is. The intended target, tidy-diff, has no case.yaml suite; its 7 legacy evals.json cases get converted, scaffolded through seed.sh (which installs the tooling) and measured. The other 61 legacy evals.json cases in code-tidying (audit-dead-code 8, batch-simplify 11, remove-comments 34, setup 3, tidy 5) stay out of scope here.
changed the title [-]test(code-tidying): convert the tidy-diff eval cases and give them the comment tooling[/-][+]test(code-tidying): give the tidy-diff comment-triage eval case the comment tooling[/+]on Nov 4, 2026
Preregistration, posted before the first paid run.
This replaces the conversion plan in my previous comment. tidy-diff already has five case.yaml cases (tidy-diff-*), so nothing is converted. The body now names the one case that needs the tooling.
Case:tidy-diff-triage-exempt-and-class-b-kept, the only changed case.
Arms, each --ablation none --runs 3, Sonnet under test, Opus judge, --allow-tools Bash Edit Write, Claude Code 2.1.318, run through docker-links.py run --, plugin copies outside git:
before: the branch tree with the comment-tooling-seed.sh line removed from the scaffold. The new narration-deleted grader is kept, so only the tooling differs.
after: the branch tree as committed.
control: the after tree with the tidy-diff SKILL.md body cut to its frontmatter, launched together with the after arm.
9 model runs, -j 3, ceiling 25 USD.
Validity: every run VALID per run-validity.py. A run with a usage-limit or auth error is rerun under this plan. No other exclusions.
Tooling reached the runs: at least one after run shows a change-shape.sh exit other than 3. If none does, the after arm does not count and the tooling seed is debugged before anything is read.
Control detected: the control's case mean falls 0.15 or more below the after arm, or a grader the after arm passes in all 3 runs fails in at least 2 control runs. If the control goes undetected, the case is reported unable to detect a loss in the comment pass. The seed still ships, since it is environment, not a text cut.
Reported: the before and after case means and the per-grader pass counts, with narration-deleted and proof-ran named. Done-when 3 asks for the effect to be shown, so a drop is reported, not reverted.
Case tidy-diff-triage-exempt-and-class-b-kept: Sonnet under test, Opus judge, --runs 3, Claude Code 2.1.318. All 9 runs are VALID per run-validity.py --scaffolded. Each arm carries one warning, skill fired: the slash-command prompt loads the skill without a Skill tool call, and the case has no skill-fired grader.
arm
mean
narration-deleted
proof-ran
code-intact
cost
before (tooling call removed)
0.792
0/3
3/3
1/3
0.62 USD
after (as committed)
0.958
3/3
3/3
2/3
0.48 USD
control (after, SKILL.md body cut)
0.458
3/3
0/3
2/3
0.21 USD
Tooling reached the runs: every after run got COMMENT-ONLY (typescript via tree_sitter_typescript) ... exit=0 from change-shape.sh and then deleted the comment.
Control: detected. It fell 0.50 below the after arm. public-doc-kept, step-comment-kept, proof-ran and deletions-tied-to-verdict passed in 3/3 after runs and failed in 3/3 control runs.
Effect: with the tooling, the case measures an applied, proven deletion (narration-deleted 0/3 to 3/3) and the mean rises 0.17.
code-intact fails in some runs of every arm. It reflects the model's other edits to src/users.ts, not the seed.
Activity
github-actions commentedon Nov 3, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
Understood:
td-seed.sh, and any other code-tidying seed whose skill path needs the comment tooling, should install it throughcomment-tooling-seed.sh, asseed.shdoes after #12703. Then a before/after eval pair on the tidy-diff cases should show what the change does to their scores. Each arm needs to be VALID perrun-validity.py, with a negative control. The seed edit can be checked in the lane. The eval pair needs a host run, which thehost-runlabel already covers. No duplicates found among the open issues.Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Nov 3, 2026
Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 3, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 3, 2026
[-]test(code-tidying): give the tidy-diff eval cases the comment tooling[/-][+]test(code-tidying): convert the tidy-diff eval cases and give them the comment tooling[/+]kyle-sexton commentedon Nov 4, 2026
Body corrected (agent choice, not an owner answer): the
td-*cases are/code-tidying:tidy's, whose comment tidying needs no parser proof, sotd-seed.shstays as it is. The intended target, tidy-diff, has nocase.yamlsuite; its 7 legacyevals.jsoncases get converted, scaffolded throughseed.sh(which installs the tooling) and measured. The other 61 legacyevals.jsoncases in code-tidying (audit-dead-code 8, batch-simplify 11, remove-comments 34, setup 3, tidy 5) stay out of scope here.[-]test(code-tidying): convert the tidy-diff eval cases and give them the comment tooling[/-][+]test(code-tidying): give the tidy-diff comment-triage eval case the comment tooling[/+]kyle-sexton commentedon Nov 4, 2026
Preregistration, posted before the first paid run.
This replaces the conversion plan in my previous comment. tidy-diff already has five
case.yamlcases (tidy-diff-*), so nothing is converted. The body now names the one case that needs the tooling.tidy-diff-triage-exempt-and-class-b-kept, the only changed case.--ablation none --runs 3, Sonnet under test, Opus judge,--allow-tools Bash Edit Write, Claude Code 2.1.318, run throughdocker-links.py run --, plugin copies outside git:comment-tooling-seed.shline removed from the scaffold. The newnarration-deletedgrader is kept, so only the tooling differs.SKILL.mdbody cut to its frontmatter, launched together with the after arm.-j 3, ceiling 25 USD.run-validity.py. A run with a usage-limit or auth error is rerun under this plan. No other exclusions.change-shape.shexit other than 3. If none does, the after arm does not count and the tooling seed is debugged before anything is read.narration-deletedandproof-rannamed. Done-when 3 asks for the effect to be shown, so a drop is reported, not reverted.kyle-sexton commentedon Nov 4, 2026
Result, measured under the preregistration.
Case
tidy-diff-triage-exempt-and-class-b-kept: Sonnet under test, Opus judge,--runs 3, Claude Code 2.1.318. All 9 runs are VALID perrun-validity.py --scaffolded. Each arm carries one warning, skill fired: the slash-command prompt loads the skill without a Skill tool call, and the case has no skill-fired grader.narration-deletedproof-rancode-intactCOMMENT-ONLY (typescript via tree_sitter_typescript) ... exit=0fromchange-shape.shand then deleted the comment.public-doc-kept,step-comment-kept,proof-rananddeletions-tied-to-verdictpassed in 3/3 after runs and failed in 3/3 control runs.narration-deleted0/3 to 3/3) and the mean rises 0.17.code-intactfails in some runs of every arm. It reflects the model's other edits tosrc/users.ts, not the seed.