You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(toolchain): #16228's text reads a Sonnet loss in the #16063 compare (not shown non-inferior) #16355
The #16063 two-version compare of #16228 reads NOT MET for toolchain on Sonnet: the paired with-arm change over 19 cases is -0.01 with a 95% interval of -0.06 to +0.04, so the lower end is below the -0.05 margin (noise-report.py, not shown non-inferior). Opus is non-inferior on the same cases. Per-case reads in the record on #16063: the after side lost single grader runs in ck-malformed-config-degrades (1.00 to .78), ck-missing-tool-incomplete (1.00 to .83), ck-skips-lint-only-surfaces (1.00 to .83) and ck-detects-changed-ecosystems (1.00 to .92), and gained in ln-missing-tool-incomplete (.67 to 1.00).
The numeric bars below (runs per side, margins) are the filing session's judgment, not measured and not owner-approved. Whoever registers the eval sets them from the measured rates.
A before and after eval of toolchain on Sonnet over the same 19 cases, 6 runs per side, both sides run together, is recorded in that pull request's body or here, and reads non-inferior at margin 0.05; or the record states that the loss is noise, with the per-case evidence.
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: in the #16063 compare of #16228, toolchain on Sonnet is not shown non-inferior (paired change -0.01, 95% interval -0.06 to +0.04, below the -0.05 margin over 19 cases). Four cases lost single grader runs. The work is to read those traces and state the cause of each loss. If the #16228 text is the cause, fix the toolchain skill text in its own PR. Then record a before/after Sonnet eval, or show per-case evidence that the loss is noise.
Nothing is missing for triage. The cause is not yet known, so the approach depends on what the traces show. The 6-runs-per-side eval is host or paid-run work, and the issue already carries host-run for that. The numeric bars are noted in the issue as unapproved, so whoever registers the eval will set them.
Related, not duplicates: #16229 and the verification issue about #16228 (a Sonnet case drop in the same #16063 compare).
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "A before and after eval of toolchain on Sonnet over the same 19 cases, 6 runs..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
Record: the Sonnet loss is noise, with per-case evidence (this session's reading, not an owner decision).
Read from the kept #16063 compare: the Sonnet toolchain result files and each named run's trace, side taken from the file stem. #16228 changed three lines in each of the check and lint skills (the Arguments line, "First add each ecosystem the repo's build files infer", "name the inferred id and its globs in the report") and added the ladder's "Covered ecosystems" section. None of the four lost runs traces to those lines.
Case
Lost run (after side)
Failed grader
Cause, with the trace text
ck-malformed-config-degrades
1 of 3
warns-malformed, meets-expectations
The agent cut its own listing with | head -40, so the record file fell off the end, and the reply says "No .claude/ecosystems/dotnet.yaml exists, so I used the bundled default."
No report was printed. The record step failed in the eval sandbox ("git update-index call failed, probably from the sandbox") and the only text is a note beginning "The check came back INCOMPLETE, and the result was not recorded."
ck-skips-lint-only-surfaces
1 of 2 counted
meets-expectations
Same shape: "The result was not recorded. The record script exited 2 because it couldn't hash the working tree"; no report text in the trace. The two regex graders passed.
ck-detects-changed-ecosystems
1 of 3
meets-expectations
The reply marked Build and Test syntax only and read Overall: PASS. The syntax only rule is the same text on both sides; the judge's criteria list "pass / FAIL / skip / —" and predate that status. Judge reasons are not stored, so this cause is inferred.
The note-only ending also occurs on the before side, in other cases: ck-gate-fails-overall run 1 ("The result was not recorded. The record script exited 2 because it couldn't hash the working tree") and ck-clean-tree-branch-diff run 3. Over the 51 Sonnet ck-* traces scanned, the last text block lacks an Overall line in 3 of 27 after-side runs and 2 of 24 before-side runs.
The head -40 slip is a tool-use choice inside one run; another after-side run used the same pipe on a shorter listing and passed.
Checked by this session directly: the quoted trace text for the first three cases and the before-side ck-gate-fails-overall run. Relayed from a worker's read and not re-read: the 51-trace scan and the fourth case.
Third done-when item: met by its second branch ("or the record states that the loss is noise, with the per-case evidence"). No before and after eval was run, and no toolchain text was changed for this issue.
Found while reading, not fixed here:
The record step (record-result.mjs) exits 2 in eval sandboxes because git update-index is blocked, and some runs then end on a note with no report. It costs about two to three runs a side in any toolchain compare.
The ck-detects-changed-ecosystems judge criteria omit the syntax only status.
Activity
github-actions commentedon Nov 15, 2026
Size: Small
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: in the #16063 compare of #16228, toolchain on Sonnet is not shown non-inferior (paired change -0.01, 95% interval -0.06 to +0.04, below the -0.05 margin over 19 cases). Four cases lost single grader runs. The work is to read those traces and state the cause of each loss. If the #16228 text is the cause, fix the toolchain skill text in its own PR. Then record a before/after Sonnet eval, or show per-case evidence that the loss is noise.
Nothing is missing for triage. The cause is not yet known, so the approach depends on what the traces show. The 6-runs-per-side eval is host or paid-run work, and the issue already carries
host-runfor that. The numeric bars are noted in the issue as unapproved, so whoever registers the eval will set them.Related, not duplicates: #16229 and the verification issue about #16228 (a Sonnet case drop in the same #16063 compare).
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 15, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "A before and after eval of toolchain on Sonnet over the same 19 cases, 6 runs..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).kyle-sexton commentedon Nov 17, 2026
Record: the Sonnet loss is noise, with per-case evidence (this session's reading, not an owner decision).
Read from the kept #16063 compare: the Sonnet toolchain result files and each named run's trace, side taken from the file stem. #16228 changed three lines in each of the check and lint skills (the Arguments line, "First add each ecosystem the repo's build files infer", "name the inferred id and its globs in the report") and added the ladder's "Covered ecosystems" section. None of the four lost runs traces to those lines.
ck-malformed-config-degradeswarns-malformed,meets-expectations| head -40, so the record file fell off the end, and the reply says "No.claude/ecosystems/dotnet.yamlexists, so I used the bundled default."ck-missing-tool-incompletelint-skip-tool-missing,overall-incomplete,overall-names-causegit update-indexcall failed, probably from the sandbox") and the only text is a note beginning "The check came back INCOMPLETE, and the result was not recorded."ck-skips-lint-only-surfacesmeets-expectationsck-detects-changed-ecosystemsmeets-expectationssyntax onlyand readOverall: PASS. Thesyntax onlyrule is the same text on both sides; the judge's criteria list "pass / FAIL / skip / —" and predate that status. Judge reasons are not stored, so this cause is inferred.Why this is noise and not the #16228 text:
ck-gate-fails-overallrun 1 ("The result was not recorded. The record script exited 2 because it couldn't hash the working tree") andck-clean-tree-branch-diffrun 3. Over the 51 Sonnetck-*traces scanned, the last text block lacks an Overall line in 3 of 27 after-side runs and 2 of 24 before-side runs.head -40slip is a tool-use choice inside one run; another after-side run used the same pipe on a shorter listing and passed.Checked by this session directly: the quoted trace text for the first three cases and the before-side
ck-gate-fails-overallrun. Relayed from a worker's read and not re-read: the 51-trace scan and the fourth case.Third done-when item: met by its second branch ("or the record states that the loss is noise, with the per-case evidence"). No before and after eval was run, and no toolchain text was changed for this issue.
Found while reading, not fixed here:
record-result.mjs) exits 2 in eval sandboxes becausegit update-indexis blocked, and some runs then end on a note with no report. It costs about two to three runs a side in any toolchain compare.ck-detects-changed-ecosystemsjudge criteria omit thesyntax onlystatus.