Skip to content

fix(toolchain): #16228's text reads a Sonnet loss in the #16063 compare (not shown non-inferior) #16355

Closed

Description

@kyle-sexton

Refs: #16063

The #16063 two-version compare of #16228 reads NOT MET for toolchain on Sonnet: the paired with-arm change over 19 cases is -0.01 with a 95% interval of -0.06 to +0.04, so the lower end is below the -0.05 margin (noise-report.py, not shown non-inferior). Opus is non-inferior on the same cases. Per-case reads in the record on #16063: the after side lost single grader runs in ck-malformed-config-degrades (1.00 to .78), ck-missing-tool-incomplete (1.00 to .83), ck-skips-lint-only-surfaces (1.00 to .83) and ck-detects-changed-ecosystems (1.00 to .92), and gained in ln-missing-tool-incomplete (.67 to 1.00).

The numeric bars below (runs per side, margins) are the filing session's judgment, not measured and not owner-approved. Whoever registers the eval sets them from the measured rates.

Done when

session: GitHub Support

Activity

  1. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  2. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  3. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor

    Size: Small
    Complexity: Medium
    Uncertainty: Knowable
    Flags: none
    Blocked by: none

    Understood: in the #16063 compare of #16228, toolchain on Sonnet is not shown non-inferior (paired change -0.01, 95% interval -0.06 to +0.04, below the -0.05 margin over 19 cases). Four cases lost single grader runs. The work is to read those traces and state the cause of each loss. If the #16228 text is the cause, fix the toolchain skill text in its own PR. Then record a before/after Sonnet eval, or show per-case evidence that the loss is noise.

    Nothing is missing for triage. The cause is not yet known, so the approach depends on what the traces show. The 6-runs-per-side eval is host or paid-run work, and the issue already carries host-run for that. The numeric bars are noted in the issue as unapproved, so whoever registers the eval will set them.

    Related, not duplicates: #16229 and the verification issue about #16228 (a Sonnet case drop in the same #16063 compare).

    Automated triage. A maintainer reviews it.

  4. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  5. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 3 (eval: "A before and after eval of toolchain on Sonnet over the same 19 cases, 6 runs..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  6. kyle-sexton commented on Nov 17, 2026

    @kyle-sexton
    MemberAuthor

    Record: the Sonnet loss is noise, with per-case evidence (this session's reading, not an owner decision).

    Read from the kept #16063 compare: the Sonnet toolchain result files and each named run's trace, side taken from the file stem. #16228 changed three lines in each of the check and lint skills (the Arguments line, "First add each ecosystem the repo's build files infer", "name the inferred id and its globs in the report") and added the ladder's "Covered ecosystems" section. None of the four lost runs traces to those lines.

    Case Lost run (after side) Failed grader Cause, with the trace text
    ck-malformed-config-degrades 1 of 3 warns-malformed, meets-expectations The agent cut its own listing with | head -40, so the record file fell off the end, and the reply says "No .claude/ecosystems/dotnet.yaml exists, so I used the bundled default."
    ck-missing-tool-incomplete 1 of 3 lint-skip-tool-missing, overall-incomplete, overall-names-cause No report was printed. The record step failed in the eval sandbox ("git update-index call failed, probably from the sandbox") and the only text is a note beginning "The check came back INCOMPLETE, and the result was not recorded."
    ck-skips-lint-only-surfaces 1 of 2 counted meets-expectations Same shape: "The result was not recorded. The record script exited 2 because it couldn't hash the working tree"; no report text in the trace. The two regex graders passed.
    ck-detects-changed-ecosystems 1 of 3 meets-expectations The reply marked Build and Test syntax only and read Overall: PASS. The syntax only rule is the same text on both sides; the judge's criteria list "pass / FAIL / skip / —" and predate that status. Judge reasons are not stored, so this cause is inferred.

    Why this is noise and not the #16228 text:

    • The note-only ending also occurs on the before side, in other cases: ck-gate-fails-overall run 1 ("The result was not recorded. The record script exited 2 because it couldn't hash the working tree") and ck-clean-tree-branch-diff run 3. Over the 51 Sonnet ck-* traces scanned, the last text block lacks an Overall line in 3 of 27 after-side runs and 2 of 24 before-side runs.
    • The head -40 slip is a tool-use choice inside one run; another after-side run used the same pipe on a shorter listing and passed.

    Checked by this session directly: the quoted trace text for the first three cases and the before-side ck-gate-fails-overall run. Relayed from a worker's read and not re-read: the 51-trace scan and the fourth case.

    Third done-when item: met by its second branch ("or the record states that the loss is noise, with the per-case evidence"). No before and after eval was run, and no toolchain text was changed for this issue.

    Found while reading, not fixed here:

  7. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions