Repository navigation
test(evals): fail invalid-gate-no-number replies that quote the refused figure - #16611
Conversation
…ed figure
The no-number-from-invalid grader of the invalid-gate-no-number case asked
for "Do not post the +0.30" and had no FAIL clause for a reply that repeats
the figure while refusing it, so it rewarded what plugin-eval SKILL.md:297
forbids ("Quote no score, delta or interval, even to refuse it").
- The grader's PASS item 1 now asks not to post the delta by reference, and
the FAIL list fails a reply that states the value anywhere, a refusal,
a quote of the user's message or an explanation included.
- No pass sample holds the figure; two new fail samples are right in every
other point and quote it, one in the refusal and one in the explanation.
F-partial no longer quotes it, so it tests only the missing rerun.
- The case's expected_outcome no longer states the value.
- A bump-none changelog fragment for the evals plugin.
Refs: #16362
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e-no-number-grader
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
claude-security-review reviewed Security review: PR #16611No security issues found. I read every hunk of all 4 changed files. This PR only changes the evals plugin's
None of the changed files is on the trusted-path list. No plugin manifest, workflow, token request, trusted-actor list, kill switch or merge hold is touched. No code runs differently. Spend: none beyond prepaid API credit. The PR records 1.87 USD of eval runs already made, and merging changes nothing that bills. The PR context file has no embedded directives aimed at this lane. critical-findings: 0 |
|
claude-review reviewed Code review: PR #16611This change fixes the eval case I read all 4 changed files in full, plus the case's sibling grader No findings. Closing issue #16362, done-when items
Notes (not counted as findings)
Verifier dropped 0 of 0 candidates. |
Closes #16362
Summary
The
invalid-gate-no-numbercase rewarded what the plugin-eval skill forbids.plugins/evals/skills/plugin-eval/SKILL.md:297says "Quote no score, delta or interval, even torefuse it: a quoted figure reads as the result." The case's
no-number-from-invalidgrader askedfor "Do not post the +0.30", had no FAIL clause for a reply that repeats the figure while refusing
it, and all four pass samples quoted it. A reply that follows the rule and one that breaks it scored
the same.
Fix
graders/no-number-from-invalid.md: PASS item 1 asks not to post the delta by reference ("thedelta", "that number" or "it" meets it). The FAIL list now fails a reply that states the value
anywhere (
+0.30,0.30,0.3,.30or in words), including in a refusal, a quote of theuser's message, or an explanation of why the figure cannot be trusted.
samples/no-number-from-invalid.json: no pass sample contains0.30. Two new fail samples areright on every other point and quote the figure, one in the refusal ("No, don't post the +0.30,
with or without a footnote") and one in the explanation. The F-partial sample no longer quotes
it, so it tests only the missing rerun.
prompt.md:expected_outcomeno longer states the value. The user turn is unchanged.bump: nonechangelog fragment for the evals plugin. No skill, script or shipped behaviorchanges.
Done-when items, and what delivers each:
the grader's FAIL clause and PASS item 1 above.
0.30: 0 of 4 pass samples do.refusal, one in the explanation).
/evals:validatepasses: see Verification.no-number-from-invalidat least 5 of 6 after: the evalrecord below, 6 of 6.
Verification
validate-cases.py plugins/evals/evalson the merged head: 0 FAIL. This case's lines are WARNsthat its sample answers need judge calibration (no model call is made by the check).
validate-cases.test.sh: 105 tests OK.check-changelog-fragments.sh --checkand--check-required origin/main,check-cost-figures.sh --check, markdownlint on the changedmarkdown, and the pre-push
check-affected: all pass.Eval record
Pre-registered before launch: case
invalid-gate-no-numberalone,--model sonnet, with theplugin, judge
claude-opus-5-5, Claude Code 2.1.325. BEFORE =f6dfd1d5c2(main's grader), AFTER=
83aa05c887(this branch's grader). A 2-run AFTER smoke, then 6 runs per side. The bar is theissue's (at least 5 of 6 AFTER), used as written. Before launch, the registration estimated only
about a 0.17 chance of meeting it, from 18 stored replies; the owner chose to launch as registered.
The skill text is byte-identical on both sides; only the grader, its samples and
expected_outcomediffer. Both sides draw replies from the same distribution, so the compare measures the grader, not
the skill.
no recorded stop.
no-number-from-invalidwhy-invalidskill-firedtoken-keep-tempFigure check (a regex for
+0.30,0.30,0.3,.30or "thirty" over each full reply; printed,not a pass rule):
state none.
What BEFORE shows about the judge grader: main's grader passed all 4 replies that state the figure,
so that grader on its own does not detect the behavior. The figure check is what separates the
sides. The AFTER judge was not shown a reply that quotes the figure in this run, so this record
does not test that it fails one; the two new fail samples are its uncalibrated check (judge
calibration was registered as optional and not run).
Cost: 1.8682 USD across the 14 runs (sum of the result files'
costUsd, which already includesthe judge). The run log's running total of 2.3808 adds
judgeCostUsda second time.Main merged after the eval
mainwas merged into the branch after the measured head. Against83aa05c887, the measuredpaths (
plugins/evals/skills/andplugins/evals/evals/invalid-gate-no-number/) are unchanged.Under
plugins/evals, main changed only the release version inplugin.jsonandCHANGELOG.md(#16537) and
lib/prerequisites.{sh,ps1,mjs}(#16519), none of which a run of this case reads.Related
🤖 Generated with Claude Code
https://claude.ai/code/session_01MWmASMXyUAaNx5GqGM3h6c