Repository navigation
fix(context-budget): correct the additivity claim and report the verdict per bucket - #3883
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Claude finished @kyle-sexton's task in 1m 6s —— View job Security review — no issues found
Skip gate met: no security-relevant surface. The diff is entirely documentation prose No security issues found. Per this lane's scope, no further action. |
|
Claude finished @kyle-sexton's task in 4m 18s —— View job
|
|
Last security-reviewed head: |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5d97b7aafa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Claude has reviewed this PR 1 time. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
…verdict The additivity record carried one verdict over both attributed buckets and computed it as a boolean, so an incomparable reading surfaced as a definite "not additive" and a consumer could not see that the deferred side composes while the prefix side does not. `additive` is now tri-state: true/false are measured verdicts and null means the reading could not be measured. A `perBucket` map carries the same three fields per attributed bucket, read off the prefixDelta/deferredDelta values the per-tool rows already carry, so no new measurement pass runs. A bucket absent from both runs is outside the binary's category vocabulary and gets no verdict row at all. Refs #3863 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ViPsHkL3ng9xWt2GjEQJob
Refs #3863 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ViPsHkL3ng9xWt2GjEQJob
The 0.6.24 entry said all three corrected homes (skill body, lever categoryBasis, README) point at `attribute --verify-additivity`. Only the skill body and the lever carry that pointer; the README states the corrected composition claim without it. The PR scopes the pointer to those two homes deliberately, so this fixes the CHANGELOG wording rather than adding a pointer to the README. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ViPsHkL3ng9xWt2GjEQJob
…listing drift Skill-listing and Skills-token mismatches poison the prefix System tools bucket, but the deferred bucket's numeric deltas remain valid under the shared mode/binary checks. Per-bucket additivity was applying systemToolsComparable to every column, so a combined-run listing change published System tools (deferred).additive as null. Gate the listing checks on the prefix bucket only, expose modeBinaryComparable for the shared checks, and cover both listing signature and Skills-token drift in the hermetic attribute suite. Refs #3863 Co-authored-by: ksextonmelodic <ksextonmelodic@gmail.com>
5d97b7a to
4b1b406
Compare
Closes #3863
Summary
context-budgetasserted flatly that the per-tool deny deltas are compositional, so a basket oftrims prices from its members. Measured readings do not support that as one statement over both
attributed buckets: the deferred side sums to the token, while the prefix side double-counts, so
the sum of prefix deltas overstates what a basket saves on the prefix. The claim lived in three
places that each stated it independently, and the engine reported a single additivity verdict
computed as a boolean, so an unmeasurable reading surfaced as a definite "not additive".
Fix
Prose, all in this change so no home keeps the old claim:
plugins/context-budget/skills/audit/SKILL.mdandplugins/context-budget/skills/audit/reference/levers.json(thedeny-bare-toollever'scategoryBasis) now say which side composes: deferred-side deltas add and a basket's deferredsaving is the sum of its members; prefix-side deltas double-count and their sum is only an upper
bound on the basket's prefix saving. Both point at
attribute --verify-additivityfor theper-bucket verdict.
plugins/context-budget/README.mdcarried the same unqualified sentence. The issue names twohomes; this is a third instance of the identical claim, so it is corrected with them rather than
left as the one surface still asserting the old wording. Called out here because it is beyond
the two homes the issue names.
reference/engine.mddocuments the new record shape;reference/report.mdtells the reporter tostate the verdict per bucket and to report a null verdict as not measurable rather than as not
additive.
Engine,
skills/audit/scripts/measure.mjs:additivity.additiveis tri-state.true/falseare measured verdicts;nullmeans thereading could not be measured. Previously
comparable && combinedSaved === sumOfParts, whichpublished every incomparable reading as
false.additivity.perBucket, one entry per attributed bucket carrying{sumOfParts, combinedSaved, additive, reasons}. It is derived from theprefixDelta/deferredDeltavalues already on the saver rows plus the combined-deny comparisonthis run already performs: no extra measurement pass, no new snapshot. Each bucket's verdict is
gated on its own measurability, so one bucket vanishing out of the combined snapshot no longer
costs the other bucket its verdict. A bucket absent from both runs is outside the binary's
category vocabulary and gets no verdict row at all.
systemBucketSaving, the null-not-zero vanished-bucket path) is untouched;the per-bucket reporting reads its
cmp.deltaoutput.Version bumped to
0.6.24with the matching CHANGELOG entry.Verification
Run in a worktree at
9f8babcdofforigin/main(912d6b3e).bash scripts/affected-tests.sh --runexits 3 (the runner's success code).plugins/context-budget/skills/audit/scripts/measure.test.shPASS at 73 passed / 0 failed (61before this change, measured on a clean
git archive origin/mainexport).plugins/context-budget/skills/audit/scripts/levers.test.shrun directly:3 passed / 0 failed.
--explainalso selectsplugins/session-flow/skills/running-retro/scripts/test_observer.py(a filename match onreport.md), which the runner reports as NOT RUN for its ecosystem; run in its own lane withpython3 -m pytest, 73 passed, 6 subtests passed. No changed file mapped to zero suites: thefour that report
no-suiteare recorded inscripts/affected-tests-no-suite.txt.--check,--check-order,--check-bump origin/main,--check-preserved origin/main(30 headings compared).ai-slopdetector over the five changed markdown surfaces: 0 findings, 5 files scanned. No emdash appears on any added line in any instruction surface in this diff (checked over the full
git diff origin/mainadded lines for the prose files).New tests, and the mutations run against them to show they are not vacuous:
additivity.additiveisnull, that the prefix bucket still getsadditive: truewithsumOfParts/combinedSavedof 1600, and that the deferred bucket's saving and verdict are bothnull.nonaddfake-binary mode: every bucket present in every run, the reading comparable, and thecombined prefix deny saving 1200 against a 1600 sum of parts. Asserts a measured
additive: false, a prefix bucket verdict offalse, a deferred bucket verdict oftrue, andexplicitly that this run's verdict is a different value from the vanish run's verdict.
novocabscenario: a bucket absent from both runs produces noperBucketrow.Mutations applied to
measure.mjs, each reverted after:additive: comparable ? … : nullreverted to the oldadditive: comparable && …. Two testsfail: "unmeasured additivity published as a boolean verdict" and "unmeasured and
measured-negative additivity verdicts are indistinguishable" (71 passed / 2 failed).
runComparable && sumOfParts !== null && combinedSaved !== nullweakened to dropthe
combinedSaved !== nullterm. One test fails: "vanished per-bucket verdict published as aboolean" (72 passed / 1 failed).
partssourced from-r.savedTokensinstead of the bucket's own row field. Threetests fail, including "per-bucket prefix sumOfParts wrong (got: 2100)" (70 passed / 3 failed).
Not verified: the sdk measurement path, which spawns a real Claude Code binary and is outside this
suite's hermetic scope, is unchanged by this PR but was not exercised. The measured 956-token
double-count figure the issue cites comes from the parent audit and is not reproduced here; this PR
does not re-measure anything.
Acceptance criteria status: all seven met. The one deviation from the issue's literal text is the
extra README correction described under Fix.
Related
🤖 Generated with Claude Code
https://claude.ai/code/session_01ViPsHkL3ng9xWt2GjEQJob
Generated by Claude Code