Skip to content

fix(context-budget): correct the additivity claim and report the verdict per bucket - #3883

Merged
kyle-sexton merged 4 commits into
mainfrom
claude/3863-additivity-claim
Sep 7, 2026
Merged

kyle-sexton merged 4 commits into
mainfrom
claude/3863-additivity-claim

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Sep 6, 2026 •

Copy link
Copy Markdown
Member

Closes #3863

Summary

context-budget asserted flatly that the per-tool deny deltas are compositional, so a basket of
trims prices from its members. Measured readings do not support that as one statement over both
attributed buckets: the deferred side sums to the token, while the prefix side double-counts, so
the sum of prefix deltas overstates what a basket saves on the prefix. The claim lived in three
places that each stated it independently, and the engine reported a single additivity verdict
computed as a boolean, so an unmeasurable reading surfaced as a definite "not additive".

Fix

Prose, all in this change so no home keeps the old claim:

  • plugins/context-budget/skills/audit/SKILL.md and
    plugins/context-budget/skills/audit/reference/levers.json (the deny-bare-tool lever's
    categoryBasis) now say which side composes: deferred-side deltas add and a basket's deferred
    saving is the sum of its members; prefix-side deltas double-count and their sum is only an upper
    bound on the basket's prefix saving. Both point at attribute --verify-additivity for the
    per-bucket verdict.
  • plugins/context-budget/README.md carried the same unqualified sentence. The issue names two
    homes; this is a third instance of the identical claim, so it is corrected with them rather than
    left as the one surface still asserting the old wording. Called out here because it is beyond
    the two homes the issue names.
  • reference/engine.md documents the new record shape; reference/report.md tells the reporter to
    state the verdict per bucket and to report a null verdict as not measurable rather than as not
    additive.

Engine, skills/audit/scripts/measure.mjs:

  • additivity.additive is tri-state. true/false are measured verdicts; null means the
    reading could not be measured. Previously comparable && combinedSaved === sumOfParts, which
    published every incomparable reading as false.
  • New additivity.perBucket, one entry per attributed bucket carrying
    {sumOfParts, combinedSaved, additive, reasons}. It is derived from the
    prefixDelta/deferredDelta values already on the saver rows plus the combined-deny comparison
    this run already performs: no extra measurement pass, no new snapshot. Each bucket's verdict is
    gated on its own measurability, so one bucket vanishing out of the combined snapshot no longer
    costs the other bucket its verdict. A bucket absent from both runs is outside the binary's
    category vocabulary and gets no verdict row at all.
  • The saturation guard (systemBucketSaving, the null-not-zero vanished-bucket path) is untouched;
    the per-bucket reporting reads its cmp.delta output.

Version bumped to 0.6.24 with the matching CHANGELOG entry.

Verification

Run in a worktree at 9f8babcd off origin/main (912d6b3e).

  • bash scripts/affected-tests.sh --run exits 3 (the runner's success code).
    plugins/context-budget/skills/audit/scripts/measure.test.sh PASS at 73 passed / 0 failed (61
    before this change, measured on a clean git archive origin/main export).
    plugins/context-budget/skills/audit/scripts/levers.test.sh run directly:
    3 passed / 0 failed. --explain also selects
    plugins/session-flow/skills/running-retro/scripts/test_observer.py (a filename match on
    report.md), which the runner reports as NOT RUN for its ecosystem; run in its own lane with
    python3 -m pytest, 73 passed, 6 subtests passed. No changed file mapped to zero suites: the
    four that report no-suite are recorded in scripts/affected-tests-no-suite.txt.
  • All four changelog-parity modes green: --check, --check-order, --check-bump origin/main,
    --check-preserved origin/main (30 headings compared).
  • ai-slop detector over the five changed markdown surfaces: 0 findings, 5 files scanned. No em
    dash appears on any added line in any instruction surface in this diff (checked over the full
    git diff origin/main added lines for the prose files).

New tests, and the mutations run against them to show they are not vacuous:

  • Vanish scenario (deferred bucket emptied out of the combined snapshot): asserts
    additivity.additive is null, that the prefix bucket still gets additive: true with
    sumOfParts/combinedSaved of 1600, and that the deferred bucket's saving and verdict are both
    null.
  • New nonadd fake-binary mode: every bucket present in every run, the reading comparable, and the
    combined prefix deny saving 1200 against a 1600 sum of parts. Asserts a measured
    additive: false, a prefix bucket verdict of false, a deferred bucket verdict of true, and
    explicitly that this run's verdict is a different value from the vanish run's verdict.
  • novocab scenario: a bucket absent from both runs produces no perBucket row.

Mutations applied to measure.mjs, each reverted after:

  1. additive: comparable ? … : null reverted to the old additive: comparable && …. Two tests
    fail: "unmeasured additivity published as a boolean verdict" and "unmeasured and
    measured-negative additivity verdicts are indistinguishable" (71 passed / 2 failed).
  2. Per-bucket gate runComparable && sumOfParts !== null && combinedSaved !== null weakened to drop
    the combinedSaved !== null term. One test fails: "vanished per-bucket verdict published as a
    boolean" (72 passed / 1 failed).
  3. Per-bucket parts sourced from -r.savedTokens instead of the bucket's own row field. Three
    tests fail, including "per-bucket prefix sumOfParts wrong (got: 2100)" (70 passed / 3 failed).

Not verified: the sdk measurement path, which spawns a real Claude Code binary and is outside this
suite's hermetic scope, is unchanged by this PR but was not exercised. The measured 956-token
double-count figure the issue cites comes from the parent audit and is not reproduced here; this PR
does not re-measure anything.

Acceptance criteria status: all seven met. The one deviation from the issue's literal text is the
extra README correction described under Fix.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01ViPsHkL3ng9xWt2GjEQJob


Generated by Claude Code

@kyle-sexton
kyle-sexton marked this pull request as ready for review September 6, 2026 21:56
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-06T21:59:43.309005Z 5d97b7a Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@claude

claude Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 1m 6s —— View job


Security review — no issues found

  • Load /review:security-review skill and gather context
  • Check skip gate (open PR, security-relevant surface, no already-passing review)
  • Fetch PR diff (gh pr diff against origin/main)
  • Review changed files for security issues
  • Report findings or clear result

Skip gate met: no security-relevant surface. The diff is entirely documentation prose
(README.md, SKILL.md, reference/engine.md, reference/report.md, reference/levers.json,
CHANGELOG.md) correcting the additivity claim, plus a pure-arithmetic addition to
measure.mjs (bucketAdditivity) and its tests. That function only recombines numbers already
produced by existing rows (prefixDelta/deferredDelta, cmp.delta) — no new I/O, no
subprocess/exec changes, no parsing of untrusted input, no auth/permissions/credential handling.

No security issues found. Per this lane's scope, no further action.

@claude

claude Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 4m 18s —— View job


/review:code-review — findings

  • Gather context (PR body, changed files)
  • Invoke /review:code-review and follow its instructions
  • Post inline findings (none met the high-signal bar)
  • Post summary comment

Scope reviewed: diff 912d6b3..5d97b7a (the 3 commits on this branch), all 9 changed files in plugins/context-budget/.

Analysis performed:

  • Traced the new bucketAdditivity() (skills/audit/scripts/measure.mjs:632-660) and its call site (scripts/measure.mjs:745-749) against the three-state delta model documented at measure.mjs#L599-L611 (number measured / null vanished-unmeasured / undefined outside vocabulary).
  • Specifically chased whether collapsing undefined into null when prefixDelta/deferredDelta are set on perTool rows (measure.mjs#L711-L712) could make bucketAdditivity's sumOfParts wrongly report null (unmeasurable) for a per-tool row where a bucket is legitimately a non-event. Traced it through: since every per-tool comparison and the combined comparison share the same baseline, a bucket can only be undefined on an individual row if it's also absent from baseline, in which case the combined comparison's own delta for that bucket is null too (present in the "after" snapshot only) — so combinedSaved is already null and the verdict is unmeasurable regardless. No path produces a wrong (as opposed to conservatively-null) verdict. Not a real bug.
  • Checked that savers/saverRows (unchanged, pre-existing filter at measure.mjs:724) gate per-tool rows on a single row-level comparable flag that conflates both buckets, so a tool whose deferred bucket vanishes in its own single-deny run drops out of the prefix bucket's sumOfParts too, even though its prefix delta was independently measurable. This is inherited, untouched logic predating this PR (confirmed against 912d6b3), and the PR's own description of scope ("derived from the values already on the saver rows... no extra measurement pass") explicitly reuses this pre-existing set. Falls under "pre-existing issue on untouched lines" per this lane's high-signal bar, so not reported as a finding — flagging here only for the author's awareness.
  • Cross-checked every doc surface (README.md, SKILL.md, levers.json, engine.md, report.md, CHANGELOG.md) against the code change and against each other: the corrected wording ("deferred-side deltas add... prefix-side deltas double-count") is consistent everywhere, and a search for the old flat claim (compositional, deltas are additive) turns up no remaining instance.
  • Verified the new tests (measure.test.sh: nonadd mode, per-bucket assertions on vanish/novocab) exercise the three-state distinction and match the implementation's branches.
  • Did not independently re-run scripts/affected-tests.sh, measure.test.sh, or the mutation testing — this sandbox's Bash tool requires interactive approval for node/test invocations that wasn't available in this lane. The PR body's test-run and mutation-testing results are author-claimed and unverified by me.

Result: no findings. The additivity-model fix, the tri-state additive verdict, and the new perBucket record are internally consistent, the docs no longer carry the old unqualified claim anywhere, and I could not construct an input that makes the new per-bucket logic report an incorrect (rather than conservatively-null) verdict.
· branch

@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Last security-reviewed head: 5d97b7aafa195aa4e5596ae2b424a678b61df7e3. On the next push, the relevance gate compares only the commits since this SHA; delete this comment to force a full re-review.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5d97b7aafa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

github-actions Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Claude has reviewed this PR 1 time. The lane skips further automatic reviews after 5; deleting this comment resets the count.

claude and others added 4 commits September 7, 2026 08:52
…verdict

The additivity record carried one verdict over both attributed buckets and
computed it as a boolean, so an incomparable reading surfaced as a definite
"not additive" and a consumer could not see that the deferred side composes
while the prefix side does not.

`additive` is now tri-state: true/false are measured verdicts and null means
the reading could not be measured. A `perBucket` map carries the same three
fields per attributed bucket, read off the prefixDelta/deferredDelta values
the per-tool rows already carry, so no new measurement pass runs. A bucket
absent from both runs is outside the binary's category vocabulary and gets no
verdict row at all.

Refs #3863

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ViPsHkL3ng9xWt2GjEQJob
The 0.6.24 entry said all three corrected homes (skill body, lever
categoryBasis, README) point at `attribute --verify-additivity`. Only
the skill body and the lever carry that pointer; the README states the
corrected composition claim without it. The PR scopes the pointer to
those two homes deliberately, so this fixes the CHANGELOG wording
rather than adding a pointer to the README.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ViPsHkL3ng9xWt2GjEQJob
…listing drift

Skill-listing and Skills-token mismatches poison the prefix System tools
bucket, but the deferred bucket's numeric deltas remain valid under the
shared mode/binary checks. Per-bucket additivity was applying
systemToolsComparable to every column, so a combined-run listing change
published System tools (deferred).additive as null.

Gate the listing checks on the prefix bucket only, expose
modeBinaryComparable for the shared checks, and cover both listing
signature and Skills-token drift in the hermetic attribute suite.

Refs #3863

Co-authored-by: ksextonmelodic <ksextonmelodic@gmail.com>
@cursor
cursor Bot force-pushed the claude/3863-additivity-claim branch from 5d97b7a to 4b1b406 Compare September 7, 2026 08:56
@kyle-sexton
kyle-sexton merged commit df58515 into main Sep 7, 2026
3 checks passed
@kyle-sexton
kyle-sexton deleted the claude/3863-additivity-claim branch September 7, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

context-budget: correct the "deltas add" claim in both homes and finish the per-bucket additivity reporting

3 participants