Skip to content

fix(session-flow): reroute audit-sessions denial and subagent findings and check their drift #6078

Closed
3 of 3 issues completed

Description

@kyle-sexton

Part of #6074

Paths below are relative to plugins/session-flow/skills/audit-sessions/ unless they start with plugins/. Line numbers are on origin/main 9671ece.

Problem

E5. tools.denials routes to a skill that addresses a different problem (medium). reference/sweep-rules.json:87-96 routes tools.denials to config-change / /fewer-permission-prompts. scripts/collect.py:368-369 records denials per toolDenialKind (stored at collect.py:654), but scripts/sweep.py:45-47 sums the kinds, so the finding cannot say which kind dominates. /fewer-permission-prompts adds allow rules (https://code.claude.com/docs/en/commands: "add a prioritized allowlist ... to reduce permission prompts"); the permissions docs say "A blocking hook also takes precedence over allow rules." and "An allow rule can't carve an exception out of a deny rule".

R1. subagents.count routes to a skill about standing instructions (medium). sweep-rules.json:97-106 routes it to unhobble-experiment / /harness-config:unhobble, whose purpose is "Bare-baseline experiment: strip a repo's standing instructions, log stumbles against the bare model". Its description has no subagent, fan-out or delegation text, and no rationale for the route is recorded (only sweep-rules.json:102 and scripts/tests/test_sweep.py:21 name it; PR #5925 and issue #5818 have no "unhobble" mention). A plausible unstated rationale may exist, so the route is undocumented, not provably wrong.

I4. The two unchecked drift rows need checks the report does not name (low). sweep.py:126-127 prints "drift not checked: no canary feeds this metric" for tools.denials and subagents.count. A plain presence canary on toolDenialKind would misfire only in a version with no denial at all, and subagents.count counts subagent files (collect.py:666), so no key canary can feed it.

Q2. The skill catalog is passed as hundreds of shell arguments (low). SKILL.md:87-94 tells the model to pass every skill in the session listing to --catalog (sweep.py:307, nargs +), but skill_label (sweep.py:96-99) only checks the five suggested_skill values in the rules file.

Evidence

Verified this pass: the lines above at 9671ece.

Measured by the audit on one Windows 11 machine, 2026-10-03 (not re-measured here):

  • E5 store denial kinds: permission-rule 2,432; automode-blocked 190; user-rejected 144; automode-unavailable 56; automode-parsing-error 3; cancelled 2; interrupted 1. Of 2,582 permission-rule denials in transcripts still on disk, 2,420 carry the PreToolUse hook-block notice and 113 the deny-rule wording. The worst session had 123 hook blocks and 0 deny-rule denials. An allowlist reduces neither. The toolDenialKind values are undocumented; their meaning beyond the notice wording is inferred.
  • R1 worst sessions: 173, 110 and 109 subagents.
  • I4: 16 sessions had Agent or Task tool calls but zero subagent files.
  • Q2: the evidence run passed 273 names to --catalog.

Proposed approach

E5:

  1. Render the per-kind split in each tools.denials finding (markdown and JSON) and name the dominant kind.
  2. Replace the /fewer-permission-prompts route. A rule has one suggested_skill. Recommended (judgment for the maintainer): /session-flow:retro (why the model kept trying blocked or unwanted calls), which fits hook blocks and human declines; name /harness-config:draft-auto-mode-rules beside the automode-blocked kind in the split. Not /harness-config:audit-permission-state as the single route: it fits only the 113 deny-rule denials.
  3. Record the kind reading's basis with an As of date and a recheck trigger beside the rule, since the kinds are undocumented.

R1: either record in the subagents.count basis why instruction ablation is the intended response, or route to /harness-ops:observability ("cross-session trends, cost, hooks"), matching tokens.sub (sweep-rules.json:23). Not /harness-ops:audit-performance: its engine never reads transcripts.

I4:

  • Add a denial_kind:<value> census key in collect.py's census key builder, following effort_value (collect.py:229), so a new or renamed kind surfaces as a "new" drift change; have a canary feed tools.denials from it.
  • Add an expect-absent invariant (style of invariant:usage_split, canaries.json:13) counting sessions with Agent or Task calls and zero subagent files, feeding subagents.count.
  • Use the same invariant-canary mechanism as the metric-accuracy child's S2 canaries.

Q2: pass --catalog only the suggested skills present in the session's listing (the five in sweep-rules.json, updated for any route E5 or R1 changes), and update SKILL.md Step 2 and evals/evals.json:8 and :13 to match. Alternative (scope review): drop the catalog step and have the model check the five names itself.

Files: reference/sweep-rules.json, reference/canaries.json, scripts/sweep.py, scripts/collect.py, scripts/census.py if the invariant needs support there, SKILL.md, evals/evals.json, scripts/tests/test_sweep.py, scripts/tests/test_drift.py, CHANGELOG and plugin.json.

Acceptance criteria

  • The tools.denials rule's suggested_skill is not /fewer-permission-prompts.
  • Each tools.denials finding, in markdown and JSON, shows the count per denial kind and names the dominant kind.
  • The tools.denials rule's basis records how the kinds were read, with an As of date.
  • Either the subagents.count basis states why /harness-config:unhobble is the intended response, or its suggested_skill is no longer /harness-config:unhobble.
  • test_sweep.py's route expectations match the chosen routes (ROUTES at test_sweep.py:21 drops unhobble-experiment if no rule uses it).
  • A fixture store whose newest version introduces a new toolDenialKind value shows that value as a "new" change in collect.py drift output.
  • The invariant reports sessions with Agent or Task calls and zero subagent files.
  • SKILL.md no longer instructs passing every listed skill to --catalog.

Constraints and gotchas

Context

Source: local handoff item 20261003-070338-session-flow-audit-sessions-accuracy-and-coverage.md (findings E5, R1, I4, Q2). Related: #5985, #6033, #4261 (guard rejections costing retry turns).

Activity

  1. kyle-sexton commented on Oct 3, 2026

    @kyle-sexton
    MemberAuthor

    Corrects the acceptance criteria of this issue: R1's route criterion is a maintainer decision, not a pass/fail check.

    The source item claimed every criterion was a pass/fail check on the defect. R1's criterion "Either the subagents.count basis states why /harness-config:unhobble is the intended response, or its suggested_skill is no longer /harness-config:unhobble" depends on a rationale only the maintainer holds: the route is undocumented, not provably wrong (this issue's own Problem section). The rule is at plugins/session-flow/skills/audit-sessions/reference/sweep-rules.json:97-106 on origin/main 9671ece (route unhobble-experiment at :102, suggested_skill at :103). R1 lives in this issue; the other decision splits from the same audit are E1 in #6075 and I1 and I5 in #6079.

    Replace that criterion with these two:

    • Maintainer decision: state whether instruction ablation (/harness-config:unhobble) is the intended response to a high subagent count, and why; if not, choose the route (the issue names /harness-ops:observability as the alternative, matching tokens.sub). Record the choice on this issue.
    • The subagents.count rule's route, suggested_skill and basis match the recorded choice, and scripts/tests/test_sweep.py ROUTES (:21) matches the routes in use.

    E5, I4 and Q2 stay agent-ready. E5 already has a recommended default and a pass/fail criterion (the route is no longer /fewer-permission-prompts); the maintainer may still override the replacement skill in review. Q2's catalog list follows whatever routes E5 and R1 end up with.

    Source: local handoff item 20261003-101457-corrections-to-6073-and-6074-6080 (retired into these comments).

  2. kyle-sexton commented on Oct 11, 2026

    @kyle-sexton
    MemberAuthor

    Owner decision (2026-10-11): Route the subagent-count finding to /harness-ops:observability and denials to /session-flow:retro (the issue's default). Lands after #6075.

    Recorded by the coordinating agent session with the owner's approval. Removing needs-human; implementation is queued and will link its branch here.

  3. removed
    needs-owner-decisionAwaiting a human or maintainer judgment call.
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Oct 11, 2026
  4. added theissue type on Oct 15, 2026
  5. kyle-sexton commented on Oct 18, 2026

    @kyle-sexton
    MemberAuthor

    This was generated by AI during backlog shaping.

    This issue is split into 3 pieces, each sized for one lane PR. Four findings over about 11 files. Part of E5 is already done: tools.denials routes to /session-flow:audit-friction, set by #6692 (merged 2026-10-09), not /fewer-permission-prompts (sweep-rules.json:87-96 on origin/main). The rest splits into three children of 3-6 files each. All three wait on #6075, whose open PR #7940 edits sweep-rules.json, sweep.py, collect.py, canaries.json, SKILL.md, test_sweep.py and test_drift.py (git diff --stat origin/main...FETCH_HEAD), and the owner put #6075 first. Owner check: the 2026-10-11 decision said route denials to /session-flow:retro, but main already rou

    Filed pieces are sub-issues of this one, and the paced queue admits them to the lane. Unfiled pieces wait on the blocker shown. The owner approved this split in an attended backlog-shaping session on 2026-10-18.

  6. kyle-sexton commented on Oct 18, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded.

    Question: Your 2026-10-11 ruling routes the audit-sessions denial finding to /session-flow:retro. Main has routed it to /session-flow:audit-friction since #6692, merged 2026-10-09, two days before that ruling. The ruling adopted the issue's 2026-10-03 default and did not mention #6692. Options: (a) keep audit-friction, as main does; (b) apply the 2026-10-11 ruling and route to retro.

    Owner's answer: (a). audit-friction is built for this: "denials, approval prompts, hook blocks, asks and hand-offs, traced to their cause". In the measured data, 2,420 of 2,582 permission-rule denials were hook blocks, and an allowlist reduces none of them. Record a short Decision recorded. comment that supersedes the denial half of the 2026-10-11 ruling. The subagent half (route to /harness-ops:observability) stands. Basis: sweep-rules.json:87-96; audit-friction SKILL.md:2; #6078 body (measured kinds).

    The owner accepted this recommendation as item D17 of a 17-decision batch in an attended backlog-shaping session on 2026-10-18. The batch question: "Accept all 17 batch 5 decisions as recommended, D1, D2, D3, D5 and D17 included? I post a quoted 'Decision recorded.' comment on each of the 11 issues." The owner answered: "Accept all 17 (Recommended)".

    Posted by AI from the owner's answer in that session.

  7. github-actions commented on Oct 25, 2026

    @github-actions
    Contributor

    The triage sweep dispatched intake triage for this issue (attempt 1 of 2).

  8. github-actions commented on Oct 25, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: scoped
    Basis: automated intake triage; the lanes App applied this class.

    Size: Large
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Decision recorded.

    Question: Your 2026-10-11 ruling routes the audit-sessions denial finding to /session-flow:retro. Main has routed it to /session-flow:audit-friction since #6692, merged 2026-10-09, two days before that ruling. [...]

    Owner's answer: (a). audit-friction is built for this: "denials, approval prompts, hook blocks, asks and hand-offs, traced to their cause". [...] The subagent half (route to /harness-ops:observability) stands.

    [quote truncated; the full decision is at the link]

    #6078 (comment)

    That decision settles the open route questions: the denial route stays /session-flow:audit-friction, and subagents.count routes to /harness-ops:observability. The work that remains is the per-kind denial split, the denial_kind drift check, the Agent/Task-without-subagent-files invariant, and the --catalog change. That is too much for one change, so it needs to be split into sub-issues, as the latest comment asks. I found no duplicates among the open issues.

    Automated triage, applied by the lanes App.

  9. melodic-automation-lanes commented on Oct 25, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane yet: Size large: the split into sub-issues at intake is pending (owner Q93).

  10. kyle-sexton commented on Oct 29, 2026

    @kyle-sexton
    MemberAuthor

    Re-triage. Not an owner stop. Both routes are decided:

    The blocker, #6075, is closed as completed.

    State on origin/main, under plugins/session-flow/skills/audit-sessions/:

    • Met: reference/sweep-rules.json:89-96 routes tools.denials to /session-flow:audit-friction (feat(session-flow): add audit-friction skill for permission and autonomy friction #6692).
    • Unmet: subagents.count still has route unhobble-experiment and suggested_skill /harness-config:unhobble (sweep-rules.json:99-106), and scripts/tests/test_sweep.py:21 still lists unhobble-experiment.
    • Unmet: no per-kind denial split in the findings, no denial_kind: census key, no invariant for sessions with Agent or Task calls but zero subagent files.
    • Unmet: SKILL.md:128 still passes every listed skill to --catalog.

    Admission refused this on 2026-10-25 as Size large with the split pending, and no sub-issues exist yet. File the three slices from the 2026-10-18 split comment as sub-issues now, each at status: ready, since their blocker #6075 is closed. The --catalog slice gets a blocked-by edge on the denial-and-route slice, as that plan had it. Done when, across them:

    • Each tools.denials finding, in markdown and JSON, shows the count per denial kind and names the dominant kind. The rule's basis records how the kinds were read, with an As of date and a recheck trigger.
    • The subagents.count rule's route, suggested_skill (/harness-ops:observability) and basis match the 2026-10-11 decision, and test_sweep.py ROUTES matches the routes in use.
    • A fixture store whose newest version introduces a new toolDenialKind value shows it as a "new" change in collect.py drift output.
    • An expect-absent invariant reports sessions with Agent or Task calls and zero subagent files, and feeds subagents.count.
    • SKILL.md passes --catalog only the skills the rules suggest, and evals/evals.json matches.
    • Each slice adds a changelog fragment.
  11. kyle-sexton commented on Oct 30, 2026

    @kyle-sexton
    MemberAuthor

    Done: every sub-issue (#10812, #10823, #10839) closed as completed via merged #10960, #10921 and #10963.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    status: readyTriaged, unblocked, and fully specified; eligible to pick up.

    Type

    Fields

    No fields configured for Bug.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions