You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
guardrails: grade every default-on guard's precision from the per-session hook event log (ADR 0003) #3756
ADR 0003 says a guard earns default-on by measured precision, and until #3750 the only instrument was a manual corpus sweep. The per-session hook event log now records every guard fire (source: "envelope", status: "blocked", subject) beside every event that did not fire, so precision can be graded from real sessions: blocked-versus-fired per guard, with the subject of each block available for a true/false-positive verdict.
The operator's original ask (2026-09-04) was to "unhobble" guards and instructions that were in place for older models and no longer apply. The pipeline is the instrument; the verdicts are not yet taken. The one guard with evidence already, block-hook-bypass (zero true positives in four fires across three sessions), is owned by the PreToolUse lane and tracked in #3721; this issue is the fleet-wide pass over the rest.
Fix
Turn session_event_log_enabled on in the operator's own sessions for two to four weeks (the switch stays off by default for consumers).
From the session files, per guard: fires, blocks, and the operator's verdict on each block's subject (true positive, false positive, scope error).
Apply ADR 0003's clauses: a guard at zero true positives is withdrawn from default-on scope and re-filed for rescoping (clause 4), one whose oracle is wrong is deleted, one with sound precision stays. Each disposition is its own PR with the numbers in the body.
Acceptance
A findings table (guard, sessions observed, fires, blocks, true positives, disposition) attached to this issue.
One PR or closing note per guard with a non-"keep" disposition.
Claimed for triage by autonomous lane instance vm-triage-202609050304. Lease expires 2026-09-06T19:40:00Z. This is a triage claim only, not an execution claim; the item stays unassigned and available to the worker lane once briefed.
The premise holds and its prerequisites are in place:
ADR 0003 is present in the ADR set.
The instrument from feat(claude-ops): per-session hook logging pipeline, default off #3750 is merged: the per-session hook event log ships as a claude-ops hook with its own test suite, the session_event_log_enabled switch is threaded through the setup and observability skills, and the observability skill reads the resulting session files.
The claude-ops README documents the switch as off by default and describes the per-event line shape the grading would read.
So the instrument exists and the verdicts genuinely are not yet taken, exactly as the issue states.
Why human-gated rather than agent-ready
Two capability blockers, neither of which a brief can remove:
The evidence does not exist yet and cannot be manufactured. Grading requires two to four weeks of the operator's own real sessions with the switch on. Synthetic or replayed sessions would defeat the purpose: ADR 0003's clauses turn on measured precision against real work, and a guard's false-positive rate is a property of what the operator actually does, not of a fixture corpus.
The verdict on each block is a human judgment. Deciding whether a given block's subject was a true positive, a false positive, or a scope error is precisely the judgment ADR 0003 reserves for the operator. An agent labeling its own guard's fires would be marking its own homework.
The dispositions the grading feeds (withdrawing a guard from default-on scope, deleting one whose oracle is wrong) are also policy changes to default-on behavior for every consumer, which is maintainer territory.
What a human needs to do to start it
Turn session_event_log_enabled on in the operator's own sessions and note the start date. The switch stays off for consumers; this is a personal-scope change only.
Let it run the two-to-four-week window.
Then the reduction is delegable: see below.
The delegable slice, if the operator wants it split out
Once a collection window has produced session files, the mechanical part can be filed as its own agent-ready item: a read-only reduction over the session event log that emits the findings table (guard, sessions observed, fires, blocks) with each block's subject quoted for the operator to adjudicate, leaving the true-positive column and the disposition column blank for the human. That item would be work-class: read-only and needs no repository mutation. It is deliberately not filed now, because filing it before the window opens would create an item blocked on an event that has no date.
Scope note
block-hook-bypass is explicitly out of scope here and stays tracked in #3721, which is open. This item is the fleet-wide pass over the remaining default-on guards.
Incidentally, and recorded only as an observation rather than as evidence for the grading: this triage pass was itself blocked once by block-hook-bypass while running a read-only reproduction script in a scratch directory. That is one more data point of the kind #3721 is collecting, not a verdict.
This was generated by AI during audit remediation. Source: the 2026-09-29 audit of the Cursor agent run (PRs opened 2026-09-27 to 2026-09-29).
Closed in error. PR #4881 closed this with Closes #3756, although the PR said "Grading waits on an operator collection window". There is no findings table and no grading. I reopened it (needs-human and work-class: read-only kept, status: needs-decision added).
Current state. The ADR 0003 section the PR added ("Session-log grading (#3756)") is removed by fix PR #5288 (Refs only). Nothing on main claims a decision or a pending task; this issue is the only tracker again.
Question: do you want the fleet-wide default-on guard grading pass (ADR 0003's measure-before-default-on, using the per-session hook event log from #3750), and when does the collection window start?
Options
A) Start now. Turn session_event_log_enabled on in your own sessions (personal scope only; off by default for consumers; documented in plugins/claude-ops/README.md line ~427 and plugins/claude-ops/skills/setup/SKILL.md lines ~86-154), comment the start date here, and let it run two to four weeks. Then ask for the delegable read-only reduction (findings table: guard, sessions, fires, blocks, quoted subject, with true-positive and disposition columns left blank for you) as a new agent-ready item; the 2026-09-06 triage deliberately did not file it before the window opens. Each non-keep disposition is then its own PR with the numbers in the body.
B) Decline and close as not planned, with a reason.
C) Defer to a named date and keep open.
RECOMMENDED (AI recommendation, not a decision): A.
Basis: docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md ("measure before shipping on the real corpus"); the instrument from #3750 is merged; your 2026-09-04 ask (issue body) was to unhobble guards for older models.
Unblocks: A starts the only path to the findings table. B or C settles the item. block-hook-bypass stays in #3721.
Group fix PR: #5288 (Refs #3756, no closing keyword).
Owner decision (2026-09-29): Option A, start now. The fleet-wide default-on guard grading pass goes ahead, and the collection window opens now. Next: the owner turns on session_event_log_enabled in their own sessions (personal scope only) and comments the start date here. After the two-to-four-week window, the read-only reduction (guard, sessions, fires, blocks, quoted subject, with true-positive and disposition columns blank) is filed as its own agent-ready item. needs-human stays until then.
Decision recorded. The owner answered in the Backlog HITL session on 2026-10-29.
Q9. Does your 10-27 "model usage is not cost" rule lift the Q5 hold on the six "paid-run" issues: #3756, #6667, #7147, #6286, #7226 and #5653? Recommendation: yes, lift it if "paid-run" meant model or API-credit spend; your section 3 now puts that outside cost.
go w/ recommendations
"Paid-run" meant model usage. The two other open items held by Q5, #7147 and #7226, are evals, and the owner resumed the subscription-eval host-run class the same day. So the Q5 hold (until the #8282 review on 2026-11-18) no longer applies here. The open questions were already decided in the 2026-10-27 record above, and this record keeps those decisions.
After 2026-10-29, append the 2026-10-28 and 2026-10-29 envelope lines from this repository's .observability/claude/sessions/ to .work/3756-session-snapshot/ on the operator's machine, reduced the same way as the 2026-10-13 to 2026-10-27 copy.
Run the read-only reduction over that copy, and post the findings table on this issue: guard, sessions observed, fires, blocks, and the quoted subject per block. The verdict and disposition columns stay blank for the owner's review, as the 2026-10-01 decision describes.
File the reduction as its own item, per the 2026-10-01 decision. Link it here.
Delete .work/3756-session-snapshot/ once the table is posted, and no later than 2026-12-18.
Host run. The 2026-10-28/29 envelope append and the read-only guard-precision reduction over this repository's session event log snapshot (2026-10-12T21:53Z to 2026-10-29T23:59Z) ran on the operator's machine.
What ran
Claude Code 2.1.314. Read-only; no repository change, no PR.
Append (criterion 1). One pass over this repository's .observability/claude/sessions/*.jsonl (452 files on 2026-10-30). Kept the source: envelope lines, reduced to the same eight fields as the first copy (ts, session_id, hook_event_name, hook, status, exit_code, subject, tool), and wrote them to a new file, .work/3756-session-snapshot/envelope-2026-10-27_29.jsonl.gz, next to the old one. The old file was not edited. Filter: ts after 2026-10-27T15:26:03Z, the last line in the 2026-10-27 copy, and before 2026-10-30T00:00Z. This also picks up the rest of 2026-10-27, which the first copy did not hold. Result: 159,371 lines, 304 blocked, 60 sessions.
Reduction (criterion 2). Read both gz files together: 1,117,580 envelope lines from 447 sessions. Every number is per hook. Sessions observed counts the distinct sessions in which the guard ran at least once. Fires counts every recorded invocation, whatever its status. Blocks counts invocations with status: blocked. Each subject value is quoted with its count. In this log a subject is <tool>:<first command word>, so it names the command and never carries its arguments. Some guards record no subject by design, and those show as empty. No subject needed redacting.
Commands, in short form: python3 -I extract.py <sessions dir> <new gz> 2026-10-27T15:26:03Z 2026-10-30T00:00:00Z, which streams each file once and keeps only the envelope lines inside those bounds, then python3 -I reduce.py envelope-2026-10-13_27.jsonl.gz envelope-2026-10-27_29.jsonl.gz, which tallies per hook. Both are one-off scripts kept in the session scratchpad.
Scope. This follows question 3 of the 2026-10-27 record: guards that were default-on through the whole window (2026-10-01 to 2026-10-29), minus block-hook-bypass. The default-on state comes from plugins/guardrails/README.md:1120-1124 (kill-switch defaults), plugins/testing/README.md:184 and plugins/source-control/README.md:411. The date a guard was added is the first commit that adds its hook file (git log --diff-filter=A).
Findings table
The Verdict and Disposition columns are left blank for the owner.
none. Partial window: its first envelope line is dated 2026-10-28
hardcoded-path-check
3
30
0
none. Fires undercounts invocations: the hook exits without an envelope line on its early paths (for example a file outside the project, plugins/guardrails/hooks/hardcoded-path-check.sh:302-354) and emits only after its scan (:412)
Not gradable from this log
Guard
Sessions
Fires
Blocks
Why this log cannot grade it
cli-flag-verify
21
1,024
0
Its session lane runs at PostToolUse and "never blocks" (plugins/guardrails/hooks/cli-flag-verify.sh:73). Its blocking lane is the git pre-push hook (exit 1), which writes nothing to the session envelope. Zero blocks here says nothing about its precision.
skill-reference-verify
21
1,015
0
Same as cli-flag-verify: the session lane emits ok (skill-reference-verify.sh:890), and the push gate is not logged.
stale-path-verify
21
1,015
0
Same as cli-flag-verify (stale-path-verify.sh:862).
block-windows-drive-tmp
0
0
0
Does not apply on this Linux host: it exits unless OSTYPE is msys, cygwin or win32 (block-windows-drive-tmp.sh:107-110).
block-exported-msys-pathconv
0
0
0
Does not apply on this Linux host: same OSTYPE gate (block-exported-msys-pathconv.sh:117-120).
Referenced or excluded
Hook
Why it is not in the table
Sessions
Fires
Blocks
block-hook-bypass
Owned by #3721 (reference only). Top subjects: Bash:cd 179, Bash:python3 132, Bash:cat 74, Bash:mkdir 62
36
38,860
524
block-unresolved-pr-merge
Added 2026-10-13, during the window. Also 5 error. Records no subject
187
1,411
250
block-fix-request-post
Added 2026-10-19, during the window
196
18,066
0
block-cross-domain
Added 2026-10-24, during the window. No envelope lines
0
0
0
block-credential-read
Off by default (plugins/guardrails/README.md:1122-1124)
0
0
0
test-weaken
Off by default (test_guards_enabled, plugins/testing/README.md:184). It ran because the operator enabled it. Also 1 error
182
3,419
0
test-scan
Not registered in plugins/testing/hooks/hooks.json on main
16
711
0
permission-denied-audit
Not a guard: it logs Claude Code permission denials as blocked
202
699
699
The formatter and audit hooks are not guards and are left out: actionlint, biome, bash, markdown, ruff, typos and eol formatting, plus the tool-failure, api-error, hook-failure, config-change, instructions-loaded, skill-usage, compact and notification hooks.
Against the acceptance criteria (2026-10-29 record)
Append the 2026-10-28 and 2026-10-29 envelope lines, reduced the same way: done (item 1 above). The rest of 2026-10-27 is included as well.
Run the reduction and post the table, with Verdict and Disposition left blank: the table is above. This record is posted as the comment that carries it.
Delete .work/3756-session-snapshot/ once the table is posted: left for the coordinator, after this comment is posted.
Before that deletion: each subject is only <tool>:<first word>. For example, Bash:cd accounts for 89 of block-root-delete-target's blocks. To judge a single block, the owner needs its session_id and ts, and only the snapshot holds those. Deleting the snapshot removes them, so whether to delete before the review is the owner's call.
Activity
kyle-sexton commentedon Sep 6, 2026
This was generated by AI during triage.
Claimed for triage by autonomous lane instance
vm-triage-202609050304. Lease expires 2026-09-06T19:40:00Z. This is a triage claim only, not an execution claim; the item stays unassigned and available to the worker lane once briefed.Generated by Claude Code
kyle-sexton commentedon Sep 6, 2026
This was generated by AI during triage.
Triage verdict
Category: Task (premise confirmed). State: briefed → human-gated. Work class:
read-only. Priority:medium.Verification (2026-09-06, against current
main)The premise holds and its prerequisites are in place:
claude-opshook with its own test suite, thesession_event_log_enabledswitch is threaded through the setup and observability skills, and the observability skill reads the resulting session files.So the instrument exists and the verdicts genuinely are not yet taken, exactly as the issue states.
Why human-gated rather than agent-ready
Two capability blockers, neither of which a brief can remove:
subjectwas a true positive, a false positive, or a scope error is precisely the judgment ADR 0003 reserves for the operator. An agent labeling its own guard's fires would be marking its own homework.The dispositions the grading feeds (withdrawing a guard from default-on scope, deleting one whose oracle is wrong) are also policy changes to default-on behavior for every consumer, which is maintainer territory.
What a human needs to do to start it
session_event_log_enabledon in the operator's own sessions and note the start date. The switch stays off for consumers; this is a personal-scope change only.The delegable slice, if the operator wants it split out
Once a collection window has produced session files, the mechanical part can be filed as its own
agent-readyitem: a read-only reduction over the session event log that emits the findings table (guard, sessions observed, fires, blocks) with each block'ssubjectquoted for the operator to adjudicate, leaving the true-positive column and the disposition column blank for the human. That item would bework-class: read-onlyand needs no repository mutation. It is deliberately not filed now, because filing it before the window opens would create an item blocked on an event that has no date.Scope note
block-hook-bypassis explicitly out of scope here and stays tracked in #3721, which is open. This item is the fleet-wide pass over the remaining default-on guards.Incidentally, and recorded only as an observation rather than as evidence for the grading: this triage pass was itself blocked once by
block-hook-bypasswhile running a read-only reproduction script in a scratch directory. That is one more data point of the kind #3721 is collecting, not a verdict.Triage claim released by
vm-triage-202609050304.Generated by Claude Code
docs(adr-0003): park fleet-wide session-log guard grading (#3756) (#4881
kyle-sexton commentedon Sep 29, 2026
This was generated by AI during audit remediation. Source: the 2026-09-29 audit of the Cursor agent run (PRs opened 2026-09-27 to 2026-09-29).
Closed in error. PR #4881 closed this with
Closes #3756, although the PR said "Grading waits on an operator collection window". There is no findings table and no grading. I reopened it (needs-humanandwork-class: read-onlykept,status: needs-decisionadded).Current state. The ADR 0003 section the PR added ("Session-log grading (#3756)") is removed by fix PR #5288 (Refs only). Nothing on main claims a decision or a pending task; this issue is the only tracker again.
Question: do you want the fleet-wide default-on guard grading pass (ADR 0003's measure-before-default-on, using the per-session hook event log from #3750), and when does the collection window start?
Options
session_event_log_enabledon in your own sessions (personal scope only; off by default for consumers; documented inplugins/claude-ops/README.mdline ~427 andplugins/claude-ops/skills/setup/SKILL.mdlines ~86-154), comment the start date here, and let it run two to four weeks. Then ask for the delegable read-only reduction (findings table: guard, sessions, fires, blocks, quoted subject, with true-positive and disposition columns left blank for you) as a new agent-ready item; the 2026-09-06 triage deliberately did not file it before the window opens. Each non-keep disposition is then its own PR with the numbers in the body.RECOMMENDED (AI recommendation, not a decision): A.
Basis:
docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md("measure before shipping on the real corpus"); the instrument from #3750 is merged; your 2026-09-04 ask (issue body) was to unhobble guards for older models.Unblocks: A starts the only path to the findings table. B or C settles the item.
block-hook-bypassstays in #3721.Group fix PR: #5288 (Refs #3756, no closing keyword).
kyle-sexton commentedon Sep 29, 2026
Owner decision (2026-09-29): Option A, start now. The fleet-wide default-on guard grading pass goes ahead, and the collection window opens now. Next: the owner turns on session_event_log_enabled in their own sessions (personal scope only) and comments the start date here. After the two-to-four-week window, the read-only reduction (guard, sessions, fires, blocks, quoted subject, with true-positive and disposition columns blank) is filed as its own agent-ready item. needs-human stays until then.
23 remaining items
kyle-sexton commentedon Oct 29, 2026
Decision recorded. The owner answered in the Backlog HITL session on 2026-10-29.
"Paid-run" meant model usage. The two other open items held by Q5, #7147 and #7226, are evals, and the owner resumed the subscription-eval host-run class the same day. So the Q5 hold (until the #8282 review on 2026-11-18) no longer applies here. The open questions were already decided in the 2026-10-27 record above, and this record keeps those decisions.
.observability/claude/sessions/to.work/3756-session-snapshot/on the operator's machine, reduced the same way as the 2026-10-13 to 2026-10-27 copy.subjectper block. The verdict and disposition columns stay blank for the owner's review, as the 2026-10-01 decision describes..work/3756-session-snapshot/once the table is posted, and no later than 2026-12-18.kyle-sexton commentedon Oct 30, 2026
Host run. The 2026-10-28/29 envelope append and the read-only guard-precision reduction over this repository's session event log snapshot (2026-10-12T21:53Z to 2026-10-29T23:59Z) ran on the operator's machine.
What ran
Claude Code 2.1.314. Read-only; no repository change, no PR.
.observability/claude/sessions/*.jsonl(452 files on 2026-10-30). Kept thesource: envelopelines, reduced to the same eight fields as the first copy (ts, session_id, hook_event_name, hook, status, exit_code, subject, tool), and wrote them to a new file,.work/3756-session-snapshot/envelope-2026-10-27_29.jsonl.gz, next to the old one. The old file was not edited. Filter:tsafter 2026-10-27T15:26:03Z, the last line in the 2026-10-27 copy, and before 2026-10-30T00:00Z. This also picks up the rest of 2026-10-27, which the first copy did not hold. Result: 159,371 lines, 304blocked, 60 sessions.hook. Sessions observed counts the distinct sessions in which the guard ran at least once. Fires counts every recorded invocation, whatever its status. Blocks counts invocations withstatus: blocked. Eachsubjectvalue is quoted with its count. In this log a subject is<tool>:<first command word>, so it names the command and never carries its arguments. Some guards record no subject by design, and those show as empty. No subject needed redacting.Commands, in short form:
python3 -I extract.py <sessions dir> <new gz> 2026-10-27T15:26:03Z 2026-10-30T00:00:00Z, which streams each file once and keeps only the envelope lines inside those bounds, thenpython3 -I reduce.py envelope-2026-10-13_27.jsonl.gz envelope-2026-10-27_29.jsonl.gz, which tallies per hook. Both are one-off scripts kept in the session scratchpad.block-hook-bypass. The default-on state comes fromplugins/guardrails/README.md:1120-1124(kill-switch defaults),plugins/testing/README.md:184andplugins/source-control/README.md:411. The date a guard was added is the first commit that adds its hook file (git log --diff-filter=A).Findings table
The Verdict and Disposition columns are left blank for the owner.
subjectper block (count)Bash:rm105,Bash:cd89,Bash:mkdir13,Bash:for8,Bash:git7,Bash:chmod7,Bash:python35,Bash:while5,Bash:ls5,Bash:pgrep3,Bash:run()3,Bash:-d3,Bash:bash2,Bash:gh2,Bash:set2,Bash:du2, one each:Bash:head,Bash:tail,Bash:&&,Bash:mv,Bash:jq,Bash:node,Bash:sed,Bash:nice,Bash:cat,Bash:which,Bash:rmdir,Bash:stat,Bash:type,Bash:-d);,Bash:sleepBash:git39,Bash:cd28,Bash:-C2,Bash:until2, one each:Bash:api,Bash:cp,Bash:cat,Bash:grep,Bash:forBash:cd15,Bash:git10,Bash:cat5,Bash:mkdir4,Bash:python33, one each:Bash:cp,Bash:kill,Bash:timeout,Bash:set,Bash:sed,Bash:printfBash:git17,Bash:bash4,Bash:cd3,Bash:npm1,Bash:mk(){1Bash:cd6,Bash:python31,Bash:-11plugins/guardrails/hooks/hardcoded-path-check.sh:302-354) and emits only after its scan (:412)Not gradable from this log
plugins/guardrails/hooks/cli-flag-verify.sh:73). Its blocking lane is the git pre-push hook (exit 1), which writes nothing to the session envelope. Zero blocks here says nothing about its precision.ok(skill-reference-verify.sh:890), and the push gate is not logged.stale-path-verify.sh:862).OSTYPEis msys, cygwin or win32 (block-windows-drive-tmp.sh:107-110).OSTYPEgate (block-exported-msys-pathconv.sh:117-120).Referenced or excluded
Bash:cd179,Bash:python3132,Bash:cat74,Bash:mkdir62error. Records no subjectplugins/guardrails/README.md:1122-1124)test_guards_enabled,plugins/testing/README.md:184). It ran because the operator enabled it. Also 1errorplugins/testing/hooks/hooks.jsonon mainblockedThe formatter and audit hooks are not guards and are left out: actionlint, biome, bash, markdown, ruff, typos and eol formatting, plus the tool-failure, api-error, hook-failure, config-change, instructions-loaded, skill-usage, compact and notification hooks.
Against the acceptance criteria (2026-10-29 record)
.work/3756-session-snapshot/once the table is posted: left for the coordinator, after this comment is posted.Before that deletion: each subject is only
<tool>:<first word>. For example,Bash:cdaccounts for 89 of block-root-delete-target's blocks. To judge a single block, the owner needs itssession_idandts, and only the snapshot holds those. Deleting the snapshot removes them, so whether to delete before the review is the owner's call.Target labels: add none; remove host-run.
melodic-automation-lanes commentedon Oct 30, 2026
Every sub-issue of this split is closed by a listed human or a merge (#11299), so the lanes App closes it as completed.