Skip to content

harness-ops/audit-native-overlap: rule on store rows for the model-disabled native surfaces with no row on Claude Code 2.1.308 #9105

Closed

Description

@kyle-sexton

This was generated by AI during attend-queue.

Refs: #4047

Problem

The intent of #4047 is that every native surface the model cannot invoke, and that overlaps a component, gets surfaced to the person. That only happens for surfaces that have a store row: a suggest sentence is baked only from a row, and a row is written only when a person rules on it (records.json note).

A read on 2026-10-22 against Claude Code 2.1.308 found model-disabled surfaces with no row:

  • inventory.py --binary-only lists run-skill-generator as a bundled skill with model_invocable: false. No store row names it.
  • overlap.py detect on that inventory exits 3 (degraded) and proposes these pairs with no row. Each is a model-disabled built-in command, except claude-code-docs, which is a model-invocable bundled skill. Similarity is 0.6 or higher:
Native surface Class, markers Candidate component Score
rename builtin-command, model-invocation-disabled docs-hygiene:rename-references 0.79
brief builtin-command, gated, model-invocation-disabled review:brief-reviewer (agent) 0.74
resume builtin-command, hidden, gated, model-invocation-disabled session-flow:continue-in-background 0.68
claude-code-docs bundled-skill, gated discovery:read-docs 0.68
stop builtin-command, gated, model-invocation-disabled session-flow:clean-stop 0.68
agents builtin-command, hidden, model-invocation-disabled docs-hygiene:write-for-agents 0.65

The scores measure name and description similarity, not shared jobs. Several pairs are probably name collisions to dismiss, for example brief against an agent.

The list is a floor, not a total. On 2.1.308 the inventory reports these limits:

  • 21 bundled-skill registrations use a computed name that the extractor did not resolve.
  • The builtin_tools, builtin_plugins and bundled_workflows lanes are broken.
  • validated_against is still 2.1.288 (inventory.py:67).

Because of these limits, other model-disabled bundled skills may exist on this build and not appear here. That extractor drift belongs to the changelog lane and #6888, not to this issue.

Desired outcome

The owner rules on each candidate. A real overlap becomes a store row with a verdict and an integration value. A model-disabled surface can take only suggest, or route when the verdict is defer (native-references README:251-261). A name collision gets a dismissal. Once that is done, detect no longer proposes these pairs.

Acceptance criteria

  • Each of the seven surfaces above (run-skill-generator and the six detect candidates) has, in docs/native-surfaces/records.json, either a store row with a person-written verdict and integration, or a dismissal with a reason.
  • Each new suggest row's skill carries its Boundary section and suggest sentence in the same change, as the convention requires.
  • docs/native-surfaces.md is regenerated, and the PR's CI runs scripts/validate-plugins.sh (native-overlap self-check and generate --check) and it passes.
  • Every touched plugin carries a changelog fragment made with scripts/new-changelog-fragment.sh, and scripts/check-changelog-fragments.sh --check passes.

Out of scope

  • Fixing the inventory extractor for 2.1.308 (the changelog lane, #6888).
  • Candidates scoring below 0.6. Rerun detect after the extractor fix to see the full set.

Related

Activity

  1. added theissue type on Oct 22, 2026
  2. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.
    on Oct 22, 2026
  3. kyle-sexton commented on Oct 24, 2026

    @kyle-sexton
    MemberAuthor

    Carried from epic #4047 (closing): six follow-ups the 2026-10-22 criterion check listed with no issue. Each is in scope here.

    • recheck subcommand
    • marker-basis field
    • name-collision keying in detect
    • durable report from a bare run
    • reverify helper
    • post-step behaviour assertion for wraps
  4. removed
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Oct 25, 2026
  5. kyle-sexton commented on Oct 25, 2026

    @kyle-sexton
    MemberAuthor

    Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.

    To hold this issue again, add needs-human (or status: needs-decision) and say why in a comment.

  6. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Oct 25, 2026
  7. kyle-sexton commented on Oct 25, 2026

    @kyle-sexton
    MemberAuthor

    Hold restored: the 2026-10-25 release took needs-human off, but this issue does need a person under the owner's stop rule. Reason (intent): "The owner rules on each candidate. A real overlap becomes a store row with a verdict" (person-written verdict required)

  8. removed
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Oct 26, 2026
  9. kyle-sexton commented on Oct 26, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded. The owner answered in the Refinement and Issue Triage sessions (2026-10-25), deciding the open items below on the facts:

    go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

    I should only ever be in the loop ONLY when necessary (intent/vision changed, something came up during implementation that impacts trajectory/alignment, OR a high security risk) - I've covered this all before

    Owner ledger: Q34 (.work/issue-triage-owner-sweep-1/interview-checklist.md:59) names this issue. It is "deferred" with "issue not touched", so it decides nothing that this record contradicts. Read against origin/main at 68d3f2b.

    1. Who may rule? The hold comment (harness-ops/audit-native-overlap: rule on store rows for the model-disabled native surfaces with no row on Claude Code 2.1.308 #9105 (comment)) and the store note say a person.
      Decided: a session acting on the owner's standing direction rules, and the store note is amended to say so.
      • The note says "every verdict here was written by a person, never by a run" (docs/native-surfaces/records.json:3). The owner's later standing rule (.work/gh-monitor/refinement/ledger-2026-10-26.md:16-21, 2026-10-26) keeps the owner out of the loop except for an intent change, a finding that alters the primary design, or a HIGH/CRITICAL security issue. A verdict on a name pair is none of these.
      • The store already works this way. Its dismissals read "Ruled 2026-09-29 by operator direction on the orchestrator's recommendation" (for example records.json:4745, :4853, :4907).
      • No security boundary rests on the note. Rows carry no field that records who wrote them (records.json row and dismissal keys: native, component, reason, as_of, date, fingerprint). Nothing enforces the note but its own text. A row only changes skill text, which merges through a PR and its CI.
      • The skill's rule that an unattended run never decides a verdict stays as it is (plugins/harness-ops/skills/audit-native-overlap/evals/evals.json:56).
    2. The four pairs that already have a dismissal.
      Decided: re-dismiss each one with its existing reason, through overlap.py dismiss, which refreshes both fingerprints (plugins/harness-ops/skills/audit-native-overlap/SKILL.md:280-284). Detect lists a dismissed pair again when either side's description changes (SKILL.md:286-288), which explains why these four came back. Each existing reason still separates the jobs:
      • rename and docs-hygiene:rename-references (records.json:4745);
      • resume and session-flow:continue-in-background (:4853);
      • stop and session-flow:clean-stop (:4907);
      • agents and docs-hygiene:write-for-agents (:3575).
    3. brief and review:brief-reviewer (agent).
      Decided: dismiss, shared word only. /brief "toggles brief-only output mode" (the ruled reason at records.json:3755). brief-reviewer is a read-only reviewer that runs the review brief it is dispatched with.
    4. claude-code-docs and discovery:read-docs.
      Decided: a verdict row, complementary, with integration route. A model-invocable bundled skill takes route or wrap (docs/conventions/native-references/README.md:254). claude-code-docs "Answers questions about Claude Code features" (records.json:3935). read-docs reads any upstream docs page through the shared cache, and it is model-invocable (plugins/discovery/skills/read-docs/SKILL.md:2-5). They share a job only on Claude Code's own pages, so the two are complementary, not duplicates. This follows the store's complementary and route precedent for the built-in Explore agent and the WebFetch and WebSearch tools (README.md:238).
    5. run-skill-generator.
      Decided: it gets a row or a dismissal only for a component that detect pairs it with at a score of 0.6 or more. The store holds pairs, so a surface with no candidate component has nothing to rule on. That matches Epic: native-overlap tooling fix and route-to-compose sweep #4047's intent as the body states it: a surface "that overlaps a component, gets surfaced".

    The six follow-ups carried from #4047 (#9105 (comment)) are not decided by this record.

    Ruling and re-dismissing need an inventory from a Claude Code install, since the fingerprints hash the live registration text (SKILL.md:280-284). A session on a machine with Claude Code builds it.

    Amended criteria (body criterion 1 replaced; criteria 2-4 stand as written):

    Acceptance criteria:

    • Each statement of the human-gated rule says a verdict is written by a person or by a session acting on the owner's recorded direction, and never by an unattended run. The statements are docs/native-surfaces/records.json:3, plugins/harness-ops/skills/audit-native-overlap/SKILL.md:8 and :288, plugins/harness-ops/skills/audit-native-overlap/reference/canonical-pairs.json:3, and docs/conventions/native-references/README.md:212. The skill's evals still require that a run never decides a verdict unattended.
    • Each pair above has the store entry decided here: four refreshed dismissals, one new dismissal for brief and review:brief-reviewer, and a complementary / route row for claude-code-docs and discovery:read-docs. run-skill-generator has an entry for each component detect pairs it with at 0.6 or more.
    • plugins/discovery/skills/read-docs/SKILL.md carries the ## Boundary section and description phrase for its route row, in the same change.

    Checks a person runs:

    • overlap.py detect on the current Claude Code release no longer proposes these pairs.
  10. github-actions commented on Oct 26, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: scoped
    Basis: automated intake triage; the lanes App applied this class.

    Size: Medium
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Decision recorded. The owner answered in the Refinement and Issue Triage sessions (2026-10-25), deciding the open items below on the facts:

    go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

    [quote truncated; the full decision is at the link]

    #9105 (comment)

    That decision settles the open questions. It covers who may rule, four refreshed dismissals, a new brief dismissal, a complementary/route row for claude-code-docs, and the run-skill-generator scope. The remaining work is the amended criteria: update the human-gated rule wording, write the store entries, and add the read-docs Boundary section. Writing the entries needs an inventory built on a machine with Claude Code installed.

    Automated triage, applied by the lanes App.

  11. 17 remaining items

  12. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    and removed
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Oct 31, 2026
  13. github-actions commented on Oct 31, 2026

    @github-actions
    Contributor
    No description provided.
  14. melodic-automation-lanes commented on Oct 31, 2026

    @melodic-automation-lanes
    Contributor

    The work-resolve-issue run that agent-ready started failed or was cancelled before it reported, or it stopped or the token broker refused it because its workflow files differ from main's. work-unblock removed and added agent-ready back to start a fresh run on main's workflow files. If this run ends the same way, it re-fires the issue again, waiting longer each time.

  15. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    and removed
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Oct 31, 2026
  16. github-actions commented on Oct 31, 2026

    @github-actions
    Contributor
    No description provided.
  17. github-actions commented on Oct 31, 2026

    @github-actions
    Contributor
    No description provided.
  18. melodic-automation-lanes commented on Oct 31, 2026

    @melodic-automation-lanes
    Contributor

    The lanes App's pull request #11512 merged with Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removed status: ready and agent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it at status: confirmed for a check of what that pull request left, until a **Decision recorded.** comment says what is left or someone closes it.

  19. removed
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Oct 31, 2026
  20. kyle-sexton commented on Oct 31, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded.

    This settles what #11433 and #11512 left: the detect check, the six follow-ups carried from #4047 and what remains. Read at origin/main f91a3910c3; 2a8767f8ce changes no file cited here. Agents decided this on evidence under standing direction §1.

    Amended criteria: all three met on main.

    • Rule wording: docs/native-surfaces/records.json:3, plugins/harness-ops/skills/audit-native-overlap/SKILL.md:25-26, :293-294, reference/canonical-pairs.json:3, and docs/contracts/native-references/README.md:212, where that convention now lives. evals/evals.json:59 still refuses a self-assigned verdict.
    • Store entries: the five dismissals from docs(harness-ops): refresh four native-overlap dismissals and dismiss brief against Claude Code 2.1.314 #11433, the claude-code-docs row at records.json:759, and no entry for run-skill-generator, which detect pairs with nothing on 2.1.316.
    • plugins/discovery/skills/read-docs/SKILL.md:2 carries the description phrase and :41 the Boundary section.

    The detect check: passed. Run 2026-10-31 15:05Z by this session on a host with Claude Code 2.1.316: inventory.py --binary-only, then overlap.py detect, exit 3 because plugin_backed is broken. builtin_commands is ok and bundled_skills degraded, so the lanes these pairs come from were read. The five dismissed pairs sit under discovery.suppressed, claude-code-docs / discovery:read-docs sits under discovery.existing with verdict complementary, and none of the six is a candidate. generate --check is in sync at 74 rows and 121 dismissals.

    The six follow-ups (comment 6386726091): all dropped, none built.

    1. recheck subcommand. Drop: the part an extraction can observe is shipped. /harness-ops:changelog Phase 5 runs native_drift.py, evaluates the store's triggers and files a work item per fired row (plugins/harness-ops/skills/changelog/SKILL.md:177-186, scripts/native_drift.py:512). A trigger that names an upstream page stays a session act (audit-native-overlap/SKILL.md:263-264). The 2026-10-22 check records no stumble against that since 2026-09-11, and a title search finds one recheck item in the skill's history, #6885, so nothing has needed one.
    2. Marker-basis field. Drop: shipped as native.docs_markers, which the self-check validates (overlap.py:654-665) and the drift pass skips (changelog/context/native-drift.md:94). The skill-doctor / context-budget:audit row uses it (docs/native-surfaces/records.json:1472).
    3. Name-collision keying in detect. Drop: the seeded-versus-observed case is handled (overlap.py:1714-1723, test_overlap.py:1391) and the same-name registration case is evidenced per registration (overlap.py:505-511). The cross-lane case has one live name on 2.1.316, fork, in builtin_commands and builtin_agents. The store holds one row for it and detect emits one candidate, at 0.0555. Scored alone, the agent registration pairs with that same component and nothing else, at 0.0436, so keying would surface nothing new today.
    4. Durable report from a bare run. Drop: detect --out already writes the candidates (audit-native-overlap/SKILL.md:77-78), Phase 5 keeps a summary baseline (native-drift.md:11), and inventory plus detect took 35 s here. The one reader of a saved detect report, native_drift.py --detect, takes the file Phase 5 writes in the same pass (native-drift.md:20-23), so nothing needs a report kept from a bare run.
    5. reverify helper. Drop: the store shows four rows stamped after 2026-10-02, and a title search for this skill's recheck items finds one, #6885, against the title form at native-drift.md:94. Hand-editing a row through a PR at that rate costs less than a helper. A row's evidence line is judgment a stamp cannot write.
    6. Post-step behaviour assertion for wraps. Drop: shipped as the resolved but degraded state (docs/contracts/native-references/README.md:298-300). Both wrap rows carry it: plugins/code-tidying/skills/batch-simplify/SKILL.md:38, :45 and plugins/testing/skills/run-e2e/context/own-driving.md:92, :107. The first case is simplify running a weaker procedure, the example Epic: native-overlap tooling fix and route-to-compose sweep #4047 named.

    What remains: one host step. plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:200 reads "Re-run on any change: a description change re-measures triggering, a body change re-measures output". #11512 changed the read-docs description and body, touched no eval and lists no run. The new phrase opens the description, as docs/contracts/native-references/README.md:65-66 requires, so the main use case moved back. All four cases in plugins/discovery/skills/read-docs/evals/evals.json name a Claude Code docs page, the ground that phrase splits. The rule has no exemption for an addition, so this applies.

    Host-run step: measure read-docs with the description and body at 5889ce96ec^ against main, run concurrently: triggering with /skill-quality:check measure-invocation, output with the skill's plugins/discovery/evals case through /evals:plugin-eval and its four evals/evals.json cases through /skill-creator:skill-creator. Judge by skill-criteria.md:190. On a pass, post the numbers and close this issue. On a loss, post the numbers and leave it open with a new record, since keeping the before text drops the phrase the route row needs.

    Seen and not part of this issue: on 2.1.316, eight other pairs at 0.6 or above have no verdict. Five are dismissals resurfaced by a description change. Three are new: cc-plugin-diff / code-tidying:tidy-diff, runner / testing:green-runner, and design / user-interface:design (0.629), which is model-disabled. Phase 5 files a work item per new candidate (changelog/SKILL.md:183); a title search finds none yet.

  21. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Oct 31, 2026
  22. kyle-sexton commented on Oct 31, 2026

    @kyle-sexton
    MemberAuthor

    Host run. Ran the three model-graded read-docs before-and-after measurements, 5889ce96ec^ against origin/main, that the earlier no-model record left waiting: triggering, the rd-unread-page-not-recalled plugin case, and the four evals.json skill cases. All three pass under skill-criteria.md, judging the description and the body separately.

    What ran (Claude Code 2.1.316, subscription login, model claude-opus-5-5 throughout):

    • Both trees were re-verified blob for blob. The before tree matches 5889ce96ec^. The after tree matches origin/main: the only commits on plugins/discovery since 49f8bd0174 touch other skills, and the plugins/discovery tree is unchanged.
    • Triggering and the plugin case. An earlier worker on this issue had already run these with claude plugin eval, then stopped before the skill suite.
      • I reused its results after checking each one: the plugin path it loaded is a byte-identical copy of the verified tree, every reused result is partial: false with no run error, and every one passes run-validity.py as VALID.
      • Flags: --ablation none for triggering, default ablation for the plugin case, --allow-tools Bash on the plugin case, -j 2, --keep-temp, HOME set to an empty directory, capped scope.
      • The first positives-train pass was INVALID (permission denials) and is excluded. Its valid rerun is used instead; the excluded pass also fired 18 of 18 on both trees.
      • Every grader in these suites is tool_used or regex, so no judge model was involved.
    • Skill suite. It ran per /skill-creator:skill-creator: one executor subagent per run, following read-docs/SKILL.md from a neutral copy of each tree.
      • Each run had its own empty docs cache and live network. It could not use the Skill tool, WebFetch, or any other copy of the plugin.
      • 4 cases × 3 runs × 2 trees = 24 runs.
      • Grading was blind: one grader per case scored all six runs under shuffled labels, then I unblinded the results.
      • I spot-checked the grades against the command logs. In every run, --cache, --max-age 0 (case 3), read --raw or slice (case 3), --profile anthropic and slice all appear where expected, and WebFetch appears in none.

    Does claude-code-docs resolve in an eval session? No.

    • It appears in the session's skill listing in 0 of 226 eval-session listings: 112 after-tree, 108 before-tree, 6 without the plugin.
    • No run invoked it.
    • The 2.1.316 binary does carry it as a bundled skill. The only mentions inside eval sessions come from the after tree's own read-docs description text.
    • So the triggering run measures the new description with its competitor absent. It shows the added clause costs no trigger, but it does not test the routing split itself.

    Results:

    Suite Before After Criterion (5-point margin, no lost trigger or case, no case down by a third)
    Triggering, 11 positives × 3 33/33 fired (1.00) 33/33 (1.00) Pass. The four Claude Code positives (pos-train-01, -03, -06, pos-val-03) fire 3/3 on both trees
    Triggering, 7 negatives × 3 0/21 false fires 0/21 Pass
    rd-unread-page-not-recalled, with plugin 1.00 (3/3) 1.00 (3/3) Pass. The without-plugin arm is 0.50 on both trees, a delta of +0.50 each, because the plain model states the recalled "3 days" with no inference label. Spot-read: both with-arms run fetch-docs.sh, report the page unread with the sandbox reason, and give no recalled figure
    Skill suite (evals.json), mean expectation pass rate 0.979 1.000 Pass

    The skill suite per case, before → after:

    • 1, reads-a-large-page: 1.00 → 1.00
    • 2, page-silent: 0.92 → 1.00. The one before-tree fail was an answer that said "doesn't give" rather than "not stated on the page".
    • 3, verification-raw: 1.00 → 1.00
    • 4, browser-url: 1.00 → 1.00

    Verdict:

    • Description: pass. Trigger rates are identical on both trees, with no lost positive and no new false trigger. The split against claude-code-docs is untested here, because that skill does not resolve in a plugin eval session.
    • Body: pass. rd-unread holds at 1.00. The skill suite holds or rises on every case, so the ## Boundary section and its reference file cost nothing measured.

    Not covered:

    • The skill suite ran as in-session subagents with self-reported command logs, not inside the plugin-eval sandbox.
    • The earlier worker's 226 kept sandboxes are still in its run directory. I removed nothing.

    No code change and no PR.

    Target labels: add none; remove host-run.

  23. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions