Skip to content

feat(harness-config): run audit-pass and audit-instructions across several repositories via the target-scope contract - #17549

Open
melodic-automation-lanes[bot] wants to merge 2 commits into
mainfrom
work-resolve-issue/issue-17245-44623155109-1
Open

melodic-automation-lanes[bot] wants to merge 2 commits into
mainfrom
work-resolve-issue/issue-17245-44623155109-1

Conversation

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

Closes #17245

Summary

Opened by the work-resolve-issue lane. The run's own summary:

/harness-config:audit-pass and /harness-config:audit-instructions now accept the target-scope contract's forms (--repo, --repos-from, --owner, --fleet, --skip). They cite the plugin's new generated copy of the contract and run one pass per repository, each with its own state key, run directory, lease, lock and report. With no target form, both behave as before. Design choice for audit-pass, based on reference/lane-catalog.md and the old arguments.md reasoning: the refusal of a non-active target is replaced. A target naming another repository runs as --repo, and delegates that accept a target (audit-instructions, harness-memory:audit, and the retirements sweep's --root) get that repository's root. Delegates that take no target (audit, audit-permission-state, audit-permission-grants, audit-prompting-postures) run only in the active project's pass. Elsewhere they are listed as skipped with 'delegate takes no target', so no finding is attributed to the wrong repository. audit-pass --fix in a non-active repository edits one worktree at that clone's HEAD and opens one PR. audit-instructions --persist-findings writes only the session's own repository's file. Eval case: audit-pass eval 30 (non-git-target-is-refused-not-half-specified) reads the changed refusal text in reference/arguments.md. No eval case reads the new multi-repository spoke files.

Not verified

  • docs/skill-cheat-sheet.md was edited by hand, not regenerated with node scripts/generate-cheatsheet.mjs (the lane runs no interpreter); the PR's CI generator check confirms it
  • scripts/cross-plugin-source-registry.txt needed no edit: its existing glob line 'docs/contracts/target-scope/README.md -> plugins/*/reference/target-scope.md' and 'reference/target-scope.md' entry already cover the new copy
  • tidy skipped: the change has 13 files, over the 10-file cutoff, and touches scripts/shared-copies.txt under the shared scripts/ path
  • check-affected skipped comment-hygiene (fetch denied), size-check (script missing in lane), typos (not installed) and changed-skills inside it (it ran separately and passed); these are left to the PR's CI
  • no build or test suite applies to these markdown and JSON edits beyond the checks above; the plugin test suites are left to the PR's CI

Fix

  • scripts/shared-copies.txt: registered plugins/harness-config/reference/target-scope.md at its alphabetical place among the target-scope lines
  • plugins/harness-config/reference/target-scope.md: new generated copy of docs/contracts/target-scope/README.md (sync-shared-copies.sh)
  • docs/contracts/target-scope/adopters/harness-config.md: new adopters file listing audit-pass and audit-instructions
  • plugins/harness-config/skills/audit-pass/SKILL.md: argument-hint shows [--repo <repo> ...] (dropped [--report-to <dir>] to stay within the 100-character hint budget; it stays in the Arguments table); the target row names the target forms and points to the new spoke; added a…
  • plugins/harness-config/skills/audit-pass/reference/multi-repository.md: new spoke covering the per-repository pass: delegate routing, user and managed scopes, per-repository state and lock via run-state.sh --root, and per-repository --fix
  • plugins/harness-config/skills/audit-pass/reference/arguments.md: replaced the not-the-active-project refusal with the --repo reading and its wrong-attribution guard; kept the non-git refusal
  • plugins/harness-config/skills/audit-pass/evals/evals.json: eval 30's expected_output and one expectation no longer cite the removed active-project-root refusal
  • plugins/harness-config/skills/audit-instructions/SKILL.md: argument-hint and full-form line show [--repo <repo> ...] (shortened <version> to <ver> to keep --unattended in the hint within budget); added a pointer to the new spoke
  • plugins/harness-config/skills/audit-instructions/context/multi-repository.md: new spoke covering the per-repository pass, user-scope merging, per-repository state via --root, one dispatch confirmation per run, and --persist-findings limited to the session's own repository
  • plugins/harness-config/README.md: the audit-pass section describes the target forms instead of the old refusal
  • docs/skill-cheat-sheet.md: the two rows' argument hints updated by hand to match the new frontmatter, since node scripts/generate-cheatsheet.mjs cannot run in this lane
  • .changes/harness-config/work-resolve-issue-issue-17245-44623155109-1-18948e15.md: minor changelog fragment

Verification

The runner ran each check from the base copy against e5d7a3f:

  • scripts/check-changelog-fragments.sh --check: passed
  • scripts/check-changed-skills.sh --merge-base: passed

The run reports:

  • check-changelog-fragments: all 16 fragments valid
  • check-changed-skills: audit-pass PASS (498/500 lines), audit-instructions PASS (499/500 lines); the first run failed on the 500-line cap, fixed by moving the new sections into spoke files
  • check-affected: 14 passed, 9 skipped; validate-plugins first failed on argument-hint budgets (109 and 102 chars), fixed
  • sync-shared-copies --check: all 989 registered copies match their canonical sources
  • check-evals-quality on audit-pass evals.json: PASS, with 5 warnings that were already present
  • fresh-context review subagent: reported 3 defects and 2 minor items; all 5 fixed (stale section link in arguments.md, retirements sweep takes --root, --fix worktree vs lock/third capture, kept --unattended in audit-instructions' hint, stale eval expected_output)
  • The lane run: https://github.com/melodic-software/claude-code-plugins/actions/runs/44623155109

Left out: the items under Not verified.

Related

#17245

Lane-Activity: work-resolve-issue/resolve-issue
@github-actions

Copy link
Copy Markdown
Contributor

Lane run report

Field Value
Activity pr-refine/fix-docs
Model rung 0
Leg handoff
Run job success
Activity step success
Commits pushed none (read lane)
Run 44627672008

Failed GitHub API calls

Method Path Status
GET /repos/{repo}/git/ref/tags/* 404

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review needs no new review of fda7d9567b76369ff22b305752b1332ac1508599: no file of this pull request changed since the review of e5d7a3f.

Security review: PR #17549

I found no security issues. I read every hunk of all 12 changed files. None of them matches a TRUSTED PATHS: glob, and the diff touches no trust control: the trusted-trigger check, the lane token, the kill switch, the broker or any merge hold.

What changed. /harness-config:audit-pass and /harness-config:audit-instructions now accept the target-scope forms (--repo, --repos-from, --owner, --fleet, --skip) and run one pass per repository. The plugin also gets a generated copy of the target-scope contract, plus an adopters file, eval text, README and cheat-sheet updates, and a changelog fragment.

Paths checked:

  • Removed refusal. audit-pass used to refuse a target outside the active project. The diff removes that refusal (plugins/harness-config/skills/audit-pass/reference/arguments.md), so this is a narrowed instruction. The rule protected correct attribution of findings, not a security property. Its intent survives: delegates that take no target are listed as skipped with delegate takes no target instead of running against the wrong repository. The non-git refusal stays. I could not read the instruction exception register because it is absent from this lane's view, so the protected-class list was not consulted. Judged by consequence, removing this rule opens no way to leak credentials, compromise the owner or create a charge.
  • Writes to other repositories. --fix in a non-active repository edits one worktree and opens one pull request. Each fix still waits for confirmation, so no fix runs unconfirmed. audit-instructions stays report-only (disallowed-tools: Edit, NotebookEdit is unchanged), and --persist-findings writes only the session's own repository. No write reaches another clone's main checkout.
  • Untrusted text. These skills run in the operator's own session over local clones, not in a credentialed CI lane. The rule that each subagent reads a repository's AGENTS.md comes from the existing shared contract, which other adopters already copy; this PR does not add it.
  • Embedded text. The PR description holds no instructions aimed at the review lane.
  • Plugin manifests. No plugin.json or marketplace.json is changed. The argument-hint frontmatter edits only change the hints users see.

Spend. Multi-repository runs mean more subagent fan-out per invocation, which spends subscription or API usage only. This is not a hold.

critical-findings: 0

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 2 inline finding(s) on fda7d9567b76369ff22b305752b1332ac1508599; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment on lines +13 to +21
- **A delegate that accepts a target is given that repository**:
`/harness-config:audit-instructions` and `/harness-memory:audit` both take `--repo <clone root>`.
A delegate that takes no target (`/harness-config:audit`, `audit-permission-state`,
`audit-permission-grants`, `audit-prompting-postures`) reads the active project only. It runs
only in the active project's pass, when that project is selected, and every other repository's
pass lists its lane in `skipped` with the reason `delegate takes no target`. Running it there
would attribute the active project's findings to the wrong repository, as
[arguments.md](arguments.md) explains. The retirements sweep takes `--root`, so it runs in every pass
with `--root <clone root>`, and the same leftover in several repositories is one row listing

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Phase 4 /doctor handoff has no routing for a pass outside the active project.

This bullet routes every Phase 3 delegate by whether it takes a target. Phase 4 (SKILL.md "The /doctor handoff", reference/doctor-handoff.md) is not covered:

  • Each repository's pass runs "Phase 0 to Phase 6" (line 9), so each one opens a delegated lane, writes an open terminator and tells the operator to run /doctor and /doctor prompt-audit.
  • Both commands are operator-run in the invoking session, so they read the active project's CLAUDE.md, not the clone's.
  • When the operator hands the output back through --resume on, say, the web pass, it lands in web's delegated section. That is the active project's findings attributed to another repository, the case lines 18 to 20 exist to prevent.
  • Each non-active pass also keeps an open lane the operator cannot close from this session.

To reproduce: run /harness-config:audit-pass --repo web from a session in api. web's report says to run /doctor and come back with --resume, and doing that records api's trim findings under web.

Fix: add a bullet saying the /doctor handoff runs only in the active project's pass. Every other pass lists it in skipped with a reason, for example "operator handoff reads the invoking session's project; open that repository to run it".

claude-lane: claude-review

Comment on lines +25 to +27
merges into one row.
- **Run state and the lock are per repository**: run `run-state.sh paths` with
`--root <clone root>`, so each repository gets its own `<state-key>`, run directory, lease, lock

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Closing-issue criteria are met in the text, but no eval or test could fail if they broke.

The criteria are #17245's "Any per-repository state (ledger, cursor, findings file, delta state) is kept per repository" and "Each listed skill accepts the target forms by citing the contract". The PR body confirms: "No eval case reads the new multi-repository spoke files."

The only changed eval is audit-pass eval 30, which covers the non-git refusal. Nothing covers:

  • --repo routing to --root <clone root> for run-state.sh paths (this file, lines 25 to 27);
  • the same routing for state-key.sh and persist-report.sh (audit-instructions/context/multi-repository.md lines 17 to 19);
  • the delegate takes no target skip (this file, lines 16 to 20; arguments.md lines 26 to 28).

If any of these regressed, every eval would still pass. For example, an edit that drops --root, so every repository shares the invoking project's state key and lock.

Fix: add one eval to each skill. Prompt with --repo <other clone> and expect:

  • run-state.sh paths (and, for audit-instructions, state-key.sh and persist-report.sh) called with that clone's --root;
  • for audit-pass, the four no-target lanes listed in skipped with delegate takes no target.

claude-lane: claude-review

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed fda7d9567b76369ff22b305752b1332ac1508599: read 12 of 12 changed file(s); 2 finding(s), 2 as inline comments.

Code review: PR #17549 (audit-pass and audit-instructions across several repositories)

I found 2 issues. Neither is blocking.

The change does what #17245 asks for:

  • Argument hints: both skills show [--repo <repo> ...] in argument-hint.
  • Contract copy: the plugin ships a generated copy of the contract. It is registered in scripts/shared-copies.txt, and the existing glob in scripts/cross-plugin-source-registry.txt (docs/contracts/target-scope/README.md -> plugins/*/reference/target-scope.md) already covers it.
  • Adopters, changelog and evals: there is an adopters file and a minor changelog fragment. The PR body names eval 30 and says no eval reads the new spoke files.

Helper flags and lanes. I checked that the helpers the spokes rely on accept --root: run-state.sh paths, lib/state-key.sh, lib/persist-report.sh and lib/check-retirements.sh all do. /harness-memory:audit already takes --repo and keeps its state per repository. The four delegates the spoke lists as taking no target match every remaining /… lane in reference/lane-catalog.md.

Cheat sheet. docs/skill-cheat-sheet.md was edited by hand. Its two rows match the new frontmatter hints, but the closing issue asks for it to be regenerated. The PR says CI's generator check confirms this; I could not run it, so that is the author's claim and unverified.

Findings (both inline):

  1. audit-pass/reference/multi-repository.md: the per-repository pass routes every delegate except the Phase 4 /doctor and /doctor prompt-audit handoff. Those are operator-run in the invoking session, so they read the active project. This is the same wrong-attribution problem the spoke guards against for the no-target delegates.
  2. Closing-issue criteria "Any per-repository state … is kept per repository" and "Each listed skill accepts the target forms": the text meets them, but no eval or test reads either new spoke, so nothing would fail if the per-repository routing regressed.

Loosened checks. audit-pass/evals/evals.json eval 30 drops the clause "in the same shape as the existing not-the-active-project-root refusal" from one expectation. That clause compared against a refusal the closing issue explicitly removes ("Replace the refusal with the target forms"). The non-git refusal's own requirements (path, reason, non-zero exit, writes nothing, suppression cost) are all kept, so I judged it as following from the spec, not as a loosening.

Loosened-check verdict: fail (default; the prompt had no loosened_check_verdict: or loosened_check_patterns: lines); patterns: 0. The prompt also had no loosened_check_decision_record: line, so the citation requirement was not checked.

Embedded text. None of the PR context asks this lane to approve, skip or drop anything.

Spend. Multi-repository runs spend subscription or prepaid usage only: one fan-out per repository, and audit-instructions asks for one ~20-dispatch confirmation per run. No billing hold.

Verifier dropped 0 of 2 candidates. No verifier subagent was dispatched; I checked both findings myself against the source.

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

The unattended merge path is about to arm native auto-merge at head fda7d9567b76369ff22b305752b1332ac1508599, the head its arming checks judged. The arm fails if the head moves first.

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor Author

Merge-Risk: max=HIGH demote=true head=fda7d9567b76369ff22b305752b1332ac1508599 rubric=10.4.1 rater=claude-opus-5-5 checker=claude-opus-5-5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(harness-config): run audit-pass and audit-instructions across several repositories via the target-scope contract

1 participant