Skip to content

performance:snapshot: report and control which code path the measured run took #4437

Description

@kyle-sexton

Problem

State left in a plugin data dir by earlier runs (a bench.launched marker) sent every harness sample for the disk-hygiene Stop hook down its Python run path. The hook's common path in real sessions (skip, no interpreter) was never measured by the harness; the result was valid only for the rarer path and nothing in the output said so.

Proposal

Snapshot requires the goal to name the code path(s) under test, and before each arm either resets the state files that select the path or records them, then reports per arm which path ran (for example from a marker, exit code or spawned-process set). An arm whose path does not match the goal is flagged rather than reported as the goal's metric.

Found during the Windows hook-latency work under #4373 (#4389/#4408, #4415/#4421, #4416/#4422).

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Sep 24, 2026
  2. kyle-sexton commented on Sep 27, 2026

    @kyle-sexton
    MemberAuthor

    This was generated by AI during triage.

    Triage: enhancement, verified gap. The goal skill names a measurement boundary and start state (cold / fresh / warm) but nothing names the code path a run must take; the snapshot skill neither resets nor records the state that selects the path, and its report carries no per-arm path. Nothing in the harness-integrity reference covers path-selecting state left by earlier runs. Priority: medium (assessed default; no blocker or deadline). Sibling #4436 touches the same skill but is a separate change (tool version/flags), not a shared decision.

    Agent Brief

    Type: Feature
    Summary: Make /performance:goal name the code path under test, and make /performance:snapshot control or record the path-selecting state and report per arm which path ran.

    Current behavior:
    A goal names metric, targets, floor, boundary and start state. Snapshot captures baseline/post arms with host qualification and interleaving, but a subject whose path is chosen by persisted state (for example a marker file in a plugin data dir left by an earlier run) can run a different path than the goal intends, and the report gives no sign of it. Observed: a stale bench.launched marker sent every sample of a Stop hook down its rare Python path, so the result described the rare path, not the common skip path.

    Desired behavior:

    • The goal records the code path(s) under test, and how to tell which one ran (a marker, exit code, spawned-process set, or other observable).
    • Before each arm, snapshot either resets the state that selects the path or records it as found; which of the two it did is stated.
    • The snapshot report states, per arm, the path that ran and the evidence for it.
    • An arm whose observed path does not match the goal's named path is flagged and not reported as the goal's metric. A path that cannot be observed is reported as unobserved, never assumed.
    • The harness-integrity checklist gains an item for path-selecting state carried over from earlier runs.

    Key interfaces:

    • /performance:goal output: new required "code path under test" element with its observable.
    • /performance:snapshot per-arm report: new "path ran" field with evidence, and a mismatch flag.
    • Harness-integrity reference: a failure-table row and checklist item for stale path-selecting state.

    Acceptance criteria:

    • The goal skill requires naming the code path(s) under test and the observable that identifies each.
    • The snapshot skill instructs reset-or-record of path-selecting state before each arm and says which was done.
    • The snapshot report template shows per-arm path and evidence, and a mismatched arm is flagged rather than reported as the metric.
    • The harness-integrity reference covers stale path-selecting state (table row and checklist item).
    • Skill bodies follow the repo's skill-body rules (verification records for volatile specifics, ## Next section intact); plugin version and CHANGELOG updated per repo convention.

    Out of scope:

  3. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Sep 27, 2026
  4. kyle-sexton commented on Sep 29, 2026

    @kyle-sexton
    MemberAuthor

    Reopened. This comes from the 2026-09-29 audit of the unattended Cursor run (report: .work/audit/REPORT.md).

    What the audit found. #4820 closed this issue with the goal skill and the harness-integrity reference untouched: 2 acceptance criteria unmet and 1 partial. There is no statement of whether the code path was reset or recorded, and no unobserved outcome for a path that could not be observed. Step 2b also split step 2's closing paragraph. Evidence: .work/audit/units/4437/pr-4820.diff.

    What #4820 delivered. A per-arm code-path report in snapshot.

    What is still missing on main. The goal skill naming the code path under test, the reset-or-record statement, the unobserved outcome, and the harness-integrity row.

    What happens next. Fixed in draft PR #5283: each arm reports a Path: line with reset or recorded state and an unobserved outcome; goal names the code path under test; harness-integrity carries the row; 2b no longer splits step 2. Version and CHANGELOG entries (0.3.0) ship with the PR. The PR is a draft and is not merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    status: readyTriaged, unblocked, and fully specified; eligible to pick up.

    Fields

    No fields configured for Feature.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions