Skip to content

silent-revert canary: verify_known_incidents proves only that a commit still fires, not which culprit or how many lines #2833

Closed

Description

@kyle-sexton

Raised in review on #2832.

verify_known_incidents in scripts/check-silent-revert.sh decides a fires
row purely on the exit status of scan_commit:

out="$(scan_commit "$sha" 2>&1)"
status=$?
case "$expect" in
  fires)
    if [[ "$status" -eq 1 ]]; then
      printf 'ok   %s fires as recorded  (%s)\n' "${sha:0:9}" "$note"

The note field is echoed back verbatim and never inspected, and nothing in
the row constrains which culprit the finding was attributed to or how many
lines it accounted for. So the replay proves only "this commit still produces
at least one finding", while printing a note that asserts much more.

The gap is widest on cc58cbc53 (#2633), whose deletions trace to two
different culprits and which the detector reports as two separate findings:

culprit PR lines
bfb66beb8 #2644 853
eda5ae5ed #2642 301

If the eda5ae5ed attribution ever regresses — a window narrowing, a blame or
rename-detection change, a threshold move — the row still passes on the
surviving bfb66beb8 finding alone, and the self-test prints
ok cc58cbc53 fires as recorded (#2633 dropped #2644's rollups (853 blamed lines) and #2642's GraphQL merge evidence (301) ...) while the run reported
nothing at all about #2642. The canary would be announcing a reproduction it
did not perform.

This is pre-existing and not specific to the two-culprit row: the 346 and
451 figures in the other two rows are equally unvalidated. It was simply not
visible until a row named two culprits.

Suggested direction

Give the corpus a machine-checkable expectation rather than a free-text note —
for example an optional culprit/count list per row that verify_known_incidents
asserts against scan_commit's findings, so a row fails when a recorded
attribution stops reproducing. Whatever shape it takes, the constraint stated
in #2808 still holds: the recorded expectations must not be relaxed to make
CI pass.

Deliberately out of scope for #2832, which corrects corpus data and is
constrained to leave the detector's behavior untouched.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageNot yet classified. Floor until a type and one priority tier are set.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions