Skip to content

work-items(work-loop): fixed telemetry title and sentinel make two lanes on one repo clobber durable state, including first_drain_complete #1295

Closed

Description

@kyle-sexton

This was generated by AI during a loop-lane launch-prompt design session.

Context

/work-items:work-loop resolves its telemetry home to the open issue titled
Lane telemetry: work-loop (exact match) and maintains exactly one status comment on it,
identified by the fixed sentinel MARKER="work-items:work-loop". That comment carries the
machine-readable durable loop state, re-read at every cycle start because "conversation context
is compaction-lossy — the comment, not the conversation, is the source of truth for these
counters."

Both the title and the marker are constants. Two worker lanes running on different machines
against the same repository therefore resolve the same telemetry issue and the same sentinel,
which violates the lane-telemetry contract's "one writer identity owns a marker" rule. Durable
state becomes last-writer-wins across machines.

The consequences are not symmetric with the item-claim story. Item claims are provider-arbitrated
and genuinely safe to run from two machines. Durable state is not:

  • item_cap, clean_streak, and rate_limit_latch are silently clobbered, so the adaptive cap
    and the rate-limit backoff stop reflecting either machine's actual experience.
  • first_drain_complete is the serious one. It is the flag that ends the first-drain C3
    ratification gate. One machine writing it true ends the earn-trust period for both machines,
    widening autonomy without any human ratification — the opposite of the gate's purpose.

The creation-race reconcile already in the upsert does not help: it converges duplicate
comments, not conflicting state written by two live lanes.

Proposed work

Make the telemetry binding distinguishable per writer, so two lanes on one repository do not
share durable state. The launch config is the likely seam — the telemetry home is already
documented as "resolved from launch config."

Whatever shape is chosen, verify it against the exclusion contract: the drain-exit snapshot, the
intake sweep, and attend-queue's attention view all exclude lane-infrastructure issues by the
Lane telemetry: <lane> title contract. A machine-suffixed title must still fall inside that
exclusion, or the telemetry issue starts counting as backlog.

Alternatively, if sharing a repository across machines is out of scope, state that constraint
explicitly in the skill so operators do not discover it by state corruption.

Acceptance criteria

  • Two worker lanes on one repository either maintain separate durable state, or the skill
    states plainly that this configuration is unsupported
  • first_drain_complete cannot be set for one machine by another machine's progress
  • Any new telemetry title shape still falls inside the lane-infrastructure exclusion used by
    the drain exit, the intake sweep, and the attention view

References

  • plugins/work-items/skills/work-loop/SKILL.md (telemetry home, sentinel marker, durable state,
    first-drain C3 ratification)
  • plugins/work-items/skills/attend-queue/SKILL.md (lane-infrastructure exclusion)
  • docs/conventions/loop-lane/README.md §4 (telemetry comment, durable loop state)

Metadata

Surfaced while planning two worker lanes across two laptops against one repository for
throughput. The plan was reduced to one lane per repository as a result.

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Jul 25, 2026
  2. added theissue type on Jul 25, 2026
  3. kyle-sexton commented on Jul 25, 2026

    @kyle-sexton
    MemberAuthor

    This was generated by AI during triage.

    Verified

    Confirmed against plugins/work-items/skills/work-loop/SKILL.md at this commit:

    • Line 57 — telemetry home resolves to "the open issue titled Lane telemetry: work-loop (exact match)".
    • Line 64 — MARKER="work-items:work-loop", a fixed constant.
    • Line 105 — the durable state block carries "rate_limit_latch":false,"first_drain_complete":false,....
    • Line 218 — the first-drain C3 gate keys on first_drain_complete in durable state.
    • Line 266 — the drain-exit path sets first_drain_complete.

    Both the title and the sentinel are constants, so two lanes on one repository resolve the same telemetry issue and the same marker. The claim holds as filed.

    Live corroboration: reading the current telemetry state on the work-loop lane issue shows {"item_cap":2,"clean_streak":0,"first_drain_complete":false} — a single shared state block with no writer identity in it. Nothing in the record distinguishes which machine wrote it.

    Classification

    • Work class: C4 structural — changes the telemetry binding contract, which the loop-lane convention (§4) and the lane-infrastructure exclusion in three separate consumers all depend on.
    • Priority: high. The first_drain_complete consequence is a silent autonomy widening: one machine completing a drain ends the earn-trust C3 ratification gate for every other machine, with no human ratification. That is a safety property failing open, and it blocks safe multi-machine operation — both priority: high criteria.

    Decision carrier

    Designated the decision carrier for the multi-lane-per-repository cluster. The one decision: is concurrent multi-lane / multi-session operation against a single repository a supported configuration, and if so what is the per-writer identity and partitioning seam?

    Members, each carrying a blocked-by edge to this item and deliberately no human-gated label:

    One human touch here resolves the shape of all three. This item states the fork explicitly in its own body ("Alternatively, if sharing a repository across machines is out of scope, state that constraint explicitly in the skill"), which is why it carries the decision rather than either member.

    Constraint for whoever resolves this

    The body's third acceptance criterion is the load-bearing one and is easy to miss: any machine-suffixed telemetry title must still fall inside the Lane telemetry: <lane> exclusion used by the drain-exit snapshot, the intake sweep, and attend-queue's attention view. This sweep excluded #1198, #1214, and #1248 by exactly that title contract — a title shape that breaks it turns lane infrastructure into backlog.

  4. kyle-sexton commented on Jul 25, 2026

    @kyle-sexton
    MemberAuthor

    This was generated by AI during autonomous /work-items:work-loop execution.

    A second /work-items:work-loop lane (this session, running oldest-first per explicit operator direction, concurrently with another session reported running newest-first) hit exactly the contention this issue tracks, in the same cycle it started:

    • The shared telemetry issue Lane telemetry: work-loop (Lane telemetry: work-loop #1248) carries a single fixed-marker comment last written by the other live lane (cycle 4, item_cap:3, clean_streak:7). Per this issue's own analysis, upserting from this lane would clobber that state under last-writer-wins.
    • Mitigation applied this cycle: this lane did not touch the Lane telemetry: work-loop #1248 comment. Cycle report and durable state for this lane's run are being kept in a local file instead (a scratchpad file, not committed to the repo), per the loop skill's own fallback ("everything else goes to a file... or a comment on the item").
    • This is a workaround for one cycle, not a resolution — it still leaves this lane's progress (clean_streak, item_cap) untracked across restarts, and duplicates no comment, so it doesn't corrupt anything, but it doesn't fix the shared-state problem either.

    Also observed live during this cycle, likely relevant to this decision: issues #815, #816, and #965 each already carry a recorded ratification directly in their issue body (attended triage 2026-07-23, operator-ratified) plus a same-day correction comment restoring them to the autonomous frontier after a prior cycle wrongly re-queued them for ratify-c3 without reading the body. As of this cycle, all three have needs-human re-applied again alongside a fresh ratify-c3 queue comment (posted today), reproducing the same mis-classification a second time. This suggests the admission gate's ratify-c3 queuing step still does not check the issue body for an existing recorded ratification before re-queuing. Filing that as a separate follow-up rather than fixing it inline here, since it is a distinct (if related) defect in the gate logic, not the telemetry-identity problem this issue tracks.

    No action needed from this comment beyond visibility -- this issue is already correctly needs-human / decision-carrier and this lane is not attempting to resolve it.

  5. kyle-sexton commented on Jul 25, 2026

    @kyle-sexton
    MemberAuthor

    This was generated by AI, delegated directly by the repository operator to resolve this blocker.

    Decision

    Concurrent multi-lane operation against a single repository is a supported configuration.
    Writer identity is partitioned in the sentinel marker, not in the issue title. The telemetry
    home stays exactly where it is — one open issue per lane titled Lane telemetry: <lane>, one per
    repository — and each lane instance owns its own sentinel-identified comment on that issue.

    The marker gains a lane-instance suffix; the existing marker becomes its prefix:

    MARKER="work-items:work-loop@<lane_instance>"
    SENT="<!-- claude-ops:lane-telemetry marker=$MARKER -->"
    

    <lane_instance> is a stable, operator-visible identifier resolved from launch config, defaulting
    to the sanitized machine hostname when unset (headless-config floor: never block on an interview,
    log the assumption). It must be stable across restarts — durable state is what survives a /loop
    expiry or a cycle-budget relaunch — and distinct across concurrently running lanes. Two lanes on
    one machine therefore must be given distinct explicit ids; that requirement is enforced by the
    collision detector below, not left to operator discipline.

    The identifier is user-influenced input that is interpolated into a shell string and a jq
    startswith filter, so it is validated before use: ^[a-z0-9][a-z0-9-]{0,31}$, rejected otherwise,
    never sanitized-and-continued.

    This is the contract the loop-lane convention already states — "one writer identity owns a
    marker"
    . Today the marker names a lane type, not a writer. The fix is to make the marker
    actually name the writer.

    Why this satisfies the three acceptance criteria

    AC1 — two lanes maintain separate durable state. Each instance's lookup matches only its own
    full sentinel, so each instance creates, reads, and edits exactly one comment that no other
    instance can match. item_cap, clean_streak, and rate_limit_latch become per-instance, which
    is the only reading that is semantically correct: they measure that lane's observed experience,
    and averaging two lanes' experience into one counter was never meaningful. The existing
    creation-race reconcile (lowest comment id canonical, duplicates tombstoned) is unchanged and
    still operates correctly — it now converges duplicates within an instance's own sentinel set.

    AC2 — first_drain_complete cannot be set by another lane's progress. It lives in the
    per-instance block, which no other instance can write. Earn-trust is therefore re-earned per lane
    instance: a new machine, or a newly named instance, runs its first drain under the C3 ratification
    gate rather than inheriting a trust period some other lane earned. The cost is bounded and by
    design — item-level ratifications already travel with the item (kind=ratify-c3 recorded on the
    item itself), so only the blanket period-end flag resets, not previously ratified items.

    AC3 — the exclusion contract is untouched. The issue title does not change, at all. The
    Lane telemetry: <lane> title contract that the drain-exit snapshot, the intake sweep, and
    attend-queue's attention view all match on is not modified, not extended, and not made
    prose-ambiguous. This criterion is satisfied vacuously rather than argued — which is precisely why
    the marker was chosen as the seam over the title.

    Collision detection (the safety property fails closed)

    Partitioning is correct only while ids are distinct, so id collision is detected rather than
    assumed away. The durable state block carries two new fields written every cycle:

    {"lane_instance":"<id>","writer_nonce":"<per-session random>","heartbeat_at":"<ISO-8601 UTC>",
     "paused_until":null}

    At cycle start, after reading the block:

    • writer_nonce equals mine → normal continuation.
    • writer_nonce differs and the block is stale (heartbeat_at older than 2 hours, and past
      paused_until when that field is set) → a previous session of this same instance restarted or
      died. Adopt the block, write my nonce, continue. This is the ordinary restart path.
    • writer_nonce differs and the block is fresh → another live lane is using my instance
      id.
      Write nothing to the block. Escalate per the convention's escalation contract (human-gated
      role label + a machine-marked escalation comment) and stop the loop cleanly.

    A lane entering a rate-limit pause writes paused_until before pausing, so a paused lane is never
    mistaken for a dead one and the staleness window stays short. Detection therefore happens on the
    first cycle, before any write — the collision case degrades to a stopped lane, never to a
    silently clobbered first_drain_complete.

    What this deliberately rejects

    • Suffixing the issue title (Lane telemetry: work-loop (laptop-a)). It puts the risk exactly
      on AC3's fragile surface: three consumers match lane infrastructure by a prose title contract,
      and one prose-matching miss turns lane infrastructure into backlog. It also multiplies
      permanently-open issues, which loop-lane telemetry: move durable state off permanently-open Issues to a GitHub Discussion (storage-mechanism reconsideration) #1250 is already filed against.
    • Declaring multi-lane-per-repository unsupported. The operator runs concurrent lanes against
      this repository as standing practice, and the item-claim protocol (assignee + lease, exit 7) is
      already provider-arbitrated and genuinely safe under concurrency. Only the telemetry binding was
      unsafe. Declaring the configuration unsupported would discard working concurrency to avoid
      fixing one identifier.

    Follow-up implementation (do NOT treat this comment as the change)

    No plugin file was edited by this decision. The implementing change is:

    1. docs/conventions/loop-lane/README.md §4 — the owner doc, first. The "Telemetry comment
      (ops: loop-lane telemetry — running per-lane status (babysit / triage / work) #502)" paragraph currently reads "Each lane maintains exactly one status comment ... never
      a second comment", which as written forbids this fix. Restate it as one comment per writer
      identity
      : a lane instance maintains exactly one sentinel-identified comment, the sentinel
      carries the instance id, and N concurrent instances of one lane legitimately hold N comments on
      that lane's telemetry issue. Add the lane-instance identity rule (stable across restarts,
      distinct across concurrent instances, charset-validated, hostname default) and the
      freshness-based takeover/collision rule, since all three lanes hold it by citation.
    2. All three lane skills — the fixed-marker pattern is identical in each, so fixing only
      work-loop leaves the same latent bug in two lanes:
      • plugins/work-items/skills/work-loop/SKILL.md (telemetry section, ~line 57 onward): marker
        construction, id resolution and validation, the new state-block fields, and the cycle-start
        takeover/collision branch.
      • plugins/work-items/skills/attend-queue/SKILL.md (~line 103): same change with
        MARKER="work-items:attend-queue@<id>".
      • plugins/source-control/skills/babysit-loop/SKILL.md (~line 225): same change with its own
        marker.
        None of these touch the Lane telemetry: <lane> title contract or the exclusion prose at
        work-loop ~line 309, attend-queue ~line 58, babysit-loop ~line 135.
    3. Manifests — add the lane-instance userConfig key to
      plugins/work-items/.claude-plugin/plugin.json and the source-control manifest, with the
      hostname-default behavior documented and a version bump on each. Note in the key's description
      that the value appears verbatim in tracker comments, so an operator who does not want a machine
      name published in a public tracker sets an opaque id instead.

    Notes for whoever implements it

    • Migration is a deliberate reset. Legacy un-suffixed comments do not match any instance's new
      sentinel, so the first cycle after the change posts a fresh block starting from defaults —
      including first_drain_complete:false for every lane. That is intended and fails closed, but it
      will produce a burst of kind=ratify-c3 queue comments on the next drain; it is not a
      regression. Existing legacy comments can be tombstoned by hand.
    • claude-ops:morning-brief keeps working, cosmetically degraded. Its telemetry reader
      iterates every comment carrying a lane: field, so N instances render as N rows all labelled
      with the same lane. Emit the instance on its own instance: line in the cycle report (its
      lane: capture is [a-z0-9_-]+ and would truncate a suffix appended there). Making
      morning-brief render the instance per row is a separate cosmetic follow-up in claude-ops,
      not a blocker for this change.
    • Storage-agnostic. This decision partitions writer identity; it does not preempt loop-lane telemetry: move durable state off permanently-open Issues to a GitHub Discussion (storage-mechanism reconsideration) #1250's
      reconsideration of whether durable state lives on an Issue comment at all. Whatever surface wins
      there carries the same per-writer-identity sentinel.

    Cluster unblocked

    Label state

    needs-human removed and agent-ready applied, since a concrete, fully specified implementation
    step now exists. One caveat for the loop: the remaining work is still C4 structural (it changes
    a cross-plugin convention contract), so /work-items:work-loop's admission gate will correctly
    decline to self-select it and will converge it back to human-gated. The intended pickup path is an
    operator-directed /work-items:work 1295, not autonomous frontier selection.

  6. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    and removed
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Jul 25, 2026
  7. kyle-sexton commented on Jul 26, 2026

    @kyle-sexton
    MemberAuthor

    This was generated by AI during an autonomous /work-items:work-loop session.

    Live reproduction, timestamped. This session PATCHed the pinned telemetry comment on #1248 (id 5071412108) at 2026-07-26T00:56:54Z, posting its own cycle 13 report. A date -u + fresh gh api read at 2026-07-26T01:05:24Z found the comment's updated_at at 2026-07-26T00:58:52Z — 2 minutes after this session's write — with body content this session never wrote: a "cycle 10" report about issues #1406/#1410, PR #1411, and a 7-day rate-limit reading of 81%, none of which match this session's own numbers or narrative at any point.

    Two independent work-items:work-loop sessions PATCHed the same single mutable comment within ~2 minutes of each other, and the later write silently discarded the earlier one — no merge, no conflict signal, no duplicate-comment fork (both sessions found the same canonical id on lookup, so the creation-race reconcile never triggered; this is a post-creation overwrite race, not the creation race that reconcile already handles).

    This session's response: stopped re-patching the canonical comment (to avoid contributing to the same clobber pattern against whichever session owns it now) and posted its own cycles 11-14 as a separate, non-sentinel comment on #1248 instead: #1248 (comment)

    Confirms the fix this issue already proposes is needed for the overwrite case too, not only the creation race the current upsert logic reconciles.

  8. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    and removed
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Jul 26, 2026
  9. kyle-sexton commented on Jul 29, 2026

    @kyle-sexton
    MemberAuthor

    This was generated by AI — observed live by autonomous /work-items:work-loop lane instance MELO-LAP-001-worker-202607290838, cycle 1. Filed here rather than as a new issue: same root cause, same fix direction.

    New evidence for this issue, plus one scope point it does not currently cover.

    The clobber this predicts has already happened, and is recorded

    #1248 comment 5081324184 (2026-07-26) is a lane reporting its own state loss:

    The canonical pinned telemetry comment (id 5071412108) was clobbered mid-session by a
    concurrent lane: this session PATCHed it at 2026-07-26T00:56:54Z …

    That is first_drain_complete-bearing durable state destroyed between instances, not a
    hypothetical.

    Practice has already diverged from the documented protocol — which makes the docs actively destructive

    plugins/work-items/skills/work-loop/SKILL.md:59,64-65 still specifies one comment,
    edited in place, keyed by a fixed sentinel:

    MARKER="work-items:work-loop"
    SENT="<!-- claude-ops:lane-telemetry marker=$MARKER -->"
    

    #1248's body repeats it: "Maintains one edited-in-place status comment (marker
    work-items:work-loop)".

    Live practice since 2026-07-27 is the opposite — new per-instance comments keyed
    <!-- work-items:telemetry lane=work-loop instance=<id> -->:

    Comment Marker
    5071412108 (07-24) claude-ops:lane-telemetry marker=work-items:work-loop
    5092113052 (07-27) work-items:telemetry lane=work-loop instance=MELO-LAP-001-worker-202607270855
    5113210738 (07-29) work-items:telemetry lane=work-loop instance=MELO-LAP-001-worker-202607290351
    5118120283 (07-29) work-items:telemetry lane=work-loop instance=MELO-LAP-001-worker-202607290838

    5113210738 states the reason inline: it posted new "rather than editing 5092113052 in place:
    that comment is keyed to MELO-LAP-001-worker-202607270855".

    The drift is the sharp edge. An instance that follows the written protocol resolves the
    sentinel, finds a comment, and PATCHes it — destroying a live sibling instance's durable state.
    The improvised convention is currently the only thing preventing that, and it is convention held
    in agent judgement, not in the skill. Up to three machines run this lane concurrently.

    Scope point this issue does not cover: the surface is per-repo, not org-wide

    Three repositories each hold their own open issue titled exactly Lane telemetry: work-loop:

    • melodic-software/claude-code-plugins#1248
    • melodic-software/dotfiles#336
    • melodic-software/ci-workflows#231

    This issue frames the problem as "two lanes, one repository". The title is also non-unique
    across repositories, so any resolution that keys durable state on the title alone must state
    which repository's issue it means. A lane procedure that reads "the telemetry issue" as a single
    org-wide surface will resolve the wrong one.

    Whatever binding is chosen, the identity needs to be (repo, lane, instance) — not (lane) — and
    the SKILL.md text and #1248's body both need to move with it, or the documented protocol keeps
    instructing the destructive PATCH.

    Related

  10. self-assigned this
    on Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.

Type

Fields

No fields configured for Bug.

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions