Skip to content

rendered-views: interactive userConfig smoke test on the current CLI (gates the fleet sweep) #3604

Closed

Description

@kyle-sexton

The userConfig mechanism has a real bug history (enable-prompt and persistence issues, roughly v2.1.86 through v2.1.210). Docs are current at v2.1.252, and a wave-1 probe on CLI 2.1.251 verified the unset path only (an unset user_config reference reaches a skill body as the literal token, the documented behavior the resolution ladders detect). The interactive set-value flow could not run in the remote container, so per the owner-approved Brief amendment the full test rides this issue and gates the fleet sweep (#3603).

Test, on a machine with the interactive /plugin dialog: set the visualization plugin's medium option to a non-default value, confirm the value substitutes into plugins/visualization/skills/visualize/SKILL.md's rung-2 line at invocation, confirm persistence across CLI restart, and confirm clearing the option restores literal-token unset behavior. Record CLI version and outcome here, then unblock #3603.

Activity

  1. added theissue type on Sep 1, 2026
  2. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Sep 1, 2026
  3. claude commented on Sep 6, 2026

    @claude
    Contributor

    Generated by Claude Code

  4. claude commented on Sep 6, 2026

    @claude
    Contributor

    This was generated by AI during triage.

    Triage outcome: human-gated (capability blocker, not a decision)

    This needs a human because of what it requires, not because anything is undecided. The test must be run through the interactive plugin dialog on a machine that has one. A remote container has no such dialog, which is exactly why the wave-1 probe could only verify the unset path and why the full test was moved onto its own issue. No amount of briefing makes this delegable to an agent in a headless environment.

    The test procedure itself is fully specified in the body and needs nothing added.

    Verification (2026-09-06)

    The setup checks out. The convention document exists with its wave-1 adoption section, the exemplar surface that migrated in wave 1 is in place and carries the rendered-views wiring, and the shared chrome reference exists in that plugin's reference directory. So the mechanism under test is real and shipped, and this test is the gate on multiplying reliance on it.

    Blocking relationship recorded

    A native dependency edge now records that #3603 is blocked by this issue. That was previously stated only in prose in both bodies. The fleet sweep will not surface as available work until this test is done, which is the intended behavior: the sweep multiplies reliance on the configuration dial across a dozen surfaces, and the dial has a real bug history through roughly v2.1.86 to v2.1.210.

    What the runner needs to do

    On a machine with the interactive plugin dialog, on the current CLI:

    1. Set the visualization plugin's medium option to a non-default value.
    2. Confirm the value substitutes into that skill's tier-2 line at invocation.
    3. Confirm the value persists across a CLI restart.
    4. Confirm clearing the option restores the unset behavior, where the reference reaches the skill body as a literal token.

    Record the CLI version and the outcome in a comment here. The version matters: the documented behavior is current as of a later version than the one the wave-1 probe used, and the bug history makes the version part of the result rather than a footnote.

    On completion, if the flow works, remove the dependency edge so #3603 becomes available. If it does not, report what failed here, because the sweep's tier ladder depends on the dial being reliable and a failure changes the sweep's design rather than merely delaying it.


    Generated by Claude Code

  5. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Sep 6, 2026
  6. kyle-sexton commented on Sep 29, 2026

    @kyle-sexton
    MemberAuthor

    Reopened: it was closed by cursor[bot] with no smoke-test result recorded; the gate is unmet.

  7. 19 remaining items

  8. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Oct 27, 2026
  9. github-actions commented on Oct 27, 2026

    @github-actions
    Contributor

    The triage sweep dispatched intake triage for this issue (attempt 1 of 2).

  10. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.
    and removed
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    on Oct 27, 2026
  11. kyle-sexton commented on Oct 28, 2026

    @kyle-sexton
    MemberAuthor

    Host run. Interactive claude under a detached tmux session with a throwaway CLAUDE_CONFIG_DIR, to drive the /plugin userConfig set, persist and clear test; blocked at the login screen before /plugin was reachable.

    CLI version: 2.1.312 (Claude Code).

    What ran:

    • tmux new-session -d with CLAUDE_CONFIG_DIR set to an empty throwaway directory under /home/kyle/worktrees/host-run-3604/, then the real claude binary by absolute path under timeout 600, driven with tmux send-keys and read with tmux capture-pane.
    • The TUI drove reliably under tmux: the welcome screen and theme picker rendered, and Enter advanced to the next screen.
    • The next screen was the first-run "Select login method" prompt. A fresh config dir carries no login, and the subscription login is a browser OAuth on the owner's account. Copying the host's existing credentials into the throwaway dir is a credential move this run is not allowed to make, so the run stopped there.

    Against the four steps of the 2026-10-01 comment:

    1. Set medium to a non-default value in /plugin: not run (login screen comes before /plugin).
    2. Value substitutes into the rung-2 line of plugins/visualization/skills/visualize/SKILL.md at invocation: not run.
    3. Value persists across a CLI restart: not run.
    4. Clearing the option restores literal-token unset behavior: not run.

    Not a TUI-driving failure: keystrokes and captures worked. The blocker is auth isolation. The test needs either an attended session that logs the throwaway config dir in once (any subscription account), or a run against an already logged-in config whose pluginConfigs the run is allowed to change and restore.

    Target labels: add none; remove none.

  12. added
    attendedNeeds the owner present: DNS, keys or secrets, elevation, non-IaC provisioning.
    and removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Oct 28, 2026
  13. kyle-sexton commented on Nov 1, 2026

    @kyle-sexton
    MemberAuthor

    #3604 host run: interactive userConfig smoke test (real config)

    • Date: 2026-11-01
    • Host: WSL2 (Linux), real logged-in ~/.claude (no throwaway CLAUDE_CONFIG_DIR), run under the owner's authorization for this run
    • CLI: claude --version → 2.1.316 (Claude Code)
    • Plugin under test: visualization@melodic-software 0.13.3, user scope (the active install for the working directory)
    • Target line: ~/.claude/plugins/cache/melodic-software/visualization/0.13.3/skills/visualize/SKILL.md:117, which reads
      2. **Configured preference**. `${user_config.medium}`. Claude Code text-substitutes

    What ran

    1. Snapshot of ~/.claude/settings.json and ~/.claude.json taken before any change. pluginConfigs existed only in
      settings.json and had no visualization@melodic-software entry. ~/.claude.json had no pluginConfigs.
    2. The real CLI ran in a detached tmux session, timeout 900 claude --permission-mode auto, working directory
      /home/kyle/worktrees. It was driven with tmux send-keys and read with tmux capture-pane -p.
    3. Each check used this prompt:
      /visualization:visualize Do not render anything and run no tools. Reply only with the exact text of the line in your skill body that begins "2. **Configured preference**", verbatim in a code block.
      The baseline check alone had a leading terminal argument. That argument is rung 1 and does not affect the text
      substitution in rung 2.
      Evidence for each check comes from two places: the model's reply on screen, and a grep of that session's transcript
      (~/.claude/projects/-home-kyle-worktrees/<session>.jsonl), which records the skill body as it was injected.
    4. Three CLI processes ran: the set session, a restart, and a final restart after the option was cleared.

    Baseline before step 1, option unset: both the reply and the transcript showed the literal token
    2. **Configured preference**. `${user_config.medium}` .

    Results

    # Step Verdict Evidence
    1 Set medium to a non-default value in /plugin pass Path: /plugin → Installed → search visualization → Configure options. The dialog showed Preferred delivery medium ◀ auto ▶ (it displays the declared default even when nothing is stored). → terminal → Save configuration. The CLI printed Configuration saved. and ran /reload-plugins on its own. settings.json then held pluginConfigs["visualization@melodic-software"] = {"options":{"medium":"terminal","thin_context_prompt":"auto"}}
    2 Value substitutes at invocation into the rung-2 line pass Same session after the save. Reply: 2. **Configured preference**. `terminal`. Claude Code text-substitutes. The transcript grep showed `terminal` in place of the token
    3 Value persists across a CLI restart pass /exit, then a new process. The reply and the transcript both showed `terminal`. The Configure options dialog in the new process also showed ◀ terminal ▶
    4 Clearing restores the literal-token unset behavior pass, with a finding The dialog has no clear or reset control; it only cycles the enum (auto, terminal, file, artifact, hosted). Choosing auto and saving stored "medium":"auto", not an unset key. To clear it, the CLI was exited and settings.json was restored from the snapshot. The semantic diff before the restore showed that only the two visualization@melodic-software option paths differed. After a fresh process started, the reply and the transcript both showed the literal 2. **Configured preference**. `${user_config.medium}` again, matching the baseline

    Findings

    • The dialog cannot unset an option. After any save, the option holds a stored value. Returning to the default
      stores "auto". The only way back to the literal-token state is editing settings.json by hand. The skill's rung 2
      already treats a literal token, an empty value and auto the same way, so the visible behavior matches. The stored
      auto was not invoked in this run.
    • One save writes every option. Saving with only medium changed also stored thin_context_prompt: "auto", the
      declared default for that option.
    • The CLI rewrites settings.json when it saves. After the save the file's key order changed (a large text
      diff, no value changes outside the plugin entry). Restoring the snapshot bytes put the original order back.

    Config diff summary

    • ~/.claude/settings.json: during the run, only pluginConfigs["visualization@melodic-software"].options.{medium,thin_context_prompt}
      differed in value. It now matches the pre-run snapshot byte for byte (cmp is clean). The file is rendered by a
      chezmoi modify_ script, which touches only the guardrails key under pluginConfigs. chezmoi apply was not run.
    • ~/.claude.json: still has no pluginConfigs. The changed keys are CLI bookkeeping that every launch writes, left as
      they are: numStartups, usage counters for this plugin and skill (pluginUsage, skillUsage), caches and tips.
    • Folder trust: projects["/home/kyle/worktrees"].hasTrustDialogAccepted went from unset to true. The first launch
      required accepting the trust dialog to run in that folder. It is the one lasting change to user config besides the
      option under test. It was left in place because live sessions write ~/.claude.json.
    • Background plugin auto-update: during the run the CLI updated three plugins on its own (knowledge 0.31.1,
      playbooks 0.25.3, harness-ops 5.1.1); the reload after the save applied them. The test did not cause this.
      visualization stayed at 0.13.3 throughout.
    • No credential files were copied. No other user config was changed by the test.

    Target labels: remove attended. #3603 is unblocked, provided its acceptance counts clearing the option by hand.
    Through the dialog alone, the nearest state is a stored auto.

  14. removed
    attendedNeeds the owner present: DNS, keys or secrets, elevation, non-IaC provisioning.
    on Nov 1, 2026
  15. kyle-sexton commented on Nov 2, 2026

    @kyle-sexton
    MemberAuthor

    Closing as completed. The attended run on 2026-11-01 (Claude Code 2.1.316) passed all four steps through the interactive /plugin dialog (#3604 (comment)):

    • the medium value saved;
    • it substituted into the rung-2 line of plugins/visualization/skills/visualize/SKILL.md;
    • it persisted across a restart;
    • clearing it brought back the literal ${user_config.medium} token.

    One finding carries into #3603: the dialog has no unset control. Choosing the default stores "auto", and only a hand edit of settings.json removes the key. Visualization's rung 2 already treats the literal token, an empty value and auto the same way (SKILL.md:117-120), so the tier ladder stands. The convention contract does not state that rule, and a stored auto was not invoked in this run, so #11361 gets a done-when line for it. As the 2026-10-01 decision says, the blocked-by edges from #3603 and #11361 to this issue are removed.

    This was generated by AI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions