You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
planning: mark interview recommendations stale when a new constraint arrives mid-interview #5769
In a long /planning:interview run (92 questions, the poteto-talk page), the user added cross-cutting rules mid-interview: one owner per behavior, own skill or extension, Anthropic as the default, no install pointers, in-flow vs on-demand, and trace Cursor-native dependencies. Each new rule invalidated recommendations already posted, and some already answered. Nothing on the page or in the skill showed which ones. The user lost confidence in the whole round:
"I want to make sure all of our questions, the answers... and the recommendations are up to date across the board... Should I skip down to question 87? Is there a certain order I should be answering these questions in now?"
The user asked for this to be filed. #5569, the earlier page-bug umbrella, is closed.
Gaps
When a new constraint enters the constraint ledger, nothing marks the recommendations and answers it may invalidate as stale. dependsOn links only question to question.
There is no "recommendations are current" all-clear and no pause banner beyond set-status, so the user cannot tell whether to keep answering while Claude re-checks.
Once the frontier grows past a few dozen questions, the page shows no recommended answering order (dependency first, then blast radius).
An answer given before a later rule changed its question's recommendation is not flagged for re-confirmation.
Suggested direction
The skill already asks each recommendation to name the constraints it was checked against (Checked against: C1, C3). Store that list per question on the page.
When a constraint is added, mark every recommendation not checked against it as stale, and show a banner until Claude has re-checked or revised each one.
Sort the open frontier by dependency, then blast radius, and show that order to the user.
Offer a re-confirm pass for answered rows whose recommendation changed after they were answered.
Acceptance
Adding a constraint marks the affected open and answered rows on the page and in the ledger, and the gate (check-open-questions.sh) refuses to pass while a stale row is unresolved.
The page shows a current/stale state for the round, and a recommended answering order once the frontier exceeds a set size.
Reported from the Poteto Workflow session and routed by the Prompting Sonnet 5.5 session, 2026-10-02.
Routing: human-gated, status: needs-decision, work-class: scoped. The direction is clear, but several design forks are open, part of the request already exists, and one part touches a mechanism that is parked on purpose. It is also wide enough (skill text, page, server, ledger sync, gate) that it likely needs /work-items:decompose into slices once the forks are settled.
What already exists:
Each recommendation carries a Checked against: line, and a new confirmed constraint is meant to re-check every open recommendation before the next round (instruction only; nothing on the page or in the gate enforces it).
Basis: verified, plugins/planning/skills/interview/context/loop.md:84-85
A changed recommendation marks the questions named in affects and its direct dependents that hold a decision as Upstream changed, and answering again clears it. That covers most of gap 4 when affects is filled.
Basis: verified, plugins/planning/skills/interview/context/surface.md:87,157
The transitive stale cascade is parked.
Basis: verified, plugins/planning/surface/DEFERRED.md:23
The open-question gate checks register status and procedure, and has no notion of a stale row.
Basis: verified, plugins/planning/scripts/check-open-questions.sh:20-27
Open decisions:
Where the per-question constraint list lives: a structured field on the page question (new schema field) or parsed from the Checked against: text.
Whether "stale against a new constraint" reuses the existing Upstream changed state or is a separate state, and whether the gate refusing on it applies to answered rows as well as open ones.
Recommended answering order: what "blast radius" means (for example, count of transitive dependents) and the frontier size that turns the order on.
Type: Feature Summary: When a constraint is added mid-interview, show which recommendations and answers it may invalidate, and keep the interview from locking until they are resolved.
Current behavior:
Adding a constraint relies on Claude re-checking open recommendations by instruction. The page shows no round-level current or stale state, no answering order for a large frontier, and the gate passes regardless.
Desired behavior:
Each question records the constraints its recommendation was checked against.
Adding a constraint marks every open and answered question not checked against it as stale, on the page and in the ledger, until Claude re-checks or revises it.
The page shows whether the round's recommendations are current or stale.
Above a set frontier size, the page shows a recommended answering order: dependencies first, then the agreed blast-radius measure.
The gate refuses to pass while a stale row is unresolved.
Acceptance criteria:
Adding a constraint in a test session marks the expected open and answered rows stale on the page and in the ledger.
The open-question gate exits non-zero while a stale row is unresolved and zero once it is resolved, covered by its test suite.
The page shows the current or stale state and, above the threshold, the order.
Existing ledgers without the new data still import and grade.
Out of scope:
The transitive stale cascade parked in the surface's deferred list.
Question: When a constraint is added mid-interview, answers not yet checked against it should be flagged, and the gate should refuse while any are unresolved. Today this is an instruction only, and the gate has no notion of stale. Options: (a) store a structured "checked against" list per question, reuse the existing "Upstream changed" state, and gate open and answered rows; (b) parse the "Checked against:" text and gate open rows only; (c) fold into #5796.
Owner's answer: (a) B cannot work on the page, which renders constraints in words rather than ids. Basis: interview SKILL.md:357; server.py:275-286; check-open-questions.sh:15-30.
The owner accepted this recommendation as item D4 of a 32-decision batch in an attended backlog-shaping session on 2026-10-18. The batch question: "Accept all 32 recommendations, including the four flagged above (D2 close the relearn, D26 cross-repo sync without per-repo confirm, D10 narrow hook, D23 delete guard only)?" The owner answered: "Accept all 32 (Recommended)".
Posted by AI from the owner's answer in that session.
Re-admit. This issue's eval runs on the host under the Claude subscription login, which is capacity, not a billing account: reaching a limit stops work and bills nothing (owner standing direction, s3). Since #12511, PAID_EVAL routes to the host queue unless a billing account is named, and none is.
Basis: #12511 (merged); owner standing direction s3.
Host run. I ran a before-and-after claude plugin eval of the interview skill text over 6 cases, comparing #12321's merge base with main. The after arm loses nothing: it scores at or above the before arm in every case.
What ran
Claude Code 2.1.316 on the subscription login, 2026-11-03 00:55Z to 01:29Z. Both arms ran at once (2 suites, -j 2 each) under the heavy-run lock.
Flags: --runs 3 -j 2 --scaffold --trust-plugin --keep-temp --no-publish --allow-tools Bash Write Edit WebFetch --judge-model opus, with the default with/without ablation. Each suite ran in a capped systemd scope, with an empty HOME, CLAUDE_CONFIG_DIR set to the subscription config, a short on-disk TMPDIR outside any checkout, and GH_TOKEN/GITHUB_TOKEN/DOCKER_CONFIG unset.
Cases: the 6 interview-* cases that exercise the register, gate statuses, constraints and recommendations, which is the text feat(planning): show a recommended answering order above an orderThreshold of open questions #12321 changed (SKILL.md Step 3 gate, loop.md Re-check and the gate's exit list). Each case is byte-identical in both arms: discuss-sets-discussing, recommendation-basis-and-flip, final-confirm-active-reply, nfr-quality-requirement, intent-asked-recorded and negative-recommendation-yes-accepts.
Loss was defined before the run. A case loses if its after with-arm mean is more than 0.34 below the before mean (one run's worth at n=3). The suite loses if the after with-arm mean over the 6 cases is more than 0.10 below the before mean. Only results that both arms got VALID count.
Validity: run-validity.py --runs 3 --scaffolded.
The after arm was VALID: 36 of 36 traces, no denials, no run errors.
The before arm was INVALID on one with-arm run of recommendation-basis-and-flip that hit its 12-turn cap; that pass scored 0.67. I reran that case alone on the before arm, and the rerun was VALID with 6 of 6 traces. Its scores replace the first pass's. The verdict does not depend on the rerun: over the 5 cases that ran VALID in both arms on the first pass, the means are 0.84 before and 0.91 after.
The tested text loaded, and it differs between the arms. Every case warns that the skill fired in 0 with-arm runs, because these cases have no skill-fired grader. feat(planning): show a recommended answering order above an orderThreshold of open questions #12321 added stale=<n> to the Step 3 gate paragraph. That string is in 10 of the 18 after-arm with-arm traces and in 0 of the 21 before-arm ones (18 first pass plus 3 rerun). The Step 3 heading "Recognize the stop condition" is in 15 of 18 after and 18 of 21 before.
Results (with-arm mean, n=3)
Case
Before
After
Without (before / after)
discuss-sets-discussing
1.00
1.00
1.00 / 1.00
final-confirm-active-reply
0.56
0.56
0.89 / 1.00
intent-asked-recorded
0.67
1.00
0.00 / 0.00
negative-recommendation-yes-accepts
1.00
1.00
0.33 / 0.44
nfr-quality-requirement
1.00
1.00
0.00 / 0.00
recommendation-basis-and-flip
1.00 (rerun)
1.00
0.17 / 0.17
Mean
0.87
0.93
No case is lower after, and the suite mean rises by 0.06, so the done-when ("a before-and-after run of the interview skill text shows no loss") is met for the changed text.
Not covered
The other 34 interview-* cases did not run. The 6 above were chosen for their relevance to the changed sections.
Activity
fix(planning): show only unanswered questions under Open and keep pos…
kyle-sexton commentedon Oct 4, 2026
This was generated by AI during triage.
Routing: human-gated,
status: needs-decision,work-class: scoped. The direction is clear, but several design forks are open, part of the request already exists, and one part touches a mechanism that is parked on purpose. It is also wide enough (skill text, page, server, ledger sync, gate) that it likely needs/work-items:decomposeinto slices once the forks are settled.What already exists:
Checked against:line, and a new confirmed constraint is meant to re-check every open recommendation before the next round (instruction only; nothing on the page or in the gate enforces it).Basis: verified, plugins/planning/skills/interview/context/loop.md:84-85
affectsand its direct dependents that hold a decision asUpstream changed, and answering again clears it. That covers most of gap 4 whenaffectsis filled.Basis: verified, plugins/planning/skills/interview/context/surface.md:87,157
Basis: verified, plugins/planning/surface/DEFERRED.md:23
Basis: verified, plugins/planning/scripts/check-open-questions.sh:20-27
Open decisions:
Checked against:text.Upstream changedstate or is a separate state, and whether the gate refusing on it applies to answered rows as well as open ones.Agent Brief
Type: Feature
Summary: When a constraint is added mid-interview, show which recommendations and answers it may invalidate, and keep the interview from locking until they are resolved.
Current behavior:
Adding a constraint relies on Claude re-checking open recommendations by instruction. The page shows no round-level current or stale state, no answering order for a large frontier, and the gate passes regardless.
Desired behavior:
Acceptance criteria:
Out of scope:
Priority: medium (assessed default).
kyle-sexton commentedon Oct 18, 2026
Decision recorded.
Question: When a constraint is added mid-interview, answers not yet checked against it should be flagged, and the gate should refuse while any are unresolved. Today this is an instruction only, and the gate has no notion of stale. Options: (a) store a structured "checked against" list per question, reuse the existing "Upstream changed" state, and gate open and answered rows; (b) parse the "Checked against:" text and gate open rows only; (c) fold into #5796.
Owner's answer: (a) B cannot work on the page, which renders constraints in words rather than ids. Basis: interview SKILL.md:357; server.py:275-286; check-open-questions.sh:15-30.
The owner accepted this recommendation as item D4 of a 32-decision batch in an attended backlog-shaping session on 2026-10-18. The batch question: "Accept all 32 recommendations, including the four flagged above (D2 close the relearn, D26 cross-repo sync without per-repo confirm, D10 narrow hook, D23 delete guard only)?" The owner answered: "Accept all 32 (Recommended)".
Posted by AI from the owner's answer in that session.
fix(planning): track proposals and read the stale mark by escape parity
fix(planning): keep a constraint-marked answer on Claude and name the…
39 remaining items
kyle-sexton commentedon Nov 2, 2026
Decision recorded.
(agent; not an owner decision)
Re-admit. This issue's eval runs on the host under the Claude subscription login, which is capacity, not a billing account: reaching a limit stops work and bills nothing (owner standing direction, s3). Since #12511, PAID_EVAL routes to the host queue unless a billing account is named, and none is.
Basis: #12511 (merged); owner standing direction s3.
kyle-sexton commentedon Nov 3, 2026
Claim. Backlog HITL host-run worker is taking this now.
kyle-sexton commentedon Nov 3, 2026
Host run. I ran a before-and-after
claude plugin evalof the interview skill text over 6 cases, comparing #12321's merge base with main. The after arm loses nothing: it scores at or above the before arm in every case.What ran
-j 2each) under the heavy-run lock.plugins/planningat4b042c64b83c1750a3fba5552eb30252a051426c(feat(planning): show a recommended answering order above an orderThreshold of open questions #12321's parent). After arm:plugins/planningat origin/mainbacba0349c84ee22bebdfe017212677f8b3df235. Each arm was a freshgit archiveexport outside any git checkout.--runs 3 -j 2 --scaffold --trust-plugin --keep-temp --no-publish --allow-tools Bash Write Edit WebFetch --judge-model opus, with the default with/without ablation. Each suite ran in a capped systemd scope, with an emptyHOME,CLAUDE_CONFIG_DIRset to the subscription config, a short on-disk TMPDIR outside any checkout, andGH_TOKEN/GITHUB_TOKEN/DOCKER_CONFIGunset.interview-*cases that exercise the register, gate statuses, constraints and recommendations, which is the text feat(planning): show a recommended answering order above an orderThreshold of open questions #12321 changed (SKILL.md Step 3 gate, loop.md Re-check and the gate's exit list). Each case is byte-identical in both arms: discuss-sets-discussing, recommendation-basis-and-flip, final-confirm-active-reply, nfr-quality-requirement, intent-asked-recorded and negative-recommendation-yes-accepts.run-validity.py --runs 3 --scaffolded.recommendation-basis-and-flipthat hit its 12-turn cap; that pass scored 0.67. I reran that case alone on the before arm, and the rerun was VALID with 6 of 6 traces. Its scores replace the first pass's. The verdict does not depend on the rerun: over the 5 cases that ran VALID in both arms on the first pass, the means are 0.84 before and 0.91 after.stale=<n>to the Step 3 gate paragraph. That string is in 10 of the 18 after-arm with-arm traces and in 0 of the 21 before-arm ones (18 first pass plus 3 rerun). The Step 3 heading "Recognize the stop condition" is in 15 of 18 after and 18 of 21 before.Results (with-arm mean, n=3)
No case is lower after, and the suite mean rises by 0.06, so the done-when ("a before-and-after run of the interview skill text shows no loss") is met for the changed text.
Not covered
interview-*cases did not run. The 6 above were chosen for their relevance to the changed sections.final-confirm-active-replyscores 0.56 with the plugin against 0.89 to 1.00 without it, in both arms.Target labels: add none; remove host-run.