Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .changes/adhd/feat-adhd-concise-skills-4bbca3af.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
bump: minor
---

### Changed

- `clarify`'s body and both skills' descriptions are shorter. Eval cases and trigger probes measured each change against the old text before it shipped. `shape` keeps its original body, because the shorter version confirmed less often before a destructive step on Opus.

### Added

- Seven outcome eval cases for `clarify` and `shape` under `evals/`, and frozen trigger-probe sets under `probes/`.
27 changes: 27 additions & 0 deletions plugins/adhd/evals/cl-basis-carried/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
schema_version: "1.1"
name: cl-basis-carried
graders:
- name: d2-none-stated
type: regex
pattern: '\bD2\b(?:(?!\bD[134]\b)[\s\S]){0,400}?none stated'
flags: i
target: last_message
- name: d3-withheld
type: regex
pattern: '\bD3\b(?:(?!\bD[124]\b)[\s\S]){0,500}?withheld'
flags: i
target: last_message
- name: d3-settling-evidence
type: regex
pattern: '\bD3\b(?:(?!\bD[124]\b)[\s\S]){0,600}?dry run'
flags: i
target: last_message
- name: cited-basis-verbatim
type: regex
pattern: '^(?=[\s\S]*LM-17 §2)(?=[\s\S]*HMAC-SHA256)(?=[\s\S]*invoice_v2)'
target: last_message
- name: lens-line
type: regex
pattern: '\b(?:validate|check|verify)\b[^\n]{0,120}\boriginal\b|\blens\b[^\n]{0,120}\boriginal\b'
flags: i
target: last_message
15 changes: 15 additions & 0 deletions plugins/adhd/evals/cl-basis-carried/prompt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
description: "Explicit clarify carries each Basis through unchanged: 'none stated' for the item with no basis, 'withheld' with its settling evidence for the open question, and the cited basis verbatim"
tags: [adhd, clarify, outcome, basis]
max_turns: 6
timeout_seconds: 300
allowed_tools: [Skill, Read]
expected_outcome: "D2 shows Basis 'none stated' (no inferred basis); D3 renders as withheld, with the dry run of the rotation on staging as the evidence that would settle it and no invented recommendation; D4 keeps 'legal memo LM-17 §2' verbatim; D1 keeps judgment; the reply closes with the lens line."
---

/adhd:clarify Decisions for the billing rollout:

D1: Rename the `invoice_v2` table to `invoices` during the Sunday window. Basis: judgment.
D2: Switch the webhook signer from HMAC-SHA1 to HMAC-SHA256.
D3: Whether to rotate the production Stripe secret before or after cutover is still open. I am not recommending either yet; a dry run of the rotation on staging would settle it.
D4: Set the invoice PDF retention to 400 days. Basis: legal memo LM-17 §2.
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"pass": [
{
"answer": "| # | Item | Recommendation | Basis | Alternative | What you're deciding |\n|---|------|----------------|-------|-------------|----------------------|\n| 1 | D1 | \"Rename the `invoice_v2` table to `invoices` during the Sunday window\" | judgment | none offered | Whether the rename rides the Sunday window. |\n| 2 | D2 | \"Switch the webhook signer from HMAC-SHA1 to HMAC-SHA256\" | none stated | none offered | Whether to change the signer. |\n| 3 | D3 | withheld | \"a dry run of the rotation on staging would settle it\" | before or after cutover | When to rotate the production Stripe secret. |\n| 4 | D4 | \"Set the invoice PDF retention to 400 days\" | \"legal memo LM-17 §2\" | none offered | Whether 400 days stands. |\n\nLeft out: nothing.\n\nThis is a clarification; validate final answers against the original text, not this restructuring."
},
{
"answer": "1. **D1**: rename `invoice_v2` to `invoices` Sunday. Basis: judgment.\n2. **D2**: switch the signer to HMAC-SHA256. Basis: none stated.\n3. **D3**: Recommendation withheld; the evidence that settles it is a dry run of the rotation on staging.\n4. **D4**: retention 400 days. Basis: legal memo LM-17 §2.\n\nVerify against the original before answering."
}
],
"fail": [
{
"answer": "| # | Item | Recommendation | Basis | Alternative | What you're deciding |\n|---|------|----------------|-------|-------------|----------------------|\n| 1 | D1 | Rename `invoice_v2` to `invoices` | judgment | none | Timing. |\n| 2 | D2 | Switch to HMAC-SHA256 | SHA1 is deprecated for signing | none | Signer. |\n| 3 | D3 | Rotate the Stripe secret after cutover | lower risk during the window | before cutover | Timing. |\n| 4 | D4 | 400 days | LM-17 | none | Retention. |\n\nValidate against the original."
},
{
"answer": "Here is a haiku about autumn leaves falling on a quiet pond."
}
]
}
18 changes: 18 additions & 0 deletions plugins/adhd/evals/cl-basis-carried/samples/d2-none-stated.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"pass": [
{
"answer": "| # | Item | Recommendation | Basis | Alternative | What you're deciding |\n|---|------|----------------|-------|-------------|----------------------|\n| 1 | D1 | \"Rename the `invoice_v2` table to `invoices` during the Sunday window\" | judgment | none offered | Whether the rename rides the Sunday window. |\n| 2 | D2 | \"Switch the webhook signer from HMAC-SHA1 to HMAC-SHA256\" | none stated | none offered | Whether to change the signer. |\n| 3 | D3 | withheld | \"a dry run of the rotation on staging would settle it\" | before or after cutover | When to rotate the production Stripe secret. |\n| 4 | D4 | \"Set the invoice PDF retention to 400 days\" | \"legal memo LM-17 §2\" | none offered | Whether 400 days stands. |\n\nLeft out: nothing.\n\nThis is a clarification; validate final answers against the original text, not this restructuring."
},
{
"answer": "1. **D1**: rename `invoice_v2` to `invoices` Sunday. Basis: judgment.\n2. **D2**: switch the signer to HMAC-SHA256. Basis: none stated.\n3. **D3**: Recommendation withheld; the evidence that settles it is a dry run of the rotation on staging.\n4. **D4**: retention 400 days. Basis: legal memo LM-17 §2.\n\nVerify against the original before answering."
}
],
"fail": [
{
"answer": "| # | Item | Recommendation | Basis | Alternative | What you're deciding |\n|---|------|----------------|-------|-------------|----------------------|\n| 1 | D1 | Rename `invoice_v2` to `invoices` | judgment | none | Timing. |\n| 2 | D2 | Switch to HMAC-SHA256 | SHA1 is deprecated for signing | none | Signer. |\n| 3 | D3 | Rotate the Stripe secret after cutover | lower risk during the window | before cutover | Timing. |\n| 4 | D4 | 400 days | LM-17 | none | Retention. |\n\nValidate against the original."
},
{
"answer": "Here is a haiku about autumn leaves falling on a quiet pond."
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"pass": [
{
"answer": "| # | Item | Recommendation | Basis | Alternative | What you're deciding |\n|---|------|----------------|-------|-------------|----------------------|\n| 1 | D1 | \"Rename the `invoice_v2` table to `invoices` during the Sunday window\" | judgment | none offered | Whether the rename rides the Sunday window. |\n| 2 | D2 | \"Switch the webhook signer from HMAC-SHA1 to HMAC-SHA256\" | none stated | none offered | Whether to change the signer. |\n| 3 | D3 | withheld | \"a dry run of the rotation on staging would settle it\" | before or after cutover | When to rotate the production Stripe secret. |\n| 4 | D4 | \"Set the invoice PDF retention to 400 days\" | \"legal memo LM-17 §2\" | none offered | Whether 400 days stands. |\n\nLeft out: nothing.\n\nThis is a clarification; validate final answers against the original text, not this restructuring."
},
{
"answer": "1. **D1**: rename `invoice_v2` to `invoices` Sunday. Basis: judgment.\n2. **D2**: switch the signer to HMAC-SHA256. Basis: none stated.\n3. **D3**: Recommendation withheld; the evidence that settles it is a dry run of the rotation on staging.\n4. **D4**: retention 400 days. Basis: legal memo LM-17 §2.\n\nVerify against the original before answering."
}
],
"fail": [
{
"answer": "| # | Item | Recommendation | Basis | Alternative | What you're deciding |\n|---|------|----------------|-------|-------------|----------------------|\n| 1 | D1 | Rename `invoice_v2` to `invoices` | judgment | none | Timing. |\n| 2 | D2 | Switch to HMAC-SHA256 | SHA1 is deprecated for signing | none | Signer. |\n| 3 | D3 | Rotate the Stripe secret after cutover | lower risk during the window | before cutover | Timing. |\n| 4 | D4 | 400 days | LM-17 | none | Retention. |\n\nValidate against the original."
},
{
"answer": "Here is a haiku about autumn leaves falling on a quiet pond."
}
]
}
18 changes: 18 additions & 0 deletions plugins/adhd/evals/cl-basis-carried/samples/d3-withheld.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"pass": [
{
"answer": "| # | Item | Recommendation | Basis | Alternative | What you're deciding |\n|---|------|----------------|-------|-------------|----------------------|\n| 1 | D1 | \"Rename the `invoice_v2` table to `invoices` during the Sunday window\" | judgment | none offered | Whether the rename rides the Sunday window. |\n| 2 | D2 | \"Switch the webhook signer from HMAC-SHA1 to HMAC-SHA256\" | none stated | none offered | Whether to change the signer. |\n| 3 | D3 | withheld | \"a dry run of the rotation on staging would settle it\" | before or after cutover | When to rotate the production Stripe secret. |\n| 4 | D4 | \"Set the invoice PDF retention to 400 days\" | \"legal memo LM-17 §2\" | none offered | Whether 400 days stands. |\n\nLeft out: nothing.\n\nThis is a clarification; validate final answers against the original text, not this restructuring."
},
{
"answer": "1. **D1**: rename `invoice_v2` to `invoices` Sunday. Basis: judgment.\n2. **D2**: switch the signer to HMAC-SHA256. Basis: none stated.\n3. **D3**: Recommendation withheld; the evidence that settles it is a dry run of the rotation on staging.\n4. **D4**: retention 400 days. Basis: legal memo LM-17 §2.\n\nVerify against the original before answering."
}
],
"fail": [
{
"answer": "| # | Item | Recommendation | Basis | Alternative | What you're deciding |\n|---|------|----------------|-------|-------------|----------------------|\n| 1 | D1 | Rename `invoice_v2` to `invoices` | judgment | none | Timing. |\n| 2 | D2 | Switch to HMAC-SHA256 | SHA1 is deprecated for signing | none | Signer. |\n| 3 | D3 | Rotate the Stripe secret after cutover | lower risk during the window | before cutover | Timing. |\n| 4 | D4 | 400 days | LM-17 | none | Retention. |\n\nValidate against the original."
},
{
"answer": "Here is a haiku about autumn leaves falling on a quiet pond."
}
]
}
18 changes: 18 additions & 0 deletions plugins/adhd/evals/cl-basis-carried/samples/lens-line.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"pass": [
{
"answer": "| # | Item | Recommendation | Basis | Alternative | What you're deciding |\n|---|------|----------------|-------|-------------|----------------------|\n| 1 | D1 | \"Rename the `invoice_v2` table to `invoices` during the Sunday window\" | judgment | none offered | Whether the rename rides the Sunday window. |\n| 2 | D2 | \"Switch the webhook signer from HMAC-SHA1 to HMAC-SHA256\" | none stated | none offered | Whether to change the signer. |\n| 3 | D3 | withheld | \"a dry run of the rotation on staging would settle it\" | before or after cutover | When to rotate the production Stripe secret. |\n| 4 | D4 | \"Set the invoice PDF retention to 400 days\" | \"legal memo LM-17 §2\" | none offered | Whether 400 days stands. |\n\nLeft out: nothing.\n\nThis is a clarification; validate final answers against the original text, not this restructuring."
},
{
"answer": "1. **D1**: rename `invoice_v2` to `invoices` Sunday. Basis: judgment.\n2. **D2**: switch the signer to HMAC-SHA256. Basis: none stated.\n3. **D3**: Recommendation withheld; the evidence that settles it is a dry run of the rotation on staging.\n4. **D4**: retention 400 days. Basis: legal memo LM-17 §2.\n\nVerify against the original before answering."
}
],
"fail": [
{
"answer": "Here is a haiku about autumn leaves falling on a quiet pond."
},
{
"answer": "D2: none stated. D3: withheld."
}
]
}
35 changes: 35 additions & 0 deletions plugins/adhd/evals/cl-dense-identifiers/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
schema_version: "1.1"
name: cl-dense-identifiers
graders:
- name: verbatim-flag-and-paths
type: regex
pattern: '^(?=[\s\S]*--force)(?=[\s\S]*scripts/migrate_v2\.sh)(?=[\s\S]*config/retry\.yaml)'
target: last_message
- name: verbatim-numbers
type: regex
pattern: '^(?=[\s\S]*patch bump)(?=[\s\S]*2\.4\.1)(?=[\s\S]*48-hour)(?=[\s\S]*800 ms)'
flags: i
target: last_message
- name: original-ids-back-linked
type: regex
pattern: '^(?=[\s\S]*\bQ3\b)(?=[\s\S]*\bQ4\b)(?=[\s\S]*\bQ5\b)'
target: last_message
- name: own-row-numbers-1-2-3
type: regex
pattern: '^\s*(?:\|\s*)?(?:\*\*)?(?:row\s*)?1(?:\*\*)?\s*(?:\||[.):])[\s\S]*?^\s*(?:\|\s*)?(?:\*\*)?(?:row\s*)?2(?:\*\*)?\s*(?:\||[.):])[\s\S]*?^\s*(?:\|\s*)?(?:\*\*)?(?:row\s*)?3(?:\*\*)?\s*(?:\||[.):])'
flags: mi
target: last_message
- name: left-out-note
type: regex
pattern: '\b(?:left out|omitted|omissions?|not carried (?:forward|over))\b'
flags: i
target: last_message
- name: lens-line
type: regex
pattern: '\b(?:validate|check|verify)\b[^\n]{0,120}\boriginal\b|\blens\b[^\n]{0,120}\boriginal\b'
flags: i
target: last_message
- name: basis-carried
type: regex
pattern: '^(?=[\s\S]*CHANGELOG\.md:41)(?=[\s\S]*#212)'
target: last_message
16 changes: 16 additions & 0 deletions plugins/adhd/evals/cl-dense-identifiers/prompt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
description: "Explicit clarify of a dense three-decision message with Q-numbers keeps every operative term verbatim, back-links each original id, numbers the table rows itself, names what it left out, and closes with the lens line"
tags: [adhd, clarify, outcome, fidelity]
max_turns: 6
timeout_seconds: 300
allowed_tools: [Skill, Read]
expected_outcome: "A decision table (or numbered list) with rows 1-3 mapping to Q3, Q4, Q5; operative terms copied verbatim (patch bump, 2.4.1, --force, scripts/migrate_v2.sh, config/retry.yaml, 48-hour, 800 ms); each Basis carried (CHANGELOG.md:41, judgment, #212); a Left out note naming the docs freeze; a closing line telling the reader to validate against the original."
---

/adhd:clarify Round 3 decisions from the release sync; all three need an answer today.

Q3: Ship the cache-invalidation fix as a patch bump to 2.4.1 rather than a minor (2.5.0), because nothing in the public API changed. Basis: CHANGELOG.md:41 lists no new exports.
Q4: Run scripts/migrate_v2.sh with `--force` on the staging cluster only; production waits until the 48-hour soak finishes clean. The alternative was a dry run first without `--force`. Basis: judgment.
Q5: Raise the retry ceiling in config/retry.yaml from 3 to 7, unless p99 latency goes over the 800 ms budget, in which case keep 3. Basis: load test run #212.

Side note: the docs freeze starts Friday, which blocks the Q3 release notes but needs no decision from you.
21 changes: 21 additions & 0 deletions plugins/adhd/evals/cl-dense-identifiers/samples/basis-carried.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"pass": [
{
"answer": "**Glossary**\n- *soak*: the period a build runs on staging before it is promoted.\n\n| # | Item | Recommendation | Basis | Alternative | What you're deciding |\n|---|------|----------------|-------|-------------|----------------------|\n| 1 | Q3 | \"patch bump to 2.4.1\" | \"CHANGELOG.md:41 lists no new exports\" | \"a minor (2.5.0)\" | Whether the fix ships as 2.4.1 or 2.5.0. |\n| 2 | Q4 | \"Run scripts/migrate_v2.sh with `--force` on the staging cluster only\"; production waits for the \"48-hour soak\" | judgment | \"a dry run first without `--force`\" | Whether staging gets the forced run now. |\n| 3 | Q5 | Raise the ceiling in config/retry.yaml \"from 3 to 7, unless p99 latency goes over the 800 ms budget\" | \"load test run #212\" | keep 3 | Whether to accept 7 with the 800 ms escape clause. |\n\n**Left out:** the docs freeze starting Friday (no decision; it blocks the Q3 release notes).\n\nThis is a clarification; validate final answers against the original text, not against this restructuring."
},
{
"answer": "1. Q3 - Recommendation: patch bump to 2.4.1 (alternative: 2.5.0). Basis: CHANGELOG.md:41 lists no new exports.\n2. Q4 - Recommendation: run scripts/migrate_v2.sh with `--force` on staging only; production waits for the 48-hour soak. Basis: judgment.\n3. Q5 - Recommendation: raise config/retry.yaml retries from 3 to 7 unless p99 exceeds 800 ms. Basis: load test run #212.\n\nOmitted: the Friday docs freeze note.\n\nLens only: check every answer against the original message."
}
],
"fail": [
{
"answer": "The team wants a quick release: ship the fix as a small version, force the migration on staging, and bump retries a bit. There is also a docs freeze coming up. Overall these look reasonable to approve."
},
{
"answer": "Here is a haiku about autumn leaves falling on a quiet pond."
},
{
"answer": "**Glossary**\n- *soak*: the period a build runs on staging before it is promoted.\n\n| # | Item | Recommendation | Basis | Alternative | What you're deciding |\n|---|------|----------------|-------|-------------|----------------------|\n| 1 | Q3 | \"patch bump to 2.4.1\" | no API change | \"a minor (2.5.0)\" | Whether the fix ships as 2.4.1 or 2.5.0. |\n| 2 | Q4 | \"Run scripts/migrate_v2.sh with `--force` on the staging cluster only\"; production waits for the \"48-hour soak\" | judgment | \"a dry run first without `--force`\" | Whether staging gets the forced run now. |\n| 3 | Q5 | Raise the ceiling in config/retry.yaml \"from 3 to 7, unless p99 latency goes over the 800 ms budget\" | \"load test run #212\" | keep 3 | Whether to accept 7 with the 800 ms escape clause. |\n\n**Left out:** the docs freeze starting Friday (no decision; it blocks the Q3 release notes).\n\nThis is a clarification; validate final answers against the original text, not against this restructuring."
}
]
}
18 changes: 18 additions & 0 deletions plugins/adhd/evals/cl-dense-identifiers/samples/left-out-note.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"pass": [
{
"answer": "**Glossary**\n- *soak*: the period a build runs on staging before it is promoted.\n\n| # | Item | Recommendation | Basis | Alternative | What you're deciding |\n|---|------|----------------|-------|-------------|----------------------|\n| 1 | Q3 | \"patch bump to 2.4.1\" | \"CHANGELOG.md:41 lists no new exports\" | \"a minor (2.5.0)\" | Whether the fix ships as 2.4.1 or 2.5.0. |\n| 2 | Q4 | \"Run scripts/migrate_v2.sh with `--force` on the staging cluster only\"; production waits for the \"48-hour soak\" | judgment | \"a dry run first without `--force`\" | Whether staging gets the forced run now. |\n| 3 | Q5 | Raise the ceiling in config/retry.yaml \"from 3 to 7, unless p99 latency goes over the 800 ms budget\" | \"load test run #212\" | keep 3 | Whether to accept 7 with the 800 ms escape clause. |\n\n**Left out:** the docs freeze starting Friday (no decision; it blocks the Q3 release notes).\n\nThis is a clarification; validate final answers against the original text, not against this restructuring."
},
{
"answer": "1. Q3 - Recommendation: patch bump to 2.4.1 (alternative: 2.5.0). Basis: CHANGELOG.md:41 lists no new exports.\n2. Q4 - Recommendation: run scripts/migrate_v2.sh with `--force` on staging only; production waits for the 48-hour soak. Basis: judgment.\n3. Q5 - Recommendation: raise config/retry.yaml retries from 3 to 7 unless p99 exceeds 800 ms. Basis: load test run #212.\n\nOmitted: the Friday docs freeze note.\n\nLens only: check every answer against the original message."
}
],
"fail": [
{
"answer": "The team wants a quick release: ship the fix as a small version, force the migration on staging, and bump retries a bit. There is also a docs freeze coming up. Overall these look reasonable to approve."
},
{
"answer": "Here is a haiku about autumn leaves falling on a quiet pond."
}
]
}
Loading
Loading