feat: add client conformance tests for SEP-2575#270
Conversation
commit: |
cd07785 to
4592cc7
Compare
pcarleton
left a comment
There was a problem hiding this comment.
Thanks for tackling this. SEP-2575 is a big one. I ran /new-sep 2575 to build the full requirement-traceability YAML and opened it as #273 so we have a shared reference for both this PR and #271.
Coverage: the YAML has 12 client-side check: rows; this PR currently covers 3 (client-populates-meta, the version-header pair). The 5 stdio-only rows can be a follow-up scenario, but the 3 client-declares-{elicitation,roots,sampling}-capability MUSTs and the client-retry-supported-version SHOULD are HTTP-reachable and would fit nicely here.
Correctness bits:
client-consistent-version— I couldn't find spec backing for this. SEP-2575 makes each request self-contained; nothing forbids a client from changingprotocolVersionbetween requests. TheflippingVersionClientnegative test would fail a conformant client. Suggest dropping.client-cancels-by-notification— spec scopesnotifications/cancelledto stdio only ("Streamable HTTP: …Nonotifications/cancelledmessage is required or expected"). This scenario is HTTP, so the check can't legitimately fire here.server/discovergap — the example's discover call (everything-client.ts:98) sends no_metaand noMCP-Protocol-Versionheader, but the harness marks it as passing.client-sends-version-headeris guarded byif (currentVersion)andclient-populates-metaruns after the discover early-return. I think we want the example to send the header and the check order to catch it if an example doesn't do it.- Check IDs — we're shifting to check ids like
sep-<NNNN>-<slug>(seesep-2164-error-codein resources.ts); renaming to match #273's slugs will let the traceability tooling line them up automatically. specReferencesURLs are empty — #273 has the per-row anchors you can lift.
|
Pushed the review fixes as a stacked PR you can merge into this branch: anubhav756#1 (one commit per item above, plus the |
18e3ecc to
efd4f4f
Compare
efd4f4f to
f560b10
Compare
Adds client-side conformance tests for stateless MCP SEP-2575.
Tests are fully aligned with the official traceability requirements in PR #273 (
src/seps/sep-2575.yaml).Note
This is a companion PR to the server-side tests in #271.
Context
SEP-2575 removes the stateful
initializehandshake and requires carrying protocol version, implementation info, and capabilities on a per-request basis in the_metaparameters and HTTP headers.This PR introduces the
request-metadataclient-side conformance scenario to verify that client implementations correctly follow these new stateless rules, incorporating all feedback from review #270.Changes
1. The
request-metadataClient ScenarioAdds the new
request-metadatascenario to verify per-request version headers, matching_metapayload specifications, and client capability obligations:toolscapability and omits the others.src/scenarios/client/request-metadata.ts):{}(SUCCESS/FAILURE).SKIPPED(vacuously conformant).everything-client.ts) declarations resolve toSUCCESS(notSKIPPED). This guarantees that we actively test the happy path of all optional capabilities in our reference implementation while keeping optionals skipped-friendly for specialized SDKs.2. Simulated Version Negotiation Retry
The mock server simulates a version negotiation flow to assert the
ClientRetrySupportedVersioncheck (severityWARNING/expectedFailureSlugscompatible):400 Bad RequestandUnsupportedProtocolVersionError(code-32001) carryingdata: { supported: ['DRAFT-2026-v1'] }.SUCCESS.How Has This Been Tested?
All 133 unit tests are fully green, warning-free, and compile cleanly using
npm testandnpm run check.E2E Verification Executions:
1. TypeScript Reference Client
npm start -- client --spec-version draft --scenario request-metadata --command "npx tsx examples/clients/typescript/everything-client.ts"Result: Passed: 7/7, 0 failed, 0 warnings (✅ OVERALL: PASSED)
SUCCESSfor all optional capabilities and version negotiation retry checks.32 Negative Unit Tests (
request-metadata.test.ts)Added extensive negative cases verifying compliance checks correctly trap clients that:
_metaparameter (sep-2575-client-populates-metaFAILURE).MCP-Protocol-VersionHTTP header (sep-2575-http-client-sends-version-headerFAILURE)._meta.protocolVersion(sep-2575-http-version-header-matches-metaFAILURE).400rejections (sep-2575-client-retry-supported-versionWARNING).roots: "string"FAILURE).Breaking Changes
None.
Types of changes
Checklist
Additional context
None.