Skip to content

feat: add client conformance tests for SEP-2575#270

Open
anubhav756 wants to merge 2 commits into
modelcontextprotocol:mainfrom
anubhav756:feat/stateless-client-tests
Open

feat: add client conformance tests for SEP-2575#270
anubhav756 wants to merge 2 commits into
modelcontextprotocol:mainfrom
anubhav756:feat/stateless-client-tests

Conversation

@anubhav756
Copy link
Copy Markdown
Contributor

@anubhav756 anubhav756 commented May 12, 2026

Adds client-side conformance tests for stateless MCP SEP-2575.

Tests are fully aligned with the official traceability requirements in PR #273 (src/seps/sep-2575.yaml).

Note

This is a companion PR to the server-side tests in #271.


Context

SEP-2575 removes the stateful initialize handshake and requires carrying protocol version, implementation info, and capabilities on a per-request basis in the _meta parameters and HTTP headers.

This PR introduces the request-metadata client-side conformance scenario to verify that client implementations correctly follow these new stateless rules, incorporating all feedback from review #270.


Changes

1. The request-metadata Client Scenario

Adds the new request-metadata scenario to verify per-request version headers, matching _meta payload specifications, and client capability obligations:

  • Optional Capability Conditional Checks:
    • Roots, sampling, and elicitation are optional client capabilities. A specialized client is 100% conformant even if it only implements the tools capability and omits the others.
    • Mock Server (src/scenarios/client/request-metadata.ts):
      • If present: verifies they are formatted as valid JSON objects {} (SUCCESS/FAILURE).
      • If absent: records SKIPPED (vacuously conformant).
  • Targeted Reference Client Assertions:
    • Inside the unit tests, we explicitly assert that the comprehensive reference client (everything-client.ts) declarations resolve to SUCCESS (not SKIPPED). This guarantees that we actively test the happy path of all optional capabilities in our reference implementation while keeping optionals skipped-friendly for specialized SDKs.

2. Simulated Version Negotiation Retry

The mock server simulates a version negotiation flow to assert the ClientRetrySupportedVersion check (severity WARNING / expectedFailureSlugs compatible):

  • On the first request, the mock server deliberately rejects it with 400 Bad Request and UnsupportedProtocolVersionError (code -32001) carrying data: { supported: ['DRAFT-2026-v1'] }.
  • The client must successfully negotiate and retry using the mutually supported version, updating the check status to SUCCESS.

How Has This Been Tested?

All 133 unit tests are fully green, warning-free, and compile cleanly using npm test and npm run check.

E2E Verification Executions:

1. TypeScript Reference Client

npm start -- client --spec-version draft --scenario request-metadata --command "npx tsx examples/clients/typescript/everything-client.ts"

Result: Passed: 7/7, 0 failed, 0 warnings (✅ OVERALL: PASSED)

  • Correctly asserts SUCCESS for all optional capabilities and version negotiation retry checks.

32 Negative Unit Tests (request-metadata.test.ts)

Added extensive negative cases verifying compliance checks correctly trap clients that:

  • Omit the _meta parameter (sep-2575-client-populates-meta $\rightarrow$ FAILURE).
  • Omit the MCP-Protocol-Version HTTP header (sep-2575-http-client-sends-version-header $\rightarrow$ FAILURE).
  • Disagree between headers and _meta.protocolVersion (sep-2575-http-version-header-matches-meta $\rightarrow$ FAILURE).
  • Fail to negotiate or retry on 400 rejections (sep-2575-client-retry-supported-version $\rightarrow$ WARNING).
  • Exit cleanly without looping/hanging on empty version intersections.
  • Send invalid (non-object) capabilities (roots: "string" $\rightarrow$ FAILURE).

Breaking Changes

None.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

Additional context

None.

@pkg-pr-new
Copy link
Copy Markdown

pkg-pr-new Bot commented May 12, 2026

Open in StackBlitz

npx https://pkg.pr.new/@modelcontextprotocol/conformance@270

commit: f560b10

@anubhav756 anubhav756 force-pushed the feat/stateless-client-tests branch from cd07785 to 4592cc7 Compare May 12, 2026 15:08
@anubhav756 anubhav756 changed the title feat: add MVP stateless client conformance tests for SEP-2575 feat: add client conformance tests for SEP-2575 May 13, 2026
Copy link
Copy Markdown
Member

@pcarleton pcarleton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling this. SEP-2575 is a big one. I ran /new-sep 2575 to build the full requirement-traceability YAML and opened it as #273 so we have a shared reference for both this PR and #271.

Coverage: the YAML has 12 client-side check: rows; this PR currently covers 3 (client-populates-meta, the version-header pair). The 5 stdio-only rows can be a follow-up scenario, but the 3 client-declares-{elicitation,roots,sampling}-capability MUSTs and the client-retry-supported-version SHOULD are HTTP-reachable and would fit nicely here.

Correctness bits:

  • client-consistent-version — I couldn't find spec backing for this. SEP-2575 makes each request self-contained; nothing forbids a client from changing protocolVersion between requests. The flippingVersionClient negative test would fail a conformant client. Suggest dropping.
  • client-cancels-by-notification — spec scopes notifications/cancelled to stdio only ("Streamable HTTP: …No notifications/cancelled message is required or expected"). This scenario is HTTP, so the check can't legitimately fire here.
  • server/discover gap — the example's discover call (everything-client.ts:98) sends no _meta and no MCP-Protocol-Version header, but the harness marks it as passing. client-sends-version-header is guarded by if (currentVersion) and client-populates-meta runs after the discover early-return. I think we want the example to send the header and the check order to catch it if an example doesn't do it.
  • Check IDs — we're shifting to check ids like sep-<NNNN>-<slug> (see sep-2164-error-code in resources.ts); renaming to match #273's slugs will let the traceability tooling line them up automatically.
  • specReferences URLs are empty — #273 has the per-row anchors you can lift.

Copy link
Copy Markdown
Member

@pcarleton pcarleton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(See above)

@pcarleton
Copy link
Copy Markdown
Member

Pushed the review fixes as a stacked PR you can merge into this branch: anubhav756#1 (one commit per item above, plus the sep-2575.yaml traceability file and a rename to request-metadata).

@anubhav756 anubhav756 force-pushed the feat/stateless-client-tests branch from 18e3ecc to efd4f4f Compare May 15, 2026 20:25
@anubhav756 anubhav756 force-pushed the feat/stateless-client-tests branch from efd4f4f to f560b10 Compare May 15, 2026 21:13
@anubhav756 anubhav756 requested a review from pcarleton May 15, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants