Achieve 100% MCP Conformance Test Coverage

[avrabe]

Goal

Build a conformance test server that passes all 28 server conformance scenarios from @modelcontextprotocol/conformance.

Current Status

• 4/28 scenarios passing (14%): server-initialize, tools-list, resources-list, prompts-list
• Protocol implementation complete: All 14 MCP methods implemented
• Gap: Missing test fixtures that conformance scenarios expect

Required Conformance Fixtures

Tools (11 scenarios)

The conformance suite expects specific tools with specific behaviors:

Scenario	Required Tool	Behavior
tools-call-simple-text	Returns text content	Basic tool call
tools-call-image	Returns base64 PNG	Image in content
tools-call-audio	Returns base64 WAV	Audio in content
tools-call-embedded-resource	Returns resource URI	Embedded resource
tools-call-mixed-content	Returns text + image + resource	Multiple content types
tools-call-with-logging	Sends log notifications during execution	notifications/message
tools-call-with-progress	Sends progress 0→100	notifications/progress
tools-call-error	Returns isError: true	Error handling
tools-call-sampling	Requests LLM response from client	sampling/createMessage
tools-call-elicitation	Requests user input	elicitation/create

Resources (6 scenarios)

Scenario	Required Resource	Behavior
resources-read-text	Text resource	Returns text content
resources-read-binary	Binary resource	Returns blob with mimeType
resources-templates-read	URI template /{id}/data	Parameter substitution
resources-subscribe	Subscribable resource	Accept subscription
resources-unsubscribe	—	Accept unsubscription

Prompts (5 scenarios)

Scenario	Required Prompt	Behavior
prompts-get-simple	test_simple_prompt	Basic prompt
prompts-get-with-args	Prompt with arg1, arg2	Argument substitution
prompts-get-embedded-resource	Prompt with resource	Embedded resource in message
prompts-get-with-image	Prompt with image	Base64 image in message

Other (4 scenarios)

Scenario	Requirement
logging-set-level	✅ Already passing
completion-complete	Return completions for refs
elicitation-sep1034-defaults	Handle elicitation with defaults
json-schema-2020-12	JSON Schema 2020-12 compliance

Implementation Plan

Phase 1: Create Conformance Test Server

Create examples/conformance-server/ with:

• All required tools with correct names and behaviors
• All required resources (text, binary, template)
• All required prompts
• Progress notification support
• Logging notification support
• Sampling request support
• Elicitation request support

Phase 2: Validate with HTTP Transport

• Run full conformance suite against SSE/HTTP endpoint
• Fix any protocol-level issues discovered
• Document any spec ambiguities found

Phase 3: CI Integration

• Add conformance tests to CI pipeline
• Require 100% pass rate for merges
• Generate conformance badge

Success Criteria

$ npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
Passed: 28/28, 0 failed, 0 warnings

References

• MCP Conformance Suite
• MCP Specification
• Issue stdio Transport Conformance Investigation Results #74 investigation results

[avrabe]

Progress Update (2025-12-13)

Current Status: 21/26 tests passing (81%)

We've made significant progress on conformance! Here's the breakdown:

✅ Passing (21 scenarios)

• server-initialize - Protocol version negotiation now works
• logging-set-level - Log level configuration
• completion-complete - Fixed ref field structure and CompletionValues type
• tools-list - Tool listing
• tools-call-simple-text - Text content
• tools-call-image - Fixed mimeType serialization
• tools-call-audio - Audio content
• tools-call-embedded-resource - Fixed Content::Resource structure
• tools-call-mixed-content - Multiple content types
• tools-call-error - Error handling
• resources-list - Resource listing
• resources-read-text - Fixed mimeType in ResourceContents
• resources-read-binary - Binary blob resources
• resources-templates-read - URI template substitution
• resources-subscribe - Subscription handling
• resources-unsubscribe - Unsubscription handling
• prompts-list - Fixed null field serialization
• prompts-get-simple - Basic prompts
• prompts-get-with-args - Argument substitution
• prompts-get-embedded-resource - Added PromptMessageContent::Resource variant
• prompts-get-with-image - Image in prompts

❌ Failing (5 scenarios) - Require Server-to-Client Communication

• tools-call-with-logging - Needs notifications/message
• tools-call-with-progress - Needs notifications/progress
• tools-call-sampling - Needs sampling/createMessage request to client
• tools-call-elicitation - Needs elicitation/create request to client
• elicitation-sep1034-defaults - Elicitation with defaults

Key Fixes Applied

1. Protocol Version Negotiation - Server now responds with client's version if supported
2. Content::Resource - Changed from string hack to proper EmbeddedResourceContents struct
3. mimeType serialization - Added #[serde(rename = "mimeType")] to Image, Audio, Resource types
4. CompletionRef - Created typed enum with ref/prompt and ref/resource variants
5. CompletionValues - Changed from Vec<CompletionInfo> to proper nested object structure
6. PromptMessageContent::Resource - Added resource variant for embedded resources in prompts
7. Optional field serialization - Added skip_serializing_if for Prompt fields

Remaining Work

The 5 remaining tests require bidirectional server-to-client communication - a significant architectural feature tracked in #87.

This includes:

• Server → Client notifications (logging, progress)
• Server → Client requests with response correlation (sampling, elicitation)
• Transport-level changes to support outbound messaging
• New ToolContext API for tool handlers

See #87 for the full implementation plan.