Skip to content

Fix large payload middleware: populate CallToolResult.Content field and add extensive logging#766

Merged
lpcox merged 5 commits intomainfrom
copilot/fix-large-payload-storage
Feb 6, 2026
Merged

Fix large payload middleware: populate CallToolResult.Content field and add extensive logging#766
lpcox merged 5 commits intomainfrom
copilot/fix-large-payload-storage

Conversation

Copy link
Contributor

Copilot AI commented Feb 6, 2026

Problem

Large payload middleware saves payloads to disk and generates metadata (queryID, payloadPath, schema) but clients receive full payloads without metadata. Test run 21762627709 showed 672KB response passed through unchanged.

Root Cause

Middleware returned transformed response in second return value, which gets discarded when wrapping for MCP SDK:

// Middleware (jqschema.go)
return result, rewrittenResponse, nil  // metadata in second param

// SDK wrapper (unified.go:363)
result, _, err := handler(ctx, req, nil)  // discarded with _
return result, err

MCP protocol sends only CallToolResult.Content to clients. Second return value is internal-only.

Changes

Modified internal/middleware/jqschema.go:

  • Create new CallToolResult with transformed response in Content field
  • Marshal metadata to JSON and wrap in TextContent
  • Second return value retained for internal use
  • Added extensive logging for payload storage and filesystem operations
rewrittenJSON, _ := json.Marshal(rewrittenResponse)
transformedResult := &sdk.CallToolResult{
    Content: []sdk.Content{
        &sdk.TextContent{Text: string(rewrittenJSON)},
    },
    IsError: result.IsError,
    Meta:    result.Meta,
}
return transformedResult, rewrittenResponse, nil

Enhanced Logging:

  • Payload directory creation with full paths and permissions (0700)
  • File write operations with size in bytes, KB, and MB
  • File verification after write with mode check (0600)
  • Processing flow: tool call start, skip reasons, marshaling, schema transformation
  • Client response details with path translation hints
  • Dual logging strategy: debug console output (logMiddleware.Printf) and operational file logging (logger.LogInfo/LogDebug/LogWarn/LogError)

Updated tests:

  • Verify Content field contains transformed response with metadata
  • Integration tests confirm end-to-end flow

Result

Clients now receive:

{
  "queryID": "abc123...",
  "payloadPath": "/tmp/jq-payloads/session/query/payload.json",
  "preview": "first 500 chars...",
  "schema": {...},
  "originalSize": 672508,
  "truncated": true
}

The enhanced logging provides comprehensive visibility into the entire payload lifecycle, making it easier to diagnose storage issues and track filesystem operations in both development and production environments.

Original prompt

This section details on the original issue you should resolve

<issue_title>[large-payload-test] Large Payload Test - 21762627709</issue_title>
<issue_description># Large MCP Payload Access Test Results

Run ID: 21762627709
Status: PARTIAL PASS
Timestamp: 2026-02-06T19:10:21Z

Test Results

  • Expected Secret: test-secret-2786167c-221f-4a0f-a034-f88ef8394279
  • Found Secret: test-secret-2786167c-221f-4a0f-a034-f88ef8394279
  • Secret Match: YES ✅
  • Payload Path: NOT PROVIDED (feature not triggered)
  • Payload Size: 672,508 bytes (656.7 KB)

Detailed Findings

✅ What Worked

  1. Successfully read secret.txt from filesystem MCP server
  2. Successfully read large-test-file.json (672KB file)
  3. Secret validation passed - secrets match correctly
  4. File content integrity verified

⚠️ What Failed

  1. Large payload storage not triggered: Despite 672KB response, MCP Gateway did not store payload to /tmp/jq-payloads/{sessionID}/{queryID}/payload.json
  2. No payload metadata returned: Response did not include payloadPath, queryID, or other expected metadata fields
  3. No stored payload files found: Search for payload.json files in /tmp returned no results

Conclusion

The filesystem MCP server is functioning correctly and can read large files. Secret verification passed. However, the MCP Gateway's large payload storage feature is not being activated despite the response size (672KB) exceeding typical thresholds. The gateway appears to be passing through the full content directly rather than storing it and returning metadata.

This suggests either:

  • The jq middleware is not enabled/configured correctly
  • The payload size threshold is set higher than 672KB
  • The feature is not triggered for filesystem MCP server responses

Run URL: https://github.com/github/gh-aw-mcpg/actions/runs/21762627709

AI generated by Large Payload Tester

<agent_instructions>Help diagnose why the fully large payload was sent by the gateway instead of sharing it through a common payloadDir path</agent_instructions>

Comments on the Issue (you are @copilot in this section)

Custom agent used: agentic-workflows
GitHub Agentic Workflows (gh-aw) - Create, debug, and upgrade AI-powered workflows with intelligent prompt routing


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 2 commits February 6, 2026 19:26
The middleware was creating the transformed response (with queryID, payloadPath, schema, etc.)
but only returning it in the second return value. When the handler was wrapped for the SDK,
this value was discarded with `_`, so the client never received the metadata.

Now the middleware creates a new CallToolResult with the transformed response in the Content field,
which is what gets sent to the client via the MCP protocol.

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
@lpcox
Copy link
Collaborator

lpcox commented Feb 6, 2026

@copilot make sure that there is extensive logging around large payloads and sharing them through the file system

Enhanced integration tests to verify that the transformed response (with queryID,
payloadPath, schema, etc.) is properly included in the CallToolResult.Content field,
not just in the internal data return value.

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix large payload storage issue in MCP test Fix large payload middleware: populate CallToolResult.Content field Feb 6, 2026
Copilot AI requested a review from lpcox February 6, 2026 19:31
…ions

Enhanced middleware logging to provide detailed visibility into:
- Payload directory creation and verification
- File write operations with size and permissions
- Payload size in multiple units (bytes, KB, MB)
- Metadata generation and client response details
- Path information for accessing payloads inside containers
- Error conditions with full context

Added both debug logs (logMiddleware.Printf) and operational logs
(logger.LogInfo/LogDebug/LogWarn/LogError) to ensure comprehensive
tracking in both console output and log files.

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 6, 2026

@copilot make sure that there is extensive logging around large payloads and sharing them through the file system

Added extensive logging in commit 27dbad2. The middleware now logs:

Payload Storage:

  • Directory creation with full paths and permissions (0700)
  • File write operations with size in bytes, KB, and MB
  • File verification after write with mode check (0600)

Processing Flow:

  • Tool call processing start with base directory
  • Reasons for skipping payload storage (errors, nil results)
  • JSON marshaling with payload size
  • Schema transformation completion

Client Response:

  • Metadata response creation with all fields
  • Path information: gateway path and container path (/workspace/mcp-payloads/{session}/{query}/payload.json)
  • Confirmation that client receives metadata

All logs use both debug console output (logMiddleware.Printf) and operational file logging (logger.LogInfo/LogDebug/LogWarn/LogError) for comprehensive tracking.

@lpcox lpcox marked this pull request as ready for review February 6, 2026 19:35
Copilot AI review requested due to automatic review settings February 6, 2026 19:35
@lpcox lpcox merged commit 71e75db into main Feb 6, 2026
3 checks passed
@lpcox lpcox deleted the copilot/fix-large-payload-storage branch February 6, 2026 19:35
Copilot AI restored the copilot/fix-large-payload-storage branch February 6, 2026 19:35
Copilot AI changed the title Fix large payload middleware: populate CallToolResult.Content field Fix large payload middleware: populate CallToolResult.Content field and add extensive logging Feb 6, 2026
Copilot stopped work on behalf of lpcox due to an error February 6, 2026 19:35
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes the large payload middleware so MCP clients actually receive payload metadata by populating CallToolResult.Content (the only part of the tool result sent over the protocol), and updates tests to validate this behavior.

Changes:

  • Populate sdk.CallToolResult.Content with JSON-encoded metadata (queryID, payloadPath, schema, preview, originalSize, truncated) as TextContent.
  • Add extensive operational/debug logging around payload persistence and metadata generation.
  • Update unit + integration tests to assert metadata is present in result.Content (and still returned via the internal data value).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File Description
internal/middleware/jqschema.go Returns rewritten metadata via CallToolResult.Content and adds detailed payload lifecycle logging.
internal/middleware/jqschema_test.go Updates unit tests to parse/validate metadata from result.Content in addition to data.
internal/middleware/jqschema_integration_test.go Updates integration tests to validate end-to-end metadata delivery through result.Content.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 286 to +289
logMiddleware.Printf("Rewritten response: tool=%s, queryID=%s, sessionID=%s, originalSize=%d, truncated=%v",
toolName, queryID, sessionID, len(payloadJSON), len(payloadStr) > 500)
toolName, queryID, sessionID, len(payloadJSON), truncated)
logger.LogInfo("payload", "Created metadata response for client: tool=%s, queryID=%s, session=%s, payloadPath=%s, originalSize=%d bytes, truncated=%v",
toolName, queryID, sessionID, filePath, len(payloadJSON), truncated)
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If savePayload fails, filePath will be empty but the middleware still builds/returns client metadata and logs a payloadPath. That can mislead clients into trying to read a non-existent file. Consider only returning/advertising payloadPath when saveErr == nil (or explicitly include storage failure info in the metadata).

See below for a potential fix:

			"preview":      preview,
			"schema":       schemaJSON,
			"originalSize": len(payloadJSON),
			"truncated":    truncated,
		}

		// Only advertise payloadPath when we actually have a stored payload
		if filePath != "" {
			rewrittenResponse["payloadPath"] = filePath
		}

		logMiddleware.Printf("Rewritten response: tool=%s, queryID=%s, sessionID=%s, originalSize=%d, truncated=%v",
			toolName, queryID, sessionID, len(payloadJSON), truncated)
		if filePath != "" {
			logger.LogInfo("payload", "Created metadata response for client: tool=%s, queryID=%s, session=%s, payloadPath=%s, originalSize=%d bytes, truncated=%v",
				toolName, queryID, sessionID, filePath, len(payloadJSON), truncated)
		} else {
			logger.LogInfo("payload", "Created metadata response for client without payloadPath: tool=%s, queryID=%s, session=%s, originalSize=%d bytes, truncated=%v",
				toolName, queryID, sessionID, len(payloadJSON), truncated)
		}

Copilot uses AI. Check for mistakes.
Comment on lines +131 to +135
logger.LogInfo("payload", "Writing large payload to filesystem: path=%s, size=%d bytes (%.2f KB, %.2f MB)",
filePath, payloadSize, float64(payloadSize)/1024, float64(payloadSize)/(1024*1024))

if err := os.WriteFile(filePath, payload, 0600); err != nil {
logger.LogError("payload", "Failed to write payload file: path=%s, size=%d bytes, error=%v",
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These LogInfo messages are emitted for every wrapped tool call (the middleware saves all payloads), and the underlying file logger fsyncs on each log write. This combination can significantly increase I/O and latency in production. Consider downgrading most of these messages to debug (or sampling), and avoid wording like "large payload" unless there is an actual size threshold.

Copilot uses AI. Check for mistakes.
Comment on lines +143 to +150
// Verify file was written correctly
if stat, err := os.Stat(filePath); err != nil {
logger.LogWarn("payload", "Could not verify payload file after write: path=%s, error=%v", filePath, err)
} else {
logger.LogDebug("payload", "Payload file verified: path=%s, size=%d bytes, mode=%s",
filePath, stat.Size(), stat.Mode())
}

Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The post-write verification does an extra os.Stat() on the hot path. Combined with always-on payload writes, this adds filesystem overhead without affecting correctness. Consider guarding this behind debug logging (or removing it) to reduce unnecessary I/O.

Suggested change
// Verify file was written correctly
if stat, err := os.Stat(filePath); err != nil {
logger.LogWarn("payload", "Could not verify payload file after write: path=%s, error=%v", filePath, err)
} else {
logger.LogDebug("payload", "Payload file verified: path=%s, size=%d bytes, mode=%s",
filePath, stat.Size(), stat.Mode())
}

Copilot uses AI. Check for mistakes.
Comment on lines +325 to +326
logger.LogInfo("payload", "Client can access full payload at: %s (inside container: /workspace/mcp-payloads/%s/%s/payload.json)",
filePath, sessionID, queryID)
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This log line hardcodes an agent/container path (/workspace/mcp-payloads/...) that depends on workflow-specific mounts and may be incorrect in most deployments. Consider removing the hardcoded path, or deriving/logging it from configuration so operational logs don’t mislead users about where the payload is accessible.

Copilot uses AI. Check for mistakes.
assert.Contains(t, contentMap, "truncated", "Content should contain truncated")

// Verify queryID format in Content
queryIDFromContent := contentMap["queryID"].(string)
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests use direct type assertions on values pulled from a map resulting from json.Unmarshal (e.g., contentMap["queryID"].(string)). If the key is missing or the type changes, the test will panic and produce a less actionable failure. Prefer require.Contains/require.True checks (or require.IsType) before asserting types/values.

Suggested change
queryIDFromContent := contentMap["queryID"].(string)
queryIDValue, ok := contentMap["queryID"]
require.True(t, ok, "Content queryID should be present")
queryIDFromContent, ok := queryIDValue.(string)
require.True(t, ok, "Content queryID should be a string")

Copilot uses AI. Check for mistakes.
Comment on lines +244 to +252
assert.True(t, contentMap["truncated"].(bool), "Should indicate truncation in Content")
preview := contentMap["preview"].(string)
assert.LessOrEqual(t, len(preview), 503, "Preview should be truncated to ~500 chars + '...'")
assert.True(t, strings.HasSuffix(preview, "..."), "Preview should end with '...'")

// Also verify in data return value
dataMap, ok := data.(map[string]interface{})
require.True(t, ok, "Data should be a map")
assert.True(t, dataMap["truncated"].(bool), "Should indicate truncation in data")
Copy link

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test directly type-asserts fields from contentMap (e.g., contentMap["truncated"].(bool), contentMap["preview"].(string)). If the JSON shape changes, the test will panic instead of failing with a clear assertion message. Consider asserting the keys exist and types are expected before using them.

Suggested change
assert.True(t, contentMap["truncated"].(bool), "Should indicate truncation in Content")
preview := contentMap["preview"].(string)
assert.LessOrEqual(t, len(preview), 503, "Preview should be truncated to ~500 chars + '...'")
assert.True(t, strings.HasSuffix(preview, "..."), "Preview should end with '...'")
// Also verify in data return value
dataMap, ok := data.(map[string]interface{})
require.True(t, ok, "Data should be a map")
assert.True(t, dataMap["truncated"].(bool), "Should indicate truncation in data")
truncatedVal, ok := contentMap["truncated"]
require.True(t, ok, "Content JSON should contain 'truncated' field")
truncated, ok := truncatedVal.(bool)
require.True(t, ok, "'truncated' field in Content JSON should be a bool")
assert.True(t, truncated, "Should indicate truncation in Content")
previewVal, ok := contentMap["preview"]
require.True(t, ok, "Content JSON should contain 'preview' field")
preview, ok := previewVal.(string)
require.True(t, ok, "'preview' field in Content JSON should be a string")
assert.LessOrEqual(t, len(preview), 503, "Preview should be truncated to ~500 chars + '...'")
assert.True(t, strings.HasSuffix(preview, "..."), "Preview should end with '...'")
// Also verify in data return value
dataMap, ok := data.(map[string]interface{})
require.True(t, ok, "Data should be a map")
dataTruncatedVal, ok := dataMap["truncated"]
require.True(t, ok, "Data JSON should contain 'truncated' field")
dataTruncated, ok := dataTruncatedVal.(bool)
require.True(t, ok, "'truncated' field in data JSON should be a bool")
assert.True(t, dataTruncated, "Should indicate truncation in data")

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[large-payload-test] Large Payload Test - 21762627709

3 participants