Add payloadDir field to MCP gateway spec for shared large payload access

[Copilot]

Add shared payload directory support to MCP gateway spec

This PR adds support for a shared directory field in the MCP gateway spec to allow large response payloads to be accessed by agents.

Changes Completed:

• Add PayloadDir field to MCPGatewayRuntimeConfig struct in pkg/workflow/tools_types.go
• Add DefaultMCPGatewayPayloadDir constant in pkg/workflow/mcp_gateway_constants.go (default: /tmp/gh-aw/res/)
• Update ensureDefaultMCPGatewayConfig to set default payload directory in pkg/workflow/mcp_gateway_config.go
• Update buildMCPGatewayConfig to include payload directory from user configuration
• Update gatewayConfig schema definition in pkg/workflow/schemas/mcp-gateway-config.schema.json
• Update public schema in docs/public/schemas/mcp-gateway-config.schema.json
• Update main workflow schema in pkg/parser/schemas/main_workflow_schema.json
• Add JSON tags to SandboxConfig, AgentSandboxConfig, and MCPGatewayRuntimeConfig for proper frontmatter parsing
• Implement payloadDir extraction in frontmatter_extraction_security.go
• Update MCP renderer in mcp_renderer.go to include payloadDir in gateway config output
• Add comprehensive tests for payloadDir field
• Fix test failure in TestCodexEngineRenderMCPConfig - updated expected output to include payloadDir
• Change default path from /tmp/jq-payloads to /tmp/gh-aw/res/ as requested
• Update MCP Gateway Specification documentation (v1.7.0 → v1.8.0) with comprehensive payloadDir field documentation
• Ensure payloadDir is created during gateway setup - added mkdir -p command in gateway setup step
• Add integration test to verify directory creation for both default and custom paths
• All tests passing

Testing:

• Created test workflows with default and custom payloadDir values
• Verified that the new default /tmp/gh-aw/res/ is correctly rendered in compiled workflows
• Verified that custom paths like /custom/payload/dir are correctly rendered
• Verified that mkdir -p command is generated in the gateway setup step for both default and custom paths
• All unit tests pass including the fixed TestCodexEngineRenderMCPConfig test
• Added integration test TestPayloadDirCreation to verify directory creation

Documentation:

• Updated MCP Gateway Specification to version 1.8.0
• Added comprehensive field documentation in Section 4.1.3 Gateway Configuration Fields table
• Updated all configuration examples throughout the specification (Section 4.1.1, 4.1.4, and Appendix A)
• Documented that the directory should be mounted as a shared volume between gateway and agent containers
• Followed W3C specification conventions and formatting

Implementation Details:

Directory Creation:
The compiler now ensures the payloadDir is created during the "Start MCP gateway" step with:

mkdir -p /tmp/gh-aw/res/  # for default
mkdir -p /custom/payload/dir  # for user-specified paths

Example Usage:

---
engine: copilot
sandbox:
  mcp:
    container: githubnext/gh-aw-mcpg
    version: latest
    port: 8080
    payloadDir: /custom/payload/dir  # optional, defaults to /tmp/gh-aw/res/
---

Compiled Output:
The gateway configuration includes:

"gateway": {
  "port": $MCP_GATEWAY_PORT,
  "domain": "${MCP_GATEWAY_DOMAIN}",
  "apiKey": "${MCP_GATEWAY_API_KEY}",
  "payloadDir": "/tmp/gh-aw/res/"
}

And the setup step creates the directory:

set -eo pipefail
mkdir -p /tmp/gh-aw/mcp-config
mkdir -p /tmp/gh-aw/res/  # payloadDir created here

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[lpcox]

@copilot please investigate why this is failing

go test -v -parallel=8 -timeout=3m -run='^Test' -tags '!integration' -coverprofile=coverage.out -json ./... > test-result-unit.json
go tool cover -html=coverage.out -o coverage.html
shell: /usr/bin/bash -e {0}
env:
GOTOOLCHAIN: local
Error: Process completed with exit code 1.

[Copilot]

@copilot please investigate why this is failing

go test -v -parallel=8 -timeout=3m -run='^Test' -tags '!integration' -coverprofile=coverage.out -json ./... > test-result-unit.json
...

Fixed in commit 7cfbf01. The test failure was caused by TestCodexEngineRenderMCPConfig expecting the old gateway configuration format without the payloadDir field. Updated the test's expected output to include the new field.

[pelikhan]

@copilot default path should be /tmp/gh-aw/res/

[Copilot]

@copilot default path should be /tmp/gh-aw/res/

Changed in commit bcc6247. Updated the default payloadDir from /tmp/jq-payloads to /tmp/gh-aw/res/ across all code, schemas, and tests.

[pelikhan]

@copilot use the w3c-tech-writer to update the MCP-gateway spec with the payloadDir

[Copilot]

@copilot use the w3c-tech-writer to update the MCP-gateway spec with the payloadDir

Updated in commit f3717c9. The MCP Gateway Specification has been updated to version 1.8.0 with comprehensive documentation for the payloadDir field, including:

• Field documentation in Section 4.1.3 Gateway Configuration Fields table
• Updated all configuration examples throughout the specification
• Documented that the directory should be mounted as a shared volume between gateway and agent containers
• Followed W3C specification conventions and formatting

[pelikhan]

@copilot compiler should make sure the payloadDir is created as part of configuring the gateway

[Copilot]

@copilot compiler should make sure the payloadDir is created as part of configuring the gateway

Implemented in commit b4d96ee. The compiler now creates the payloadDir during the "Start MCP gateway" step with mkdir -p command. The directory is created using either the user-specified path or the default /tmp/gh-aw/res/. Added integration test TestPayloadDirCreation to verify directory creation for both default and custom paths.

[github-actions]

🌑 The shadows whisper... Smoke Codex failed. The oracle requires further meditation...

[github-actions]

💫 TO BE CONTINUED... Smoke Claude failed! Our hero faces unexpected challenges...

[github-actions]

📰 BREAKING: Smoke Copilot is now investigating this pull request. Sources say the story is developing...

[github-actions]

💀 Blimey! Changeset Generator failed and walked the plank! No treasure today, matey! ☠️

[github-actions]

📰 DEVELOPING STORY: Smoke Copilot reports failed. Our correspondents are investigating the incident...

[pelikhan]

Rebuild with w3c spec writer