API proxy token exchange fails on GHE Cloud with data residency (400/403)

[lpcox]

Summary

The AWF api-proxy cannot obtain a Copilot session token on GitHub Enterprise Cloud with data residency instances. The proxy's token exchange calls /copilot_internal/v2/token which returns 403 on data residency, causing the Copilot CLI inside the container to get 400 Bad Request when listing models — and immediately exit with code 1.

The Copilot CLI works perfectly when run directly on the host (bypassing the api-proxy), proving that Copilot itself is fully functional on data residency. The issue is exclusively in the api-proxy's token exchange mechanism.

Environment

• Platform: GHE Cloud with data residency (EU)
• Domain: contoso-aw.ghe.com / api.contoso-aw.ghe.com
• gh-aw version: v0.59.0
• AWF version: v0.24.2
• Copilot CLI: 1.0.6
• Workflow: repo-assist (from githubnext/agentics)
• Copilot license: Enterprise plan, CLI enabled, seat assigned to user

Reproduction

1. Set up a repo on a GHE Cloud data residency instance
2. Configure api-target: "api.contoso-aw.ghe.com" in workflow frontmatter
3. Create a fine-grained PAT with Copilot Requests: Read-only permission
4. Run the workflow — agent step fails immediately (5 seconds, exit code 1)

Evidence

Inside AWF container (FAILS) — run 22234276

From sandbox/agent/logs/process-*.log:

Using COPILOT_API_URL from environment: http://172.30.0.30:10002
No auth info available, skipping custom agents load
Starting Copilot CLI: 1.0.6
Error loading models: Error: Failed to list models: 400 Bad Request
MCP transport for github closed
MCP client for github closed
runPromptMode: executePromptDirectly returned succeeded=false
runPromptMode: exiting with code 1

The CLI connects to the api-proxy at 172.30.0.30:10002, which returns 400 because it failed its own upstream token exchange.

Outside AWF container (WORKS) — same run, diagnostic step

executePromptDirectly completing: hasError=false, will return true
runPromptMode: executePromptDirectly returned succeeded=true
runPromptMode: exiting with code 0
[shutdown] Shutdown complete

The same Copilot CLI binary, same token, same host — but running directly on the runner (not through the api-proxy) — authenticates and completes successfully.

Token exchange curl test (confirms proxy root cause)

curl -H "Authorization: token <COPILOT_GITHUB_TOKEN>" \
  "https://api.contoso-aw.ghe.com/copilot_internal/v2/token"
→ HTTP 403: "Resource not accessible by personal access token"

This is the exact endpoint the api-proxy calls. It returns 403 on data residency instances for both fine-grained PATs (with Copilot Requests permission) and GitHub App tokens (ghs_...).

Firewall logs confirm minimal traffic

▼ 2 requests | 2 allowed | 0 blocked | 1 unique domain
| api.contoso-aw.ghe.com | 2 | 0 |

Only 2 requests made before failure — the token exchange attempt and the model list.

Root Cause Analysis

The Copilot CLI has two different authentication paths:

1. Direct (works): When running natively, the CLI uses its own internal auth mechanism (likely device-flow or a different token endpoint) that works on data residency instances. Requires GH_HOST to be set.

2. Via api-proxy (fails): The AWF api-proxy intercepts Copilot requests and performs its own token exchange at /copilot_internal/v2/token. This endpoint returns 403 on data residency instances, even with correctly scoped PATs.

The api-proxy was designed for api.github.com where /copilot_internal/v2/token works with PATs. On data residency instances (api.<slug>.ghe.com), this endpoint either doesn't exist or doesn't accept PAT-based authentication.

What needs to change

The api-proxy needs to support GHE Cloud data residency token exchange. Options:

1. Use the same auth mechanism as the Copilot CLI — whatever internal auth the CLI uses that works on data residency, the api-proxy should use the same approach
2. Pass-through mode — allow the Copilot CLI to handle its own auth directly when --copilot-api-target points to a non-github.com host, instead of the proxy doing token exchange
3. Support the data residency token endpoint — if data residency has a different token exchange endpoint or requires a different auth flow, the proxy should detect and use it

Related issues

• gh-aw-firewall#1300 — API proxy routes Copilot requests to GHES API instead of Copilot API (broader GHES routing issue)
• gh-aw-firewall#1302 — Auto-inject GH_HOST in agent container (partially addressed in gh-aw v0.59.0)
• gh-aw#20966 — Audit GHES/GHE Cloud support gaps

Workaround

Currently none. The api-proxy is the mandatory path for Copilot requests inside the AWF sandbox. We confirmed the CLI works outside the sandbox but there is no way to bypass the proxy for production workflow runs.

Timeline of debugging (20+ workflow runs)

Run	Change	Result
22224791–22232891	Various token/license fixes	403 on token exchange
22234076	Added GH_HOST to diagnostic step	CLI works directly! (exit 0)
22234276	Added GH_HOST to Execute step env	CLI still fails inside container (400 from proxy)

The breakthrough in run 22234076 proved the problem is isolated to the api-proxy, not Copilot configuration.

[github-actions]

👋 Hello! I noticed this issue might be related to an existing issue:

• GHEC: API proxy does not forward GHES token exchange to enterprise Copilot endpoints #1315 — GHEC: API proxy does not forward GHES token exchange to enterprise Copilot endpoints

Both issues describe the AWF api-proxy failing to exchange a Copilot session token on GHE Cloud (*.ghe.com) instances, with the same 403 response from /copilot_internal/v2/token and a resulting 400 Bad Request when the Copilot CLI tries to list models. They share key terms: API proxy token exchange, ghe.com, copilot_internal, fine-grained PAT, and 403/400 errors.

Your issue adds valuable detail around the data residency variant specifically, which may represent a distinct sub-case or a continuation of the same root cause.

If #1315 covers your scenario, please consider whether this is a duplicate. Otherwise, feel free to clarify how data residency differs from the general GHE Cloud case described there!

This is an automated message from the issue duplication detector.

Generated by Issue Duplication Detector for issue #1330

[lpcox]

🎯 HTTP Traffic Capture — Complete Auth Flow Revealed

Using UNDICI_DEBUG=full (Node 24 fetch() debugging), we captured every HTTP request the Copilot CLI makes when it successfully authenticates on a GHE Cloud data residency instance.

Key Finding: The CLI does NOT use /copilot_internal/v2/token

The api-proxy is calling the wrong endpoint. The data residency Copilot API uses a completely different auth and routing model:

4 Unique Hosts Contacted

Host	Purpose	Status
api.contoso-aw.ghe.com	Auth check: GET /copilot_internal/user	✅ 200
copilot-api.contoso-aw.ghe.com	Inference, models, MCP	✅ 200
copilot-telemetry-service.contoso-aw.ghe.com	Telemetry	✅ 200
api.githubcopilot.com	Memory check (cross-domain)	401 (expected)

Complete Request Sequence (chronological)

1. GET  api.contoso-aw.ghe.com/copilot_internal/user                    → 200
2. GET  copilot-telemetry-service.contoso-aw.ghe.com/telemetry?...      → 200
3. POST copilot-api.contoso-aw.ghe.com/mcp/readonly                     → 200
4. POST copilot-api.contoso-aw.ghe.com/mcp/readonly                     → 202
5. GET  copilot-api.contoso-aw.ghe.com/models                           → 200
6. POST copilot-api.contoso-aw.ghe.com/chat/completions                 → 200
7. GET  copilot-api.contoso-aw.ghe.com/agents/swe/custom-agents/...     → 404
8. GET  api.githubcopilot.com/agents/swe/internal/memory/v0/...         → 401
9. GET  copilot-api.contoso-aw.ghe.com/agents/swe/internal/memory/...   → 404
10. POST copilot-telemetry-service.contoso-aw.ghe.com/telemetry         → 200
11. DELETE copilot-api.contoso-aw.ghe.com/mcp/readonly                  → 204

Root Cause

The api-proxy currently does token exchange via:

POST api.contoso-aw.ghe.com/copilot_internal/v2/token → 403 ❌

But the CLI never calls that endpoint. Instead it:

1. Authenticates via GET api.contoso-aw.ghe.com/copilot_internal/user (not /v2/token)
2. Routes all inference to copilot-api.contoso-aw.ghe.com (separate subdomain from api.contoso-aw.ghe.com)

What the api-proxy needs

For data residency instances (where api-target is api.<slug>.ghe.com):

1. Auth endpoint: Use GET /copilot_internal/user instead of POST /copilot_internal/v2/token
2. Inference routing: Route to copilot-api.<slug>.ghe.com (not api.<slug>.ghe.com)
3. Additional domains needed in firewall allow-list:
   • copilot-api.contoso-aw.ghe.com (inference, models, MCP)
   • copilot-telemetry-service.contoso-aw.ghe.com (telemetry)

Evidence

Run 22235251 — diagnostic step "Test C: HTTP request capture (NODE_DEBUG)" with UNDICI_DEBUG=full.

The CLI completed successfully: exit code 0, used gpt-4.1 (13.7k input tokens, 7 output), printed expected response.