feat: add smoke test for Copilot CLI offline BYOK mode#2003
Conversation
Add a new smoke-copilot-byok workflow that validates the COPILOT_OFFLINE code path through the AWF api-proxy sidecar. - Sets COPILOT_API_KEY to a dummy value to trigger AWF's BYOK detection (COPILOT_OFFLINE=true, COPILOT_PROVIDER_BASE_URL) - Sidecar authenticates to api.githubcopilot.com using the real COPILOT_GITHUB_TOKEN — no new secrets required - Post-step verifies firewall logs show traffic to the expected Copilot API target - Compiled with gh-aw v0.68.3 and post-processed for local builds Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
There was a problem hiding this comment.
Pull request overview
Adds a new Agentic Workflow smoke test to exercise the Copilot CLI “offline BYOK” code path (dummy COPILOT_API_KEY triggers AWF offline routing through the api-proxy sidecar) and to record results back to PRs via safe outputs.
Changes:
- Added
smoke-copilot-byokworkflow source defining the BYOK/offline smoke test and post-run validations. - Added the compiled workflow lock file for the new workflow.
- Updated the action pin lock to include
github/gh-aw-actions/setup@v0.68.3.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/smoke-copilot-byok.md | New BYOK/offline Copilot smoke workflow definition, including post-step validations. |
| .github/workflows/smoke-copilot-byok.lock.yml | Compiled/locked workflow generated from the .md source. |
| .github/aw/actions-lock.json | Adds SHA pin entry for github/gh-aw-actions/setup@v0.68.3. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comments suppressed due to low confidence (1)
.github/workflows/smoke-copilot-byok.md:117
- The “BYOK Inference Test” as written is not actually validating that offline/BYOK routing is in effect; the agent being able to respond would also be true in the normal (non-BYOK) path. To make this test meaningful, have the agent explicitly verify runtime signals like
COPILOT_OFFLINE=trueandCOPILOT_PROVIDER_BASE_URL(or equivalent) from within the sandbox, and/or check that the Copilot CLI is configured to use the proxy base URL.
### 3. BYOK Inference Test
You are running in offline BYOK mode right now. The fact that you can read this prompt and respond means the BYOK inference path (agent → api-proxy sidecar → api.githubcopilot.com) is working. Confirm ✅.
- Files reviewed: 3/3 changed files
- Comments generated: 2
| if [ -d "$LOGS_DIR" ]; then | ||
| echo "::group::Checking firewall logs for offline BYOK traffic" | ||
| if find "$LOGS_DIR" -name '*.log' -exec grep -l "api.githubcopilot.com" {} + 2>/dev/null; then | ||
| echo "✅ Detected traffic to api.githubcopilot.com via api-proxy (BYOK offline mode)" | ||
| else | ||
| echo "::warning::No traffic to api.githubcopilot.com found in firewall logs" | ||
| fi | ||
| echo "::endgroup::" | ||
| fi |
There was a problem hiding this comment.
The BYOK traffic verification is non-blocking: if the firewall logs directory is missing, or if no matching traffic is found, this step still exits 0 (it only emits a warning). That means the workflow can report success without actually validating the offline BYOK path. Consider failing the job when logs are missing or when expected Copilot API traffic is not detected (optionally gating only on PR/workflow_dispatch if flakiness is a concern).
This issue also appears on line 116 of the same file.
| if [ -d "$LOGS_DIR" ]; then | |
| echo "::group::Checking firewall logs for offline BYOK traffic" | |
| if find "$LOGS_DIR" -name '*.log' -exec grep -l "api.githubcopilot.com" {} + 2>/dev/null; then | |
| echo "✅ Detected traffic to api.githubcopilot.com via api-proxy (BYOK offline mode)" | |
| else | |
| echo "::warning::No traffic to api.githubcopilot.com found in firewall logs" | |
| fi | |
| echo "::endgroup::" | |
| fi | |
| if [ ! -d "$LOGS_DIR" ]; then | |
| echo "::error::Firewall logs directory not found: $LOGS_DIR" | |
| exit 1 | |
| fi | |
| echo "::group::Checking firewall logs for offline BYOK traffic" | |
| MATCHING_LOGS="$(find "$LOGS_DIR" -name '*.log' -exec grep -l "api.githubcopilot.com" {} + 2>/dev/null || true)" | |
| if [ -z "$MATCHING_LOGS" ]; then | |
| echo "::error::No traffic to api.githubcopilot.com found in firewall logs" | |
| echo "::endgroup::" | |
| exit 1 | |
| fi | |
| printf '%s\n' "$MATCHING_LOGS" | |
| echo "✅ Detected traffic to api.githubcopilot.com via api-proxy (BYOK offline mode)" | |
| echo "::endgroup::" |
| if find "$LOGS_DIR" -name '*.log' -exec grep -l "api.githubcopilot.com" {} + 2>/dev/null; then | ||
| echo "✅ Detected traffic to api.githubcopilot.com via api-proxy (BYOK offline mode)" | ||
| else | ||
| echo "::warning::No traffic to api.githubcopilot.com found in firewall logs" |
There was a problem hiding this comment.
The firewall log grep is hardcoded to api.githubcopilot.com, but Copilot traffic may legitimately hit other Copilot API hostnames (e.g., enterprise/business/individual endpoints) depending on environment/config. To avoid false negatives (and then passing due to the warning), consider matching against the effective target (e.g., ${COPILOT_API_TARGET} if set) or a broader pattern/set of allowed Copilot API hosts.
| if find "$LOGS_DIR" -name '*.log' -exec grep -l "api.githubcopilot.com" {} + 2>/dev/null; then | |
| echo "✅ Detected traffic to api.githubcopilot.com via api-proxy (BYOK offline mode)" | |
| else | |
| echo "::warning::No traffic to api.githubcopilot.com found in firewall logs" | |
| TARGET_HOST="${COPILOT_API_TARGET:-}" | |
| TARGET_HOST="${TARGET_HOST#http://}" | |
| TARGET_HOST="${TARGET_HOST#https://}" | |
| TARGET_HOST="${TARGET_HOST%%/*}" | |
| COPILOT_API_HOST_PATTERN='(^|[[:space:]])(api|enterprise-api|business-api|individual-api)\.githubcopilot\.com([[:space:]]|$)' | |
| if [ -n "$TARGET_HOST" ]; then | |
| if find "$LOGS_DIR" -name '*.log' -exec grep -Fl "$TARGET_HOST" {} + 2>/dev/null; then | |
| echo "✅ Detected traffic to $TARGET_HOST via api-proxy (BYOK offline mode)" | |
| else | |
| echo "::warning::No traffic to $TARGET_HOST found in firewall logs" | |
| fi | |
| elif find "$LOGS_DIR" -name '*.log' -exec grep -El "$COPILOT_API_HOST_PATTERN" {} + 2>/dev/null; then | |
| echo "✅ Detected traffic to a Copilot API host via api-proxy (BYOK offline mode)" | |
| else | |
| echo "::warning::No traffic to known Copilot API hosts found in firewall logs" |
|
Smoke Test Results ✅ GitHub MCP — #1995 fix(api-proxy): fix Gemini API_KEY_INVALID with credential isolation | #1991 Activate smoke-opencode workflow via copilot engine workaround Overall: PASS
|
🤖 Smoke Test: OpenCode Engine Validation
Overall status: PASS
|
🔍 Smoke Test Results
Overall: PR by @lpcox | No assignees
|
|
Smoke test matrix:
|
Smoke Test: GitHub Actions Services Connectivity ✅All connectivity checks passed:
All GitHub Actions service containers are reachable from the AWF sandbox via
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS
|
Summary
Add a new
smoke-copilot-byokworkflow that validates the Copilot CLI offline BYOK code path through the AWF api-proxy sidecar.How it works
COPILOT_API_KEY: dummy-byok-key-for-offline-mode— a dummy value that triggers AWF's BYOK detectionCOPILOT_API_KEYis present → setsCOPILOT_OFFLINE=true+COPILOT_PROVIDER_BASE_URLpointing to the api-proxy sidecarCOPILOT_GITHUB_TOKENand uses it forAuthorization: Bearertoapi.githubcopilot.com(token resolution:COPILOT_GITHUB_TOKEN || COPILOT_API_KEY)api.githubcopilot.comKey design decisions
COPILOT_GITHUB_TOKENfor real auth; the dummyCOPILOT_API_KEYjust activates offline modeapi.githubcopilot.com(default), not a third-party provider, so the test validates the offline code path without requiring external credentialsdefaults+github(same domains as regular copilot smoke)New secrets and actions
dummy-byok-key-for-offline-modein workflow env. Used solely to trigger AWF's BYOK/offline mode detection.ba90f21): New SHA pin added toactions-lock.jsonby the compiler.Files
.github/workflows/smoke-copilot-byok.md— Workflow source.github/workflows/smoke-copilot-byok.lock.yml— Compiled lock file (post-processed for local builds).github/aw/actions-lock.json— Updated action pin