Skip to content

GHES: Replace opaque Copilot errors with actionable messages#20976

Closed
Claude wants to merge 5 commits intomainfrom
claude/replace-opaque-copilot-errors
Closed

GHES: Replace opaque Copilot errors with actionable messages#20976
Claude wants to merge 5 commits intomainfrom
claude/replace-opaque-copilot-errors

Conversation

@Claude
Copy link
Copy Markdown
Contributor

@Claude Claude AI commented Mar 14, 2026

Summary

This PR adds comprehensive GHES-specific error detection for the Copilot engine, replacing opaque error messages with clear, actionable guidance for common GHES deployment issues.

Changes

Detection Script

  • Added detect_ghes_copilot_errors.sh to detect 4 common GHES error patterns:
    • 403 on token exchange - Copilot not licensed or GitHub Connect issues
    • 400 on model loading - API proxy routing problems
    • Firewall blocks - GHES API domain not in allowed domains
    • gh CLI misconfiguration - CLI targeting github.com instead of GHES

Error Message Templates

  • Created actionable error message templates for each error type:
    • ghes_token_exchange_error.md - Licensing and GitHub Connect resolution steps
    • ghes_model_loading_error.md - API proxy troubleshooting
    • ghes_firewall_block_error.md - Firewall configuration guidance
    • ghes_gh_cli_error.md - gh CLI configuration fixes

Compiler Integration

  • Added GHES error detection step to Copilot engine workflows (runs after agent execution)
  • Added 4 new job outputs for GHES errors (ghes_token_exchange_403, ghes_model_loading_400, ghes_firewall_block, ghes_gh_cli_misconfigured)
  • Pass GHES error outputs to conclusion job for error reporting
  • Updated failure message templates to include GHES error context

Tests

  • Added comprehensive tests in ghes_copilot_errors_test.go:
    • Verifies GHES error detection step is generated for Copilot workflows
    • Verifies error outputs are exposed in agent job
    • Verifies error environment variables are passed to conclusion job
    • Verifies non-Copilot engines don't include GHES error detection
  • Updated golden test files for workflow compilation

Testing

All tests pass:

  • ✅ GHES error detection tests
  • ✅ Golden file tests (updated for new outputs)
  • ✅ Build and formatting validation

Example Error Messages

When a GHES-specific error is detected, users will see actionable guidance like:

Token Exchange 403:

🔑 Copilot Token Exchange Failed (GHES): The Copilot CLI received a 403 error when attempting to exchange your GitHub token for a Copilot access token. This usually means Copilot is not licensed for this enterprise or organization. Ask your GHES site admin to: 1) Enable GitHub Connect, 2) Enable Copilot at the enterprise level, 3) Assign a Copilot seat to the token owner.

Model Loading 400:

⚠️ Copilot Model Loading Failed (GHES): The Copilot CLI received a 400 error when attempting to load available models. On GHES, this often indicates an API proxy routing issue.

Firewall Block:

🔥 GHES API Domain Blocked by Firewall: The workflow firewall is blocking access to your GHES API domain, preventing the Copilot CLI from communicating with the server.

gh CLI Misconfiguration:

🔧 gh CLI Misconfigured for GHES: The gh CLI is targeting github.com instead of your GHES instance, causing repository resolution errors.

Claude AI and others added 3 commits March 14, 2026 18:28
- Create detect_ghes_copilot_errors.sh script to detect common GHES errors
- Add actionable error message templates for each GHES error type
- Integrate GHES error detection into Copilot engine workflow generation
- Add error outputs to agent job and pass to conclusion job
- Update failure message templates to include GHES error context
- Support detection of: token exchange 403, model loading 400, firewall blocks, gh CLI misconfiguration

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
- Add comprehensive tests for GHES error detection in workflow compilation
- Update golden test files to include new GHES error outputs
- Verify tests pass for Copilot engine workflows

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
@lpcox
Copy link
Copy Markdown
Collaborator

lpcox commented Mar 14, 2026

@claude[agent] fix the pr title and summary

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
@Claude
Copy link
Copy Markdown
Contributor Author

Claude AI commented Mar 14, 2026

@claude[agent] fix the pr title and summary

Updated PR title to "GHES: Replace opaque Copilot errors with actionable messages" and added comprehensive description with implementation details, testing information, and example error messages. (8a960c1)

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh gh pr view --json title,body /home/REDACTED/.nvm/bash_completion /home/REDACTED/.nvm/install.sh /home/REDACTED/.nvm/nvm-exec /home/REDACTED/.nvm/nvm.sh /home/REDACTED/.nvm/package.json /home/REDACTED/.nvm/rename_test.sh /home/REDACTED/.nvm/test (http block)
    • Triggering command: /usr/bin/gh gh pr list --head claude/replace-opaque-copilot-errors --json number,title,body --limit 1 /home/REDACTED/.nvm/package.json /home/REDACTED/.nvm/rename_test.sh /home/REDACTED/.nvm/test (http block)
    • Triggering command: `/usr/bin/gh gh pr edit --title GHES: Replace opaque Copilot errors with actionable messages --body ## Summary

This PR adds comprehensive GHES-specific error detection for the Copilot engine, replacing opaque error messages with clear, actionable guidance for common GHES deployment issues.

Changes

Detection Script

  • Added detect_ghes_copilot_ /home/REDACTED/.nvm/nvm-exec /home/REDACTED/.nvm/nvm.sh /home/REDACTED/.nvm/package.json /home/REDACTED/.nvm/rename_test.sh /home/REDACTED/.nvm/test (http block)

If you need me to access, download, or install something from one of these locations, you can either:

@Claude Claude AI changed the title Completing task GHES: Replace opaque Copilot errors with actionable messages Mar 14, 2026
@Claude Claude AI requested a review from lpcox March 14, 2026 19:07
@pelikhan pelikhan marked this pull request as ready for review March 15, 2026 01:44
Copilot AI review requested due to automatic review settings March 15, 2026 01:44
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds GHES-specific GitHub Copilot error detection to generated workflows and surfaces those signals in the agent-failure issue/comment templates so failures on GitHub Enterprise Server can be diagnosed with clearer remediation guidance.

Changes:

  • Add a new post-run agent step (detect-ghes-errors) plus job outputs for several GHES Copilot failure modes (token exchange 403, model loading 400, firewall block, gh CLI misconfig).
  • Plumb the new outputs into the conclusion job env and extend the agent-failure handler to render GHES-specific failure context from new prompt templates.
  • Add a new shell detector script, new markdown prompt templates, golden/lock workflow regenerations, and compiler tests.

Reviewed changes

Copilot reviewed 131 out of 131 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
pkg/workflow/copilot_engine_execution.go Introduces generator for the GHES error detection step.
pkg/workflow/compiler_yaml_main_job.go Emits the GHES detection step into Copilot engine main job steps.
pkg/workflow/compiler_main_job.go Exposes GHES error flags as main-job outputs for Copilot workflows.
pkg/workflow/notify_comment.go Passes GHES error outputs into conclusion job env for failure reporting.
pkg/workflow/ghes_copilot_errors_test.go Adds compiler tests asserting step/output/env plumbing for Copilot vs non-Copilot engines.
actions/setup/sh/detect_ghes_copilot_errors.sh Adds log-scraping script that sets GHES-specific error outputs for GitHub Actions.
actions/setup/js/handle_agent_failure.cjs Renders GHES-specific failure context into agent-failure issues/comments using new prompt templates.
actions/setup/md/ghes_token_exchange_error.md New prompt: GHES Copilot token exchange 403 guidance.
actions/setup/md/ghes_model_loading_error.md New prompt: GHES Copilot model loading 400 guidance.
actions/setup/md/ghes_firewall_block_error.md New prompt: firewall block guidance for GHES API domain access.
actions/setup/md/ghes_gh_cli_error.md New prompt: gh CLI misconfiguration guidance for GHES.
actions/setup/md/agent_failure_issue.md Adds {ghes_error_context} placeholder to failure issue template.
actions/setup/md/agent_failure_comment.md Adds {ghes_error_context} placeholder to failure comment template.
pkg/workflow/testdata/wasm_golden/TestWasmGolden_CompileFixtures/basic-copilot.golden Updates golden output to include GHES detection step + outputs.
pkg/workflow/testdata/wasm_golden/TestWasmGolden_CompileFixtures/smoke-copilot.golden Updates golden output to include GHES detection step + outputs.
pkg/workflow/testdata/wasm_golden/TestWasmGolden_CompileFixtures/with-imports.golden Updates golden output to include GHES detection step + outputs.
.github/workflows/workflow-skill-extractor.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/workflow-normalizer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/workflow-health-manager.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/workflow-generator.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/weekly-safe-outputs-spec-review.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/weekly-issue-summary.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/weekly-editors-health-check.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/video-analyzer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/ubuntu-image-analyzer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/tidy.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/test-workflow.lock.yml Regenerated lock workflow to include GHES detection step and outputs.
.github/workflows/test-project-url-default.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/test-dispatcher.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/terminal-stylist.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/technical-doc-writer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/super-linter.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/sub-issue-closer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/stale-repo-identifier.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/smoke-workflow-call.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/smoke-workflow-call-with-inputs.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/smoke-update-cross-repo-pr.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/smoke-test-tools.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/smoke-temporary-id.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/smoke-project.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/smoke-multi-pr.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/smoke-create-cross-repo-pr.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/smoke-copilot.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/smoke-copilot-arm.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/slide-deck-maintainer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/security-review.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/security-compliance.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/research.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/repository-quality-improver.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/repo-tree-map.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/repo-audit-analyzer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/release.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/refiner.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/q.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/python-data-charts.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/pr-triage-agent.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/pr-nitpick-reviewer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/portfolio-analyst.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/poem-bot.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/plan.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/pdf-summary.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/org-health-report.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/notion-issue-summary.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/metrics-collector.lock.yml Regenerated lock workflow to include GHES detection step and outputs.
.github/workflows/mergefest.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/mcp-inspector.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/layout-spec-maintainer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/jsweep.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/issue-triage-agent.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/issue-monster.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/hourly-ci-cleaner.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/grumpy-reviewer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/gpclean.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/glossary-maintainer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/github-remote-mcp-auth-test.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/functional-pragmatist.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/firewall.lock.yml Regenerated lock workflow to include GHES detection step and outputs.
.github/workflows/firewall-escape.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/example-permissions-warning.lock.yml Regenerated lock workflow to include GHES detection step and outputs.
.github/workflows/draft-pr-cleanup.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/docs-noob-tester.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/discussion-task-miner.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/dictation-prompt.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/dev.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/dev-hawk.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/dependabot-go-checker.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/dependabot-burner.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/delight.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/dead-code-remover.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-workflow-updater.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-testify-uber-super-expert.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-team-status.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-syntax-error-quality.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-semgrep-scan.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-secrets-analysis.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-repo-chronicle.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-regulatory.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-performance-summary.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-news.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-mcp-concurrency-analysis.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-malicious-code-scan.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-firewall-report.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-file-diet.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-copilot-token-report.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-compiler-quality.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-cli-tools-tester.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-cli-performance.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-assign-issue-to-user.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/daily-architecture-diagram.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/copilot-pr-prompt-analysis.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/copilot-pr-nlp-analysis.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/copilot-pr-merged-report.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/copilot-cli-deep-research.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/constraint-solving-potd.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/code-simplifier.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/code-scanning-fixer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/cli-consistency-checker.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/ci-doctor.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/ci-coach.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/chroma-issue-indexer.lock.yml Regenerated lock workflow to include GHES detection step and outputs.
.github/workflows/breaking-change-checker.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/brave.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/bot-detection.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/auto-triage-issues.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/artifacts-summary.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/archie.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/agent-persona-explorer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/agent-performance-analyzer.lock.yml Regenerated lock workflow to include GHES detection step, outputs, and conclusion env vars.
.github/workflows/ace-editor.lock.yml Regenerated lock workflow to include GHES detection step and outputs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment on lines +53 to +57
# Check for gh CLI misconfiguration (targeting github.com instead of GHES)
# Pattern: "Could not resolve to a Repository" or "gh.*github.com" when GHES is expected
if grep -qE "(Could not resolve to a Repository|gh.*targeting.*github\.com|GH_HOST.*not set)" "$LOG_FILE"; then
echo "Detected gh CLI misconfiguration (targeting github.com instead of GHES)"
echo "ghes_gh_cli_misconfigured=true" >> "$GITHUB_OUTPUT"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GHES: Replace opaque Copilot errors with actionable messages

4 participants