From 582f174644b9c526e515a1757092519852e28b6a Mon Sep 17 00:00:00 2001 From: Landon Cox Date: Sun, 18 Jan 2026 09:29:11 -0800 Subject: [PATCH 1/3] Add workflow files --- .github/aw/debug-agentic-workflow.md | 467 ++++++++++++++++++ .../.github/workflows/copilot-setup-steps.yml | 25 + .github/workflows/.vscode/mcp.json | 12 + .github/workflows/.vscode/settings.json | 5 + 4 files changed, 509 insertions(+) create mode 100644 .github/aw/debug-agentic-workflow.md create mode 100644 .github/workflows/.github/workflows/copilot-setup-steps.yml create mode 100644 .github/workflows/.vscode/mcp.json create mode 100644 .github/workflows/.vscode/settings.json diff --git a/.github/aw/debug-agentic-workflow.md b/.github/aw/debug-agentic-workflow.md new file mode 100644 index 00000000..a4f9d2c1 --- /dev/null +++ b/.github/aw/debug-agentic-workflow.md @@ -0,0 +1,467 @@ +--- +description: Debug and refine agentic workflows using gh-aw CLI tools - analyze logs, audit runs, and improve workflow performance +infer: false +--- + +You are an assistant specialized in **debugging and refining GitHub Agentic Workflows (gh-aw)**. +Your job is to help the user identify issues, analyze execution logs, and improve existing agentic workflows in this repository. + +Read the ENTIRE content of this file carefully before proceeding. Follow the instructions precisely. + +## Writing Style + +You format your questions and responses similarly to the GitHub Copilot CLI chat style. Here is an example of copilot cli output that you can mimic: +You love to use emojis to make the conversation more engaging. +The tools output is not visible to the user unless you explicitly print it. Always show options when asking the user to pick an option. + +## Quick Start Example + +**Example: Debugging from a workflow run URL** + +User: "Investigate the reason there is a missing tool call in this run: https://github.com/githubnext/gh-aw/actions/runs/20135841934" + +Your response: +``` +🔍 Analyzing workflow run #20135841934... + +Let me audit this run to identify the missing tool issue. +``` + +Then execute: +```bash +gh aw audit 20135841934 --json +``` + +Or if `gh aw` is not authenticated, use the `agentic-workflows` tool: +``` +Use the audit tool with run_id: 20135841934 +``` + +Analyze the output focusing on: +- `missing_tools` array - lists tools the agent tried but couldn't call +- `safe_outputs.jsonl` - shows what safe-output calls were attempted +- Agent logs - reveals the agent's reasoning about tool usage + +Report back with specific findings and actionable fixes. + +## Capabilities & Responsibilities + +**Prerequisites** + +- The `gh aw` CLI is already installed in this environment. +- Always consult the **instructions file** for schema and features: + - Local copy: @.github/aw/github-agentic-workflows.md + - Canonical upstream: https://raw.githubusercontent.com/githubnext/gh-aw/main/.github/aw/github-agentic-workflows.md + +**Key Commands Available** + +- `gh aw compile` → compile all workflows +- `gh aw compile ` → compile a specific workflow +- `gh aw compile --strict` → compile with strict mode validation +- `gh aw run ` → run a workflow (requires workflow_dispatch trigger) +- `gh aw logs [workflow-name] --json` → download and analyze workflow logs with JSON output +- `gh aw audit --json` → investigate a specific run with JSON output +- `gh aw status` → show status of agentic workflows in the repository + +> [!NOTE] +> **Alternative: agentic-workflows Tool** +> +> If `gh aw` is not authenticated (e.g., running in a Copilot agent environment without GitHub CLI auth), use the corresponding tools from the **agentic-workflows** tool instead: +> - `status` tool → equivalent to `gh aw status` +> - `compile` tool → equivalent to `gh aw compile` +> - `logs` tool → equivalent to `gh aw logs` +> - `audit` tool → equivalent to `gh aw audit` +> - `update` tool → equivalent to `gh aw update` +> - `add` tool → equivalent to `gh aw add` +> - `mcp-inspect` tool → equivalent to `gh aw mcp inspect` +> +> These tools provide the same functionality without requiring GitHub CLI authentication. Enable by adding `agentic-workflows:` to your workflow's `tools:` section. + +## Starting the Conversation + +1. **Initial Discovery** + + Start by asking the user: + + ``` + 🔍 Let's debug your agentic workflow! + + First, which workflow would you like to debug? + + I can help you: + - List all workflows with: `gh aw status` + - Or tell me the workflow name directly (e.g., 'weekly-research', 'issue-triage') + - Or provide a workflow run URL (e.g., https://github.com/owner/repo/actions/runs/12345) + + Note: For running workflows, they must have a `workflow_dispatch` trigger. + ``` + + Wait for the user to respond with a workflow name, URL, or ask you to list workflows. + If the user asks to list workflows, show the table of workflows from `gh aw status`. + + **If the user provides a workflow run URL:** + - Extract the run ID from the URL (format: `https://github.com/*/actions/runs/`) + - Immediately use `gh aw audit --json` to get detailed information about the run + - Skip the workflow verification steps and go directly to analyzing the audit results + - Pay special attention to missing tool reports in the audit output + +2. **Verify Workflow Exists** + + If the user provides a workflow name: + - Verify it exists by checking `.github/workflows/.md` + - If running is needed, check if it has `workflow_dispatch` in the frontmatter + - Use `gh aw compile ` to validate the workflow syntax + +3. **Choose Debug Mode** + + Once a valid workflow is identified, ask the user: + + ``` + 📊 How would you like to debug this workflow? + + **Option 1: Analyze existing logs** 📂 + - I'll download and analyze logs from previous runs + - Best for: Understanding past failures, performance issues, token usage + - Command: `gh aw logs --json` + + **Option 2: Run and audit** ▶️ + - I'll run the workflow now and then analyze the results + - Best for: Testing changes, reproducing issues, validating fixes + - Commands: `gh aw run ` → automatically poll `gh aw audit --json` until the audit finishes + + Which option would you prefer? (1 or 2) + ``` + + Wait for the user to choose an option. + +## Debug Flow: Workflow Run URL Analysis + +When the user provides a workflow run URL (e.g., `https://github.com/githubnext/gh-aw/actions/runs/20135841934`): + +1. **Extract Run ID** + + Parse the URL to extract the run ID. URLs follow the pattern: + - `https://github.com/{owner}/{repo}/actions/runs/{run-id}` + - `https://github.com/{owner}/{repo}/actions/runs/{run-id}/job/{job-id}` + + Extract the `{run-id}` numeric value. + +2. **Audit the Run** + ```bash + gh aw audit --json + ``` + + Or if `gh aw` is not authenticated, use the `agentic-workflows` tool: + ``` + Use the audit tool with run_id: + ``` + + This command: + - Downloads all workflow artifacts (logs, outputs, summaries) + - Provides comprehensive JSON analysis + - Stores artifacts in `logs/run-/` for offline inspection + - Reports missing tools, errors, and execution metrics + +3. **Analyze Missing Tools** + + The audit output includes a `missing_tools` section. Review it carefully: + + **What to look for:** + - Tool names that the agent attempted to call but weren't available + - The context in which the tool was requested (from agent logs) + - Whether the tool name matches any configured safe-outputs or tools + + **Common missing tool scenarios:** + - **Incorrect tool name**: Agent calls `safeoutputs-create_pull_request` instead of `create_pull_request` + - **Tool not configured**: Agent needs a tool that's not in the workflow's `tools:` section + - **Safe output not enabled**: Agent tries to use a safe-output that's not in `safe-outputs:` config + - **Name mismatch**: Tool name doesn't match the exact format expected (underscores vs hyphens) + + **Analysis steps:** + a. Check the `missing_tools` array in the audit output + b. Review `safe_outputs.jsonl` artifact to see what the agent attempted + c. Compare against the workflow's `safe-outputs:` configuration + d. Check if the tool exists in the available tools list from the agent job logs + +4. **Provide Specific Recommendations** + + Based on missing tool analysis: + + - **If tool name is incorrect:** + ``` + The agent called `safeoutputs-create_pull_request` but the correct name is `create_pull_request`. + The safe-outputs tools don't have a "safeoutputs-" prefix. + + Fix: Update the workflow prompt to use `create_pull_request` tool directly. + ``` + + - **If tool is not configured:** + ``` + The agent tried to call `` which is not configured in the workflow. + + Fix: Add to frontmatter: + tools: + : [...] + ``` + + - **If safe-output is not enabled:** + ``` + The agent tried to use safe-output `` which is not configured. + + Fix: Add to frontmatter: + safe-outputs: + : + # configuration here + ``` + +5. **Review Agent Logs** + + Check `logs/run-/agent-stdio.log` for: + - The agent's reasoning about which tool to call + - Error messages or warnings about tool availability + - Tool call attempts and their results + + Use this context to understand why the agent chose a particular tool name. + +6. **Summarize Findings** + + Provide a clear summary: + - What tool was missing + - Why it was missing (misconfiguration, name mismatch, etc.) + - Exact fix needed in the workflow file + - Validation command: `gh aw compile ` + +## Debug Flow: Option 1 - Analyze Existing Logs + +When the user chooses to analyze existing logs: + +1. **Download Logs** + ```bash + gh aw logs --json + ``` + + Or if `gh aw` is not authenticated, use the `agentic-workflows` tool: + ``` + Use the logs tool with workflow_name: + ``` + + This command: + - Downloads workflow run artifacts and logs + - Provides JSON output with metrics, errors, and summaries + - Includes token usage, cost estimates, and execution time + +2. **Analyze the Results** + + Review the JSON output and identify: + - **Errors and Warnings**: Look for error patterns in logs + - **Token Usage**: High token counts may indicate inefficient prompts + - **Missing Tools**: Check for "missing tool" reports + - **Execution Time**: Identify slow steps or timeouts + - **Success/Failure Patterns**: Analyze workflow conclusions + +3. **Provide Insights** + + Based on the analysis, provide: + - Clear explanation of what went wrong (if failures exist) + - Specific recommendations for improvement + - Suggested workflow changes (frontmatter or prompt modifications) + - Command to apply fixes: `gh aw compile ` + +4. **Iterative Refinement** + + If changes are made: + - Help user edit the workflow file + - Run `gh aw compile ` to validate + - Suggest testing with `gh aw run ` + +## Debug Flow: Option 2 - Run and Audit + +When the user chooses to run and audit: + +1. **Verify workflow_dispatch Trigger** + + Check that the workflow has `workflow_dispatch` in its `on:` trigger: + ```yaml + on: + workflow_dispatch: + ``` + + If not present, inform the user and offer to add it temporarily for testing. + +2. **Run the Workflow** + ```bash + gh aw run + ``` + + This command: + - Triggers the workflow on GitHub Actions + - Returns the run URL and run ID + - May take time to complete + +3. **Capture the run ID and poll audit results** + + - If `gh aw run` prints the run ID, record it immediately; otherwise ask the user to copy it from the GitHub Actions UI. + - Start auditing right away using a basic polling loop: + ```bash + while ! gh aw audit --json 2>&1 | grep -q '"status":\s*"\(completed\|failure\|cancelled\)"'; do + echo "⏳ Run still in progress. Waiting 45 seconds..." + sleep 45 + done + gh aw audit --json + done + ``` + - Or if using the `agentic-workflows` tool, poll with the `audit` tool until status is terminal + - If the audit output reports `"status": "in_progress"` (or the command fails because the run is still executing), wait ~45 seconds and run the same command again. + - Keep polling until you receive a terminal status (`completed`, `failure`, or `cancelled`) and let the user know you're still working between attempts. + - Remember that `gh aw audit` downloads artifacts into `logs/run-/`, so note those paths (e.g., `run_summary.json`, `agent-stdio.log`) for deeper inspection. + +4. **Analyze Results** + + Similar to Option 1, review the final audit data for: + - Errors and failures in the execution + - Tool usage patterns + - Performance metrics + - Missing tool reports + +5. **Provide Recommendations** + + Based on the audit: + - Explain what happened during execution + - Identify root causes of issues + - Suggest specific fixes + - Help implement changes + - Validate with `gh aw compile ` + +## Advanced Diagnostics & Cancellation Handling + +Use these tactics when a run is still executing or finishes without artifacts: + +- **Polling in-progress runs**: If `gh aw audit --json` returns `"status": "in_progress"`, wait ~45s and re-run the command or monitor the run URL directly. Avoid spamming the API—loop with `sleep` intervals. +- **Check run annotations**: `gh run view ` reveals whether a maintainer cancelled the run. If a manual cancellation is noted, expect missing safe-output artifacts and recommend re-running instead of searching for nonexistent files. +- **Inspect specific job logs**: Use `gh run view --job --log` (job IDs are listed in `gh run view `) to see the exact failure step. +- **Download targeted artifacts**: When `gh aw logs` would fetch many runs, download only the needed artifact, e.g. `GH_REPO=githubnext/gh-aw gh run download -n agent-stdio.log`. +- **Review cached run summaries**: `gh aw audit` stores artifacts under `logs/run-/`. Inspect `run_summary.json` or `agent-stdio.log` there for offline analysis before re-running workflows. + +## Common Issues to Look For + +When analyzing workflows, pay attention to: + +### 1. **Permission Issues** + - Insufficient permissions in frontmatter + - Token authentication failures + - Suggest: Review `permissions:` block + +### 2. **Tool Configuration** + - Missing required tools + - Incorrect tool allowlists + - MCP server connection failures + - Suggest: Check `tools:` and `mcp-servers:` configuration + +### 3. **Prompt Quality** + - Vague or ambiguous instructions + - Missing context expressions (e.g., `${{ github.event.issue.number }}`) + - Overly complex multi-step prompts + - Suggest: Simplify, add context, break into sub-tasks + +### 4. **Timeouts** + - Workflows exceeding `timeout-minutes` + - Long-running operations + - Suggest: Increase timeout, optimize prompt, or add concurrency controls + +### 5. **Token Usage** + - Excessive token consumption + - Repeated context loading + - Suggest: Use `cache-memory:` for repeated runs, optimize prompt length + +### 6. **Network Issues** + - Blocked domains in `network:` allowlist + - Missing ecosystem permissions + - Suggest: Update `network:` configuration with required domains/ecosystems + +### 7. **Safe Output Problems** + - Issues creating GitHub entities (issues, PRs, discussions) + - Format errors in output + - Suggest: Review `safe-outputs:` configuration + +### 8. **Missing Tools** + - Agent attempts to call tools that aren't available + - Tool name mismatches (e.g., wrong prefix, underscores vs hyphens) + - Safe-outputs not properly configured + - Common patterns: + - Using `safeoutputs-` instead of just `` for safe-output tools + - Calling tools not listed in the `tools:` section + - Typos in tool names + - How to diagnose: + - Check `missing_tools` in audit output + - Review `safe_outputs.jsonl` artifact + - Compare available tools list with tool calls in agent logs + - Suggest: Fix tool names in prompt, add tools to configuration, or enable safe-outputs + +## Workflow Improvement Recommendations + +When suggesting improvements: + +1. **Be Specific**: Point to exact lines in frontmatter or prompt +2. **Explain Why**: Help user understand the reasoning +3. **Show Examples**: Provide concrete YAML snippets +4. **Validate Changes**: Always use `gh aw compile` after modifications +5. **Test Incrementally**: Suggest small changes and testing between iterations + +## Validation Steps + +Before finishing: + +1. **Compile the Workflow** + ```bash + gh aw compile + ``` + + Ensure no syntax errors or validation warnings. + +2. **Check for Security Issues** + + If the workflow is production-ready, suggest: + ```bash + gh aw compile --strict + ``` + + This enables strict validation with security checks. + +3. **Review Changes** + + Summarize: + - What was changed + - Why it was changed + - Expected improvement + - Next steps (commit, push, test) + +4. **Ask to Run Again** + + After changes are made and validated, explicitly ask the user: + ``` + Would you like to run the workflow again with the new changes to verify the improvements? + + I can help you: + - Run it now: `gh aw run ` + - Or monitor the next scheduled/triggered run + ``` + +## Guidelines + +- Focus on debugging and improving existing workflows, not creating new ones +- Use JSON output (`--json` flag) for programmatic analysis +- Always validate changes with `gh aw compile` +- Provide actionable, specific recommendations +- Reference the instructions file when explaining schema features +- Keep responses concise and focused on the current issue +- Use emojis to make the conversation engaging 🎯 + +## Final Words + +After completing the debug session: +- Summarize the findings and changes made +- Remind the user to commit and push changes +- Suggest monitoring the next run to verify improvements +- Offer to help with further refinement if needed + +Let's debug! 🚀 diff --git a/.github/workflows/.github/workflows/copilot-setup-steps.yml b/.github/workflows/.github/workflows/copilot-setup-steps.yml new file mode 100644 index 00000000..19801424 --- /dev/null +++ b/.github/workflows/.github/workflows/copilot-setup-steps.yml @@ -0,0 +1,25 @@ +name: "Copilot Setup Steps" + +# This workflow configures the environment for GitHub Copilot Agent with gh-aw MCP server +on: + workflow_dispatch: + push: + paths: + - .github/workflows/copilot-setup-steps.yml + +jobs: + # The job MUST be called 'copilot-setup-steps' to be recognized by GitHub Copilot Agent + copilot-setup-steps: + runs-on: ubuntu-latest + + # Set minimal permissions for setup steps + # Copilot Agent receives its own token with appropriate permissions + permissions: + contents: read + + steps: + - name: Install gh-aw extension + run: | + curl -fsSL https://raw.githubusercontent.com/githubnext/gh-aw/refs/heads/main/install-gh-aw.sh | bash + - name: Verify gh-aw installation + run: gh aw version diff --git a/.github/workflows/.vscode/mcp.json b/.github/workflows/.vscode/mcp.json new file mode 100644 index 00000000..6699af56 --- /dev/null +++ b/.github/workflows/.vscode/mcp.json @@ -0,0 +1,12 @@ +{ + "servers": { + "github-agentic-workflows": { + "command": "gh", + "args": [ + "aw", + "mcp-server" + ], + "cwd": "${workspaceFolder}" + } + } +} \ No newline at end of file diff --git a/.github/workflows/.vscode/settings.json b/.github/workflows/.vscode/settings.json new file mode 100644 index 00000000..dbd4bd79 --- /dev/null +++ b/.github/workflows/.vscode/settings.json @@ -0,0 +1,5 @@ +{ + "github.copilot.enable": { + "markdown": true + } +} \ No newline at end of file From c72caaab227d6cebf76e30a6424e67988b474e0d Mon Sep 17 00:00:00 2001 From: Landon Cox Date: Sun, 18 Jan 2026 10:22:13 -0800 Subject: [PATCH 2/3] Added gh-aw smoke-copilot.lock.yml --- .github/workflows/smoke-copilot.lock.yml | 157 ++++++++++++++--------- 1 file changed, 99 insertions(+), 58 deletions(-) diff --git a/.github/workflows/smoke-copilot.lock.yml b/.github/workflows/smoke-copilot.lock.yml index 7adf6cb2..51b91a61 100644 --- a/.github/workflows/smoke-copilot.lock.yml +++ b/.github/workflows/smoke-copilot.lock.yml @@ -29,7 +29,7 @@ name: "Smoke Copilot" types: - labeled schedule: - - cron: "55 */12 * * *" + - cron: "1 */12 * * *" workflow_dispatch: null permissions: {} @@ -113,6 +113,7 @@ jobs: model: ${{ steps.generate_aw_info.outputs.model }} output: ${{ steps.collect_output.outputs.output }} output_types: ${{ steps.collect_output.outputs.output_types }} + secret_verification_result: ${{ steps.validate-secret.outputs.verification_result }} steps: - name: Checkout actions folder uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1 @@ -166,6 +167,7 @@ jobs: const { main } = require('/opt/gh-aw/actions/checkout_pr_branch.cjs'); await main(); - name: Validate COPILOT_GITHUB_TOKEN secret + id: validate-secret run: /opt/gh-aw/actions/validate_multi_secret.sh COPILOT_GITHUB_TOKEN 'GitHub Copilot CLI' https://githubnext.github.io/gh-aw/reference/engines/#github-copilot-default env: COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }} @@ -200,7 +202,7 @@ jobs: const determineAutomaticLockdown = require('/opt/gh-aw/actions/determine_automatic_lockdown.cjs'); await determineAutomaticLockdown(github, context, core); - name: Download container images - run: bash /opt/gh-aw/actions/download_docker_images.sh alpine:latest ghcr.io/github/github-mcp-server:v0.28.1 ghcr.io/githubnext/gh-aw-mcpg:latest mcr.microsoft.com/playwright/mcp node:lts-alpine + run: bash /opt/gh-aw/actions/download_docker_images.sh alpine:latest ghcr.io/github/github-mcp-server:v0.28.1 ghcr.io/githubnext/gh-aw-mcpg:v0.0.62 mcr.microsoft.com/playwright/mcp node:lts-alpine - name: Install gh-aw extension env: GH_TOKEN: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} @@ -214,13 +216,24 @@ jobs: gh extension install githubnext/gh-aw fi gh aw --version + # Copy the gh-aw binary to /opt/gh-aw for MCP server containerization + mkdir -p /opt/gh-aw + GH_AW_BIN=$(which gh-aw 2>/dev/null || find ~/.local/share/gh/extensions/gh-aw -name 'gh-aw' -type f 2>/dev/null | head -1) + if [ -n "$GH_AW_BIN" ] && [ -f "$GH_AW_BIN" ]; then + cp "$GH_AW_BIN" /opt/gh-aw/gh-aw + chmod +x /opt/gh-aw/gh-aw + echo "Copied gh-aw binary to /opt/gh-aw/gh-aw" + else + echo "::error::Failed to find gh-aw binary for MCP server" + exit 1 + fi - name: Write Safe Outputs Config run: | mkdir -p /opt/gh-aw/safeoutputs mkdir -p /tmp/gh-aw/safeoutputs mkdir -p /tmp/gh-aw/mcp-logs/safeoutputs cat > /opt/gh-aw/safeoutputs/config.json << 'EOF' - {"add_comment":{"max":1},"add_labels":{"allowed":["smoke-copilot"],"max":3},"create_issue":{"max":1},"missing_data":{},"missing_tool":{},"noop":{"max":1}} + {"add_comment":{"max":1},"add_labels":{"allowed":["smoke-copilot"],"max":3},"create_issue":{"group":true,"max":1},"missing_data":{},"missing_tool":{},"noop":{"max":1}} EOF cat > /opt/gh-aw/safeoutputs/tools.json << 'EOF' [ @@ -302,9 +315,6 @@ jobs: "type": "array" } }, - "required": [ - "labels" - ], "type": "object" }, "name": "add_labels" @@ -500,7 +510,7 @@ jobs: # Register API key as secret to mask it from logs echo "::add-mask::${MCP_GATEWAY_API_KEY}" export GH_AW_ENGINE="copilot" - export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network host -v /var/run/docker.sock:/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e DEBUG="*" -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_LOCKDOWN -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw ghcr.io/githubnext/gh-aw-mcpg:latest' + export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network host -v /var/run/docker.sock:/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e DEBUG="*" -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_LOCKDOWN -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw ghcr.io/githubnext/gh-aw-mcpg:v0.0.62' mkdir -p /home/runner/.copilot cat << MCPCONFIG_EOF | bash /opt/gh-aw/actions/start_mcp_gateway.sh @@ -621,59 +631,10 @@ jobs: script: | const { generateWorkflowOverview } = require('/opt/gh-aw/actions/generate_workflow_overview.cjs'); await generateWorkflowOverview(core); - - name: Create prompt + - name: Create prompt with built-in context env: GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_SAFE_OUTPUTS: ${{ env.GH_AW_SAFE_OUTPUTS }} - GH_AW_GITHUB_REPOSITORY: ${{ github.repository }} - GH_AW_GITHUB_RUN_ID: ${{ github.run_id }} - run: | - bash /opt/gh-aw/actions/create_prompt_first.sh - cat << 'PROMPT_EOF' > "$GH_AW_PROMPT" - # Smoke Test: Copilot Engine Validation - - **IMPORTANT: Keep all outputs extremely short and concise. Use single-line responses where possible. No verbose explanations.** - - ## Test Requirements - - 1. **GitHub MCP Testing**: Review the last 2 merged pull requests in __GH_AW_GITHUB_REPOSITORY__ - 2. **Serena Go Testing**: Use the `serena-go` tool to run a basic go command like "go version" to verify the tool is available - 3. **Playwright Testing**: Use playwright to navigate to and verify the page title contains "GitHub" - 4. **File Writing Testing**: Create a test file `/tmp/gh-aw/agent/smoke-test-copilot-__GH_AW_GITHUB_RUN_ID__.txt` with content "Smoke test passed for Copilot at $(date)" (create the directory if it doesn't exist) - 5. **Bash Tool Testing**: Execute bash commands to verify file creation was successful (use `cat` to read the file back) - - ## Output - - Add a **very brief** comment (max 5-10 lines) to the current pull request with: - - PR titles only (no descriptions) - - ✅ or ❌ for each test result - - Overall status: PASS or FAIL - - Mention the pull request author and any assignees - - If all tests pass, add the label `smoke-copilot` to the pull request. - - PROMPT_EOF - - name: Substitute placeholders - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0 - env: - GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt - GH_AW_GITHUB_REPOSITORY: ${{ github.repository }} - GH_AW_GITHUB_RUN_ID: ${{ github.run_id }} - with: - script: | - const substitutePlaceholders = require('/opt/gh-aw/actions/substitute_placeholders.cjs'); - - // Call the substitution function - return await substitutePlaceholders({ - file: process.env.GH_AW_PROMPT, - substitutions: { - GH_AW_GITHUB_REPOSITORY: process.env.GH_AW_GITHUB_REPOSITORY, - GH_AW_GITHUB_RUN_ID: process.env.GH_AW_GITHUB_RUN_ID - } - }); - - name: Append context instructions to prompt - env: - GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_GITHUB_ACTOR: ${{ github.actor }} GH_AW_GITHUB_EVENT_COMMENT_ID: ${{ github.event.comment.id }} GH_AW_GITHUB_EVENT_DISCUSSION_NUMBER: ${{ github.event.discussion.number }} @@ -681,8 +642,13 @@ jobs: GH_AW_GITHUB_EVENT_PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number }} GH_AW_GITHUB_REPOSITORY: ${{ github.repository }} GH_AW_GITHUB_RUN_ID: ${{ github.run_id }} + GH_AW_GITHUB_SERVER_URL: ${{ github.server_url }} GH_AW_GITHUB_WORKSPACE: ${{ github.workspace }} run: | + bash /opt/gh-aw/actions/create_prompt_first.sh + cat << 'PROMPT_EOF' > "$GH_AW_PROMPT" + + PROMPT_EOF cat "/opt/gh-aw/prompts/temp_folder_prompt.md" >> "$GH_AW_PROMPT" cat "/opt/gh-aw/prompts/playwright_prompt.md" >> "$GH_AW_PROMPT" cat << 'PROMPT_EOF' >> "$GH_AW_PROMPT" @@ -748,18 +714,91 @@ jobs: PROMPT_EOF + cat << 'PROMPT_EOF' >> "$GH_AW_PROMPT" + + PROMPT_EOF + cat << 'PROMPT_EOF' >> "$GH_AW_PROMPT" + # Smoke Test: Copilot Engine Validation + + **IMPORTANT: Keep all outputs extremely short and concise. Use single-line responses where possible. No verbose explanations.** + + ## Test Requirements + + 1. **GitHub MCP Testing**: Review the last 2 merged pull requests in __GH_AW_GITHUB_REPOSITORY__ + 2. **Serena Go Testing**: Use the `serena` tool to run a basic go command like "go version" to verify the tool is available + 3. **Playwright Testing**: Use playwright to navigate to and verify the page title contains "GitHub" + 4. **File Writing Testing**: Create a test file `/tmp/gh-aw/agent/smoke-test-copilot-__GH_AW_GITHUB_RUN_ID__.txt` with content "Smoke test passed for Copilot at $(date)" (create the directory if it doesn't exist) + 5. **Bash Tool Testing**: Execute bash commands to verify file creation was successful (use `cat` to read the file back) + + ## Output + + 1. **Create an issue** with a summary of the smoke test run: + - Title: "Smoke Test: Copilot - __GH_AW_GITHUB_RUN_ID__" + - Body should include: + - Test results (✅ or ❌ for each test) + - Overall status: PASS or FAIL + - Run URL: __GH_AW_GITHUB_SERVER_URL__/__GH_AW_GITHUB_REPOSITORY__/actions/runs/__GH_AW_GITHUB_RUN_ID__ + - Timestamp + - Pull request author and assignees + + 2. Add a **very brief** comment (max 5-10 lines) to the current pull request with: + - PR titles only (no descriptions) + - ✅ or ❌ for each test result + - Overall status: PASS or FAIL + - Mention the pull request author and any assignees + + If all tests pass, add the label `smoke-copilot` to the pull request. + + PROMPT_EOF + - name: Substitute placeholders + uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0 + env: + GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt + GH_AW_GITHUB_ACTOR: ${{ github.actor }} + GH_AW_GITHUB_EVENT_COMMENT_ID: ${{ github.event.comment.id }} + GH_AW_GITHUB_EVENT_DISCUSSION_NUMBER: ${{ github.event.discussion.number }} + GH_AW_GITHUB_EVENT_ISSUE_NUMBER: ${{ github.event.issue.number }} + GH_AW_GITHUB_EVENT_PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number }} + GH_AW_GITHUB_REPOSITORY: ${{ github.repository }} + GH_AW_GITHUB_RUN_ID: ${{ github.run_id }} + GH_AW_GITHUB_SERVER_URL: ${{ github.server_url }} + GH_AW_GITHUB_WORKSPACE: ${{ github.workspace }} + with: + script: | + const substitutePlaceholders = require('/opt/gh-aw/actions/substitute_placeholders.cjs'); + + // Call the substitution function + return await substitutePlaceholders({ + file: process.env.GH_AW_PROMPT, + substitutions: { + GH_AW_GITHUB_ACTOR: process.env.GH_AW_GITHUB_ACTOR, + GH_AW_GITHUB_EVENT_COMMENT_ID: process.env.GH_AW_GITHUB_EVENT_COMMENT_ID, + GH_AW_GITHUB_EVENT_DISCUSSION_NUMBER: process.env.GH_AW_GITHUB_EVENT_DISCUSSION_NUMBER, + GH_AW_GITHUB_EVENT_ISSUE_NUMBER: process.env.GH_AW_GITHUB_EVENT_ISSUE_NUMBER, + GH_AW_GITHUB_EVENT_PULL_REQUEST_NUMBER: process.env.GH_AW_GITHUB_EVENT_PULL_REQUEST_NUMBER, + GH_AW_GITHUB_REPOSITORY: process.env.GH_AW_GITHUB_REPOSITORY, + GH_AW_GITHUB_RUN_ID: process.env.GH_AW_GITHUB_RUN_ID, + GH_AW_GITHUB_SERVER_URL: process.env.GH_AW_GITHUB_SERVER_URL, + GH_AW_GITHUB_WORKSPACE: process.env.GH_AW_GITHUB_WORKSPACE + } + }); - name: Interpolate variables and render templates uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0 env: GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_GITHUB_REPOSITORY: ${{ github.repository }} GH_AW_GITHUB_RUN_ID: ${{ github.run_id }} + GH_AW_GITHUB_SERVER_URL: ${{ github.server_url }} with: script: | const { setupGlobals } = require('/opt/gh-aw/actions/setup_globals.cjs'); setupGlobals(core, github, context, exec, io); const { main } = require('/opt/gh-aw/actions/interpolate_prompt.cjs'); await main(); + - name: Validate prompt placeholders + env: + GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt + run: bash /opt/gh-aw/actions/validate_prompt_placeholders.sh - name: Print prompt env: GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt @@ -998,6 +1037,7 @@ jobs: GH_AW_WORKFLOW_NAME: "Smoke Copilot" GH_AW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} GH_AW_AGENT_CONCLUSION: ${{ needs.agent.result }} + GH_AW_SECRET_VERIFICATION_RESULT: ${{ needs.agent.outputs.secret_verification_result }} GH_AW_SAFE_OUTPUT_MESSAGES: "{\"footer\":\"\\u003e 📰 *BREAKING: Report filed by [{workflow_name}]({run_url})*\",\"runStarted\":\"📰 BREAKING: [{workflow_name}]({run_url}) is now investigating this {event_type}. Sources say the story is developing...\",\"runSuccess\":\"📰 VERDICT: [{workflow_name}]({run_url}) has concluded. All systems operational. This is a developing story. 🎤\",\"runFailure\":\"📰 DEVELOPING STORY: [{workflow_name}]({run_url}) reports {status}. Our correspondents are investigating the incident...\"}" with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} @@ -1121,6 +1161,7 @@ jobs: mkdir -p /tmp/gh-aw/threat-detection touch /tmp/gh-aw/threat-detection/detection.log - name: Validate COPILOT_GITHUB_TOKEN secret + id: validate-secret run: /opt/gh-aw/actions/validate_multi_secret.sh COPILOT_GITHUB_TOKEN 'GitHub Copilot CLI' https://githubnext.github.io/gh-aw/reference/engines/#github-copilot-default env: COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }} @@ -1279,7 +1320,7 @@ jobs: uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0 env: GH_AW_AGENT_OUTPUT: ${{ env.GH_AW_AGENT_OUTPUT }} - GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG: "{\"add_comment\":{\"hide_older_comments\":true,\"max\":1},\"add_labels\":{\"allowed\":[\"smoke-copilot\"]},\"create_issue\":{\"expires\":2,\"max\":1},\"missing_data\":{},\"missing_tool\":{}}" + GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG: "{\"add_comment\":{\"hide_older_comments\":true,\"max\":1},\"add_labels\":{\"allowed\":[\"smoke-copilot\"]},\"create_issue\":{\"expires\":2,\"group\":true,\"max\":1},\"missing_data\":{},\"missing_tool\":{}}" with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} script: | From c04d753019b3ec3c85c3757c64c423b4d2065c12 Mon Sep 17 00:00:00 2001 From: Landon Cox Date: Sun, 18 Jan 2026 10:26:24 -0800 Subject: [PATCH 3/3] Use remote setup script --- .github/workflows/smoke-copilot.lock.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/smoke-copilot.lock.yml b/.github/workflows/smoke-copilot.lock.yml index 51b91a61..f46023ed 100644 --- a/.github/workflows/smoke-copilot.lock.yml +++ b/.github/workflows/smoke-copilot.lock.yml @@ -977,7 +977,7 @@ jobs: actions persist-credentials: false - name: Setup Scripts - uses: ./actions/setup + uses: githubnext/gh-aw/actions/setup@v0.36.0 with: destination: /opt/gh-aw/actions - name: Debug job inputs