fix: make smoke-claude safe outputs trigger-aware for workflow_dispatch by lpcox · Pull Request #2036 · github/gh-aw-firewall

lpcox · 2026-04-16T21:46:33Z

Problem

The smoke-claude workflow fails on workflow_dispatch (manual) and schedule triggers because the prompt unconditionally tells the agent to add comments and labels to "the pull request."

add_comment gracefully skips when there's no PR context
add_labels hard-fails with: No issue/PR number available

This causes the safe_outputs job to fail, marking the entire run as failed even though the agent job succeeded.

Failed run: https://github.com/github/gh-aw-firewall/actions/runs/24535395752

Fix

Updated the prompt to be trigger-aware:

PR triggers: add comment + label as before
Non-PR triggers (workflow_dispatch, schedule): use noop to report results instead of attempting add_comment/add_labels

Also recompiled the lock file and ran post-processing.

The smoke-claude prompt unconditionally instructs the agent to add comments and labels to 'the pull request', but workflow_dispatch and schedule triggers have no PR context. The add_labels handler hard-fails (unlike add_comment which skips gracefully), causing the safe_outputs job to fail on manual runs. Update the prompt to only emit add_comment/add_labels when triggered by a pull_request, and use noop for non-PR triggers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-04-16T21:47:33Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	84.33%	84.41%	📈 +0.08%
Statements	83.56%	83.64%	📈 +0.08%
Functions	87.39%	87.39%	➡️ +0.00%
Branches	74.78%	74.82%	📈 +0.04%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/docker-manager.ts`	86.8% → 87.1% (+0.30%)	86.4% → 86.7% (+0.29%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-04-16T21:49:28Z

Smoke Test Results

Test	Status
GitHub MCP (last 2 PRs: "chore: recompile workflows for gh-aw v0.68.5", "chore: upgrade gh-aw to v0.68.4 and recompile workflows")	✅
Playwright (github.com title contains "GitHub")	✅
File write (`smoke-test-claude-24535683334.txt`)	✅
Bash verification (cat file)	✅

Overall: PASS

💥 [THE END] — Illustrated by Smoke Claude

github-actions · 2026-04-16T21:49:35Z

✅ Smoke Test PASSED

Test Results:

✅ GitHub MCP: Retrieved PR feat: add AWF JSON/YAML config ingestion with schema validation and CLI precedence #2018 ("feat: add AWF JSON/YAML config ingestion...")
✅ GitHub.com: HTTP 200
✅ File I/O: /tmp/gh-aw/agent/smoke-test-copilot-24535683253.txt verified

Status: PASS
Author: @lpcox

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-04-16T21:49:45Z

🔥 Smoke Test: Copilot BYOK (Offline) — PASS ✅

Test	Result
GitHub MCP (list merged PRs)	✅ #2018: feat: add AWF JSON/YAML config ingestion
GitHub.com connectivity	✅ (gh CLI authenticated)
File write/read (`smoke-test-copilot-byok-24535683298.txt`)	✅
BYOK inference (agent responding)	✅

Running in BYOK offline mode (COPILOT_OFFLINE=true) via api-proxy → api.githubcopilot.com

cc @lpcox

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-04-16T21:50:08Z

Smoke Test: GitHub Actions Services Connectivity ✅

All checks passed:

Check	Result
Redis `PING` at `host.docker.internal:6379`	✅ `PONG`
PostgreSQL `pg_isready` at `host.docker.internal:5432`	✅ accepting connections
PostgreSQL `SELECT 1` on `smoketest` db as `postgres`	✅ returned `1`

Note: redis-cli was not available; Redis was tested via nc (raw Redis protocol), which returned +PONG.

🔌 Service connectivity validated by Smoke Services

Copilot

Pull request overview

Updates the “Smoke Claude” agentic workflow to avoid failing safe-outputs processing on non-PR triggers (manual/scheduled runs) by making the reporting instructions trigger-aware, then recompiles the generated lock workflow and updates the pinned setup action reference.

Changes:

Adjusts the smoke-claude prompt to differentiate PR vs non-PR (workflow_dispatch/schedule) reporting behavior.
Recompiles .github/workflows/smoke-claude.lock.yml with newer gh-aw tooling (v0.68.4) and associated generated updates.
Updates .github/aw/actions-lock.json to pin github/gh-aw-actions/setup to v0.68.4.

Show a summary per file

File	Description
.github/workflows/smoke-claude.md	Makes output instructions trigger-aware, introducing a non-PR reporting path via noop safe output.
.github/workflows/smoke-claude.lock.yml	Regenerates the compiled workflow with updated gh-aw versions and generated steps/config.
.github/aw/actions-lock.json	Updates the pinned `github/gh-aw-actions/setup` action version/SHA used by compilation.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 3/3 changed files
Comments generated: 2

Copilot · 2026-04-16T21:51:15Z

-If all tests pass, add the label `smoke-claude` to the pull request.
+If all tests pass, add the label `smoke-claude` to the pull request.
+
+**If triggered by workflow_dispatch or schedule** (no PR context), use a noop safe output to report the test results summary instead. Do NOT attempt to add comments or labels when there is no pull request.


The new non-PR trigger path relies on the agent producing a safe-output entry, but the instruction says "use a noop safe output" without explicitly naming the tool (noop) or making it unambiguous that it must be invoked as a safe-output tool call. Consider wording this as an explicit requirement to call the noop safe-output tool with the short test summary, so workflow_dispatch/schedule runs don't fail the "Validate safe outputs were invoked" post-step due to the agent only printing text.

Suggested change

**If triggered by workflow_dispatch or schedule** (no PR context), use a noop safe output to report the test results summary instead. Do NOT attempt to add comments or labels when there is no pull request.

**If triggered by workflow_dispatch or schedule** (no PR context), you must call the `noop` safe-output tool with a very brief test results summary. Do NOT only print the summary as plain text. Do NOT attempt to add comments or labels when there is no pull request.

Copilot · 2026-04-16T21:51:16Z

      - name: Upload agent artifacts
        if: always()
        continue-on-error: true
-        uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:


The workflow no longer uploads a dedicated firewall-audit-logs artifact (firewall logs/audit are now bundled into the agent artifact). Many other workflows in this repo still publish firewall-audit-logs, so removing it here can break any tooling that expects that artifact name and makes logs harder to consume independently. Consider restoring the separate firewall-audit-logs upload step (or uploading both) for consistency/backward-compatibility.

github-actions · 2026-04-16T21:52:15Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	all passed	✅ PASS
Go	env	✅	all passed	✅ PASS
Go	uuid	✅	all passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	all passed	✅ PASS
Node.js	execa	✅	all passed	✅ PASS
Node.js	p-limit	✅	all passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #2036 · ● 1.1M · ◷

github-actions · 2026-04-16T21:52:38Z

Smoke test report:

feat: add AWF JSON/YAML config ingestion with schema validation and CLI precedence
fix: add explicit model for Copilot BYOK smoke test
GitHub MCP ✅ | safeinputs-gh CLI ❌ | Playwright ✅ | Tavily ❌
File write ✅ | Bash cat ✅ | Discussion interaction ❌ | Build (npm ci && npm run build) ✅
Overall status: FAIL

🔮 The oracle has spoken through Smoke Codex

lpcox requested a review from Mossaka as a code owner April 16, 2026 21:46

Copilot AI review requested due to automatic review settings April 16, 2026 21:46

github-actions Bot added the smoke-claude label Apr 16, 2026

github-actions Bot added the smoke-copilot label Apr 16, 2026

github-actions Bot added smoke-copilot-byok smoke-services labels Apr 16, 2026

Copilot AI reviewed Apr 16, 2026

View reviewed changes

github-actions Bot added the build-test label Apr 16, 2026

lpcox merged commit d3e9ceb into main Apr 16, 2026
59 of 62 checks passed

lpcox deleted the copilot/fix-smoke-claude-dispatch branch April 16, 2026 22:02

This was referenced Apr 16, 2026

fix: allow package.json/lock in dep security monitor PRs #2041

Merged

[aw] No-Op Runs #2042

Closed

chore: upgrade all workflows to gh-aw v0.68.6 #2043

Closed

Fix BYOK smoke workflow COPILOT_MODEL fallback override in postprocessing #2049

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: make smoke-claude safe outputs trigger-aware for workflow_dispatch#2036

fix: make smoke-claude safe outputs trigger-aware for workflow_dispatch#2036
lpcox merged 1 commit intomainfrom
copilot/fix-smoke-claude-dispatch

lpcox commented Apr 16, 2026

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	If triggered by workflow_dispatch or schedule (no PR context), use a noop safe output to report the test results summary instead. Do NOT attempt to add comments or labels when there is no pull request.
	If triggered by workflow_dispatch or schedule (no PR context), you must call the `noop` safe-output tool with a very brief test results summary. Do NOT only print the summary as plain text. Do NOT attempt to add comments or labels when there is no pull request.

Conversation

lpcox commented Apr 16, 2026

Problem

Fix

Uh oh!

github-actions Bot commented Apr 16, 2026

✅ Coverage Check Passed

Overall Coverage

Uh oh!

github-actions Bot commented Apr 16, 2026

Smoke Test Results

Uh oh!

github-actions Bot commented Apr 16, 2026

✅ Smoke Test PASSED

Uh oh!

github-actions Bot commented Apr 16, 2026

🔥 Smoke Test: Copilot BYOK (Offline) — PASS ✅

Uh oh!

github-actions Bot commented Apr 16, 2026

Smoke Test: GitHub Actions Services Connectivity ✅

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 16, 2026

🏗️ Build Test Suite Results

Uh oh!

github-actions Bot commented Apr 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants