fix: add explicit execute directive to smoke-codex to prevent noop#1078
Conversation
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Updates the Codex smoke-test prompt to explicitly instruct the agent to execute the task (rather than classifying the content as repository documentation and no-op’ing), improving reliability of the Smoke Codex agentic workflow.
Changes:
- Adds an “EXECUTE NOW” directive at the top of the
smoke-codex.mdtask body to force action execution. - Keeps the existing 1–8 smoke test steps unchanged; the directive is intended to ensure they run.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| echo "Safe output validation passed" | ||
| --- | ||
|
|
||
| > **⚡ EXECUTE NOW:** Run all steps 1–8 below and call `add_comment` on this pull request with results. This is a mandatory smoke test execution—not documentation to read. |
There was a problem hiding this comment.
This directive unconditionally tells the agent to call add_comment “on this pull request”, but this workflow also runs on schedule and workflow_dispatch where there may be no PR context. Consider making the instruction conditional (e.g., only require add_comment when ${{ github.event_name }} is pull_request), aligning with the post-step validation that only enforces add_comment for PR triggers.
The Codex smoke test was failing because
gpt-5.2-codexcallednoopwith "No task requested beyond providing repository instructions; no actions taken" — skipping all smoke tests entirely.Root cause: Codex CLI reads
AGENTS.md(26KB) from the working directory before processing the prompt. SinceAGENTS.mdopens with "This file provides guidance to coding agent when working with code in this repository," and the prompt itself is dense with instructional content (security policy, tool usage guides, report structure), the model classifies the entire context as documentation and finds no executable task — even though# Smoke Test: Codex Engine Validationwith 8 explicit steps is present at the end.Confirmed via artifact analysis: the agent listed MCP tools from all 5 servers (safeinputs, safeoutputs, playwright, github, tavily) then called
noopwithin ~3 seconds, never attempting any test.Fix:
Added a blockquote directive at the top of the
smoke-codex.mdtask body that explicitly breaks the documentation-classification pattern:> **⚡ EXECUTE NOW:** Run all steps 1–8 below and call `add_comment` on this pull request with results. This is a mandatory smoke test execution—not documentation to read.The lock file uses
{{#runtime-import .github/workflows/smoke-codex.md}}so this takes effect at workflow runtime without recompiling.Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
https://api.github.com/graphql/usr/bin/gh gh auth status(http block)/usr/bin/gh gh run list -w Smoke Codex --limit 10 --json databaseId,conclusion,headBranch,event,createdAt(http block)https://api.github.com/repos/github/gh-aw-firewall/actions/runs/22468322854/artifacts/usr/bin/gh gh run download 22468322854 -R github/gh-aw-firewall --name safe-output --dir ./safe-output-22468322854(http block)/usr/bin/gh gh run download 22468322854 -R github/gh-aw-firewall --name agent-artifacts --dir ./agent-artifacts-22468322854(http block)If you need me to access, download, or install something from one of these locations, you can either:
Original prompt
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.