fix: surface Codex model access blocked errors in step summary#18699
fix: surface Codex model access blocked errors in step summary#18699
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Surfaces previously dropped Codex plain-text error lines (e.g., model access blocked / policy violations) in the rendered markdown step summary by extracting them into a dedicated Errors section.
Changes:
- Added
extractCodexErrorMessages(lines)to detectERROR:andReconnecting... N/M (...)patterns, de-duplicate messages, and track reconnect attempts. - Injected a
## ⚠️ Errorssection into the Codex markdown output before## 🤖 Reasoningwhen errors are detected. - Exported the new helper and added unit tests covering extraction and markdown ordering.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| actions/setup/js/parse_codex_log.cjs | Adds error extraction + renders an Errors section in the Codex markdown output and exports the helper for testing. |
| actions/setup/js/parse_codex_log.test.cjs | Adds tests for the new error extractor and verifies the Errors section appears and is ordered correctly. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| for (const line of lines) { | ||
| // Match: ERROR: <message> (final error after all retries exhausted) | ||
| const errorMatch = line.match(/^ERROR:\s*(.+)$/); | ||
| if (errorMatch) { | ||
| messages.add(errorMatch[1].trim()); | ||
| } | ||
|
|
||
| // Match: Reconnecting... N/M (error message) - reconnect attempts with error details | ||
| const reconnectMatch = line.match(/^Reconnecting\.\.\.\s+(\d+)\/(\d+)\s*\((.+)\)$/); | ||
| if (reconnectMatch) { | ||
| const attempt = parseInt(reconnectMatch[1]); | ||
| const total = parseInt(reconnectMatch[2]); | ||
| if (attempt > reconnectCount) reconnectCount = attempt; | ||
| if (total > maxReconnects) maxReconnects = total; | ||
| messages.add(reconnectMatch[3].trim()); | ||
| } |
There was a problem hiding this comment.
extractCodexErrorMessages uses an end-anchored reconnect regex (...\((.+)\)$). If log lines have trailing whitespace or CRLF (\r), reconnect lines like Reconnecting... 1/5 (...)\r won't match and retries won’t be surfaced. Consider normalizing each line with trimEnd() before matching, or relaxing the pattern to allow optional whitespace/\r after the closing ) (e.g. \)\s*$).
| const attempt = parseInt(reconnectMatch[1]); | ||
| const total = parseInt(reconnectMatch[2]); |
There was a problem hiding this comment.
extractCodexErrorMessages uses parseInt(...) without a radix. It’s safer and consistent to pass base 10 (or use Number(...)) even though the regex only matches digits.
| const attempt = parseInt(reconnectMatch[1]); | |
| const total = parseInt(reconnectMatch[2]); | |
| const attempt = parseInt(reconnectMatch[1], 10); | |
| const total = parseInt(reconnectMatch[2], 10); |
| // Extract error messages (e.g., model access blocked, cyber_policy_violation) | ||
| const errorInfo = extractCodexErrorMessages(lines); | ||
| if (errorInfo.hasErrors) { | ||
| markdown += "## ⚠️ Errors\n\n"; | ||
| for (const message of errorInfo.messages) { | ||
| markdown += `> ${message}\n\n`; | ||
| } | ||
| if (errorInfo.reconnectCount > 0) { | ||
| markdown += `> Reconnect attempts: ${errorInfo.reconnectCount}/${errorInfo.maxReconnects}\n\n`; | ||
| } | ||
| } |
There was a problem hiding this comment.
parseCodexLog now surfaces ERROR: / Reconnecting... lines in an Errors section, but those same lines can still be treated as “thinking” content if they appear after a thinking marker (they’re not in the skip list). That can duplicate error text under ## 🤖 Reasoning. Consider skipping these patterns in the main parsing loop (e.g., extend the metadata skip condition to include line.startsWith("ERROR:") / line.startsWith("Reconnecting...")) once they’ve been extracted.
| beforeEach(async () => { | ||
| const module = await import("./parse_codex_log.cjs"); | ||
| extractCodexErrorMessages = module.extractCodexErrorMessages; | ||
| }); |
There was a problem hiding this comment.
The outer beforeEach already imports ./parse_codex_log.cjs for every test; this nested beforeEach re-imports the same module again, doubling work for this describe block. Consider wiring extractCodexErrorMessages into the existing outer setup (add a let extractCodexErrorMessages; alongside the other helpers) and remove this nested import.
When Codex hits a
cyber_policy_violation(or similar API error), the agent log emits plain-text error lines that the parser silently dropped — leaving the step summary with no explanation for the failure.Affected log patterns:
These lines weren't matched by the skip conditions (which filter timestamped debug/info lines) but also had no handler, so they fell through.
Changes
parse_codex_log.cjs: AddedextractCodexErrorMessages(lines)that scans for^ERROR:and^Reconnecting... N/M (...)patterns, de-duplicates the error message (same text repeats on each retry), and tracks retry count## ⚠️ Errorssection into the markdown output when errors are detected, positioned before## 🤖 ReasoningextractCodexErrorMessagesand added 9 tests covering extraction, de-duplication, retry count tracking, false-positive avoidance, and section orderingWarning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
https://api.github.com/graphql/usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GO111MODULE 64/bin/go git conf�� --get remote.origin.url /usr/bin/git pvF5/gNK6rERQOWhgit GO111MODULE 64/bin/go git(http block)https://api.github.com/repos/actions/ai-inference/git/ref/tags/v1/usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq .object.sha -json GO111MODULE 6444111/b285/vet.cfg GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v3/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq .object.sha agent-persona-explorer.md GO111MODULE 1/x64/bin/node GOINSECURE GOMOD GOMODCACHE go t-ha�� vaScript2764459125/001/test-simple-frontmatter.md GO111MODULE ache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v5/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha k/gh-aw/gh-aw/pkg/styles/theme.go k/gh-aw/gh-aw/pkg/styles/theme_test.go /usr/bin/git GOSUMDB GOWORK 64/bin/go git rev-�� --show-toplevel rtcfg /usr/bin/gh b136a0101c461533git GO111MODULE 64/bin/go gh(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq .object.sha --git-dir sh /usr/bin/git "prettier" --chegit node 64/bin/go git init�� 64/bin/go PIlg48W/J-s6rDiV_9LLtYheNxUI /usr/bin/git ty-test.md GO111MODULE cec38b61a80db28f--show-toplevel /usr/bin/git(http block)https://api.github.com/repos/actions/checkout/git/ref/tags/v6/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq .object.sha -json GO111MODULE 6444111/b411/repoutil.test GOINSECURE GOMOD GOMODCACHE 6444111/b411/repoutil.test e=/t�� 2957-29248/test-1218850741 GO111MODULE(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq .object.sha -json GO111MODULE /opt/hostedtoolcache/go/1.25.0/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env runs/20260227-142957-29248/test-4028165192/.github/workflows GO111MODULE 1/x64/bin/node GOINSECURE GOMOD GOMODCACHE 1/x64/bin/node(http block)/usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq .object.sha --show-current node /usr/bin/git prettier --check 64/bin/go git conf�� user.email test@example.com /usr/bin/git GOPATH sh 64/bin/go git(http block)https://api.github.com/repos/actions/github-script/git/ref/tags/v8/usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE go m/_n�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha ck 'scripts/**/*GOINSECURE GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE 4OUXOOJWc0iX env -json GO111MODULE 47092b0907500211-d GOINSECURE GOMOD GOMODCACHE sh(http block)/usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq .object.sha ript formatting GOINSECURE GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE wCOxz1AFAzDu env -json GO111MODULE 808bd37381275d0a-d GOINSECURE GOMOD GOMODCACHE node(http block)https://api.github.com/repos/actions/setup-go/git/ref/tags/v4/usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq .object.sha -json GO111MODULE 1/x64/bin/node GOINSECURE GOMOD GOMODCACHE go -has�� ithub/workflows/bot-detection.md GO111MODULE ache/node/24.13.1/x64/bin/node l GOMOD GOMODCACHE go(http block)https://api.github.com/repos/actions/setup-node/git/ref/tags/v4/usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq .object.sha -json GO111MODULE ache/go/1.25.0/x64/pkg/tool/linu-test.short=true GOINSECURE GOMOD GOMODCACHE ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet env -json GO111MODULE ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE ache/go/1.25.0/x64/pkg/tool/linux_amd64/vet(http block)https://api.github.com/repos/github/gh-aw/actions/runs/1/artifacts/usr/bin/gh gh run download 1 --dir test-logs/run-1 GO111MODULE x_amd64/link GOINSECURE GOMOD GOMODCACHE x_amd64/link env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE uV/nLUya0DMdbzlh8afRszc/J-5ptimqu69dtnPwupbS(http block)https://api.github.com/repos/github/gh-aw/actions/runs/12345/artifacts/usr/bin/gh gh run download 12345 --dir test-logs/run-12345 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/12346/artifacts/usr/bin/gh gh run download 12346 --dir test-logs/run-12346 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/2/artifacts/usr/bin/gh gh run download 2 --dir test-logs/run-2 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/3/artifacts/usr/bin/gh gh run download 3 --dir test-logs/run-3 GO111MODULE x_amd64/link GOINSECURE GOMOD GOMODCACHE x_amd64/link env -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet(http block)https://api.github.com/repos/github/gh-aw/actions/runs/4/artifacts/usr/bin/gh gh run download 4 --dir test-logs/run-4 GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/runs/5/artifacts/usr/bin/gh gh run download 5 --dir test-logs/run-5 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go estl�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/github/gh-aw/actions/workflows/usr/bin/gh gh workflow list --json name,state,path GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE erignore env e=false GO111MODULE 64/bin/go GOINSECURE %H %ct %D GOMODCACHE go(http block)/usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 100 GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE node(http block)/usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 6 GOMOD GOMODCACHE x_amd64/link env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE RS/30TOoEVfShYOYdEkq8TH/T8gtgSrY2uJORVhNLsE9(http block)https://api.github.com/repos/github/gh-aw/contents/.github%2Fworkflows%2Faudit-workflows.md/opt/hostedtoolcache/node/24.13.1/x64/bin/node /opt/hostedtoolcache/node/24.13.1/x64/bin/node --conditions node --conditions development --experimental-import-meta-resolve --require /home/REDACTED/work/gh-aw/gh-aw/actions/setup/js/node_modules/vitest/suppress-warnings.cjs /home/REDACTED/work/gh-aw/gh-aw/actions/setup/js/node_modules/vitest/dist/workers/forks.js Initial force chfetch it git conf�� user.name Test User cal/bin/git --amend m it git(http block)https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.0.0/usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.0.0 --jq .object.sha -json GO111MODULE At,event,headBranch,headSha,displayTitle GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/nonexistent/action/git/ref/tags/v999.999.999/usr/bin/gh gh api /repos/nonexistent/action/git/ref/tags/v999.999.999 --jq .object.sha -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env(http block)https://api.github.com/repos/nonexistent/repo/actions/runs/12345/usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion GOINSECURE GOMOD GOMODCACHE x_amd64/link env -json GO111MODULE x_amd64/vet GOINSECURE GOMOD GOMODCACHE x_amd64/vet(http block)https://api.github.com/repos/owner/repo/actions/workflows/usr/bin/gh gh workflow list --json name,state,path --repo owner/repo 64/bin/go GOINSECURE GOMOD GOMODCACHE ache/go/1.25.0/xGO111MODULE env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)/usr/bin/gh gh workflow list --json name,state,path --repo owner/repo 64/bin/go GOINSECURE GOMOD GOMODCACHE ache/go/1.25.0/xGO111MODULE env 971469/b385/_pkgGOINSECURE GO111MODULE 64/bin/go GOINSECURE b/gh-aw/pkg/consenv GOMODCACHE go(http block)https://api.github.com/repos/owner/repo/contents/file.md/tmp/go-build3946444111/b381/cli.test /tmp/go-build3946444111/b381/cli.test -test.testlogfile=/tmp/go-build3946444111/b381/testlog.txt -test.paniconexit0 -test.v=true -test.parallel=4 -test.timeout=10m0s -test.run=^Test -test.short=true GOINSECURE GOMOD GOMODCACHE go env ck 'scripts/**/*GOINSECURE GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)https://api.github.com/repos/test-owner/test-repo/actions/secrets/usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE erignore env tformat GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go(http block)If you need me to access, download, or install something from one of these locations, you can either:
Original prompt
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.