Skip to content

Add automated test failure reporting to CI workflows#14302

Merged
pelikhan merged 3 commits intomainfrom
copilot/investigate-ci-failure-34180
Feb 7, 2026
Merged

Add automated test failure reporting to CI workflows#14302
pelikhan merged 3 commits intomainfrom
copilot/investigate-ci-failure-34180

Conversation

Copy link
Contributor

Copilot AI commented Feb 7, 2026

CI run #34180 failed with a package-level test failure but no individual test failures, requiring manual JSON log parsing to investigate. This occurs when tests panic during initialization, race conditions are detected, build errors happen, or tests timeout.

Changes

New script: scripts/report-test-failures.sh

  • Parses go test -json output for both individual and package-level failures
  • Extracts failure context (test name, package, elapsed time, recent output)
  • Provides diagnostic hints for package-level failures

CI integration: .github/workflows/ci.yml

  • Runs failure report script automatically when unit or integration tests fail
  • Displays structured failure analysis in $GITHUB_STEP_SUMMARY

Example output for package-level failures:

📦 Package-level Failure:
   Package: github.com/github/gh-aw/pkg/workflow
   Elapsed: 10.837s
   
   ⚠️  No individual test marked as failed!
   This could indicate:
   - A test panicked during initialization
   - A race condition detected by -race flag
   - A build/compilation issue in test code
   - A test timeout

Future CI test failures will surface immediately with actionable debugging information instead of requiring JSON log analysis.

Original prompt

This section details on the original issue you should resolve

<issue_title>[CI Failure Doctor] CI Failure Investigation - Run #34180</issue_title>
<issue_description># 🏥 CI Failure Investigation - Run github/gh-aw#34180

Summary

The integration job Integration: Workflow Misc Part 2 fails because go test for github.com/github/gh-aw/pkg/workflow exits with FAIL, but the JSON log contains only passing tests and general warnings about missing GitHub toolset permissions, so the exact failure point is not visible in the current output.

Failure Details

  • Run: 21774366589
  • Commit: a25956906dbbb7385ebe4402f51cf073d7a3df75
  • Trigger: push

Root Cause Analysis

go test completes but emits a single Action:"fail" event for the package summary; no individual test is marked as failed, and there are no panic/fatal messages in the log. The striking output consists only of warnings about "Missing required permissions for github toolsets", so from the log alone we cannot pin down which test or code change caused the non-zero exit status.

Failed Jobs and Errors

  • Integration: Workflow Misc Part 2: go test -tags integration prints FAIL github.com/github/gh-aw/pkg/workflow 10.837s and the test-result-integration-Workflow Misc Part 2.json log contains no per-test failures, only the package-level failure entry.

Investigation Findings

  • Every recorded test run in the JSON log reports Action:"pass", and the only Action:"fail" entry is the final summary for the pkg/workflow package.
  • No panic, FAIL, or fatal strings appear anywhere in the log, so nothing identifies the failing test or step.
  • Many tests emit warnings about missing permissions for GitHub toolsets (issues: read / pull-requests: read), but the warnings themselves are not flagged as failures.
  • A local attempt to rerun the same go test command fails before any tests run because proxy.golang.org returns 403 Forbidden when fetching dependencies, so I could not rerun the suite in this environment to collect additional details.

Recommended Actions

  • Re-run the failing workflow with debug/verbose output (or rerun the Go command manually with the same -tags, -run, and -skip flags) while teeing the JSON output so the first Action:"fail" is visible before the command exits.
  • Add a helper step that scans test-result-integration-*.json and prints any Action:"fail" entries when the job fails, so investigators immediately know which test or package tripped the failure.
  • If the failure reappears, examine whether recent documentation or permission-related changes introduced additional warnings that are now causing the suite to treat the build as failed.

Prevention Strategies

Document and surface the failing test name (and failure message) by parsing test-result-integration-*.json whenever go test exits non-zero, so future CI failures are unambiguous even when the JSON log is long.

AI Team Self-Improvement

  • Always inspect the test-result-*.json output for "Action":"fail" before concluding a job failure.
  • If the JSON log only has a package-level failure, rerun the exact go test command locally with -run/-skip filters to narrow down the culprit.

Historical Context

No prior investigation exists for this workflow run, and this is the first recorded failure of the Workflow Misc Part 2 integration matrix in run 21774366589.

AI generated by CI Failure Doctor

To add this workflow in your repository, run gh aw add githubnext/agentics/workflows/ci-doctor.md@ea350161ad5dcc9624cf510f134c6a9e39a6f94d. See usage guide.

  • expires on Feb 8, 2026, 5:04 AM UTC

Comments on the Issue (you are @copilot in this section)


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 2 commits February 7, 2026 07:24
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Investigate CI failure for run #34180 Add automated test failure reporting to CI workflows Feb 7, 2026
Copilot AI requested a review from pelikhan February 7, 2026 07:31
@pelikhan pelikhan marked this pull request as ready for review February 7, 2026 13:26
Copilot AI review requested due to automatic review settings February 7, 2026 13:26
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds automated parsing and surfacing of go test -json failures in CI so that package-level failures (with no individual test failures) are immediately diagnosable from the GitHub Actions job summary.

Changes:

  • Add scripts/report-test-failures.sh to extract individual test failures and package-level failures (plus nearby output) from go test -json logs.
  • Add a lightweight bash test harness scripts/report-test-failures_test.sh with fixture JSON inputs.
  • Integrate failure reporting into .github/workflows/ci.yml so reports are appended to $GITHUB_STEP_SUMMARY when unit/integration tests fail.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
scripts/report-test-failures.sh New parser/report script for go test -json failures (individual + package-level).
scripts/report-test-failures_test.sh New bash test harness validating expected reporting behavior across scenarios.
.github/workflows/ci.yml Runs the report script automatically on test failures and posts results to the step summary.
Comments suppressed due to low confidence (1)

scripts/report-test-failures.sh:72

  • The grep patterns that interpolate $TEST_NAME are treated as regular expressions. Subtest names can contain regex metacharacters (and package names contain dots), which can cause over-matching or missed matches and lead to showing the wrong “Recent test output”. Use fixed-string matching (e.g., grep -F) or escape the variable before passing it to grep.
      grep "\"Test\":\"$TEST_NAME\"" "$file" | grep '"Action":"output"' | tail -10 | while IFS= read -r output_line; do
        OUTPUT=$(echo "$output_line" | sed 's/.*"Output":"\(.*\)".*/\1/' | sed 's/\\n/\n/g' | sed 's/\\t/\t/g')
        echo "   $OUTPUT"
      done

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +89 to +92
grep "\"Package\":\"$PACKAGE\"" "$file" | grep '"Action":"output"' | tail -20 | while IFS= read -r output_line; do
OUTPUT=$(echo "$output_line" | sed 's/.*"Output":"\(.*\)".*/\1/' | sed 's/\\n/\n/g' | sed 's/\\t/\t/g')
echo " $OUTPUT"
done
Copy link

Copilot AI Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This grep interpolates $PACKAGE into a regex; package paths contain '.' which will match any character in regex mode. That can pull output from the wrong package into the “Recent package output” section. Use fixed-string matching (grep -F) or escape $PACKAGE before building the pattern.

This issue also appears on line 69 of the same file.

Copilot uses AI. Check for mistakes.
Comment on lines +44 to +55
if "$REPORT_SCRIPT" "$TEST_DIR/individual-failure.json" > /tmp/test2-output.txt 2>&1; then
echo "❌ FAIL: Should exit 1 when failures found"
exit 1
else
if grep -q "TestSomething" /tmp/test2-output.txt && grep -q "test.go:123" /tmp/test2-output.txt; then
echo "✅ PASS: Correctly detected and reported individual test failure"
else
echo "❌ FAIL: Missing expected failure details"
cat /tmp/test2-output.txt
exit 1
fi
fi
Copy link

Copilot AI Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test script writes outputs to fixed filenames under /tmp (e.g., /tmp/test2-output.txt). This can collide across parallel runs and leaves stray files behind. Prefer mktemp for output files (and remove them in the existing cleanup trap).

Copilot uses AI. Check for mistakes.
@pelikhan pelikhan merged commit e7d9b96 into main Feb 7, 2026
161 of 163 checks passed
@pelikhan pelikhan deleted the copilot/investigate-ci-failure-34180 branch February 7, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI Failure Doctor] CI Failure Investigation - Run #34180

3 participants