Add automated test failure reporting to CI workflows#14302
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds automated parsing and surfacing of go test -json failures in CI so that package-level failures (with no individual test failures) are immediately diagnosable from the GitHub Actions job summary.
Changes:
- Add
scripts/report-test-failures.shto extract individual test failures and package-level failures (plus nearby output) fromgo test -jsonlogs. - Add a lightweight bash test harness
scripts/report-test-failures_test.shwith fixture JSON inputs. - Integrate failure reporting into
.github/workflows/ci.ymlso reports are appended to$GITHUB_STEP_SUMMARYwhen unit/integration tests fail.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
scripts/report-test-failures.sh |
New parser/report script for go test -json failures (individual + package-level). |
scripts/report-test-failures_test.sh |
New bash test harness validating expected reporting behavior across scenarios. |
.github/workflows/ci.yml |
Runs the report script automatically on test failures and posts results to the step summary. |
Comments suppressed due to low confidence (1)
scripts/report-test-failures.sh:72
- The grep patterns that interpolate $TEST_NAME are treated as regular expressions. Subtest names can contain regex metacharacters (and package names contain dots), which can cause over-matching or missed matches and lead to showing the wrong “Recent test output”. Use fixed-string matching (e.g., grep -F) or escape the variable before passing it to grep.
grep "\"Test\":\"$TEST_NAME\"" "$file" | grep '"Action":"output"' | tail -10 | while IFS= read -r output_line; do
OUTPUT=$(echo "$output_line" | sed 's/.*"Output":"\(.*\)".*/\1/' | sed 's/\\n/\n/g' | sed 's/\\t/\t/g')
echo " $OUTPUT"
done
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| grep "\"Package\":\"$PACKAGE\"" "$file" | grep '"Action":"output"' | tail -20 | while IFS= read -r output_line; do | ||
| OUTPUT=$(echo "$output_line" | sed 's/.*"Output":"\(.*\)".*/\1/' | sed 's/\\n/\n/g' | sed 's/\\t/\t/g') | ||
| echo " $OUTPUT" | ||
| done |
There was a problem hiding this comment.
This grep interpolates $PACKAGE into a regex; package paths contain '.' which will match any character in regex mode. That can pull output from the wrong package into the “Recent package output” section. Use fixed-string matching (grep -F) or escape $PACKAGE before building the pattern.
This issue also appears on line 69 of the same file.
| if "$REPORT_SCRIPT" "$TEST_DIR/individual-failure.json" > /tmp/test2-output.txt 2>&1; then | ||
| echo "❌ FAIL: Should exit 1 when failures found" | ||
| exit 1 | ||
| else | ||
| if grep -q "TestSomething" /tmp/test2-output.txt && grep -q "test.go:123" /tmp/test2-output.txt; then | ||
| echo "✅ PASS: Correctly detected and reported individual test failure" | ||
| else | ||
| echo "❌ FAIL: Missing expected failure details" | ||
| cat /tmp/test2-output.txt | ||
| exit 1 | ||
| fi | ||
| fi |
There was a problem hiding this comment.
The test script writes outputs to fixed filenames under /tmp (e.g., /tmp/test2-output.txt). This can collide across parallel runs and leaves stray files behind. Prefer mktemp for output files (and remove them in the existing cleanup trap).
CI run #34180 failed with a package-level test failure but no individual test failures, requiring manual JSON log parsing to investigate. This occurs when tests panic during initialization, race conditions are detected, build errors happen, or tests timeout.
Changes
New script:
scripts/report-test-failures.shgo test -jsonoutput for both individual and package-level failuresCI integration:
.github/workflows/ci.yml$GITHUB_STEP_SUMMARYExample output for package-level failures:
Future CI test failures will surface immediately with actionable debugging information instead of requiring JSON log analysis.
Original prompt
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.