Skip to content

ci: parallelize fuzz tests into 4 matrix groups (~3 min savings)#18879

Merged
pelikhan merged 4 commits intomainfrom
copilot/parallelize-fuzz-tests
Feb 28, 2026
Merged

ci: parallelize fuzz tests into 4 matrix groups (~3 min savings)#18879
pelikhan merged 4 commits intomainfrom
copilot/parallelize-fuzz-tests

Conversation

Copy link
Contributor

Copilot AI commented Feb 28, 2026

  • Replace single sequential fuzz job with 4 parallel matrix groups
  • Each matrix job uploads its own artifact fuzz-results-{group-name}
  • Updated concurrency group per matrix group
  • Added fail-fast: false
  • Added if: always() to upload step so fuzz logs are captured even on failure
Original prompt

This section details on the original issue you should resolve

<issue_title>[ci-coach] ci: parallelize fuzz tests into 4 matrix groups</issue_title>
<issue_description>### Summary

The fuzz job currently runs 24 tests sequentially, taking ~4.4 minutes. This PR splits them into 4 parallel matrix groups, reducing fuzz job duration to ~1.5 minutes — saving ~3 minutes per main branch push.

Optimization

Parallelize Fuzz Tests

Type: Matrix Strategy
Impact: ~3 minutes saved per main branch push (4.4 min → ~1.5 min, 3x speedup)
Risk: Low — same tests, same parameters, same logic; only execution order changes

Current structure (sequential):

fuzz job: 24 tests × ~11s each = ~264s (~4.4 min)
```

**Proposed structure** (4 parallel matrix groups):
```
Fuzz: Parser          (4 tests): ~44s  ─┐
Fuzz: Workflow Core   (6 tests): ~66s   ├─ run in parallel → ~88s total
Fuzz: Workflow Parsing(6 tests): ~66s   │
Fuzz: Workflow Triggers(8 tests): ~88s ─┘

Groups:

  • Parser: FuzzParseFrontmatter, FuzzScheduleParser, FuzzRuntimeImportExpressionValidation, FuzzRuntimeImportProcessExpressions
  • Workflow Core: FuzzExpressionParser, FuzzMentionsFiltering, FuzzSanitizeOutput, FuzzSanitizeIncomingText, FuzzSanitizeLabelContent, FuzzWrapExpressionsInTemplateConditionals
  • Workflow Parsing: FuzzYAMLParsing, FuzzTemplateRendering, FuzzInputValidation, FuzzNetworkPermissions, FuzzSafeJobConfig, FuzzParseLabelTriggerShorthand
  • Workflow Triggers: FuzzExpandLabelTriggerShorthand, FuzzValidateNoTemplateInjection, FuzzRemoveHeredocContent, FuzzMarkdownCodeRegionBalancer, FuzzParseTriggerShorthand, FuzzTriggerIRToYAMLMap, FuzzParseInputDefinition, FuzzParseInputDefinitions

Each matrix job uploads its own artifact (fuzz-results-{group-name}) so results are preserved per group.

Expected Impact

  • Total Time Savings: ~3 minutes per main branch push
  • Risk Level: Low — identical test logic, just parallelized

CI Analysis Summary (Run #88)

Full Analysis Results
  • Runs analyzed: 100 recent workflow runs
  • Success rate: 49% (49/100)
  • Avg successful run duration: ~23 minutes
  • Test coverage: ✅ All 4,739 tests executed — no gaps found
  • Fuzz tests: 24 tests, all passing, all running ~11s each

Other findings (no action needed):

  • CI is already highly parallelized — test and integration jobs run independently
  • 22 integration test matrix groups provide good coverage
  • validate-yamljs dependency is an intentional quality gate

Testing Plan

  • Verify all 4 fuzz matrix jobs run on next main push
  • Confirm fuzz-results-{Parser,Workflow-Core,Workflow-Parsing,Workflow-Triggers} artifacts are uploaded
  • Monitor first run to validate timing improvement

Proposed by CI Coach workflow run #88

Generated by CI Optimization Coach

  • expires on Mar 1, 2026, 1:38 PM UTC

[!NOTE]
This was originally intended as a pull request, but the git push operation failed.

Workflow Run: View run details and download patch artifact

The patch file is available in the agent-artifacts artifact in the workflow run linked above.

To create a pull request with the changes:

# Download the artifact from the workflow run
gh run download 22488012589 -n agent-artifacts -D /tmp/agent-artifacts-22488012589

# Create a new branch
git checkout -b ci-coach/parallelize-fuzz-tests-f93e69d7e21b242f

# Apply the patch (--3way handles cross-repo patches where files may already exist)
git am --3way /tmp/agent-artifacts-22488012589/aw-ci-coach-parallelize-fuzz-tests.patch

# Push the branch to origin
git push origin ci-coach/parallelize-fuzz-tests-f93e69d7e21b242f

# Create the pull request
gh pr create --title '[ci-coach] ci: parallelize fuzz tests into 4 matrix groups' --base main --head ci-coach/parallelize-fuzz-tests-f93e69d7e21b242f --repo github/gh-aw
Show patch preview (143 of 143 lines)
From 86c64181dcc6ad0b9374fbd3ba1432f8baa209e8 Mon Sep 17 00:00:00 2001
From: GitHub Copilot <copilot@github.com>
Date: Fri, 27 Feb 2026 13:36:16 +0000
Subject: [PATCH] ci: parallelize fuzz tests into 4 matrix groups (~3 min
 saved)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Split the single sequential fuzz job (24 tests × ~11s = ~4.4 min) into
4 parallel matrix groups, red...

</details>


> **Custom agent used: technical-doc-writer**
> AI technical documentation writer for GitHub Actions library using GitHub Docs voice



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes github/gh-aw#18696

<!-- START COPILOT CODING AGENT TIPS -->
---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Parallelize fuzz tests into 4 matrix groups ci: parallelize fuzz tests into 4 matrix groups (~3 min savings) Feb 28, 2026
@pelikhan pelikhan marked this pull request as ready for review February 28, 2026 18:03
Copilot AI review requested due to automatic review settings February 28, 2026 18:03
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Parallelizes the main-branch fuzzing job by splitting 24 fuzz targets into 4 matrix groups to reduce end-to-end CI time.

Changes:

  • Added a strategy.matrix to run fuzz tests in 4 parallel groups with fail-fast: false.
  • Replaced 24 hardcoded run_fuzz_test calls with a loop over a per-matrix tests list.
  • Scoped concurrency and artifact names per matrix group to avoid cross-group cancellation and keep outputs separated.
Comments suppressed due to low confidence (1)

.github/workflows/ci.yml:1370

  • The loop calls run_fuzz_test but does not check/accumulate its return status. Because the script continues after the loop (copying corpus data, etc.) and there is no set -e, an unexpected fuzz failure can be ignored and the step can still exit successfully. Track failures inside the loop (while continuing to run remaining fuzz targets) and ensure the step exits non-zero if any target fails unexpectedly; if you still want artifacts on failure, pair this with if: always() on the upload step.
          # Run fuzz tests for this matrix group
          for entry in ${{ matrix.tests }}; do
            fuzz_name="${entry%%:*}"
            package="${entry##*:}"
            run_fuzz_test "${fuzz_name}" "${package}"
          done

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan pelikhan merged commit 9402dde into main Feb 28, 2026
@pelikhan pelikhan deleted the copilot/parallelize-fuzz-tests branch February 28, 2026 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants