Skip to content

feat: use docker in frontend GHA to parallelize work#31490

Merged
mistercrunch merged 26 commits into
masterfrom
faster_frontend_builds
Jan 8, 2025
Merged

feat: use docker in frontend GHA to parallelize work#31490
mistercrunch merged 26 commits into
masterfrom
faster_frontend_builds

Conversation

@mistercrunch
Copy link
Copy Markdown
Member

@mistercrunch mistercrunch commented Dec 17, 2024

Optimizing/parallelizing work around frontend builds.

  • docker build is typically much faster than CI builds, able to reuse layers
  • build happening only once, before parallelizing the work on top
  • sharding jest tests, running 8 in parallel (!)
  • maintained the codecov coverage reporting though upload is broken both in master and here, requires more attention

Clock time savings from 25 to 5-10 minutes depending on docker layer cache

Machine time savings also significant VS parallelizing the build steps.

Oh also, bycatch fixing the multiple alembic/db migrations heads currently in master

workflow topology

Screenshot 2025-01-07 at 3 20 27 PM

@github-actions github-actions Bot added doc Namespace | Anything related to documentation github_actions Pull requests that update GitHub Actions code labels Dec 17, 2024
@mistercrunch mistercrunch force-pushed the faster_frontend_builds branch from c310511 to 042e732 Compare December 17, 2024 01:52
@github-actions github-actions Bot added preset-io and removed doc Namespace | Anything related to documentation labels Dec 17, 2024
@mistercrunch mistercrunch force-pushed the faster_frontend_builds branch from b6b3a99 to 6f9bd98 Compare December 17, 2024 19:49
@mistercrunch mistercrunch force-pushed the faster_frontend_builds branch from ad8b086 to 06f044c Compare December 19, 2024 07:30
@mistercrunch mistercrunch force-pushed the faster_frontend_builds branch from 904ec1e to 49d8fa9 Compare December 21, 2024 19:32
@mistercrunch mistercrunch marked this pull request as ready for review January 5, 2025 01:25
Copy link
Copy Markdown

@korbit-ai korbit-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've completed my review and didn't find any issues... but I did find this unicorn.

                    /
               ,.. /
             ,'   ';
  ,,.__    _,' /';  .
 :','  ~~~~    '. '~
:' (   )         )::,
'. '. .=----=..-~  .;'
 '  ;'  ::   ':.  '"
   (:   ':    ;)
    \\   '"  ./
     '"      '"

Need a new review? Comment /korbit-review on this PR and I'll review your latest changes.

Korbit Guide: Usage and Customization

Interacting with Korbit

  • You can manually ask Korbit to review your PR using the /korbit-review command in a comment at the root of your PR.
  • You can ask Korbit to generate a new PR description using the /korbit-generate-pr-description command in any comment on your PR.
  • Too many Korbit comments? I can resolve all my comment threads if you use the /korbit-resolve command in any comment on your PR.
  • Chat with Korbit on issues we post by tagging @korbit-ai in your reply.
  • Help train Korbit to improve your reviews by giving a 👍 or 👎 on the comments Korbit posts.

Customizing Korbit

  • Check out our docs on how you can make Korbit work best for you and your team.
  • Customize Korbit for your organization through the Korbit Console.

Current Korbit Configuration

General Settings
Setting Value
Review Schedule Automatic excluding drafts
Max Issue Count 10
Automatic PR Descriptions
Issue Categories
Category Enabled
Naming
Database Operations
Documentation
Logging
Error Handling
Systems and Environment
Objects and Data Structures
Readability and Maintainability
Asynchronous Processing
Design Patterns
Third-Party Libraries
Performance
Security
Functionality

Feedback and Support

Note

Korbit Pro is free for open source projects 🎉

Looking to add Korbit to your team? Get started with a free 2 week trial here

@mistercrunch mistercrunch force-pushed the faster_frontend_builds branch from eeb5fb2 to e782250 Compare January 5, 2025 08:40
@mistercrunch
Copy link
Copy Markdown
Member Author

NOTE: this needs to work in forks/without dockerhub creds to be viable

@mistercrunch mistercrunch force-pushed the faster_frontend_builds branch 2 times, most recently from 55fcae1 to 4c54dc4 Compare January 7, 2025 23:15
@mistercrunch mistercrunch force-pushed the faster_frontend_builds branch from 4c54dc4 to b61e546 Compare January 7, 2025 23:47
@github-actions github-actions Bot added the risk:db-migration PRs that require a DB migration label Jan 8, 2025
@mistercrunch mistercrunch merged commit 0eca79c into master Jan 8, 2025
@mistercrunch mistercrunch deleted the faster_frontend_builds branch January 8, 2025 00:36
mistercrunch added a commit that referenced this pull request Jan 8, 2025
I just merged #31490 and realize
it fails when no frontend changes are found. It will run the first
workflow and fail on subsequent.

This PR makes it such that the first workflow emits an output to tell
the other workflows to skip if needed.
alexandrusoare pushed a commit to alexandrusoare/superset that referenced this pull request Jun 19, 2025
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 5.0.0 First shipped in 5.0.0 labels Jul 29, 2025
qfcwell pushed a commit to qfcwell/superset that referenced this pull request May 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels dependencies:npm github_actions Pull requests that update GitHub Actions code preset-io risk:db-migration PRs that require a DB migration size/L 🚢 5.0.0 First shipped in 5.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants