-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Switch to building images in parallel #24580
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4b87729 to
319bada
Compare
Member
Author
|
This one will save quite a lot of elapsed and build time on building our images in CI - especially for main builds. |
2897f0f to
07b6fe0
Compare
d890b9a to
cc33e91
Compare
Member
Author
|
All right got some numbers: For builds with "full tests" (main builds)
|
In the new Breeze, switching to using parallelism is a ... breeze. This PR adds the capability of building the images in parallel in Breeze locally - for breeze command, but also uses this capability to build the images in parallel in our CI. Our builds are always executed on powerful, big machines with lots of CPU and docker run in memory filesystem with 32GB RAM, so it should be possible to run all builds in parallel on a single machine rather then spin off parallel machines to run the builds using the matrix strategy of Github Actions. Generally speaking - this will either speed up or get 4x cost saving for the build steps for all the "full test needed" PRs as well as all the main builds. There are a number of savings and improvements we can achieve this way: 1) less overhead for starting and runnning the machines 2) seems that with the new buildkit, the parallel builds are not suffering from some sequential locks (as it used to be, so we are basically do the same job using 25% resources for building the images. 3) we will stop having random "one image failed to build" cases - they will all either fail or succeed. 4) Less checks in the output 5) Production builds will additionally gain from single CI image pulled in order to perform the preparation of the packages and single package preparation step - it will save 4-5 minutes per image. The disadvantage is a less clear output of such parallel build where outputs from multiple builds will be interleaved in one CI output.
cc33e91 to
8b847ed
Compare
ashb
approved these changes
Jun 21, 2022
Member
Author
|
occasional failures only |
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Jun 29, 2022
In the new Breeze, switching to using parallelism is a ... breeze. This PR adds the capability of building the images in parallel in Breeze locally - for breeze command, but also uses this capability to build the images in parallel in our CI. Our builds are always executed on powerful, big machines with lots of CPU and docker run in memory filesystem with 32GB RAM, so it should be possible to run all builds in parallel on a single machine rather then spin off parallel machines to run the builds using the matrix strategy of Github Actions. Generally speaking - this will either speed up or get 4x cost saving for the build steps for all the "full test needed" PRs as well as all the main builds. There are a number of savings and improvements we can achieve this way: 1) less overhead for starting and runnning the machines 2) seems that with the new buildkit, the parallel builds are not suffering from some sequential locks (as it used to be, so we are basically do the same job using 25% resources for building the images. 3) we will stop having random "one image failed to build" cases - they will all either fail or succeed. 4) Less checks in the output 5) Production builds will additionally gain from single CI image pulled in order to perform the preparation of the packages and single package preparation step - it will save 4-5 minutes per image. The disadvantage is a less clear output of such parallel build where outputs from multiple builds will be interleaved in one CI output. (cherry picked from commit 893d935)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area:dev-tools
changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
full tests needed
We need to run full set of tests for this PR to merge
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the new Breeze, switching to using parallelism is a ... breeze.
This PR adds the capability of building the images in parallel in Breeze
locally - for breeze command, but also uses this capability to build the
images in parallel in our CI. Our builds are always executed on
powerful, big machines with lots of CPU and docker run in memory
filesystem with 32GB RAM, so it should be possible to run all builds in
parallel on a single machine rather then spin off parallel machines to
run the builds using the matrix strategy of Github Actions.
Generally speaking - this will either speed up or get 4x cost saving for
the build steps for all the "full test needed" PRs as well as all the
main builds.
There are a number of savings and improvements we can achieve this way:
suffering from some sequential locks (as it used to be, so
we are basically do the same job using 25% resources for building
the images.
will all either fail or succeed.
pulled in order to perform the preparation of the packages
and single package preparation step - it will save 4-5 minutes
per image.
The disadvantage is a less clear output of such parallel build where
outputs from multiple builds will be interleaved in one CI output.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragement file, named
{pr_number}.significant.rst, in newsfragments.