Fix optimization of PROD image building side-effect #36106
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hussein-awala
approved these changes
Dec 7, 2023
potiuk
force-pushed
the
fix-optimization-of-prod-image-building-for-non-main-branches
branch
from
a43bd09 to
5c5096b
Compare
vincbeck
approved these changes
Dec 7, 2023
potiuk
force-pushed
the
fix-optimization-of-prod-image-building-for-non-main-branches
branch
from
5c5096b to
43400d5
Compare
The change #35856 optimized waiting time before PROD image builds start - rather than waiting for full constratints generation, the PROD image building just used source constraints generated right after building the CI image quickly. This is fine for main because there we install airflow and packages using constraints from sources, but for release branches we use the provider constraints - in order to be able to install providers from PyPI rather than from sources. This means that we have to wait for constraints generation to complete before we start building PROD images - because we need to download the constraints generated there to use them. Unfortunately GitHub Actions do not have conditional dependencies depending on where the workflow is run - so instead we have to effectively duplicate PROD build steps and skip steps in them instead.
potiuk
force-pushed
the
fix-optimization-of-prod-image-building-for-non-main-branches
branch
from
43400d5 to
be46114
Compare
Taragolis
deleted the
fix-optimization-of-prod-image-building-for-non-main-branches
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The change #35856 optimized waiting time before PROD image builds start - rather than waiting for full constratints generation, the PROD image building just used source constraints generated right after building the CI image quickly. This is fine for main because there we install airflow and packages using constraints from sources, but for release branches we use the provider constraints - in order to be able to install providers from PyPI rather than from sources.
This means that we have to wait for constraints generation to complete before we start building PROD images - because we need to download the constraints generated there to use them.
Unfortunately GitHub Actions do not have conditional dependencies depending on where the workflow is run - so instead we have to effectively duplicate PROD build steps and skip steps in them instead.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.