Use constraints generated at preview time rather than regenerate #35013
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
potiuk
requested review from
Taragolis,
amoghrajesh,
eladkal and
pierrejeambrun
Member
Author
|
"Final" improvement to constraint generation process -> making it more robust and faster and clearer in case |
potiuk
force-pushed
the
faster-and-more-robust-constraint-update
branch
from
864ec4b to
0b8d985
Compare
potiuk
commented
Oct 18, 2023
.github/workflows/ci.yml
Outdated
Member
Author
There was a problem hiding this comment.
It's also faster as we do not need to install breeze/pull image since we are using constraints generated earlier and download them from artifacts
potiuk
force-pushed
the
faster-and-more-robust-constraint-update
branch
4 times, most recently
from
e6fea01 to
d7ef1e7
Compare
eladkal
approved these changes
Oct 18, 2023
potiuk
force-pushed
the
faster-and-more-robust-constraint-update
branch
from
d7ef1e7 to
5fa3512
Compare
So far, we regenerated the constraints in order to get them updated in constraint branches at the end of the CI build when we knew all the builds succeded. However this was not perfect for two reasons: * there was a race condition that a dependency had been released between the time images were built and tests completed. While it had no impact on "source" constraints (they reflect what is in the image), it could change the PyPI constraints generated or "no-providers" constraints, because they use "current" set of dependencies in PyPI to generate them. * generating constraints (for PyPI and "no-providers") takes time because `pip` has to resolve dependencies again - taking into account what is in the PyPI registry, and this can take a long time (minutes or even it can lead to long `pip` backtracking. We generally cancel running builds when new commit is merged in main, so this could lead to such constraint job being canceled by subsequent merge This PR uses the fact that we have now "preview-constraints" job that uploads constraints as artifacts in CI workflow, and instead of regenerating the constraints, we can download the constraints from uploaded artifact and use it - this is very quick and we can also make a clear dependency between "preview" and "update" constraints jobs - making it clear in case of PIP backtracking that we have problem with constraints, not with the image. This will make it far clearer when we will have problems with constraints and backtracking that this is the real issue we have (allowing such PRs to get merged while constraints backtracking problem is being worked on.
potiuk
force-pushed
the
faster-and-more-robust-constraint-update
branch
from
5fa3512 to
bad3806
Compare
ephraimbuddy
added
the
changelog:skip
potiuk
added a commit
that referenced
this pull request
Oct 29, 2023
) So far, we regenerated the constraints in order to get them updated in constraint branches at the end of the CI build when we knew all the builds succeded. However this was not perfect for two reasons: * there was a race condition that a dependency had been released between the time images were built and tests completed. While it had no impact on "source" constraints (they reflect what is in the image), it could change the PyPI constraints generated or "no-providers" constraints, because they use "current" set of dependencies in PyPI to generate them. * generating constraints (for PyPI and "no-providers") takes time because `pip` has to resolve dependencies again - taking into account what is in the PyPI registry, and this can take a long time (minutes or even it can lead to long `pip` backtracking. We generally cancel running builds when new commit is merged in main, so this could lead to such constraint job being canceled by subsequent merge This PR uses the fact that we have now "preview-constraints" job that uploads constraints as artifacts in CI workflow, and instead of regenerating the constraints, we can download the constraints from uploaded artifact and use it - this is very quick and we can also make a clear dependency between "preview" and "update" constraints jobs - making it clear in case of PIP backtracking that we have problem with constraints, not with the image. This will make it far clearer when we will have problems with constraints and backtracking that this is the real issue we have (allowing such PRs to get merged while constraints backtracking problem is being worked on. (cherry picked from commit f115ec1)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
So far, we regenerated the constraints in order to get them updated in constraint branches at the end of the CI build when we knew all the builds succeded. However this was not perfect for two reasons:
there was a race condition that a dependency had been released between the time images were built and tests completed. While it had no impact on "source" constraints (they reflect what is in the image), it could change the PyPI constraints generated or "no-providers" constraints, because they use "current" set of dependencies in PyPI to generate them.
generating constraints (for PyPI and "no-providers") takes time because
piphas to resolve dependencies again - taking into account what is in the PyPI registry, and this can take a long time (minutes or even it can lead to longpipbacktracking. We generally cancel running builds when new commit is merged in main, so this could lead to such constraint job being canceled by subsequent mergeThis PR uses the fact that we have now "preview-constraints" job that uploads constraints as artifacts in CI workflow, and instead of regenerating the constraints, we can download the constraints from uploaded artifact and use it - this is very quick and we can also make a clear dependency between "preview" and "update" constraints jobs - making it clear in case of PIP backtracking that we have problem with constraints, not with the image. This will make it far clearer when we will have problems with constraints and backtracking that this is the real issue we have (allowing such PRs to get merged while constraints backtracking problem is being worked on.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.