Move constraints generation to separate job in CI workflow #34990
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
potiuk
requested review from
amoghrajesh,
eladkal,
ephraimbuddy,
hussein-awala and
pierrejeambrun
Member
Author
|
This one should prevent the case we had few days ago - which prevented tests from running few days ago (workarounded by #34918). After new providers have been released today by @eladkal, PyPI constraints generation has a chance to succeed today - the "preview-constraints" job in this PR will be green if it does, then i will also restore the actual constraints generation in the final step. |
potiuk
force-pushed
the
move-early-constrints-generation-to-separate-steps
branch
2 times, most recently
from
ed8282f to
c158122
Compare
eladkal
approved these changes
Oct 17, 2023
hussein-awala
approved these changes
Oct 17, 2023
Member
Author
|
Yes. It looks like releasing the new providers unblocked |
potiuk
force-pushed
the
move-early-constrints-generation-to-separate-steps
branch
2 times, most recently
from
8f8a9b6 to
f35c9b3
Compare
Member
Author
Well. Not really - I need to look further :( |
potiuk
force-pushed
the
move-early-constrints-generation-to-separate-steps
branch
2 times, most recently
from
49c8cda to
25b7ad6
Compare
We generate constraints for preview right after images are build, in order to be able to see the output and to diagnose which dependencies have been updated later in the process. So far we were doing it in CI workflow and When constraints generation failed however (for example because pip backtracking takes a log ot time), it faied build image workflow and did not allow tests to complete. This PR extracts constraint generation to CI separate job which does not block tests from running in case constraints generation fails or times out. It also moves out the steps from the composite action which allows to better see which step failed and allows to see the constraints used in each job more easily.
potiuk
force-pushed
the
move-early-constrints-generation-to-separate-steps
branch
from
25b7ad6 to
d68f6cb
Compare
Member
Author
|
Seems I got it fixed. For whatever reason Not obvious at all, but at least the outputs we get when similart things happen in the future should be more readable and it should not block the PRs. |
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Oct 18, 2023
The PROD image uses source constraints to get built. The apache#34990 had moved the constraint generation to CI workflow which works for main builds, but for regular PRs we also need to upload them in the build-image workflow - otherwise PROD image build cannot find them as artifacts. This PR restores only source constraints upload as artifacts (this is very fast)
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Oct 18, 2023
…#35004) The PROD image uses source constraints to get built. The apache#34990 had moved the constraint generation to CI workflow which works for main builds, but for regular PRs we also need to upload them in the build-image workflow - otherwise PROD image build cannot find them as artifacts. This PR restores only source constraints upload as artifacts (this is very fast)
ephraimbuddy
added
the
changelog:skip
potiuk
added a commit
that referenced
this pull request
Oct 29, 2023
We generate constraints for preview right after images are build, in order to be able to see the output and to diagnose which dependencies have been updated later in the process. So far we were doing it in CI workflow and When constraints generation failed however (for example because pip backtracking takes a log ot time), it faied build image workflow and did not allow tests to complete. This PR extracts constraint generation to CI separate job which does not block tests from running in case constraints generation fails or times out. It also moves out the steps from the composite action which allows to better see which step failed and allows to see the constraints used in each job more easily. (cherry picked from commit a450b81)
potiuk
added a commit
that referenced
this pull request
Oct 29, 2023
The PROD image uses source constraints to get built. The #34990 had moved the constraint generation to CI workflow which works for main builds, but for regular PRs we also need to upload them in the build-image workflow - otherwise PROD image build cannot find them as artifacts. This PR restores only source constraints upload as artifacts (this is very fast) (cherry picked from commit 40b9d2b)
potiuk
deleted the
move-early-constrints-generation-to-separate-steps
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We generate constraints for preview right after images are build, in order to be able to see the output and to diagnose which dependencies have been updated later in the process.
So far we were doing it in CI workflow and When constraints generation failed however (for example because pip backtracking takes a log ot time), it faied build image workflow and did not allow tests to complete.
This PR extracts constraint generation to CI separate job which does not block tests from running in case constraints generation fails or times out.
It also moves out the steps from the composite action which allows to better see which step failed and allows to see the constraints used in each job more easily.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.