Synchronize and fix ARM CI workflows #56856
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
potiuk
requested review from
amoghrajesh,
ashb,
bugraoz93,
eladkal,
gopidesupavan,
jason810496,
jscheffl and
kaxil
as code owners
boring-cyborg
bot
added
area:dev-tools
backport-to-v3-1-test
Member
Author
|
I also run ARM job manually here from that branch: https://github.com/apache/airflow/actions/runs/18633758589 - because ARM job is not executed automatically on PR |
potiuk
force-pushed
the
fix-arm-tests
branch
2 times, most recently
from
36e8901 to
d050c22
Compare
Member
Author
|
ok. as suspected not all integration tests will run (no multi-platform images) https://github.com/apache/airflow/actions/runs/18634153359 -> ARM run without those. |
There were a few issues with ARM workflows: * not all jobs were run in ARM tests - we do not want to run mysql of course, but other tests should be fine to run on ARM * some conditions were not updated (we have to somehow duplicate amd and arm job definition because we run out of composite workflows - so sometimes conditions are not synced) * most importantly - we uploaded prek cache in build-info job, but that job only run on AMD, not on ARM so the ARM cache was really an AMD one (and it caused unterminated strings in doctoc installation It's not possible to upload same artifact twice in the same job and since we use prek in several jobs we should make sure that the cache is only uploaded once per job. This was the reason why it was initially uploaded in build-info job (and save-cache was set to false elsewhere). With this PR, we have save-cache in 3 places: * basic checks * static CI-image bound checks * in octopin (Python 3.11) Basic checks and static checks are mutually exclusive (controlled by basic-checks-only flag) - so we can safely upload cache in both. In all other places we only install prek with cache, but we do not save the cache as artifact.
Member
Author
|
And found some other small issues - we have multi-platform RAT image ... but we forced amd even on ARM 😱 |
github-actions bot
pushed a commit
that referenced
this pull request
Oct 19, 2025
There were a few issues with ARM workflows: * not all jobs were run in ARM tests - we do not want to run mysql of course, but other tests should be fine to run on ARM * some conditions were not updated (we have to somehow duplicate amd and arm job definition because we run out of composite workflows - so sometimes conditions are not synced) * most importantly - we uploaded prek cache in build-info job, but that job only run on AMD, not on ARM so the ARM cache was really an AMD one (and it caused unterminated strings in doctoc installation It's not possible to upload same artifact twice in the same job and since we use prek in several jobs we should make sure that the cache is only uploaded once per job. This was the reason why it was initially uploaded in build-info job (and save-cache was set to false elsewhere). With this PR, we have save-cache in 3 places: * basic checks * static CI-image bound checks * in octopin (Python 3.11) Basic checks and static checks are mutually exclusive (controlled by basic-checks-only flag) - so we can safely upload cache in both. In all other places we only install prek with cache, but we do not save the cache as artifact. (cherry picked from commit 59089cd) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Backport successfully created: v3-1-test
|
potiuk
added a commit
that referenced
this pull request
Oct 19, 2025
There were a few issues with ARM workflows: * not all jobs were run in ARM tests - we do not want to run mysql of course, but other tests should be fine to run on ARM * some conditions were not updated (we have to somehow duplicate amd and arm job definition because we run out of composite workflows - so sometimes conditions are not synced) * most importantly - we uploaded prek cache in build-info job, but that job only run on AMD, not on ARM so the ARM cache was really an AMD one (and it caused unterminated strings in doctoc installation It's not possible to upload same artifact twice in the same job and since we use prek in several jobs we should make sure that the cache is only uploaded once per job. This was the reason why it was initially uploaded in build-info job (and save-cache was set to false elsewhere). With this PR, we have save-cache in 3 places: * basic checks * static CI-image bound checks * in octopin (Python 3.11) Basic checks and static checks are mutually exclusive (controlled by basic-checks-only flag) - so we can safely upload cache in both. In all other places we only install prek with cache, but we do not save the cache as artifact. (cherry picked from commit 59089cd) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Member
|
fantastic thanks for syncing :) |
Member
Author
BTW. It's likely we could extract it to a single, composit workflow - despite limitations in the number of workflows. Maybe we should try. |
nailo2c
pushed a commit
to nailo2c/airflow
that referenced
this pull request
Oct 20, 2025
There were a few issues with ARM workflows: * not all jobs were run in ARM tests - we do not want to run mysql of course, but other tests should be fine to run on ARM * some conditions were not updated (we have to somehow duplicate amd and arm job definition because we run out of composite workflows - so sometimes conditions are not synced) * most importantly - we uploaded prek cache in build-info job, but that job only run on AMD, not on ARM so the ARM cache was really an AMD one (and it caused unterminated strings in doctoc installation It's not possible to upload same artifact twice in the same job and since we use prek in several jobs we should make sure that the cache is only uploaded once per job. This was the reason why it was initially uploaded in build-info job (and save-cache was set to false elsewhere). With this PR, we have save-cache in 3 places: * basic checks * static CI-image bound checks * in octopin (Python 3.11) Basic checks and static checks are mutually exclusive (controlled by basic-checks-only flag) - so we can safely upload cache in both. In all other places we only install prek with cache, but we do not save the cache as artifact.
TyrellHaywood
pushed a commit
to TyrellHaywood/airflow
that referenced
this pull request
Oct 22, 2025
There were a few issues with ARM workflows: * not all jobs were run in ARM tests - we do not want to run mysql of course, but other tests should be fine to run on ARM * some conditions were not updated (we have to somehow duplicate amd and arm job definition because we run out of composite workflows - so sometimes conditions are not synced) * most importantly - we uploaded prek cache in build-info job, but that job only run on AMD, not on ARM so the ARM cache was really an AMD one (and it caused unterminated strings in doctoc installation It's not possible to upload same artifact twice in the same job and since we use prek in several jobs we should make sure that the cache is only uploaded once per job. This was the reason why it was initially uploaded in build-info job (and save-cache was set to false elsewhere). With this PR, we have save-cache in 3 places: * basic checks * static CI-image bound checks * in octopin (Python 3.11) Basic checks and static checks are mutually exclusive (controlled by basic-checks-only flag) - so we can safely upload cache in both. In all other places we only install prek with cache, but we do not save the cache as artifact.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There were a few issues with ARM workflows:
It's not possible to upload same artifact twice in the same job and since we use prek in several jobs we should make sure that the cache is only uploaded once per job. This was the reason why it was initially uploaded in build-info job (and save-cache was set to false elsewhere). With this PR, we have save-cache in 3 places:
Basic checks and static checks are mutually exclusive (controlled by basic-checks-only flag) - so we can safely upload cache in both.
In all other places we only install prek with cache, but we do not save the cache as artifact.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.