Final fix for image caching #22618
Merged
Final fix for image caching #22618
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is - I hope - final fix for our image caching. I fought with it for quite a while and it turned out that I was fighting with buidkit bug when preparing a multiplatform image. I think I finally got it under control, the main problem was that when we prepared multi-platform image, seems like the image that was prepared "last" was overriding the cache that was prepared by the other platform build. This cause numerous problems with investigating it, because sometimes it looked like it worked, when the "other" platform was faster to build and someties it did not when it was slower. It almost drove me mad. It looks like however, that the inline cache works better in this case (and I got it repetetively working) so finally we might go back to faster builds on CI and fast pull for ./breeze. I also opened an issue at buildkit, hoping that they will be able to fix it: moby/buildkit#2758
Member
Author
Member
Author
|
BTW. @mik-laj - this MIGHT MEAN that we can get the |
mik-laj
approved these changes
Mar 30, 2022
github-actions
bot
added
the
full tests needed
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
Member
Author
|
FYI: The problem with caching multi-node builds has been confirmed and it looks like it's a buildx plugin bug, I closed the buildkit one and opened docker/buildx#1044 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is - I hope - final fix for our image caching. I fought with
it for quite a while and it turned out that I was fighting with
buidkit bug when preparing a multiplatform image.
I think I finally got it under control, the main problem was that
when we prepared multi-platform image, seems like the image that
was prepared "last" was overriding the cache that was prepared
by the other platform build. This cause numerous problems with
investigating it, because sometimes it looked like it worked,
when the "other" platform was faster to build and someties it
did not when it was slower. It almost drove me mad.
It looks like however, that the inline cache works better in
this case (and I got it repetetively working) so finally we might
go back to faster builds on CI and fast pull for ./breeze.
I also opened an issue at buildkit, hoping that they will be able
to fix it:
moby/buildkit#2758
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.