Update create_emr_on_eks_cluster method to try when "cluster is not reachable as its connection is currently being updated"
#46497
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… reachable as its connection is currently being updated"
boring-cyborg
bot
added
area:providers
provider:amazon
o-nikolas
approved these changes
Feb 6, 2025
ferruzzi
approved these changes
Feb 6, 2025
potiuk
approved these changes
Feb 6, 2025
insomnes
pushed a commit
to insomnes/airflow
that referenced
this pull request
Feb 6, 2025
… reachable as its connection is currently being updated" (apache#46497)
niklasr22
pushed a commit
to niklasr22/airflow
that referenced
this pull request
Feb 8, 2025
… reachable as its connection is currently being updated" (apache#46497)
ambika-garg
pushed a commit
to ambika-garg/airflow
that referenced
this pull request
Feb 17, 2025
… reachable as its connection is currently being updated" (apache#46497)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The AWS system test
example_emr_eks.pystarted to fail because of the error below:This error happens when trying to create the virtual cluster on EMR. The creation of this virtual cluster happens after creating the EKS cluster and waiting for its status to be active. After some testing, the creation succeeds after waiting few seconds between the creation of the EKS cluster and the virtual cluster. It must be a race condition. Retrying is the only option here since there is no notion of cluster connection status in EKS.
I also revert the change I did in #46441 because it did not help (it just happened to succeed when I changed the version, but it was a fluke ...).
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.