Enable EksPodOperator when deferrable is true

[jx2lee]

(TBD) closes: #39685

[jx2lee]

#41377 was closed because branch was renamed

[vincbeck]

You are waiting before actually executing the task? Also, you need to check wait_for_completion, if wait_for_completion is False you do not want to wait. wait_for_completion configures whether the operator wait for a given status, deferrable configures how to wait (if wait_for_completion is True): in poke mode or deferrable mode.

[jx2lee]

Thank you for review. I'm still getting familiar with deferrable concept and will work on resolving the issue as I understand it better.
I'll proceed accordingly, and mention if I need any help!

[vincbeck]

Perfect! Thanks for doing it :)

[o-nikolas]

I was under the impression this was already fixed by #41178 and the deferrable logic was inherited through the k8s pod operator itself. Were you testing with an older version of the provider packages?

[jx2lee]

@o-nikolas you're right.

• Deferrable logic was inherited throught the k8s pod operator. (execute in eks.py EksPodOperator)
• It seems that issue in this PR was resolved by the fix in #41178 . The direction to address the issue was incorrect. I have two options: either close the issue or update the relevant test code in #39685. What do you think?

[o-nikolas]

@o-nikolas you're right.

* Deferrable logic was inherited through the k8s pod operator. ([execute](https://github.com/apache/airflow/blob/c0ffa9c5d96625c68ded9562632674ed366b5eb3/airflow/providers/amazon/aws/operators/eks.py#L1075) in eks.py EksPodOperator)

* It seems that issue in this PR was resolved by the fix in [#41178](https://github.com/apache/airflow/pull/41178) . The direction to address the issue was incorrect. I have two options: either close the issue or update the relevant test code in [#39685](https://github.com/apache/airflow/issues/39685). What do you think?

In particular reference to this:

I have two options: either close the issue or update the relevant test code in #39685. What do you think?

What exactly is still broken? Can you describe that more, it would help me provide further direction. Thanks for sticking with this!

[jx2lee]

Sorry for late reply! 🥲

What exactly is still broken? Can you describe that more, it would help me provide further direction. Thanks for sticking with this!

The issue is no longer reproducible. EksPodOperator created with the deferrable parameter runs without any problems. Initially, I thought issue was due to it not being deferrable, but that was my mistake.

• Error in #39695 is not reproducible in commits prior to #41178.
• Since this PR inherits from the PodOperator and modifies it to a deferrable mode, the approach is incorrect. Deferrable tests work fine in commits prior to Pass content of kube/config file to triggerer as a dictionary #41178.

and, look at full logs in Anything else #39695
kubernetes.config.config_exception.ConfigException: Invalid kube-config file. No configuration found. often occurred. i think that kube config file in created pod was broken, because It could be related to network issue or others. I don’t think fix in #41178 resolved #39695.
-> Please let me know if I’m mistaken.

I’d like to dig deeper into the problem and request more information from the reporter (such as the full log when deferrable=False, etc).

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.