-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Move Pod*Exceptions to separate module to avoid unnecessary slow imports in CLI #45759
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
7997f2f to
c98b7d9
Compare
potiuk
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice one! Love the timing results
c98b7d9 to
4444c0d
Compare
|
I applied "full tests needed" and rebased it - I think that one should run full suite of tests due to potential compatibility issues. |
|
Looks like the only failure is unrelated one (already failing on parent commit, same failure as here: #45727 (comment) - |
o-nikolas
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahh, I love these kinds of PRs :)
|
Added an issue for the flaky stuff #45774 |
|
It looks like another "caplog issue" |
Co-authored-by: Igor Kholopov <ikholopov@google.com>
Co-authored-by: Igor Kholopov <ikholopov@google.com>
Co-authored-by: Igor Kholopov <ikholopov@google.com>
Pod*Exceptions has been moved from core to providers back in de92a81, but kept in
airflow/exceptions.pyunder try-case for backwards compatibility and allow for usage of an older k8s provider with a newer core airflow.This resulted in 2 unintended side-effects:
airflowresulted in importingkubernetesclient. It is effectively the most expensive import out of all CLI does for any command, even though it is used by very few commands. Here are the timings of a trivialairflow dag-processor --helppod_generator.py, the import fromexceptions.pyfailed even if providers were up to date with recursive import. But it was caught by exception handling, resulting infrom airflow.exceptions import PodMutationHookExceptionandfrom airflow.providers.cncf.kubernetes.pod_generator import PodMutationHookExceptionpointing to different classes defeating the purpose of the fallback.This PR addresses both by moving PodGenerator exceptions to the separate module that only import its base class. This keeps imports backwards compatible, doesn't attempt to load k8s modules and fixes the divergence of exception classes in
airflow.providers.cncf.kubernetes.pod_generatorandairflow.exceptions.It is worth noting, that all usages of those exceptions in core Airflow also has been cleaned up since then, so if you believe that we need to remove those exceptions from
airflow/exceptions.py, please let me know.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.