Move all k8S classes to cncf.kubernetes provider #32767
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
potiuk
added
the
full tests needed
boring-cyborg
bot
added
area:CLI
area:dev-tools
provider:cncf-kubernetes
potiuk
force-pushed
the
move-kubernetes-classes-to-provider
branch
3 times, most recently
from
0489b31 to
2b5f5c9
Compare
potiuk
force-pushed
the
move-kubernetes-classes-to-provider
branch
2 times, most recently
from
d3a18e9 to
4e281b6
Compare
potiuk
requested review from
XD-DENG,
ashb,
bolkedebruin,
dstandish,
jedcunningham,
kaxil,
o-nikolas and
pierrejeambrun
as code owners
potiuk
force-pushed
the
move-kubernetes-classes-to-provider
branch
2 times, most recently
from
83e9b83 to
dcc849d
Compare
Member
Author
|
Based on #32775 so only last commit counts. |
potiuk
force-pushed
the
move-kubernetes-classes-to-provider
branch
from
dcc849d to
5bff371
Compare
potiuk
changed the title
Member
Author
|
Only last commit counts. |
potiuk
force-pushed
the
move-kubernetes-classes-to-provider
branch
2 times, most recently
from
76afae1 to
4e56e6a
Compare
potiuk
force-pushed
the
move-kubernetes-classes-to-provider
branch
3 times, most recently
from
17c3838 to
e2ed30e
Compare
potiuk
force-pushed
the
move-kubernetes-classes-to-provider
branch
3 times, most recently
from
205c9a4 to
d7d4bbb
Compare
Member
Author
|
docs/apache-airflow/core-concepts/executor/celery_kubernetes.rst
Outdated
Show resolved
Hide resolved
docs/apache-airflow/core-concepts/executor/local_kubernetes.rst
Outdated
Show resolved
Hide resolved
potiuk
force-pushed
the
move-kubernetes-classes-to-provider
branch
from
d7d4bbb to
c486137
Compare
Member
Author
|
Applied this round of comments from @jedcunningham . Also added newsfragment |
potiuk
force-pushed
the
move-kubernetes-classes-to-provider
branch
from
c486137 to
eb6573a
Compare
This is the big move of all Kubenetes classes to go to provider. The changes that are implemented in this move: * replaced all imports from airflow.kubernetes to cncf.kubernetes Swith PEP-563 dynamic import rediretion and deprecation messages those messages now support overriding the "replacement" hints to make K8s deprecations more accurate * pre_7_4_0_compatibility package with classes used by past providerrs have been "frozen" and stored in the package with import redirections from airflow.kubernetes(with deprecation warnings) * kubernetes configuration is moved to kubernetes provider * mypy started complaining about conf and set used in configuration. so better solution to handle deprecations and hinting conf returning AirlfowConfigParsing was added. * example_kuberntes_executor uses configuration reading not in top level but in execute method * PodMutationHookException and PodReconciliationError have been moved to cncf.kubernetes provider and they are imported from there with fallback to an airflow.exception ones in case old provider is used in Airflow 2.7.0 * k8s methods in task_instance have been deprecated and reolaced with functions in "cncf.kubernetes` template_rendering module the old way still works but raise deprecaton warnings. * added extras with versions for celery and k8s * raise AirflowOptionalProviderFeatureException in case there is attempt to use CeleryK8sExecutor and cncf.k8s is not installed. * added few "new" core utils to k8s (hashlib_wrapper etc) * both warnings and errors indicate minimum versions for both cncf.k8s and Celery providers.
potiuk
force-pushed
the
move-kubernetes-classes-to-provider
branch
from
eb6573a to
b7af1c4
Compare
jedcunningham
approved these changes
Jul 25, 2023
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Jul 27, 2023
The apache#32767 missed imports of imports for a few kubernetes modules. This PR adds the missing ones.
jedcunningham
pushed a commit
that referenced
this pull request
Jul 27, 2023
The #32767 missed imports of imports for a few kubernetes modules. This PR adds the missing ones.
69 tasks
ephraimbuddy
added
the
type:misc/internal
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Sep 13, 2023
The apache#32767 has moved all k8s classes to cncf.kubernetes provider, however there was a mistake with location of Pod*Exceptions - rather than in pod_manager they remained defined in the kubernetes_executor package - which has the side-effect that trying to import them in Airflow Pre 2.7 raised the "You should not use the provider's executors in this version of Airflow." error. This change moves the exceptions to the pod_generator package to fix the problem.
potiuk
added a commit
that referenced
this pull request
Sep 13, 2023
The #32767 has moved all k8s classes to cncf.kubernetes provider, however there was a mistake with location of Pod*Exceptions - rather than in pod_manager they remained defined in the kubernetes_executor package - which has the side-effect that trying to import them in Airflow Pre 2.7 raised the "You should not use the provider's executors in this version of Airflow." error. This change moves the exceptions to the pod_generator package to fix the problem.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the big move of all Kubenetes classes to go to provider.
The changes that are implemented in this move:
Swith PEP-563 dynamic import rediretion and deprecation messages
those messages now support overriding the "replacement" hints
to make K8s deprecations more accurate
providerrs have been "frozen" and stored in the package with
import redirections from airflow.kubernetes(with deprecation warnings)
so better solution to handle deprecations and hinting conf
returning AirlfowConfigParsing was added.
top level but in execute method
been moved to cncf.kubernetes provider and they are imported
from there with fallback to an airflow.exception ones in case
old provider is used in Airflow 2.7.0
with functions in "cncf.kubernetes` template_rendering module
the old way still works but raise deprecaton warnings.
attempt to use CeleryK8sExecutor and cncf.k8s is not installed.
and Celery providers.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.