Skip to content

Conversation

@moiseenkov
Copy link
Contributor

Bump missing version for apache-airflow-providers-cncf-kubernetes up to 8.0.0 required for the #36847.

related: #36847, #37890

@moiseenkov moiseenkov force-pushed the bump_google_provider_dependency branch from fd8606a to b1332fa Compare March 5, 2024 15:38
@eladkal
Copy link
Contributor

eladkal commented Mar 5, 2024

We don't normally bump min versions for each feature, that would cause all providers to work only with latest version. I think where required we are raising exception to let use know they need newer version of the depended provider.

@moiseenkov
Copy link
Contributor Author

@eladkal , thank you for fast reply.
Could you please clarify this for me, because I haven't seen these kinds of exceptions before and I'm not aware of this approach? In which cases we should (and shouldn't) bump the provider package's dependencies? Is there any docs?

@eladkal
Copy link
Contributor

eladkal commented Mar 5, 2024

Example:

raise AirflowException(
f"You are trying to use common-sql with {hook.__class__.__name__},"
f" but the Hook class comes from provider {provider} that does not support it."
f" Please upgrade provider {provider} to at least {min_version}."
)

raise AirflowException(
f"You are trying to use `common-sql` with {hook.__class__.__name__},"
" but its provider does not support it. Please upgrade the provider to a version that"
" supports `common-sql`. The hook class should be a subclass of"
" `airflow.providers.common.sql.hooks.sql.DbApiHook`."
f" Got {hook.__class__.__name__} Hook with class hierarchy: {hook.__class__.mro()}"
)

@potiuk
Copy link
Member

potiuk commented Mar 5, 2024

And to add - what @eladkal explained is even better protection and more user-friendly way of adding the version dependency. The problem with the cross-provider dependencies is that they are soft. They are not - and cannot be enforced, so they are mostly informative. The only way you can make such cross-dependency works is when you do

pip install apache-airflow-providers-google[cncf.kubernetes]

When you do:

pip install apache-airflow-providers-google
pip install apache-airflow-providers-cncf-kubernetes

or

pip install apache-airflow[google,cncf.kubernetes]

there is absolutely no way to enforce this extra limit based on what is written in the extra. So checking the version of provider (or failing on importing non-existing value and raising exception is way better and more user-friendly approach..

BTW. We hava a dedicated exception OptionalProviderFeatureException - that can be used for that purpose.

@moiseenkov
Copy link
Contributor Author

Thank you so much for your explanation! Our team will follow these guides.

@moiseenkov moiseenkov closed this Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants