Skip to content

Conversation

@kacpermuda
Copy link
Contributor

Some providers, such as Snowflake and DBT Cloud, do not require an OpenLineage provider but may offer optional features that depend on it. These features are generally available starting from a specific version of the OpenLineage provider or client. This decorator helps ensure compatibility, preventing import errors and providing clear logs about version requirements.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@mobuchowski mobuchowski merged commit a993db2 into apache:main Mar 18, 2025
60 checks passed
Comment on lines +44 to +46
Raises:
ValueError: If neither `provider_min_version` nor `client_min_version` is provided.
TypeError: If the decorator is used without parentheses (e.g., `@require_openlineage_version`).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes openlineage behave differently from other providers.
We always raise AirflowOptionalProviderFeatureException when there is problem with version.
Even in openlineage itself

try:
from airflow.providers.openlineage.conf import namespace
except ModuleNotFoundError as e:
from airflow.exceptions import AirflowOptionalProviderFeatureException
msg = "Please install `apache-airflow-providers-openlineage>=1.7.0`"
raise AirflowOptionalProviderFeatureException(e, msg)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wonder if we can change require_openlineage_version to require_provider_version(provider=...)
and create a more generic interface for all providers that implement the function

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So these two errors ValueError and TypeError are related to how this decorator is used. If the OL version if incompatible, we simply log and return None. Do you think we should raise this AirflowOptionalProviderFeatureException in such cases?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any Exceptions raised during OL's extraction are caught by OL anyway, so the result would be similar.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is a provider version incompatibility we should raise AirflowOptionalProviderFeatureException

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed it and added generic checker in this PR: #47909. Could you check if that's satisfying solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants