Skip to content

Conversation

@bolkedebruin
Copy link
Contributor

This adds low level support for open lineage to ObjectStorage and integrates it into common.io.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@bolkedebruin
Copy link
Contributor Author

should I just add openlineage to the list to 'always' test @potiuk ?

Copy link
Member

@hussein-awala hussein-awala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I just added a comment about the change on key extraction.

@uranusjr
Copy link
Member

uranusjr commented Dec 25, 2023

Pulling @mobuchowski to the conversation since he mentioned he wants to investigate adding this some time ago.

@bolkedebruin
Copy link
Contributor Author

@mobuchowski if there is any way to emit lineage events and openlineage can reconcile those we can add those events at the lower level so lineage becomes available for TaskFlow as well without manual intervention. Not sure if we can though?

@mobuchowski
Copy link
Contributor

mobuchowski commented Dec 27, 2023

@bolkedebruin working on it 🙂 f15a1e0
Main use would be instrumenting hooks - but would work with any other relevant code like object storage.

@bolkedebruin
Copy link
Contributor Author

ptal @hussein-awala @potiuk @uranusjr. The doc caching update has been split from this pr, I've kept the resolution of a key relative per examples and best practices. Integrating 'emitting' open lineage events requires work on the openlineage side per @mobuchowski.

Copy link
Member

@hussein-awala hussein-awala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants