Skip to content

openlineage, snowflake: add OpenLineage support for Snowflake #31696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mobuchowski merged 2 commits into from
Jul 21, 2023
Merged

openlineage, snowflake: add OpenLineage support for Snowflake #31696

mobuchowski merged 2 commits into from
Jul 21, 2023

Conversation

@JDarDagran
Copy link
Contributor

@JDarDagran JDarDagran commented Jun 3, 2023

This PR adds OpenLineage support for SnowflakeOperator.

depends on: #31398

sunank200
sunank200 reviewed Jun 13, 2023
View reviewed changes
Copy link
Collaborator

@sunank200 sunank200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation changes for the operator should be added for open lineage

airflow/providers/common/sql/hooks/sql.py Outdated
Copy link
Collaborator

@sunank200 sunank200 Jun 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have we tested this?

airflow/providers/common/sql/hooks/sql.py Outdated
Copy link
Collaborator

@sunank200 sunank200 Jun 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't there specific lineage in case of snowflake - eg. query run time

Copy link
Contributor Author

@JDarDagran JDarDagran Jun 20, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now I'm trying to reflect what's currently sent from openlineage-airflow package.
I might add some more information but I'd really appreciate what would be useful and available to retrieve.

Copy link
Contributor Author

@JDarDagran JDarDagran Jul 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the other hand - https://docs.snowflake.com/en/sql-reference/account-usage/query_history#usage-notes: latency for QUERY_HISTORY is up to 45 minutes so it looks like it's not sustainable to always get_ query run time

@JDarDagran JDarDagran force-pushed the aip-53-snowflake branch 2 times, most recently from edb89d4 to 3c4844f Compare July 7, 2023 07:02
@JDarDagran
Copy link
Contributor Author

JDarDagran commented Jul 7, 2023

One failing test looking rather flaky.

@pankajkoti pankajkoti removed their request for review July 12, 2023 12:23
@JDarDagran
Add OpenLineage support for SnowflakeOperator.
6d40c1c 
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
@JDarDagran
Change how default schema is retrieved.
dae1ba3 
Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
@mobuchowski mobuchowski merged commit 5b082c3 into apache:main Jul 21, 2023
@eladkal eladkal mentioned this pull request Jul 29, 2023
Closed
69 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mobuchowski mobuchowski mobuchowski approved these changes

@potiuk potiuk Awaiting requested review from potiuk

@ashb ashb Awaiting requested review from ashb

@jedcunningham jedcunningham Awaiting requested review from jedcunningham

@eladkal eladkal Awaiting requested review from eladkal

@mik-laj mik-laj Awaiting requested review from mik-laj

@phanikumv phanikumv Awaiting requested review from phanikumv

@pankajastro pankajastro Awaiting requested review from pankajastro

+1 more reviewer

@sunank200 sunank200 sunank200 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:dev-tools area:providers provider:common-sql provider:openlineage AIP-53 provider:snowflake Issues related to Snowflake provider

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@JDarDagran @mobuchowski @sunank200