Skip to content

openlineage, gcs: add openlineage methods for GcsToGcsOperator #31350

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
eladkal merged 1 commit into from
Jul 27, 2023
Merged

openlineage, gcs: add openlineage methods for GcsToGcsOperator #31350

eladkal merged 1 commit into from
Jul 27, 2023

Conversation

@mobuchowski
Copy link
Contributor

@mobuchowski mobuchowski commented May 17, 2023 •
edited
Loading

This PR adds OpenLineage support for GcsToGcsOperator.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels May 17, 2023
@mobuchowski mobuchowski force-pushed the openlineage-gcs-operator branch from fa7a3c0 to ec0ae89 Compare May 17, 2023 11:35
@mobuchowski mobuchowski force-pushed the openlineage-gcs-operator branch from ec0ae89 to 49167ab Compare May 18, 2023 09:56
sunank200
sunank200 reviewed May 18, 2023
View reviewed changes
Copy link
Collaborator

@sunank200 sunank200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should add documentation around this along with the following changes

airflow/providers/google/cloud/transfers/gcs_to_gcs.py Outdated
Copy link
Collaborator

@sunank200 sunank200 May 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about DataSource dataset facet as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#dataset-facets?

Copy link
Collaborator

@sunank200 sunank200 Jul 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this not required @mobuchowski ?

Copy link
Contributor Author

@mobuchowski mobuchowski Jul 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's required here

@mobuchowski
Copy link
Contributor Author

mobuchowski commented May 25, 2023

@sunank200

Add run facet which captures the error as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#run-facets
Add test case for error scenario where runfacet is populated

In general case it's hard to capture errors now, since on_task_instance_failed does not pass actual error message.
We could store it somewhere on task_instance maybe.

@mobuchowski mobuchowski force-pushed the openlineage-gcs-operator branch 2 times, most recently from 9af4382 to edf62f9 Compare May 25, 2023 12:24
@mobuchowski mobuchowski force-pushed the openlineage-gcs-operator branch 2 times, most recently from 65afad8 to cf86fb7 Compare June 15, 2023 15:02
@pankajkoti pankajkoti removed their request for review July 12, 2023 12:22
@mobuchowski mobuchowski force-pushed the openlineage-gcs-operator branch 4 times, most recently from 26d29bb to 0b83011 Compare July 24, 2023 11:52
sunank200
sunank200 approved these changes Jul 24, 2023
View reviewed changes
airflow/providers/google/cloud/transfers/gcs_to_gcs.py Outdated
Copy link
Collaborator

@sunank200 sunank200 Jul 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this not required @mobuchowski ?

sunank200
sunank200 approved these changes Jul 24, 2023
View reviewed changes
Copy link
Collaborator

@sunank200 sunank200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once test passes it should be good to be merged.

@mobuchowski
openlineage, gcs: add openlineage methods for GcsToGcsOperator
5158070 
Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
@mobuchowski mobuchowski force-pushed the openlineage-gcs-operator branch from 0b83011 to 5158070 Compare July 24, 2023 20:02
@eladkal eladkal merged commit b733667 into apache:main Jul 27, 2023
@eladkal eladkal mentioned this pull request Jul 29, 2023
Closed
69 tasks
@kacpermuda kacpermuda mentioned this pull request Apr 16, 2024
Merged
@eladkal eladkal mentioned this pull request May 1, 2024
Closed
@eladkal eladkal mentioned this pull request May 12, 2024
Closed
66 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eladkal eladkal eladkal approved these changes

@potiuk potiuk Awaiting requested review from potiuk potiuk is a code owner

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

@phanikumv phanikumv Awaiting requested review from phanikumv

@pankajastro pankajastro Awaiting requested review from pankajastro

+2 more reviewers

@sunank200 sunank200 sunank200 approved these changes

@ahidalgob ahidalgob ahidalgob approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:providers provider:google Google (including GCP) related issues provider:openlineage AIP-53

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mobuchowski @sunank200 @ahidalgob @eladkal