[YAML] Implement a consistent BigQuery read and write transform. #28169
Conversation
|
R: @Polber |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
Codecov Report
@@ Coverage Diff @@
## master #28169 +/- ##
==========================================
+ Coverage 72.30% 72.36% +0.05%
==========================================
Files 678 680 +2
Lines 99853 100398 +545
==========================================
+ Hits 72197 72650 +453
- Misses 26077 26169 +92
Partials 1579 1579
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 27 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
* Use the schema provided, rather than just re-ordering. * Allow for unspecified optional fields. * Allow skipping of external transforms with unparsable (e.g. language-specific) schemas.
This allows one to remap configs for a cosistant API across languages (including differences like camelCase vs. snake_case).
This can be invoked from either Python or Java.
| type: beamJar | ||
| transforms: | ||
| 'ReadFromBigQuery': 'beam:schematransform:org.apache.beam:bigquery_storage_read:v1' | ||
| 'WriteToBigQuery': 'beam:schematransform:org.apache.beam:bigquery_storage_write:v1' |
There was a problem hiding this comment.
Oh, yes. (I changed this at the last minute just to be safe...)
|
Java_Examples_Dataflow_Java17 PreCommit |
|
Run Python_Integration PreCommit |
|
Python failures are timeout flakes seen elsewhere, Java failures also unrelated (only java change is BQ write provider, which is unused in WordCount). |
2 participants
This adapts the Java and Python BigQuery IOs to have the same interface and be callable from Yaml.
Several bugfixes were required along the way to make this work smoothly which are divided out in the commit history.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.