[BEAM-11772] Integrate BigQuery sink file loads with GroupIntoBatches #13859
Conversation
|
R: @reuvenlax, @pabloem |
nehsyc
changed the title
|
Run Java_Examples_Dataflow PreCommit |
pabloem
left a comment
pabloem
left a comment
There was a problem hiding this comment.
This mostly looks reasonable. I just have one question on the test utilty change.
| Map<String, List<String>> insertErrors = Maps.newHashMap(); | ||
|
|
||
| // The counter for the number of insertions performed. | ||
| static AtomicInteger insertCount; |
There was a problem hiding this comment.
why is this static? shouldn't it belong to the instance?
There was a problem hiding this comment.
It seems that the instance can be reconstructed for multiple times. If not using a static variable, the counter will be zero at the verification time. There is explanation for the static table map above:
...each ParDo will get a separate instance of FakeDatasetServices...
|
Run Java PostCommit |
pabloem
left a comment
pabloem
left a comment
There was a problem hiding this comment.
LGTM
@reuvenlax PTAL - I will merge in a couple days if you don't have any more comments.
|
thanks @nehsyc ! |
2 participants
Use GroupIntoBatches.WithShardedKey API to enable dynamic sharding when using FILE_LOADS for unbounded data. This is a follow-up change to #13496.
This PR
withAutoShardingoption in BigQueryIO builder toBatchLoads.numFileShardsto be 0 when performing streaming FILE_LOADS.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.