Sharding vortex

[Naireen]

Enable autosharding for BQ: #22818

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---					---
Java
Python	---					---
XLang						---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[Naireen]

Run Java PreCommit

[Naireen]

R: @reuvenlax

[codecov]

Codecov Report

Merging #16795 (09c4bd5) into master (c7f6426) will increase coverage by 9.64%.
The diff coverage is n/a.

❗ Current head 09c4bd5 differs from pull request most recent head ec3bcf3. Consider uploading reports for the commit ec3bcf3 to get more accurate results

@@            Coverage Diff             @@
##           master   #16795      +/-   ##
==========================================
+ Coverage   73.98%   83.62%   +9.64%     
==========================================
  Files         713      453     -260     
  Lines       94048    62471   -31577     
==========================================
- Hits        69582    52242   -17340     
+ Misses      23180    10229   -12951     
+ Partials     1286        0    -1286     

Flag	Coverage Δ
go	?
python	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/internal/pickler.py	77.27% <0.00%> (-18.19%)	⬇️
...beam/runners/portability/local_job_service_main.py	0.00% <0.00%> (-14.44%)	⬇️
...cp/internal/clients/bigquery/bigquery_v2_client.py	53.36% <0.00%> (-13.47%)	⬇️
sdks/python/apache_beam/io/gcp/bigquery.py	63.59% <0.00%> (-10.65%)	⬇️
sdks/python/apache_beam/io/gcp/gcsio_overrides.py	60.00% <0.00%> (-10.00%)	⬇️
sdks/python/apache_beam/runners/job/utils.py	69.23% <0.00%> (-7.70%)	⬇️
sdks/python/apache_beam/typehints/row_type.py	89.47% <0.00%> (-7.55%)	⬇️
...thon/apache_beam/runners/worker/sdk_worker_main.py	73.22% <0.00%> (-5.27%)	⬇️
sdks/python/apache_beam/transforms/cy_combiners.py	58.40% <0.00%> (-4.51%)	⬇️
...ks/python/apache_beam/runners/worker/operations.py	70.01% <0.00%> (-4.32%)	⬇️
... and 467 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[aaltay]

I triggered the tests. I am not sure what is the next step after that.

@Naireen - it would be good to add a jira in the issue title with some explanation of what you are trying to do.

[reuvenlax]

LGTM, however please attach a JIRA to the title

[github-actions]

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

[Naireen]

Run Java PreCommit

[johnjcasey]

run java precommit

[johnjcasey]

Does autosharding default to true or false?

[Naireen]

The Write method sets autosharding as false by default:

beam/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

Line 1768 in 0fbecde

.setAutoSharding(false)

. The user has to explicitly enable it

[johnjcasey]

can we push this to a helper method to improve readability?

[Naireen]

Refactored, PTAL.

[reuvenlax]

LGTM

[Naireen]

Run Spotless PreCommit

[Naireen]

Run Java PreCommit