Add partitioned reads to JDBC SchemaIO #25240

byronellis · 2023-01-31T18:03:24Z

Addresses #25011 by adding partitioned reads to the existing SchemaIO.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

…elieve numeric and datetime are currently supported). Start adding a JdbcPartitionedReadSchemaTransformProvider as a more generic SchemaTransform. This fits better with the SchemaTransform approach as the partitioned read is actually a different transform entirely from the non-partitioned version.

…scussion. Added a test to the Python side that should exercise this pathway (though this is difficult to fully verify). Verified that it is actually run during tests and that it will fail if something is very wrong though.

…011-jdbc-java

codecov · 2023-01-31T18:24:45Z

Codecov Report

Merging #25240 (10e5483) into master (05dc71a) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #25240      +/-   ##
==========================================
- Coverage   72.97%   72.97%   -0.01%     
==========================================
  Files         743      743              
  Lines       99037    99037              
==========================================
- Hits        72273    72271       -2     
- Misses      25398    25400       +2     
  Partials     1366     1366

Flag	Coverage Δ
python	`82.51% <ø> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/io/jdbc.py	`78.26% <ø> (ø)`

... and 7 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Abacn · 2023-01-31T18:38:19Z

sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcSchemaIOProvider.java

-          if (outputParallelization != null) {
-            readRows = readRows.withOutputParallelization(outputParallelization);
+          // If we define a partition column we need to go a different route
+          @Nullable String partitionColumn = config.getString("partitionColumn");


Just note that for newly introduced parameters it may need to check the existence of the field first, similar to #25062 otherwise may causing error in go SDK's xlang jdbc wrapper

Hi @byronellis if you do not have time, I am willing to fix the comments in order to make it in before Beam 2.46.0 cut.

opened #25577 fixed a couple of things and test passed there

github-actions · 2023-01-31T19:10:08Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @pabloem for label python.
R: @apilloud for label java.
R: @johnjcasey for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

ahmedabu98 · 2023-01-31T22:46:08Z

sdks/python/apache_beam/io/jdbc.py

        ('output_parallelization', typing.Optional[bool]),
        ('autosharding', typing.Optional[bool]),
+        ('partitionColumn', typing.Optional[str]),
+        ('tableName', typing.Optional[str]),


Was the tableName configuration field meant to be added?

github-actions · 2023-02-08T12:13:59Z

Reminder, please take a look at this pr: @pabloem @apilloud @johnjcasey

ahmedabu98 · 2023-02-08T12:17:32Z

R: @ahmedabu98

ahmedabu98 · 2023-02-08T12:17:51Z

Waiting on author

github-actions · 2023-02-08T12:18:51Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

github-actions · 2023-07-22T12:39:48Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

github-actions · 2023-09-21T12:41:30Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

github-actions · 2023-09-29T12:55:52Z

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

Byron Ellis and others added 5 commits January 24, 2023 18:34

Merge branch 'apache:master' into beam25011-jdbc-java

b1e8bc7

Merge remote-tracking branch 'origin/beam25011-jdbc-java' into beam25…

1e8ebd1

…011-jdbc-java

Merge branch 'apache:master' into beam25011-jdbc-java

10e5483

Abacn reviewed Jan 31, 2023

View reviewed changes

github-actions bot added io java jdbc python labels Jan 31, 2023

github-actions bot added the Next Action: Reviewers label Jan 31, 2023

ahmedabu98 reviewed Jan 31, 2023

View reviewed changes

github-actions bot added the slow-review label Feb 8, 2023

github-actions bot added slow-review Next Action: Author and removed slow-review Next Action: Reviewers labels Feb 8, 2023

Abacn mentioned this pull request Feb 21, 2023

Add partitioned reads to JDBC SchemaIO #25577

Merged

3 tasks

tvalentyn removed the slow-review label May 22, 2023

github-actions bot added stale and removed stale labels Jul 22, 2023

github-actions bot added the stale label Sep 21, 2023

github-actions bot closed this Sep 29, 2023

Add partitioned reads to JDBC SchemaIO #25240

Add partitioned reads to JDBC SchemaIO #25240

Uh oh!

Conversation

byronellis commented Jan 31, 2023

GitHub Actions Tests Status (on master branch)

Uh oh!

codecov bot commented Jan 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Abacn Jan 31, 2023

Choose a reason for hiding this comment

Uh oh!

Abacn Feb 15, 2023

Choose a reason for hiding this comment

Uh oh!

Abacn Feb 22, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 31, 2023

Uh oh!

ahmedabu98 Jan 31, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 8, 2023

Uh oh!

ahmedabu98 commented Feb 8, 2023

Uh oh!

ahmedabu98 commented Feb 8, 2023

Uh oh!

github-actions bot commented Feb 8, 2023

Uh oh!

github-actions bot commented Jul 22, 2023

Uh oh!

github-actions bot commented Sep 21, 2023

Uh oh!

github-actions bot commented Sep 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Jan 31, 2023 •

edited

Loading