Add driverJars parameter to JdbcIO. #25824

pranavbhandari24 · 2023-03-13T15:38:11Z

Add driverJars parameter to JdbcIO.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

pranavbhandari24 · 2023-03-13T15:41:43Z

This change allows users to use driver jars in GCS. With this change, Dataflow templates will be able to migrate to JdbcIO instead of DynamicJdbcIO.

sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java

pranavbhandari24 · 2023-03-21T14:48:06Z

R: @bvolpato

pranavbhandari24 · 2023-03-21T14:49:20Z

R: @johnjcasey

github-actions · 2023-03-21T14:49:20Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

Abacn · 2023-03-21T19:15:23Z

Does setDriverClassName work? This is the way to set jdbc driver for jdbcIO. It also works for xlang jdbcio. imo the jar should be available in classpath which should be handled by the build system instead of the IO connector. CC: @chamikaramj may have an idea.

chamikaramj

I think it should work as a way to allow JdbcIO to more dynamically choose driver jars. Added some comments.

sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java

pranavbhandari24 · 2023-03-29T19:50:40Z

I changed the implementation to accept a comma separated list of driver Jars. These files will be saved locally and passed to the Datasource using a URL classloader

chamikaramj

Thanks.

sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcSchemaIOProvider.java

sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java

sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java

sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java

chamikaramj · 2023-03-30T16:11:23Z

sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java

I think we should also add an end-to-end integration test for this feature.

codecov · 2023-04-20T17:18:29Z

Codecov Report

Merging #25824 (f9dd2ae) into master (f549fd3) will decrease coverage by 0.04%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #25824      +/-   ##
==========================================
- Coverage   81.12%   81.08%   -0.04%     
==========================================
  Files         469      469              
  Lines       67185    67199      +14     
==========================================
- Hits        54501    54486      -15     
- Misses      12684    12713      +29

Flag	Coverage Δ
python	`81.08% <ø> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/io/jdbc.py	`79.16% <ø> (ø)`

... and 14 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

chamikaramj

LGTM. Thanks.

chamikaramj · 2023-04-20T23:17:15Z

Retest this please

chamikaramj · 2023-04-21T18:39:48Z

Please fix spotless:

Execution failed for task ':sdks:java:io:jdbc:spotlessJavaCheck'.
16:20:23 > The following files had format violations:
16:20:23 sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
16:20:23 @@ -451,6 +451,7 @@

This change allows users to use driver jars saved in GCS. With this change, Dataflow templates will be able to migrate to JdbcIO instead of DynamicJdbcIO.

pranavbhandari24 · 2023-04-26T14:05:51Z

Run Python PreCommit

johnjcasey · 2023-04-28T15:59:30Z

sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java

    }

-    DataSource buildDatasource() {
+    public DataSource buildDatasource() {


Nit: should this be public?

johnjcasey · 2023-04-28T16:00:03Z

sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java


  /** Infers the Beam {@link Schema} from {@link ResultSetMetaData}. */
-  static Schema toBeamSchema(ResultSetMetaData md) throws SQLException {
+  public static Schema toBeamSchema(ResultSetMetaData md) throws SQLException {


johnjcasey · 2023-04-28T16:00:08Z

sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java

 /** Provides utility functions for working with Beam {@link Schema} types. */
 @Experimental(Kind.SCHEMAS)
-class SchemaUtil {
+public class SchemaUtil {


johnjcasey · 2023-04-28T16:01:10Z

LGTM

This change allows users to use driver jars saved in GCS. With this change, Dataflow templates will be able to migrate to JdbcIO instead of DynamicJdbcIO.

* Add driverJars parameter to JdbcIO. (#25824) This change allows users to use driver jars saved in GCS. With this change, Dataflow templates will be able to migrate to JdbcIO instead of DynamicJdbcIO. * Fix jdbc xlang schema type mismatch (#26480) * Fix jdbc xlang schema type mismatch * Also fix fetch_size type mismatch * Add new fields in the end --------- Co-authored-by: Pranav Bhandari <bhandari.pranav22@gmail.com> Co-authored-by: Yi Hu <yathu@google.com>

This change allows users to use driver jars saved in GCS. With this change, Dataflow templates will be able to migrate to JdbcIO instead of DynamicJdbcIO.

* Add channel split separations. * Use channel leases for parallel processing. * move data wait to chan + atomic * Progress and split on execute goroutine. * Convert metrics tests to short ids. * Add Provision handler w/ capabilities. * Add artifact validation to run_rc_validation (#26407) * Add artifact validation to run_rc_validation * config file for artifacts * use java bom contents for validation * add pip_pre = True * Update tox.ini * Remove out of date experimental warning on SDFs (#26450) * [ToBF] Refinement 26.04.23 (#26428) * content tree rebuilds on sdk change * expandable parent node widget * comment fix * comment fix --------- Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> * Allow implicit flattening for yaml inputs. (#26423) * Add field annotations for high-priority Syndeo schema transforms (#26384) add changes * [Go SDK] Timers with new datalayer (#26101) * added timer package * add timer changes and merged with rebo's pr * timer fired in stateful * error setting new timer in ontimer * looping timers work * send fv instead of bytes * changes to coder/pardo * works for all cases, only cleanup left * remove comments and validate onTimer * generic coder for user key * fixes coder end to end * remove logs * add unit test and refactor * add docs * new example * fix static lint * support emitters * allow input col of CoGBK as well * unit tests, periodic impulse, minor refactor * update PipelineTimer interface, minor refactor, doc comment for example * add warn message * single edge timer coder, rm kv coder check, cache encoder,decoder * Update chromedriver-binary requirement in /sdks/python Updates the requirements on [chromedriver-binary](https://github.com/danielkaiser/python-chromedriver-binary) to permit the latest version. - [Release notes](https://github.com/danielkaiser/python-chromedriver-binary/releases) - [Commits](danielkaiser/python-chromedriver-binary@v100.0.4896.20.0...v113.0.5672.24.0) --- updated-dependencies: - dependency-name: chromedriver-binary dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> * Automation: Tour of Beam infrastructure deployment (#25793) * Added Terraform scripts for TOB infra * ToB Frontend related updates * Update settings.gradle.kts * Deleted redundant file and minor README change * Addressing comments in the PR * Added newline at the end of variables.tf file * Update README.md * Updates related to Tour of Beam infrastructure * Update locals.tf * Output.tf updates * Update output.tf * Updates * Update main.tf * Updates to cloudfunctions_bucket variable * service_account_id changes * Update main.tf * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Bulk update of terraform scripts * Update README.md * Update README.md * Datastore_namespace updates * Update README.md * Update README.md * Update README.md * Update README.md * Update main.tf * Update README.md * Update README.md * Update README.md * Some minor TF updates * Update README.md * Modify batch IT to use count instead of hash (#26327) * Modify batch IT to use count instead of hash * remove unused varaiable * run spotless, update pipeline state checking * Update timeout to 45 minutes * revert timeout, add additional counter to try and pinpoint missing records * add a log to notify ranges used when workers restart * change counts from metrics to combiners * add a window to streaming test * move the passert to the correct place * Remove extra counter, apply spotless * add additional metric to KafkaWriter * Remove debugging metrics * verify pipeline is not failed * remove extra newline * Revert "Modify batch IT to use count instead of hash (#26327)" (#26466) This reverts commit 9903b2f. * Bump Java Dataflow container images (#26459) * keep retrying mass_comment until it has started all jobs (#26457) * keep retrying mass_comment until it has started all jobs * fix lookups * Add driverJars parameter to JdbcIO. (#25824) This change allows users to use driver jars saved in GCS. With this change, Dataflow templates will be able to migrate to JdbcIO instead of DynamicJdbcIO. * [Roll Fwd PR] Rename _namespace to _get_display_data_namespace"" (#26470) * Update parquetio and textio to work with -beam_strict (#26469) * use wheel sdk location for PostCommit_Py_Examples (#26473) * Move back the timeout of Python PostCommit to 4h * Minor fix on Python PostCommit description strings * Add recent postcommits to jenkins README * More user-friendly providers. * Add yaml preprocessing phases. * Add flexible windowing syntax to yaml. * Implement flatten in terms of preprocessor phase. This composes better with windowing. * Reword SQL note. * Make linter happy. * Survive errors in size estimation in MongoDbIO * Fix jdbc xlang schema type mismatch (#26480) * Fix jdbc xlang schema type mismatch * Also fix fetch_size type mismatch * Add new fields in the end * [Python] Add saved_weights example to tf notebook (#26472) * add saved_weights example to tf notebook * add description * updated text blocks * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> --------- Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> * Revert "Revert "Modify batch IT to use count instead of hash (#26327)" (#26466)" (#26467) This reverts commit 5c67668. * Bump github.com/tetratelabs/wazero from 1.0.3 to 1.1.0 in /sdks (#26486) Bumps [github.com/tetratelabs/wazero](https://github.com/tetratelabs/wazero) from 1.0.3 to 1.1.0. - [Release notes](https://github.com/tetratelabs/wazero/releases) - [Commits](wazero/wazero@v1.0.3...v1.1.0) --- updated-dependencies: - dependency-name: github.com/tetratelabs/wazero dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * tour of beam integration tests (#25925) * Integration test to load the default example of the default SDK and change the example (#24730) (#24729) * Fix formatting and README (#24730) * Support collection v1.17.0 (#24730) * LoadingIndicator on chaning examples, remove duplicating licenses (#24730) * Add a missing license header (#24730) * Integration test for changing SDK and running code (#24779) (#382) * Integration test for changing SDK and running code (#24779) * Rename an integration test (#24779) * Use enum to switch SDK in integration test (#24779) * Find SDK in a dropdown by key (#24779) * Add a TODO (#24779) * Fix exports (#24779) * Issue24779 integration changing sdk from 24370 (#387) * Integration test for changing SDK and running code (#24779) * Rename an integration test (#24779) * Use enum to switch SDK in integration test (#24779) * Find SDK in a dropdown by key (#24779) * Add a TODO (#24779) * Fix exports (#24779) * Integration tests miscellaneous UI (#383) * miscellaneous ui integration tests * reverted pubspec.lock * gradle tasks ordered alhpabetically * integration tests refactoring * clean code * integration tests miscellaneous ui fix pr * rename method * added layout adaptivity * A minor cleanup (#24779) Co-authored-by: Dmitry Repin <mr.malarg@gmail.com> * integration tests run and editing * example selector test * minor fixes * rat * fix pr * minor * minor * rat * integration test finder written * integration test minor fixes * minor fixes * removed comment * minor fixes * playground integration tests minor fixes * integration test pumpAnSettleNoException * integration test shortcut refactor * integration test another changing shortcuts running * upgrade to flutter 3.7.1 * workaround comment * playground frontend updated major versions * issues 25329 25331 25336 * 25329 extract connectivity extension to separate file * Upgrade Flutter to 3.7.3 in integration tests (#24730) * Fix integration test (#24730) * fix cors issue and added mouse scroll to tags * Upgrade Flutter in Dockerfile (#24720) * sorting moved to model * sorting moved to model * sorting moved to model * bugs fix * issue 25278 * fix pr * quites fix in en.yaml * Fix not loading default example (#25528) * fix compile error * Refactor output tabs, test embedded playground (#25136) (#439) * Refactor output tabs, test embedded playground (#25136) * Clean up (#25136) * Change example paths to IDs in integration tests * issue25640 tob ci * fix tob ci * rename ci process * test add new line to main * test add new line to main * commented unit test run * issue25640 changed server path * issue25640 tests on welcome page * deleted config.g.dart * issue25640 pr fixes * Update .github/workflows/tour_of_beam_frontend_test.yml Co-authored-by: alexeyinkin <leha@inkin.ru> * Update learning/tour-of-beam/frontend/integration_test/welcome_page_test.dart Co-authored-by: alexeyinkin <leha@inkin.ru> * Improve tests (#25640) * issue25640 tour page tests * pr fix * removed import * pr fix * fix test * 25640 fixed pubspec.lock * issue25640 fix readme * updated readme * issue25640 fixed after master merge * issue25483 ToB pipeline options * removed unnecesary variable * pr fix * Update learning/tour-of-beam/frontend/assets/translations/en.yaml Co-authored-by: alexeyinkin <leha@inkin.ru> * playground hides when snippet does not exists * pipeline options extracted to playground components * issue25483 pipeline options * added errors handling, fix pr * refactoring * Revert "refactoring" This reverts commit 1540961. * removed unnecessary constants * playground controller in tour notifier becomes nullable * playground controller returned to non nullable in tour notifier * playground controller actions * removed unnecessary code * tob scaffold wrapped with animated builder * minor fixes * partially fixed tests * Upgrade flutter_code_editor to v0.2.19 (#25640) * Replace output SelectableText with a CodeField instance (#25640) * Trigger ToB integration tests (#25640) * Clean up (#25640) * Enable manual workflow runs for Playground and ToB integration tests (#25640) --------- Co-authored-by: Alexey Inkin <alexey.inkin@akvelon.com> Co-authored-by: alexeyinkin <leha@inkin.ru> * Eliminate nullness errors from MongoDbIO * Move provisioned options outside of harness.Main (#26476) Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com> * Add SDF adjacent element test * Consolidate residual processing * fix race condition on split boundaries * deflake fixed window combine tests * Add comments to job functions. * Make int64check per window. * Update sdks/go/pkg/beam/runners/prism/internal/worker/bundle.go Co-authored-by: Ritesh Ghorse <riteshghorse@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Anand Inguva <anandinguva98@gmail.com> Co-authored-by: Anand Inguva <34158215+AnandInguva@users.noreply.github.com> Co-authored-by: Darkhan Nausharipov <31556582+nausharipov@users.noreply.github.com> Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> Co-authored-by: Robert Bradshaw <robertwb@gmail.com> Co-authored-by: Andrei Gurau <andreigurau@google.com> Co-authored-by: Ritesh Ghorse <riteshghorse@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: ruslan-ikhsan <114978215+ruslan-ikhsan@users.noreply.github.com> Co-authored-by: johnjcasey <95318300+johnjcasey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: Pranav Bhandari <bhandari.pranav22@gmail.com> Co-authored-by: Svetak Sundhar <svetaksundhar@google.com> Co-authored-by: Jeremy Edwards <jeremyje@gmail.com> Co-authored-by: Kenneth Knowles <klk@google.com> Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> Co-authored-by: Dmitry Repin <mr.malarg@gmail.com> Co-authored-by: Alexey Inkin <alexey.inkin@akvelon.com> Co-authored-by: alexeyinkin <leha@inkin.ru>

* Add channel split separations. * Use channel leases for parallel processing. * move data wait to chan + atomic * Progress and split on execute goroutine. * Convert metrics tests to short ids. * Add Provision handler w/ capabilities. * Add artifact validation to run_rc_validation (apache#26407) * Add artifact validation to run_rc_validation * config file for artifacts * use java bom contents for validation * add pip_pre = True * Update tox.ini * Remove out of date experimental warning on SDFs (apache#26450) * [ToBF] Refinement 26.04.23 (apache#26428) * content tree rebuilds on sdk change * expandable parent node widget * comment fix * comment fix --------- Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> * Allow implicit flattening for yaml inputs. (apache#26423) * Add field annotations for high-priority Syndeo schema transforms (apache#26384) add changes * [Go SDK] Timers with new datalayer (apache#26101) * added timer package * add timer changes and merged with rebo's pr * timer fired in stateful * error setting new timer in ontimer * looping timers work * send fv instead of bytes * changes to coder/pardo * works for all cases, only cleanup left * remove comments and validate onTimer * generic coder for user key * fixes coder end to end * remove logs * add unit test and refactor * add docs * new example * fix static lint * support emitters * allow input col of CoGBK as well * unit tests, periodic impulse, minor refactor * update PipelineTimer interface, minor refactor, doc comment for example * add warn message * single edge timer coder, rm kv coder check, cache encoder,decoder * Update chromedriver-binary requirement in /sdks/python Updates the requirements on [chromedriver-binary](https://github.com/danielkaiser/python-chromedriver-binary) to permit the latest version. - [Release notes](https://github.com/danielkaiser/python-chromedriver-binary/releases) - [Commits](danielkaiser/python-chromedriver-binary@v100.0.4896.20.0...v113.0.5672.24.0) --- updated-dependencies: - dependency-name: chromedriver-binary dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> * Automation: Tour of Beam infrastructure deployment (apache#25793) * Added Terraform scripts for TOB infra * ToB Frontend related updates * Update settings.gradle.kts * Deleted redundant file and minor README change * Addressing comments in the PR * Added newline at the end of variables.tf file * Update README.md * Updates related to Tour of Beam infrastructure * Update locals.tf * Output.tf updates * Update output.tf * Updates * Update main.tf * Updates to cloudfunctions_bucket variable * service_account_id changes * Update main.tf * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Bulk update of terraform scripts * Update README.md * Update README.md * Datastore_namespace updates * Update README.md * Update README.md * Update README.md * Update README.md * Update main.tf * Update README.md * Update README.md * Update README.md * Some minor TF updates * Update README.md * Modify batch IT to use count instead of hash (apache#26327) * Modify batch IT to use count instead of hash * remove unused varaiable * run spotless, update pipeline state checking * Update timeout to 45 minutes * revert timeout, add additional counter to try and pinpoint missing records * add a log to notify ranges used when workers restart * change counts from metrics to combiners * add a window to streaming test * move the passert to the correct place * Remove extra counter, apply spotless * add additional metric to KafkaWriter * Remove debugging metrics * verify pipeline is not failed * remove extra newline * Revert "Modify batch IT to use count instead of hash (apache#26327)" (apache#26466) This reverts commit 9903b2f. * Bump Java Dataflow container images (apache#26459) * keep retrying mass_comment until it has started all jobs (apache#26457) * keep retrying mass_comment until it has started all jobs * fix lookups * Add driverJars parameter to JdbcIO. (apache#25824) This change allows users to use driver jars saved in GCS. With this change, Dataflow templates will be able to migrate to JdbcIO instead of DynamicJdbcIO. * [Roll Fwd PR] Rename _namespace to _get_display_data_namespace"" (apache#26470) * Update parquetio and textio to work with -beam_strict (apache#26469) * use wheel sdk location for PostCommit_Py_Examples (apache#26473) * Move back the timeout of Python PostCommit to 4h * Minor fix on Python PostCommit description strings * Add recent postcommits to jenkins README * More user-friendly providers. * Add yaml preprocessing phases. * Add flexible windowing syntax to yaml. * Implement flatten in terms of preprocessor phase. This composes better with windowing. * Reword SQL note. * Make linter happy. * Survive errors in size estimation in MongoDbIO * Fix jdbc xlang schema type mismatch (apache#26480) * Fix jdbc xlang schema type mismatch * Also fix fetch_size type mismatch * Add new fields in the end * [Python] Add saved_weights example to tf notebook (apache#26472) * add saved_weights example to tf notebook * add description * updated text blocks * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> --------- Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> * Revert "Revert "Modify batch IT to use count instead of hash (apache#26327)" (apache#26466)" (apache#26467) This reverts commit 5c67668. * Bump github.com/tetratelabs/wazero from 1.0.3 to 1.1.0 in /sdks (apache#26486) Bumps [github.com/tetratelabs/wazero](https://github.com/tetratelabs/wazero) from 1.0.3 to 1.1.0. - [Release notes](https://github.com/tetratelabs/wazero/releases) - [Commits](wazero/wazero@v1.0.3...v1.1.0) --- updated-dependencies: - dependency-name: github.com/tetratelabs/wazero dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * tour of beam integration tests (apache#25925) * Integration test to load the default example of the default SDK and change the example (apache#24730) (apache#24729) * Fix formatting and README (apache#24730) * Support collection v1.17.0 (apache#24730) * LoadingIndicator on chaning examples, remove duplicating licenses (apache#24730) * Add a missing license header (apache#24730) * Integration test for changing SDK and running code (apache#24779) (apache#382) * Integration test for changing SDK and running code (apache#24779) * Rename an integration test (apache#24779) * Use enum to switch SDK in integration test (apache#24779) * Find SDK in a dropdown by key (apache#24779) * Add a TODO (apache#24779) * Fix exports (apache#24779) * Issue24779 integration changing sdk from 24370 (apache#387) * Integration test for changing SDK and running code (apache#24779) * Rename an integration test (apache#24779) * Use enum to switch SDK in integration test (apache#24779) * Find SDK in a dropdown by key (apache#24779) * Add a TODO (apache#24779) * Fix exports (apache#24779) * Integration tests miscellaneous UI (apache#383) * miscellaneous ui integration tests * reverted pubspec.lock * gradle tasks ordered alhpabetically * integration tests refactoring * clean code * integration tests miscellaneous ui fix pr * rename method * added layout adaptivity * A minor cleanup (apache#24779) Co-authored-by: Dmitry Repin <mr.malarg@gmail.com> * integration tests run and editing * example selector test * minor fixes * rat * fix pr * minor * minor * rat * integration test finder written * integration test minor fixes * minor fixes * removed comment * minor fixes * playground integration tests minor fixes * integration test pumpAnSettleNoException * integration test shortcut refactor * integration test another changing shortcuts running * upgrade to flutter 3.7.1 * workaround comment * playground frontend updated major versions * issues 25329 25331 25336 * 25329 extract connectivity extension to separate file * Upgrade Flutter to 3.7.3 in integration tests (apache#24730) * Fix integration test (apache#24730) * fix cors issue and added mouse scroll to tags * Upgrade Flutter in Dockerfile (apache#24720) * sorting moved to model * sorting moved to model * sorting moved to model * bugs fix * issue 25278 * fix pr * quites fix in en.yaml * Fix not loading default example (apache#25528) * fix compile error * Refactor output tabs, test embedded playground (apache#25136) (apache#439) * Refactor output tabs, test embedded playground (apache#25136) * Clean up (apache#25136) * Change example paths to IDs in integration tests * issue25640 tob ci * fix tob ci * rename ci process * test add new line to main * test add new line to main * commented unit test run * issue25640 changed server path * issue25640 tests on welcome page * deleted config.g.dart * issue25640 pr fixes * Update .github/workflows/tour_of_beam_frontend_test.yml Co-authored-by: alexeyinkin <leha@inkin.ru> * Update learning/tour-of-beam/frontend/integration_test/welcome_page_test.dart Co-authored-by: alexeyinkin <leha@inkin.ru> * Improve tests (apache#25640) * issue25640 tour page tests * pr fix * removed import * pr fix * fix test * 25640 fixed pubspec.lock * issue25640 fix readme * updated readme * issue25640 fixed after master merge * issue25483 ToB pipeline options * removed unnecesary variable * pr fix * Update learning/tour-of-beam/frontend/assets/translations/en.yaml Co-authored-by: alexeyinkin <leha@inkin.ru> * playground hides when snippet does not exists * pipeline options extracted to playground components * issue25483 pipeline options * added errors handling, fix pr * refactoring * Revert "refactoring" This reverts commit 1540961. * removed unnecessary constants * playground controller in tour notifier becomes nullable * playground controller returned to non nullable in tour notifier * playground controller actions * removed unnecessary code * tob scaffold wrapped with animated builder * minor fixes * partially fixed tests * Upgrade flutter_code_editor to v0.2.19 (apache#25640) * Replace output SelectableText with a CodeField instance (apache#25640) * Trigger ToB integration tests (apache#25640) * Clean up (apache#25640) * Enable manual workflow runs for Playground and ToB integration tests (apache#25640) --------- Co-authored-by: Alexey Inkin <alexey.inkin@akvelon.com> Co-authored-by: alexeyinkin <leha@inkin.ru> * Eliminate nullness errors from MongoDbIO * Move provisioned options outside of harness.Main (apache#26476) Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com> * Add SDF adjacent element test * Consolidate residual processing * fix race condition on split boundaries * deflake fixed window combine tests * Add comments to job functions. * Make int64check per window. * Update sdks/go/pkg/beam/runners/prism/internal/worker/bundle.go Co-authored-by: Ritesh Ghorse <riteshghorse@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Anand Inguva <anandinguva98@gmail.com> Co-authored-by: Anand Inguva <34158215+AnandInguva@users.noreply.github.com> Co-authored-by: Darkhan Nausharipov <31556582+nausharipov@users.noreply.github.com> Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> Co-authored-by: Robert Bradshaw <robertwb@gmail.com> Co-authored-by: Andrei Gurau <andreigurau@google.com> Co-authored-by: Ritesh Ghorse <riteshghorse@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: ruslan-ikhsan <114978215+ruslan-ikhsan@users.noreply.github.com> Co-authored-by: johnjcasey <95318300+johnjcasey@users.noreply.github.com> Co-authored-by: Yi Hu <yathu@google.com> Co-authored-by: Pranav Bhandari <bhandari.pranav22@gmail.com> Co-authored-by: Svetak Sundhar <svetaksundhar@google.com> Co-authored-by: Jeremy Edwards <jeremyje@gmail.com> Co-authored-by: Kenneth Knowles <klk@google.com> Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> Co-authored-by: Dmitry Repin <mr.malarg@gmail.com> Co-authored-by: Alexey Inkin <alexey.inkin@akvelon.com> Co-authored-by: alexeyinkin <leha@inkin.ru>

github-actions bot added io java jdbc labels Mar 13, 2023

pranavbhandari24 marked this pull request as draft March 13, 2023 15:41

pranavbhandari24 force-pushed the master branch from d898f6b to 32a882f Compare March 13, 2023 17:02

Abacn reviewed Mar 14, 2023

View reviewed changes

sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java Outdated Show resolved Hide resolved

pranavbhandari24 force-pushed the master branch from 32a882f to 2aa24a2 Compare March 21, 2023 14:46

pranavbhandari24 marked this pull request as ready for review March 21, 2023 14:47

pranavbhandari24 requested a review from Abacn March 21, 2023 14:49

chamikaramj requested changes Mar 22, 2023

View reviewed changes

pranavbhandari24 force-pushed the master branch from 2aa24a2 to c69f71e Compare March 29, 2023 19:41

pranavbhandari24 changed the title ~~Add driverJarDirectory parameter to JdbcIO.~~ Add driverJars parameter to JdbcIO. Mar 29, 2023

pranavbhandari24 requested review from chamikaramj and removed request for Abacn March 29, 2023 19:50

chamikaramj requested changes Mar 30, 2023

View reviewed changes

pranavbhandari24 force-pushed the master branch 2 times, most recently from d329a34 to c3b18d4 Compare April 20, 2023 15:26

github-actions bot added the python label Apr 20, 2023

pranavbhandari24 requested a review from chamikaramj April 20, 2023 15:43

pranavbhandari24 force-pushed the master branch from c3b18d4 to 969b813 Compare April 20, 2023 15:48

chamikaramj reviewed Apr 20, 2023

View reviewed changes

pranavbhandari24 force-pushed the master branch 3 times, most recently from be9a837 to ceac109 Compare April 24, 2023 15:28

github-actions bot added the examples label Apr 24, 2023

Add driverJars parameter to JdbcIO.

f9dd2ae

This change allows users to use driver jars saved in GCS. With this change, Dataflow templates will be able to migrate to JdbcIO instead of DynamicJdbcIO.

pranavbhandari24 force-pushed the master branch from ceac109 to f9dd2ae Compare April 24, 2023 19:20

github-actions bot removed the examples label Apr 24, 2023

pranavbhandari24 requested a review from chamikaramj April 24, 2023 19:28

johnjcasey reviewed Apr 28, 2023

View reviewed changes

johnjcasey merged commit fb770c3 into apache:master Apr 28, 2023

bvolpato mentioned this pull request Apr 28, 2023

Cherry-pick: Add driverJars parameter to JdbcIO. (#25824) #26474

Merged

Abacn mentioned this pull request Apr 28, 2023

Fix jdbc xlang schema type mismatch #26480

Merged

3 tasks

Abacn mentioned this pull request May 27, 2023

[Task]: Add maxConnections to SchemaIO field builder and expose it in xlang wrappers #25168

Closed

15 tasks

Add driverJars parameter to JdbcIO. #25824

Add driverJars parameter to JdbcIO. #25824

Uh oh!

Conversation

pranavbhandari24 commented Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!