Skip to content

Conversation

@Amar3tto
Copy link
Collaborator

@Amar3tto Amar3tto commented Jul 17, 2022

Resolves #24961
Resolves #22315

  • Added test CDAP Plugins for reading/writing from/to PostgreSQL database and Input/Output Formats and FormatProviders for these plugins.
  • Added CdapIOIT class on HadoopFormatIOIT with integration test that performs the following:
    • Writes TestRowDBWritables to the PostgreSQL database using DBBatchSink CDAP plugin.
    • Reads test rows from the PostgreSQL database using DBBatchSource CDAP plugin.
    • Asserts expected and actual hashcodes for given rowCount.
  • Added Jenkins job called job_PerformanceTests_CdapIO for running CdapIOIT tests.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@Amar3tto
Copy link
Collaborator Author

Run Spotless PreCommit

@Amar3tto Amar3tto changed the title Add integration tests for CdapIO [CdapIO] Add integration tests for CdapIO (Batch) Jul 18, 2022
@Amar3tto
Copy link
Collaborator Author

Run Java HadoopFormatIO Performance Test

@aromanenko-dev
Copy link
Contributor

Run Seed Job

@Amar3tto Amar3tto marked this pull request as ready for review August 1, 2022 06:09
@github-actions
Copy link
Contributor

github-actions bot commented Aug 1, 2022

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @apilloud for label java.
R: @johnjcasey for label io.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@aromanenko-dev
Copy link
Contributor

Run Seed Job

@aromanenko-dev
Copy link
Contributor

Run Java CdapIO Performance Test

@aromanenko-dev
Copy link
Contributor

Run Seed Job

@aromanenko-dev aromanenko-dev self-requested a review August 1, 2022 17:09
@elizaveta-lomteva
Copy link
Contributor

@chamikaramj Cham, PR is ready for review. Thanks!

@aromanenko-dev
Copy link
Contributor

Run Java CdapIO Performance Test

Copy link
Contributor

@chamikaramj chamikaramj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also make sure that the benchmark results are available in http://metrics.beam.apache.org/.

LGTM.

@chamikaramj chamikaramj merged commit 74a534b into apache:master Aug 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Implement CdapIO connector for Java [Cdap IO][Task]: Implement integration tests for CDAP IO batch interface

4 participants