Skip to content

[BEAM-9444] Use GCP Libraries BOM for Google Cloud Dependencies #11156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
suztomo wants to merge 18 commits into from
Closed

[BEAM-9444] Use GCP Libraries BOM for Google Cloud Dependencies #11156

suztomo wants to merge 18 commits into from

Conversation

@suztomo
Copy link
Contributor

@suztomo suztomo commented Mar 18, 2020 •
edited
Loading

As per Subject: Proposal: Beam to use GCP Libraries BOM, specifying library versions through GCP Libraries BOM.

  • To use BOM, declaring the BOM dependency with "platform" (Gradle: Importing Maven BOMs)
  • Removing version part from Google Cloud dependencies that is in the BOM.
  • Maven artifact POM generation logic showing "platform" dependency in dependnecyManagement section.

Linkage Check Result

No change. https://gist.github.com/suztomo/eadcdcac2b465a771cdefe27a29bbbc8

Artifact Version Changes

Resolved artifact version changes as part of this PR:

Unchanged

com.google.api:gax:1.54.0
com.google.api:gax-grpc:1.54.0
com.google.api:api-common:1.8.1
com.google.cloud:google-cloud-bigquerystorage:0.125.0-beta
com.google.cloud:google-cloud-datacatalog:0.32.1
io.grpc:grpc-auth:1.27.2
io.grpc:grpc-core:1.27.2
io.grpc:grpc-context:1.27.2
io.grpc:grpc-grpclb:1.27.2
io.grpc:grpc-netty:1.27.2
io.grpc:grpc-stub:1.27.2
com.google.api.grpc:proto-google-common-protos:1.17.0

Upgrade

com.google.auth:google-auth-library-credentials:0.20.0 from 0.19.0
com.google.auth:google-auth-library-oauth2-http:0.20.0 from 0.19.0
com.google.cloud:google-cloud-core:1.93.0 from 1.92.2
com.google.cloud:google-cloud-core-grpc:1.93.0 from 1.92.2
com.google.cloud:google-cloud-spanner:1.49.2 from 1.49.1
com.google.http-client:google-http-client:1.34.2 from 1.34.0
com.google.http-client:google-http-client-jackson2:1.34.2 from 1.34.0
com.google.http-client:google-http-client-protobuf:1.34.2 from 1.34.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.90.0 from 0.85.1
com.google.api.grpc:proto-google-cloud-bigtable-v2:1.10.0 from 1.9.1
com.google.api.grpc:proto-google-cloud-datastore-v1:0.85.2 from 0.85.0
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:1.49.2 from 1.49.1

Downgrade

com.google.cloud:google-cloud-bigquery:1.107.0 from 1.108.0 // GCP Libraries BOM 4.3.0 will have 1.108.1
com.google.api.grpc:grpc-google-cloud-pubsub-v1:1.85.0 from 1.85.1 // GCP Librareis BOM 4.3.0 will have 1.85.1
com.google.api.grpc:proto-google-cloud-pubsub-v1:1.85.0 from 1.85.1 // GCP Librareis BOM 4.3.0 will have 1.85.1

Removal

Removed unused fields

io.grpc:grpc-protobuf:1.27.2
io.grpc:grpc-protobuf-lite:1.27.2

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status
Build Status		 Build Status
Build Status
Build Status		 Build Status Build Status Build Status
Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status		 --- Build Status
Build Status
Build Status		 Build Status
Build Status		 --- --- Build Status
XLang --- --- --- Build Status --- --- Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status		 Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@iemejia
Copy link
Member

iemejia commented Mar 18, 2020

bang

@iemejia
Copy link
Member

iemejia commented Mar 18, 2020

Run Java PostCommit

@iemejia
Copy link
Member

iemejia commented Mar 18, 2020

Run Java HadoopFormatIO Performance Test

@iemejia
Copy link
Member

iemejia commented Mar 18, 2020

Run BigQueryIO Streaming Performance Test Java

@iemejia
Copy link
Member

iemejia commented Mar 18, 2020

Run Dataflow ValidatesRunner

@iemejia
Copy link
Member

iemejia commented Mar 18, 2020

Run Spark ValidatesRunner

@iemejia
Copy link
Member

iemejia commented Mar 18, 2020

Run SQL Postcommit

@suztomo
Copy link
Contributor Author

suztomo commented Mar 18, 2020

@iemejia Thank you. Sonatype is unhealthy now. I'll check later today.

10:51:05             > Could not get resource 'https://oss.sonatype.org/content/repositories/staging/io/grpc/grpc-core/maven-metadata.xml'.
10:51:05                > Could not GET 'https://oss.sonatype.org/content/repositories/staging/io/grpc/grpc-core/maven-metadata.xml'.
10:51:05                   > Read timed out

@aaltay
Copy link
Member

aaltay commented Mar 24, 2020

Run Java PostCommit

@aaltay
Copy link
Member

aaltay commented Mar 24, 2020

Run Java HadoopFormatIO Performance Test

@aaltay
Copy link
Member

aaltay commented Mar 24, 2020

Run BigQueryIO Streaming Performance Test Java

@aaltay
Copy link
Member

aaltay commented Mar 24, 2020

Run Dataflow ValidatesRunner

@aaltay
Copy link
Member

aaltay commented Mar 24, 2020

Run Spark ValidatesRunner

@aaltay
Copy link
Member

aaltay commented Mar 24, 2020

Run SQL Postcommit

@aaltay
Copy link
Member

aaltay commented Mar 24, 2020

Run Java PostCommit

1 similar comment
@aaltay
Copy link
Member

aaltay commented Mar 27, 2020

Run Java PostCommit

@aaltay
Copy link
Member

aaltay commented Mar 27, 2020

Run Java HadoopFormatIO Performance Test

@aaltay
Copy link
Member

aaltay commented Mar 27, 2020

Run BigQueryIO Streaming Performance Test Java

@aaltay
Copy link
Member

aaltay commented Mar 27, 2020

Run Dataflow ValidatesRunner

@aaltay
Copy link
Member

aaltay commented Mar 27, 2020

Run Spark ValidatesRunner

@aaltay
Copy link
Member

aaltay commented Mar 27, 2020

Run SQL Postcommit

@kamilwu
Copy link
Contributor

kamilwu commented Mar 30, 2020

Run Dataflow ValidatesRunner

@kamilwu
Copy link
Contributor

kamilwu commented Mar 30, 2020

Run Java PreCommit

@suztomo
Copy link
Contributor Author

suztomo commented Mar 30, 2020 •
edited
Loading

:runners:google-cloud-dataflow-java:validatesRunnerLegacyWorkerTest failed: https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/252/

11:12:20 > Task :runners:google-cloud-dataflow-java:validatesRunnerLegacyWorkerTest
11:13:55 
11:13:55 org.apache.beam.sdk.transforms.ParDoTest$TimerFamilyTests > testTimerWithMultipleTimerFamilyUnbounded FAILED
11:13:55     java.lang.RuntimeException at ParDoTest.java:4770
11:13:55         Caused by: java.lang.IllegalArgumentException at ParDoTest.java:4770
11:29:11 
11:29:11 org.apache.beam.sdk.transforms.ParDoTest$TimerFamilyTests > testTimerFamilyEventTimeUnbounded FAILED
11:29:11     java.lang.RuntimeException at ParDoTest.java:4708
11:29:11         Caused by: java.lang.IllegalArgumentException at ParDoTest.java:4708

It seems other post commit jobs are also failing:
https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/4738/console

08:01:43 > Task :runners:google-cloud-dataflow-java:validatesRunnerLegacyWorkerTest
08:03:14 
08:03:14 org.apache.beam.sdk.transforms.ParDoTest$TimerFamilyTests > testTimerWithMultipleTimerFamilyUnbounded FAILED
08:03:14     java.lang.RuntimeException at ParDoTest.java:4770
08:03:14         Caused by: java.lang.IllegalArgumentException at ParDoTest.java:4770
08:16:38 
08:16:38 org.apache.beam.sdk.transforms.ParDoTest$TimerFamilyTests > testTimerFamilyEventTimeUnbounded FAILED
08:16:38     java.lang.RuntimeException at ParDoTest.java:4708
08:16:38         Caused by: java.lang.IllegalArgumentException at ParDoTest.java:4708
08:33:31 
08:33:31 org.apache.beam.sdk.transforms.FlattenTest > testFlattenPCollectionsThenParDo FAILED
08:33:31     java.lang.RuntimeException at FlattenTest.java:118

Checking which PR started this failure. -> Found #11216 (comment)

chamikaramj
chamikaramj reviewed Apr 2, 2020
View reviewed changes
Copy link
Contributor

@chamikaramj chamikaramj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
google_auth_library_oauth2_http : "com.google.auth:google-auth-library-oauth2-http",
google_cloud_bigquery : "com.google.cloud:google-cloud-bigquery",
google_cloud_bigquery_storage : "com.google.cloud:google-cloud-bigquerystorage",
google_cloud_bigtable_client_core : "com.google.cloud.bigtable:bigtable-client-core:1.13.0",
Copy link
Contributor

@chamikaramj chamikaramj Apr 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have we confirmed that GCS artifacts not included in this BOM do not conflict with artifacts included in the BOM ?

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
force project.library.java.values()
// Filtering versionless coordinates that depend on BOM
def librariesWithVersion = project.library.java.values().findAll { it.split(':').size() > 2 }
force librariesWithVersion
Copy link
Contributor

@chamikaramj chamikaramj Apr 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please force findings here just for the record.

suztomo
suztomo commented Apr 3, 2020
View reviewed changes
Copy link
Contributor Author

@suztomo suztomo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chamikaramj PTAL. Added source code comment on force.

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
force project.library.java.values()
// Filtering versionless coordinates that depend on BOM
def librariesWithVersion = project.library.java.values().findAll { it.split(':').size() > 2 }
force librariesWithVersion
Copy link
Contributor Author

@suztomo suztomo Apr 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added comment with ticket ID BEAM-9542.

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
google_auth_library_oauth2_http : "com.google.auth:google-auth-library-oauth2-http",
google_cloud_bigquery : "com.google.cloud:google-cloud-bigquery",
google_cloud_bigquery_storage : "com.google.cloud:google-cloud-bigquerystorage",
google_cloud_bigtable_client_core : "com.google.cloud.bigtable:bigtable-client-core:1.13.0",
Copy link
Contributor Author

@suztomo suztomo Apr 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, Linkage Checker detects the discrepancy if there's any.

Note that our existing integration tests are not that useful here since they mostly check one GCP service at a time while users may use a combination.

This change is not intended to fix compatibility with users' libraries. It's about how Beam maintains the versions of these Google libraries with ease.

This was referenced Apr 10, 2020
Closed
Closed
@suztomo
Copy link
Contributor Author

suztomo commented Apr 27, 2020

Gradle's updateOfflineRepository task (via https://github.com/mdietrichstein/gradle-offline-dependencies-plugin) does not work with Maven BOMs. The plugin is marked as not maintained. Closing this for now.

@suztomo suztomo closed this Apr 27, 2020
suztomo
suztomo commented Oct 8, 2020
View reviewed changes
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
configuration: (configuration.shadowClosure ? 'shadow' : 'compile'), scope: 'compile')
generateDependenciesFromConfiguration(configuration: 'provided', scope: 'provided')

if (!boms.isEmpty()) {
Copy link
Contributor Author

@suztomo suztomo Oct 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When this Beam module uses a BOM, we need to set dependencyManagent element

@kileys kileys mentioned this pull request Oct 12, 2020
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aaltay aaltay aaltay left review comments

@lukecwik lukecwik lukecwik left review comments

@chamikaramj chamikaramj chamikaramj left review comments

Assignees

No one assigned

Labels

build dataflow examples extensions gcp google-cloud-platform-core io java runners sql

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@suztomo @iemejia @aromanenko-dev @aaltay @kamilwu @lukecwik @chamikaramj