Skip to content

[BEAM-11626] Guava version 25.1-jre for Hadoop/Cassandra and Guava version 30.1 (latest) for the rest #13740

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
suztomo wants to merge 6 commits into from
Closed

[BEAM-11626] Guava version 25.1-jre for Hadoop/Cassandra and Guava version 30.1 (latest) for the rest #13740

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
35f5b05
Latest Guava version
suztomo Jan 13, 2021
851f8ce
Special guava version for hadoop projects
suztomo Jan 14, 2021
8fb92ed
sdks:java:io:cassandra also uses old guava
suztomo Jan 14, 2021
9006814
Declaring guava25projects variable
suztomo Jan 14, 2021
0ec2187
Merge remote-tracking branch 'origin/master' into guava_split
suztomo Jan 22, 2021
0183d98
Breaking change notice
suztomo Jan 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,11 @@

## Breaking Changes

* The Java artifacts "beam-sdks-java-io-kinesis", "beam-sdks-java-io-google-cloud-platform", and
"beam-sdks-java-extensions-sql-zetasql" declare Guava 30.1-jre dependency (It was 25.1-jre in Beam 2.27.0).
This new Guava version may introduce dependency conflicts if your project or dependencies rely
on removed APIs. If affected, ensure to use an appropriate Guava version via `dependencyManagement` in Maven and
`force` in Gradle.
Comment on lines +70 to +74
Copy link
Contributor Author

@suztomo suztomo Jan 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aaltay I added this note for potential impact to Beam users. The potential risk described here is not special to this Guava version. Every dependency upgrade, in general, carries a risk of introducing dependency conflicts if a user relies on removed methods or classes. (Therefore this note might not be needed.)

* X behavior was changed ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).

## Deprecations
Expand Down
10 changes: 9 additions & 1 deletion buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,14 @@ class BeamModulePlugin implements Plugin<Project> {
"""
static AtomicInteger startingExpansionPortNumber = new AtomicInteger(18091)

/** List of paths to the projects that require Guava 25. Hadoop and Cassandra have dependency to
* methods available in the old Guava version (BEAM-11626) */
static List<String> guava25Projects = [
":sdks:java:io:cassandra",
":sdks:java:io:hadoop-format",
":sdks:java:io:hadoop-file-system",
]

/** A class defining the set of configurable properties accepted by applyJavaNature. */
class JavaNatureConfiguration {
/** Controls whether the spotbugs plugin is enabled and configured. */
Expand Down Expand Up @@ -436,7 +444,7 @@ class BeamModulePlugin implements Plugin<Project> {
def google_oauth_clients_version = "1.31.0"
// Try to keep grpc_version consistent with gRPC version in google_cloud_platform_libraries_bom
def grpc_version = "1.32.2"
def guava_version = "25.1-jre"
def guava_version = guava25Projects.contains(project.path) ? "25.1-jre" : "30.1-jre"
Copy link
Member

@aaltay aaltay Jan 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would not this be problematic, causing Beam to depend on 2 different versions? Which version, users of Beam will be depending if they need to use Beam with one of these 3 projects?

Copy link
Contributor Author

@suztomo suztomo Jan 15, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which version, users of Beam will be depending if they need to use Beam with one of these 3 projects?

There's no impact to the Beam Cassandra and Hadoop artifacts. The Maven artifact org.apache.beam:beam-sdks-java-io-hadoop-format:2.27.0, org.apache.beam:beam-sdks-java-io-cassandra:2.27.0, or org.apache.beam:beam-sdks-java-io-hadoop-file-system:2.27.0 does not declare Guava dependency.

However, if Beam Cassandra / Hadoop users use Beam with beam-sdks-java-io-kinesis, beam-sdks-java-io-google-cloud-platform, or beam-sdks-java-extensions-sql-zetasql (they declare Guava dependency), then the users need to pin Guava version to 25.1-jre. They can use <dependencyManagement> for Maven and force for Gradle.

If the Beam users don't depend on any of beam-sdks-java-io-kinesis, beam-sdks-java-io-google-cloud-platform, or beam-sdks-java-extensions-sql-zetasql, then this change does not have any effect to them.

Copy link
Member

@aaltay aaltay Jan 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. I think this would be an undocumented hurdle for the impacted users. I am not sure what is the best course of action. Hopefully @kennknowles would have a recommendation.

Copy link
Contributor Author

@suztomo suztomo Jan 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think I should document that condition ("if Beam Cassandra / Hadoop users use Beam with beam-sdks-java-io-kinesis, ...") somewhere.

Copy link

@ibzib ibzib Jan 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We always treat library.java as a global constant. In all existing cases where a project requires a library version that deviates from library.java, we don't use library.java and instead hard-code that dependency in the project's build.gradle.

IMO making library.java conditional on the project being compiled defeats the purpose of declaring a common version in the first place.

Copy link
Contributor Author

@suztomo suztomo Jan 25, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In all existing cases where a project requires a library version that deviates from library.java, we don't use library.java and instead hard-code that dependency in the project's build.gradle.

That's great information. Let me try that. I see hadoop-common does that with force. Thanks.

Memo for myself in hadoop-common:

hadoopVersions.each {kv ->
  configurations."hadoopVersion$kv.key" {
    resolutionStrategy {
      force "org.apache.hadoop:hadoop-client:$kv.value"
      force "org.apache.hadoop:hadoop-common:$kv.value"
      force "org.apache.hadoop:hadoop-mapreduce-client-core:$kv.value"
    }
  }
}

def hadoop_version = "2.10.1"
def hamcrest_version = "2.1"
def influxdb_version = "2.19"
Expand Down