Skip to content

ARROW-16996: [Java] Configure Netty/GRPC/Protobuf base on BOM configuration + upgrade of dependencies by CVE #13544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lidavidm merged 3 commits into from
Jul 8, 2022
Merged

ARROW-16996: [Java] Configure Netty/GRPC/Protobuf base on BOM configuration + upgrade of dependencies by CVE #13544

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
28774c6
Configure Netty/GRPC/Protobuf base on BOM configuration + upgrade of …
davisusanibar Jul 8, 2022
a8ac2b3
Deleting mac/linux netty-native profiles
davisusanibar Jul 8, 2022
338a8ab
Deleting mac/linux netty-native profiles from documentation
davisusanibar Jul 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions docs/source/developers/java/building.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,11 +186,6 @@ Arrow repository, and update the following settings:
Settings > Build, Execution, Deployment > Compiler > Java Compiler and disable
"Use '--release' option for cross-compilation (Java 9 and later)". Otherwise
you will get an error like "package sun.misc does not exist".
* You may need to disable the ``linux-netty-native`` or ``mac-netty-native``
profile in the Maven tool window if you get an error like the following::

Unresolved dependency: 'io.netty:netty-transport-native-unix-common:jar:4.1.72.Final'

* If using IntelliJ's Maven integration to build, you may need to change
``<fork>`` to ``false`` in the pom.xml files due to an `IntelliJ bug
<https://youtrack.jetbrains.com/issue/IDEA-278903>`__.
Expand Down
49 changes: 3 additions & 46 deletions java/flight/flight-core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,27 +53,22 @@
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty</artifactId>
<version>${dep.grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-core</artifactId>
<version>${dep.grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-context</artifactId>
<version>${dep.grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
<version>${dep.grpc.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>${dep.netty-tcnative.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
Expand All @@ -82,12 +77,10 @@
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-handler</artifactId>
<version>${dep.netty.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport</artifactId>
<version>${dep.netty.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
Expand All @@ -96,17 +89,14 @@
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
<version>${dep.grpc.version}</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${dep.protobuf.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-api</artifactId>
<version>${dep.grpc.version}</version>
</dependency>

<dependency>
Expand Down Expand Up @@ -224,10 +214,10 @@
<artifactId>protobuf-maven-plugin</artifactId>
<version>0.6.1</version>
<configuration>
<protocArtifact>com.google.protobuf:protoc:${dep.protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
<protocArtifact>com.google.protobuf:protoc:${dep.protobuf-bom.version}:exe:${os.detected.classifier}</protocArtifact>
<clearOutputDirectory>false</clearOutputDirectory>
<pluginId>grpc-java</pluginId>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:${dep.grpc.version}:exe:${os.detected.classifier}</pluginArtifact>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:${dep.grpc-bom.version}:exe:${os.detected.classifier}</pluginArtifact>
</configuration>
<executions>
<execution>
Expand Down Expand Up @@ -257,6 +247,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>3.3.0</version>
<executions>
<execution>
<id>analyze</id>
Expand Down Expand Up @@ -311,38 +302,4 @@
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>linux-netty-native</id>
<activation>
<os>
<family>linux</family>
</os>
</activation>
<dependencies>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-unix-common</artifactId>
<version>${dep.netty.version}</version>
Copy link
Member

@lidavidm lidavidm Jul 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems this is required, at least judging by the build?

Copy link
Member

@lidavidm lidavidm Jul 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From some searching, it seems dependencyManagement may not get inherited by profiles, so we'll need to explicitly reference the version here still.

Copy link
Contributor Author

@davisusanibar davisusanibar Jul 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What will be the impact if we only use io.netty:netty-transport-native-unix-common:4.1.78.Final instead of io.netty:netty-transport-native-unix-common:osx-x86_64:4.1.78.Final? Is there some reason to why we use that?

I am building and testing code without -osx / -linux and it finished without problems.

Copy link
Member

@lidavidm lidavidm Jul 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need the platform specific code since this is a JNI library.

Copy link
Member

@lidavidm lidavidm Jul 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That said our tests might not actually hit any of the platform-specific paths

Copy link
Member

@kiszk kiszk Jul 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That said our tests might not actually hit any of the platform-specific paths

Does it mean that we still need the platform-specific JNI library for actual usages? Or, do we ever need no the JNI libraries for both test and actual usages?

Copy link
Member

@lidavidm lidavidm Jul 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We currently do not need the Netty JNI libraries for tests, because we aren't properly testing those paths (ARROW-17024). For actual use, they are only required if the application is using domain sockets, or wants to manually configure Netty to use epoll/kqueue. The application can always explicitly add the necessary dependencies.

Copy link
Member

@kiszk kiszk Jul 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I will remove this code to download the Netty JNI binaries from the build script for s390x.

<classifier>${os.detected.name}-${os.detected.arch}</classifier>
</dependency>
</dependencies>
</profile>
<profile>
<id>mac-netty-native</id>
<activation>
<os>
<family>mac</family>
</os>
</activation>
<dependencies>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-unix-common</artifactId>
<version>${dep.netty.version}</version>
<classifier>${os.detected.name}-${os.detected.arch}</classifier>
</dependency>
</dependencies>
</profile>
</profiles>
</project>
9 changes: 2 additions & 7 deletions java/flight/flight-grpc/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,10 @@
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-core</artifactId>
<version>${dep.grpc.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
<version>${dep.grpc.version}</version>
</dependency>
<dependency>
<groupId>org.apache.arrow</groupId>
Expand All @@ -72,7 +70,6 @@
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
<version>${dep.grpc.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
Expand All @@ -81,12 +78,10 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${dep.protobuf.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-api</artifactId>
<version>${dep.grpc.version}</version>
</dependency>
</dependencies>

Expand All @@ -105,10 +100,10 @@
<artifactId>protobuf-maven-plugin</artifactId>
<version>0.6.1</version>
<configuration>
<protocArtifact>com.google.protobuf:protoc:${dep.protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
<protocArtifact>com.google.protobuf:protoc:${dep.protobuf-bom.version}:exe:${os.detected.classifier}</protocArtifact>
<clearOutputDirectory>false</clearOutputDirectory>
<pluginId>grpc-java</pluginId>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:${dep.grpc.version}:exe:${os.detected.classifier}</pluginArtifact>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:${dep.grpc-bom.version}:exe:${os.detected.classifier}</pluginArtifact>
</configuration>
<executions>
<execution>
Expand Down
1 change: 0 additions & 1 deletion java/flight/flight-integration-tests/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,6 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${dep.protobuf.version}</version>
</dependency>
<dependency>
<groupId>commons-cli</groupId>
Expand Down
4 changes: 0 additions & 4 deletions java/flight/flight-sql/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
<version>${dep.grpc.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
Expand All @@ -69,17 +68,14 @@
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
<version>${dep.grpc.version}</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${dep.protobuf.version}</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-api</artifactId>
<version>${dep.grpc.version}</version>
</dependency>
<dependency>
<groupId>org.apache.arrow</groupId>
Expand Down
10 changes: 2 additions & 8 deletions java/flight/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,6 @@

<packaging>pom</packaging>

<properties>
<dep.grpc.version>1.44.1</dep.grpc.version>
<dep.netty-tcnative.version>2.0.46.Final</dep.netty-tcnative.version>
<dep.protobuf.version>3.19.4</dep.protobuf.version>
</properties>

<modules>
<module>flight-core</module>
<module>flight-grpc</module>
Expand All @@ -46,10 +40,10 @@
<version>0.6.1</version>
<configuration>
<protocArtifact>
com.google.protobuf:protoc:${dep.protobuf.version}:exe:${os.detected.classifier}
com.google.protobuf:protoc:${dep.protobuf-bom.version}:exe:${os.detected.classifier}
</protocArtifact>
<pluginId>grpc-java</pluginId>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:${dep.grpc.version}:exe:${os.detected.classifier}
<pluginArtifact>io.grpc:protoc-gen-grpc-java:${dep.grpc-bom.version}:exe:${os.detected.classifier}
</pluginArtifact>
</configuration>
</plugin>
Expand Down
40 changes: 24 additions & 16 deletions java/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,9 @@
<dep.junit.jupiter.version>5.4.0</dep.junit.jupiter.version>
<dep.slf4j.version>1.7.25</dep.slf4j.version>
<dep.guava.version>30.1.1-jre</dep.guava.version>
<dep.netty.version>4.1.72.Final</dep.netty.version>
<dep.netty-bom.version>4.1.78.Final</dep.netty-bom.version>
<dep.grpc-bom.version>1.47.0</dep.grpc-bom.version>
<dep.protobuf-bom.version>3.21.2</dep.protobuf-bom.version>
<dep.jackson-bom.version>2.13.2.20220328</dep.jackson-bom.version>
<dep.hadoop.version>2.7.1</dep.hadoop.version>
<dep.fbs.version>1.12.0</dep.fbs.version>
Expand Down Expand Up @@ -524,21 +526,6 @@
<artifactId>guava</artifactId>
<version>${dep.guava.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-handler</artifactId>
<version>${dep.netty.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-buffer</artifactId>
<version>${dep.netty.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-common</artifactId>
<version>${dep.netty.version}</version>
</dependency>
<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
Expand Down Expand Up @@ -578,6 +565,27 @@
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-bom</artifactId>
<version>${dep.netty-bom.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-bom</artifactId>
<version>${dep.grpc-bom.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-bom</artifactId>
<version>${dep.protobuf-bom.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

Expand Down