-
Notifications
You must be signed in to change notification settings - Fork 4k
ARROW-16996: [Java] Configure Netty/GRPC/Protobuf base on BOM configuration + upgrade of dependencies by CVE #13544
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
davisusanibar
commented
Jul 8, 2022
- Configure Netty/GRPC/Protobuf base on BOM Bill Of Material configuration to dependencies versions be added by configuration (Bill Of Materials (BOM) pom netty/netty#5994).
- Upgrade Netty/GRPC/Protobuf dependencies. Netty CVE
…Netty/GRPC/Protobuf dependencies
lidavidm
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
|
Hmm, something about the build is off though. |
| <dependency> | ||
| <groupId>io.netty</groupId> | ||
| <artifactId>netty-transport-native-unix-common</artifactId> | ||
| <version>${dep.netty.version}</version> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems this is required, at least judging by the build?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From some searching, it seems dependencyManagement may not get inherited by profiles, so we'll need to explicitly reference the version here still.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What will be the impact if we only use io.netty:netty-transport-native-unix-common:4.1.78.Final instead of io.netty:netty-transport-native-unix-common:osx-x86_64:4.1.78.Final? Is there some reason to why we use that?
I am building and testing code without -osx / -linux and it finished without problems.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need the platform specific code since this is a JNI library.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That said our tests might not actually hit any of the platform-specific paths
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That said our tests might not actually hit any of the platform-specific paths
Does it mean that we still need the platform-specific JNI library for actual usages? Or, do we ever need no the JNI libraries for both test and actual usages?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We currently do not need the Netty JNI libraries for tests, because we aren't properly testing those paths (ARROW-17024). For actual use, they are only required if the application is using domain sockets, or wants to manually configure Netty to use epoll/kqueue. The application can always explicitly add the necessary dependencies.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. I will remove this code to download the Netty JNI binaries from the build script for s390x.
|
If we're going to remove the profiles please update the docs too. |
|
Is github-actions crossbow submit java-jars run needed? How do you evaluate if this is needed? |
|
It's usually fine unless we're changing something about how the JARs are built |
Added |
|
I just noticed. Why are all tests skipped? |
|
Never mind, they get run at the end. However: The test that exercises the Netty-with-native-utils path is getting skipped |
|
That said they're getting skipped on HEAD too so I'll file a ticket to investigate that. |
|
@kiszk Could you upload protoc-gen-grpc-java 1.47.0 for s390x to https://apache.jfrog.io/ui/native/arrow/protoc-gen-grpc-java-binary/ to fix a Travis CI job failure? e.g.: https://app.travis-ci.com/github/apache/arrow/jobs/576822591#L2933 |
|
I will do it this weekend. |
|
When I prepare the required binaries at my end, the build process on s390x goes forward. But, I got the following error while the build on x86_64 did not fail. I am investigating why this error occurs. |
|
Oh wait. I read it backwards, sorry. Hmm, it looks like something is generating gRPC/Protobuf code inside flight-sql? Or maybe we need to make those dependencies platform-dependent. |
|
When I execute only analyze-only, an error does not occur... |
|
Sorry for my bad. After executing |
|
Submitted a PR #13632 |
|
Now, the test passed. |
…ration + upgrade of dependencies by CVE (apache#13544) - Configure Netty/GRPC/Protobuf base on BOM Bill Of Material configuration to dependencies versions be added by configuration (netty/netty#5994). - Upgrade Netty/GRPC/Protobuf dependencies. Netty [CVE](GHSA-269q-hmxg-m83q) Authored-by: david dali susanibar arce <davi.sarces@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com>
… configuration + upgrade of dependencies by CVE (apache#13544)" This reverts commit 9b18459.
* DX-86156: Cherry pick a4a3d3f for apacheGH-39265 * Update README_DREMIO.md * ARROW-16996: [Java] Configure Netty/GRPC/Protobuf base on BOM configuration + upgrade of dependencies by CVE (apache#13544) - Configure Netty/GRPC/Protobuf base on BOM Bill Of Material configuration to dependencies versions be added by configuration (netty/netty#5994). - Upgrade Netty/GRPC/Protobuf dependencies. Netty [CVE](GHSA-269q-hmxg-m83q) Authored-by: david dali susanibar arce <davi.sarces@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com> * Revert "ARROW-16996: [Java] Configure Netty/GRPC/Protobuf base on BOM configuration + upgrade of dependencies by CVE (apache#13544)" This reverts commit 9b18459. * Update pom.xml Remove unused dependency. * apacheGH-38996: [Java] Update dependencies and plugins for JPMS modules (apache#38994) Update dependencies and plugins to versions that work correctly with JPMS modules. * Update several plugins to use module-enabled versions: * maven-compiler-plugin is updated to 3.11.0 * maven-shade-plugin is updated to 3.2.4 * maven-dependency-plugin is updated to 3.1.2 * CycloneDX is updated to 2.7.10 * Update grpc-java to 1.59 for module support * Update jackson to 2.16.0 as 2.15.1 had corrupt module-info.class files that broke module support. Existing tests only Users may need to update their own dependencies if they have the same ones. * Closes: apache#38996 Authored-by: James Duong <james.duong@improving.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com> * Update README_DREMIO.md --------- Signed-off-by: David Li <li.davidm96@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com> Co-authored-by: panbingkun <pbk1982@gmail.com> Co-authored-by: david dali susanibar arce <davi.sarces@gmail.com> Co-authored-by: James Duong <james.duong@improving.com>
…ration + upgrade of dependencies by CVE (apache#13544) - Configure Netty/GRPC/Protobuf base on BOM Bill Of Material configuration to dependencies versions be added by configuration (netty/netty#5994). - Upgrade Netty/GRPC/Protobuf dependencies. Netty [CVE](GHSA-269q-hmxg-m83q) Authored-by: david dali susanibar arce <davi.sarces@gmail.com> Signed-off-by: David Li <li.davidm96@gmail.com>