Skip to content

Conversation

@lhotari
Copy link
Member

@lhotari lhotari commented Jun 14, 2024

Fixes #22909

Motivation

Netty's optimized direct byte buffer support is needed for high performance. Currently these debug messages will get logged when Netty library gets initialized:

  • "direct buffer constructor: unavailable: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module"
  • "jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$7 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module"
  • "java.nio.DirectByteBuffer.(long, {int,long}): unavailable {}"

Another detail is that Pulsar uses com.scurrilous.circe.checksum.Crc32cIntChecksum class for checksum calculation. For optimized checksum calculation Pulsar attempts to load libcirce-checksum native library. When that isn't available, com.scurrilous.circe.checksum.Java9IntHash is used. That only works when "--add-opens java.base/java.util.zip=ALL-UNNAMED" is passed on the command line.

Modifications

Pass these parameters in the Java runtime command line:

 -Dio.netty.tryReflectionSetAccessible=true
 -Dorg.apache.pulsar.shade.io.netty.tryReflectionSetAccessible=true
 -Dio.grpc.netty.shaded.io.netty.tryReflectionSetAccessible=true
 --add-opens java.base/java.nio=ALL-UNNAMED
 --add-opens java.base/jdk.internal.misc=ALL-UNNAMED
 --add-opens java.base/java.util.zip=ALL-UNNAMED

remove --add-opens java.base/sun.net=ALL-UNNAMED added in #16423 since it's no longer needed after #22329

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Copy link
Contributor

@cbornet cbornet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@lhotari lhotari merged commit f3d4d5a into apache:master Jun 17, 2024
lhotari added a commit that referenced this pull request Jun 19, 2024
lhotari added a commit that referenced this pull request Jun 19, 2024
lhotari added a commit that referenced this pull request Jun 19, 2024
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 21, 2024
…r Function runtimes (apache#22910)

(cherry picked from commit f3d4d5a)
(cherry picked from commit 20de952)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 24, 2024
…r Function runtimes (apache#22910)

(cherry picked from commit f3d4d5a)
(cherry picked from commit 20de952)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 25, 2024
…r Function runtimes (apache#22910)

(cherry picked from commit f3d4d5a)
(cherry picked from commit 20de952)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 1, 2024
…r Function runtimes (apache#22910)

(cherry picked from commit f3d4d5a)
(cherry picked from commit 20de952)
hanmz pushed a commit to hanmz/pulsar that referenced this pull request Feb 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Pulsar Functions Runtime doesn't properly enable direct byte buffer access for Netty on Java 17+

3 participants