GC Counter is not enabled on JDK 11 and JDK 17

[didip]

Affected Version

Druid 0.22.1 and 0.23.0

Description

I got this WARN message on JDK 11 and JDK 17.

2022-06-25T16:41:52,529 WARN [main] org.apache.druid.java.util.metrics.JvmMonitor - Cannot initialize GC counters. If running JDK11 and above, add --add-exports java.base/jdk.internal.perf=ALL-UNNAMED to the JVM arguments to enable GC counters.

But even after I added that, the WARN persist.

So, I printed out the stacktrace:

java.lang.RuntimeException: Cannot perf data for process 61 - java.lang.IllegalAccessError: class org.gridkit.lab.jvm.perfdata.JStatData$PerfIntr (in unnamed module @0x4f209819) cannot access class sun.management.counter.Units (in module java.management) because module java.management does not export sun.management.counter to unnamed module @0x4f209819
        at org.gridkit.lab.jvm.perfdata.JStatData.connect(JStatData.java:49) ~[jvm-attach-api-1.5.jar:?]
        at org.apache.druid.java.util.metrics.JvmMonitor$GcCounters.<init>(JvmMonitor.java:191) ~[druid-core-0.23.0.jar:0.23.0]
        at org.apache.druid.java.util.metrics.JvmMonitor.tryCreateGcCounters(JvmMonitor.java:167) [druid-core-0.23.0.jar:0.23.0]
        at org.apache.druid.java.util.metrics.JvmMonitor.<init>(JvmMonitor.java:80) [druid-core-0.23.0.jar:0.23.0]
        at org.apache.druid.java.util.metrics.JvmMonitor.<init>(JvmMonitor.java:70) [druid-core-0.23.0.jar:0.23.0]
        at org.apache.druid.java.util.metrics.JvmMonitor.<init>(JvmMonitor.java:65) [druid-core-0.23.0.jar:0.23.0]
        at org.apache.druid.server.metrics.MetricsModule.getJvmMonitor(MetricsModule.java:143) [druid-server-0.23.0.jar:0.23.0]
        at org.apache.druid.server.metrics.MetricsModule$$FastClassByGuice$$99ddce1b.invoke(<generated>) [druid-server-0.23.0.jar:0.23.0]
        at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264) [guice-4.1.0.jar:?]
        at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401) [guice-4.1.0.jar:?]
        at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376) [guice-4.1.0.jar:?]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [guice-4.1.0.jar:?]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [guice-4.1.0.jar:?]
        at org.apache.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68) [druid-core-0.23.0.jar:0.23.0]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054) [guice-4.1.0.jar:?]
        at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:102) [druid-server-0.23.0.jar:0.23.0]
        at org.apache.druid.server.metrics.MetricsModule$$FastClassByGuice$$99ddce1b.invoke(<generated>) [druid-server-0.23.0.jar:0.23.0]
        at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264) [guice-4.1.0.jar:?]
        at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401) [guice-4.1.0.jar:?]
        at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376) [guice-4.1.0.jar:?]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [guice-4.1.0.jar:?]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [guice-4.1.0.jar:?]
        at org.apache.druid.guice.LifecycleScope$1.get(LifecycleScope.java:68) [druid-core-0.23.0.jar:0.23.0]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [guice-4.1.0.jar:?]
        at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) [guice-4.1.0.jar:?]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [guice-4.1.0.jar:?]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [guice-4.1.0.jar:?]
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180) [guice-4.1.0.jar:?]
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110) [guice-4.1.0.jar:?]
        at com.google.inject.Guice.createInjector(Guice.java:99) [guice-4.1.0.jar:?]
        at com.google.inject.Guice.createInjector(Guice.java:73) [guice-4.1.0.jar:?]
        at com.google.inject.Guice.createInjector(Guice.java:62) [guice-4.1.0.jar:?]
        at org.apache.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:449) [druid-server-0.23.0.jar:0.23.0]
        at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:93) [druid-services-0.23.0.jar:0.23.0]
        at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:62) [druid-services-0.23.0.jar:0.23.0]
        at org.apache.druid.cli.Main.main(Main.java:114) [druid-services-0.23.0.jar:0.23.0]

It looks like java.management/sun.management=ALL-UNNAMED is needed.

But unfortunately, even after I added it as either --add-exports or --add-opens, the problem persists. Anyone else got other ideas?

[dampcake]

In the Druid 0.18.0 release notes this is called out specifically. Setting:
--add-exports jdk.management/com.sun.management.internal=ALL-UNNAMED
should resolve the issue. See: https://github.com/apache/druid/releases?q=0.18.0&expanded=true#java-11

Edit: after reading it more it looks like it was a different issue that suggested that export. Though I can say I am running with the all the flags mentioned there and am not seeing this error.

[gianm]

I am using JvmMonitor on Java 11 too, and it works OK with these options:

--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED
--add-exports=java.base/jdk.internal.perf=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED

Like @dampcake, I'm not sure which one exactly is making this particular thing work.

I think it'd be good to add them to the bundled jvm.configs. But then we should also develop an ability to select a different jvm.config for different java versions. Or we could also use a single config with -XX:+IgnoreUnrecognizedVMOptions.

[gianm]

Btw, another thing. #12469 is needed for ZGC, at least. It's not the issue you're running into (you are running into some module issue). But it is another issue you may run into if you use a non-G1 collector.

[gianm]

#12481 may be helpful too, although it isn't merged yet.

[FrankChen021]

#12481 has been merged into the master branch. You can have a try if you're interested in it. @didip

[didip]

@FrankChen021 Thanks for the heads up, I will give that a try when I have the chance.