Weblogic exception on agent startup : Class not Found weblogic.security.SSL.jsseadapter.JaSSLContextImpl

[SylvainJuge]

Discovered with 1.28.4 agent + Weblogic 12c 12.2.1.3.0, very likely to impact other versions as well.

Description

The following stack trace is visible in Weblogic standard output when the agent starts

2022-01-20 17:23:11,235 [elastic-apm-server-healthcheck] ERROR co.elastic.apm.agent.util.ExecutorUtils - [Security:090461]Class not Found weblogic.security.SSL.jsseadapter.JaSSLContextImpl
java.lang.IllegalArgumentException: [Security:090461]Class not Found weblogic.security.SSL.jsseadapter.JaSSLContextImpl
        at weblogic.security.utils.SSLSetup.getDelegateClass(SSLSetup.java:146) ~[?:?]
        at weblogic.security.utils.SSLSetup.getSSLDelegateInstance(SSLSetup.java:103) ~[?:?]
        at weblogic.security.utils.SSLContextWrapper.<init>(SSLContextWrapper.java:82) ~[?:?]
        at weblogic.security.utils.SSLContextWrapper.getInstance(SSLContextWrapper.java:68) ~[?:?]
        at weblogic.security.utils.SSLSetup.getSSLContext(SSLSetup.java:368) ~[?:?]
        at weblogic.security.utils.SSLSetup.getSSLContext(SSLSetup.java:360) ~[?:?]
        at weblogic.security.SSL.SSLClientInfo.getSSLSocketFactory(SSLClientInfo.java:111) ~[?:?]
        at weblogic.security.SSL.SSLSocketFactory.setSSLClientInfo(SSLSocketFactory.java:243) ~[?:?]
        at weblogic.security.SSL.SSLSocketFactory.<init>(SSLSocketFactory.java:46) ~[?:?]
        at weblogic.security.SSL.SSLSocketFactory.getInstance(SSLSocketFactory.java:79) ~[?:?]
        at weblogic.net.http.HttpsClient.getInternalSocketFactory(HttpsClient.java:263) ~[?:?]
        at weblogic.net.http.HttpsClient.openServer(HttpsClient.java:272) ~[?:?]
        at weblogic.net.http.HttpsClient.openServer(HttpsClient.java:362) ~[?:?]
        at weblogic.net.http.HttpsClient.New(HttpsClient.java:565) ~[?:?]
        at weblogic.net.http.HttpsClient.New(HttpsClient.java:546) ~[?:?]
        at weblogic.net.http.HttpsURLConnection.connect(HttpsURLConnection.java:235) ~[?:?]
        at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:685) ~[?:?]
        at weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:42) ~[?:?]
        at co.elastic.apm.agent.report.HttpUtils.consumeAndClose(HttpUtils.java:62) ~[elastic-apm-agent-1.28.4.jar:?]
        at co.elastic.apm.agent.report.ApmServerClient.executeForAllUrls(ApmServerClient.java:280) ~[elastic-apm-agent-1.28.4.jar:?]
        at co.elastic.apm.agent.report.ApmServerHealthChecker.call(ApmServerHealthChecker.java:65) ~[elastic-apm-agent-1.28.4.jar:?]
        at co.elastic.apm.agent.report.ApmServerHealthChecker.call(ApmServerHealthChecker.java:42) ~[elastic-apm-agent-1.28.4.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_311]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_311]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_311]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_311]
Caused by: java.lang.ClassNotFoundException: weblogic/security/SSL/jsseadapter/JaSSLContextImpl
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_311]
        at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_311]
        at weblogic.security.utils.SSLSetup.loadDelegateClass(SSLSetup.java:161) ~[?:?]
        at weblogic.security.utils.SSLSetup.getDelegateClass(SSLSetup.java:133) ~[?:?]
        ... 25 more
2022-01-20 17:23:12,102 [elastic-apm-agent-initialization] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state

Impact

Low, does not prevent the agent to send data to APM server
Could be quite high, as it works "by accident" of the agent being fast on JVM startup, and it can prevent the agent to communicate with APM server. Would also happen systematically when using runtime attach.

[SylvainJuge]

As it was really easy to reproduce with a local WL setup, I spend some time on fixing this.
When this happens, it prevents the agent from communicating with APM server, thus the impact could be quite high.

It does not happen when the agent starts without delay because the agent is fast enough to create the URL object before the WL server overrides the default handler.

The issue is generic and happens whenever the default URL protocol handlers are overridden in the JVM.

[SylvainJuge]

Unfortunately, the fix provided by #2429 proved to not be reliable and had to be reverted.

• on Java 11, it adds an extra JVM warning due to the (ab)use of Class.forName
• on Java 17, it is not possible anymore to use the Class.forName.

An alternative approach is required here:

• instrumentation of the URL constructor to be able to call the Class.forName from a place where it's OK.
• agent might have created URL objects before the instrumentation is available.
• maybe instrument the URLStreamHandler implementations themselves to gracefully delegate to default implementations when another implementation is registered. This might not be possible if the implementation class could not be loaded from the bootstrap/agent classpath.

Also, the chosen implementation should try to avoid relying on implementation details as it is required to work on JDK from 7 to 17.

[SylvainJuge]

A few more details on why fixing this might be tedious:

• before any other URLStreamHandler is registered, the URL instances created will use the default implementation, so when the agent is configured through -javaagent it might have time to create the URL object early with the default. However if configuration is changed afterwards, or if the agent startup is delayed, the agent is unable to communicate properly.
• other implementations of URLStreamHandler can be registered through multiple ways: through ServiceLoader SPI, through system properties and programatically through URL.setURLStreamHandlerFactory (javadoc) which can be called only once for the whole JVM process lifetime
• once an instance of URLStreamHandler has been resolved for a given protocol, it is stored in a map, so the factory is called only once per protocol.
• using an alternate protocol handler for the URL objects created by the agent could be an option, for example agenthttp and agenthttps. Doing so should be done without calling URL.setURLStreamHandlerFactory as it might not be called anymore afterwards, so maybe the URL constructor is still the right place to do that, also we should still find a way to delegate to default URLStreamHandler implementations with the "right" protocol afterwards.
• the SecurityManager might prevent calling URL constructor with a non-default handler, so maybe intercepting the end of the constructor instead of the beginning could be relevant. Also, checking a thread-local state might be used instead of a marker implementation of URLStreamHandler passed through arguments, and intercepting the end of the constructor execution to alter its returned value would allow to avoid the SecurityManager check.

[SylvainJuge]

Another alternative suggested by @eyalkoren : try to always create the URL object from the application classloader instead of the agent CL: this CL is expected to have access to the Weblogic classes and dependencies.