SSL connections causing high CPU usage during high QPS

[NikhilCollooru]

We want to support high QPS ssl connections. We depend on Jetty to handle incoming connections. By default, Jetty uses the JSSE provider from the JVM for SSL. Since this is in-efficient, we recently made changes to use Conscypt in Jetty.

We still continue to see high CPU during high QPS ssl connections. When we took a flamechart, we observed that a significant CPU is being spent on __lll_lock_wait_private and __lll_unlock_wake_private. These locks seemed to be acquired/released from org/conscrypt/NativeCrypto.ENGINE_SSL_read_direct or org/conscrypt/NativeCrypto.d2i_X509_bio .

Were these locks introduced by PR by @flooey ? Can this be the cause ?

The flamechart svg file can be found here: (open with chrome)
https://drive.google.com/file/d/1ZxcKQhld2C1mV64pvdNCLEP3copI7J3b/view?usp=sharing

The code for our httpserver can be found here:
https://github.com/prestodb/airlift/blob/master/http-server/src/main/java/com/facebook/airlift/http/server/HttpServer.java

To use conscrypt, I changed
from:
slContextFactory sslContextFactory = new SslContextFactory();
at here

to:

    static {
        Security.addProvider(new OpenSSLProvider());
    }

SslContextFactory.Server sslContextFactory = new SslContextFactory.Server();
sslContextFactory.addExcludeProtocols("TLSv1.3");
sslContextFactory.setProvider("Conscrypt");

Can someone help us understand the flamechart and figure out the reason for high CPU ?