Performance degradation when I use async requests in LdapConnectionPool in comparison with LdapConnection

[gredwhite]

Hey Neal!

I am investigating performance. So I've added 100_000 entries into my ldap server.
The aim is - to read 10_000 entries from ldap server(I use different size batches for that)

1. Using async api and independent LdapConnections (size is 20)
2. Using async api and LdapConnection from LdapConnectionPool(size is 20)

Let me provide peaces of code:

case 1:

    @ParameterizedTest
    @EnumSource(BatchSize::class)
    fun getByBatchAsyncBasicListener(batchSize: BatchSize) {
        val areWholeBatches = ENTRY_COUNT % batchSize.size == 0
        val batchCount = if (areWholeBatches) {
            ENTRY_COUNT / batchSize.size
        } else {
            ENTRY_COUNT / batchSize.size + 1
        }
        val entries = HashSet<SearchResultEntry>()
      
            IntRange(0, batchCount - 1).map { batchIndex ->
                IntRange(1, batchSize.size)
                    .filter { entryIndex -> entryIndex + (batchIndex * batchSize.size) <= ENTRY_COUNT }
                    .map { entryIndex ->
                        getEntryDn(entryIndex + (batchIndex * batchSize.size))
                    }.let { idList ->
                        getEntriesBatchAsync(baseSearchDn, idList)
                    }
            }.map { (asyncRequestId, listener) ->
                asyncRequestId.get()
                entries.addAll(listener.searchEntries)
            }

    }

    fun getEntriesBatchAsync(
        baseDn: String,
        dns: List<String>,
        vararg attributes: String,
    ): Pair<AsyncRequestID, BasicAsyncSearchResultListener> {
        val resultListener = BasicAsyncSearchResultListener()
        val searchRequest = SearchRequest(
            resultListener,
            baseDn,
            SearchScope.ONE,
            DereferencePolicy.NEVER,
            0,
            0,
            false,
            getIdsFilter(dns),
            *attributes,
        )
        val asyncSearch = getConnection().asyncSearch(searchRequest)
        return Pair(asyncSearch, resultListener)
    }

some details about getConnection():

At this case I create up to 20 connections and return them in random manner.

The result is:

case 2:

    @ParameterizedTest
    @EnumSource(BatchSize::class)
    fun getByBatchAsyncBasicListenerConnectionPoolNew(batchSize: BatchSize) {
        require(ENTRY_COUNT > batchSize.size)
        val areWholeBatches = ENTRY_COUNT % batchSize.size == 0
        val batchCount = if (areWholeBatches) {
            ENTRY_COUNT / batchSize.size
        } else {
            ENTRY_COUNT / batchSize.size + 1
        }
        val entries = HashSet<SearchResultEntry>()
        try {
            IntRange(0, batchCount - 1).map { batchIndex ->
                IntRange(1, batchSize.size)
                    .filter { entryIndex -> entryIndex + (batchIndex * batchSize.size) <= ENTRY_COUNT }
                    .map { entryIndex ->
                        getEntryDn(entryIndex + (batchIndex * batchSize.size))
                    }.let { idsBatch ->
                        getEntriesBatchAsyncPoolNew(baseSearchDn, idsBatch)
                    }
            }.flatMap { (asyncReq, listener) ->
                asyncReq.get()
                listener.searchEntries
            }.let { result -> entries.addAll(result) }       
    }

    fun getEntriesBatchAsyncPoolNew(
        baseDn: String,
        dns: List<String>,
        vararg attributes: String,
    ): Pair<AsyncRequestID, BasicAsyncSearchResultListener> =
        SearchRequest(
            BasicAsyncSearchResultListener(),
            baseDn,
            SearchScope.ONE,
            DereferencePolicy.NEVER,
            0,
            0,
            false,
            getIdsFilter(dns),
            *attributes,
        )
            .let { searchReq ->
                getConnectionPool().processRequestsAsync(listOf(searchReq), -1)
                    .map { asyncRequestID ->
                        Pair(asyncRequestID, searchReq.searchResultListener as BasicAsyncSearchResultListener)
                    }[0]
            }

at this case getConnectionPool() returns the same object of type LDAPConnectionPool which is created like this:

        val startTLSPostConnectProcessor = StartTLSPostConnectProcessor(SSLUtil(TrustAllTrustManager()).createSSLContext())
        val ldapConnectionPool = LDAPConnectionPool(exampleConnection, 1, 20, startTLSPostConnectProcessor)
        ldapConnectionPool.setBindRequest(simpleBindRequest)

The result is:

I tried to make Thread dump during test run and I see a lot of Reader Threads. Maybe it could help ti understand the root cause.

---

As you can see LDAPConnectionPool make performance much worse.

Do I use API in a wrong way ? is there way to improve that ?

[gredwhite]

@dirmgr Did you have a chance to take a look here ?

[dirmgr]

There should not be any reason that an individual connection not associated with a connection pool should yield substantially different performance or behavior than a connection that has been checked out of a pool (and not released back to the pool until all associated processing has completed).

A connection pool is basically just a mechanism for maintaining a set of connections. Any connections that have been created and aren't actively in use will be held in a LinkedBlockingQueue, and when a connection is needed (either for checkout or for internal processing), the one at the head of the queue will be selected. Only connections in the queue will be subject to extra processing like health checking or maximum age processing, so as long as a connection is checked out of the pool, then it should behave exactly the same as a connection that isn't associated with a pool at all.

However, if you check out a connection from a pool, kick off one or more asynchronous operations, and then immediately release the connection back to the pool before the asynchronous operations have completed, then the connection will be immediately available for use in processing other requests. So if you have checked out a connection, issued asynchronous requests, returned the connection back to the pool, and it's then used for processing other requests while the initial requests are also still active, then that could create a lot of contention within that connection. In general, if you're using a pooled connection for processing asynchronous operations, then you shouldn't release the connection back to the pool until those operations have all completed.

The thread you have highlighted in whatever tool you're using is a connection reader thread. That thread is dedicated to constantly reading from a connection so that as soon as the server returns a response, it will be passed to the thread that issued the request so that it can be processed accordingly. Any time that you create a connection that isn't configured to operate in synchronous mode, there will be a listener thread associated with it (connections that operate in synchronous mode don't use the extra thread, but also only attempt to read responses when they're actually waiting on a response, which means that they can't be used for asynchronous processing). And this use of a reader thread is the same regardless of whether it's an individual connection or one created by a connection pool.

[gredwhite]

I see that default value for LDAPConnectionOptions.useSynchronousMode is false and I don't override it anywhere

So I can't explain the difference in performance

[dirmgr]

Is this perhaps related to the other issue in which you were releasing the same connection back to the pool multiple times, so that the same connection might be used concurrently for multiple operations, or so that the connection is getting terminated and requires re-establishing new connections?

[gredwhite]

Is this perhaps related to the other issue in which you were releasing the same connection back to the pool multiple times, so that the same connection might be used concurrently for multiple operations, or so that the connection is getting terminated and requires re-establishing new connections?

No, there is no relation at all. For that investigation I've prepared separated codebase.