[Java][Flight] Flight SQL tests are flaky

[laurentgo]

Describe the bug, including details regarding any error messages, version, and platform.

Several test failures in flight-sql module have been observed in multiple job executions:

• https://github.com/apache/arrow/actions/runs/9185953424/job/25260768750
• https://github.com/apache/arrow/actions/runs/9185714602/job/25260156899?pr=41772

The reported issue is

Error:  Errors: 
Error:    TestFlightSqlStreams.tearDown:224 » IllegalState Memory was leaked by query. Memory leaked: (250384)
Allocator(ROOT) 0/250384/250896/2147483647 (res/actual/peak/limit)

Note that there are also multiple messages about unclosed ManagedChannels in flight-core module:

• org.apache.arrow.flight.auth2.TestBasicAuth2
• org.apache.arrow.flight.core/org.apache.arrow.flight.TestFlightGrpcUtils.testMultipleGrpcServices
• org.apache.arrow.flight.core/org.apache.arrow.flight.TestDoExchange.setUp
• org.apache.arrow.flight.core/org.apache.arrow.flight.TestServerOptions.addHealthCheckService

May 22, 2024 5:50:41 AM io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Previous channel ManagedChannelImpl{logId=505, target=directaddress:///localhost/127.0.0.1:5555} was garbage collected without being shut down! ~*~*~*
    Make sure to call shutdown()/shutdownNow()
java.lang.RuntimeException: ManagedChannel allocation site
	at io.grpc.internal@1.63.0/io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:102)
	at io.grpc.internal@1.63.0/io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:60)
	at io.grpc.internal@1.63.0/io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:51)
	at io.grpc.internal@1.63.0/io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:672)
	at io.grpc@1.63.0/io.grpc.ForwardingChannelBuilder2.build(ForwardingChannelBuilder2.java:260)
	at org.apache.arrow.flight.core/org.apache.arrow.flight.TestServerOptions.addHealthCheckService(TestServerOptions.java:191)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)

but those seems to only cause warnings, not errors

Component(s)

FlightRPC, Java

[vibhatha]

cc @lidavidm

[laurentgo]

I wonder if the issue is somehow similar to the one hinted at https://github.com/apache/arrow/blob/9ba9253e8527a7f3e2c6e47e631e278b8ca84e53/java/flight/flight-core/src/test/java/org/apache/arrow/flight/TestDoExchange.java#L407 where somehow data is not fully reclaimed? (in my local test for TestDoExchange it seems the leak is related to allocations triggered by Producer)

[lidavidm]

Can anyone reproduce it locally?

[laurentgo]

I cannot for TestFlightSqlStreams

[lidavidm]

I think we need to do two things:

• tearDown needs to run after each test to narrow down the scope of the leaked memory.
• We need to properly add debug information to print out the leaked buffer's retain/release history and where it was allocated. We optionally track this info and it should be enabled in CI and the error message should be enhanced.

@vibhatha

[vibhatha]

I will take a look. But need sometime, probably earliest next week.

[vibhatha]

We optionally track this info and it should be enabled in CI and the error message should be enhanced.

@lidavidm could I get a pointer in how to enable this in CIs?

[lidavidm]

You'll have to look. We neutered it after performance complaints.

[laurentgo]

Another occurence of a build failure: https://github.com/apache/arrow/actions/runs/9210096683/job/25336267356?pr=41800

[vibhatha]

Hard thing is replicating this locally. I tried a few times yesterday, no luck. Need to enable what @lidavidm mentioned in the CIs. I need to look into that.