mem-leak in historical 0.12

[sixtus]

it's been haunting me since the 0.12 deploy, because it's very slow to grow. It looks triggered by new segments. Just my preliminary findings.

I just restarted the cluster with druid.query.groupBy.numParallelCombineThreads=1 and I can finally seen GC in mem usage again.

[sixtus]

it's very nasty because the nodes don't oom. they get stuck in limbo land and make queries stall.

[gianm]

Can you tell what kinds of objects are being retained that shouldn't be? Maybe from looking through a heap histogram on an idle node for things that "shouldn't" be there.

[gianm]

Can you tell if you had any exceptions that occurred in the ParallelCombine.combine method (or callees of it)? It looks like there is a resource leak that occurs if that happens.

[gianm]

#5630 fixes that particular leak. I'm not sure if it's the one you saw or not.

[sixtus]

one of the nodes just doubled the ram usage after setting it to 1.

Apr 11 23:10:00 alexandria2 druid-historical[22230]: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException: Cancelled
Apr 11 23:10:00 alexandria2 druid-historical[22230]: at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:177) ~[druid-services-0.12.0-selfcontained.jar:0.12.0]

within 20min the ram usage doubled afterwards.

[jihoonson]

This is probably because ParallelCombiner is using processing buffers instead of merge buffers. I'll make a patch shortly.

[sixtus]

I guess I have to update the ticket. Setting parallelCombiner to one did not solve the mem leak. More when I know more.

[gianm]

@sixtus It might be helpful to analyze a heap dump from a time when the historicals are misbehaving?

[sixtus]

status update: we need to restart our historicals every ~12h atm. Heap dumps point at http://www.evanjones.ca/java-bytebuffer-leak.html - we are currently testing the change on one server, looks very promising.

[gianm]

@sixtus Do you mean you added -Djdk.nio.maxCachedBufferSize=262144?

What JVM brand / version are you running?

[sixtus]

it looks like it was just the cruft of 6 years old recipes. It went away after tuning jvm and druid options.