[BUG] java.lang.OutOfMemoryError: unable to create new native thread

[zz-jason]

Describe the bug

Got java.lang.OutOfMemoryError: unable to create new native thread when creating numerous SmartRawKVClient

What did you do

Create numous SmartRawKVClient like the following:

@Test
public void testMultiClients() throws InterruptedException {
  for (int i = 0; i < 10240; i++) {
    client = session.createSmartRawClient();
  }
}

What do you expect

no errors

What happens instead

testMultiClients(org.tikv.raw.SmartRawKVClientTest)  Time elapsed: 0.998 sec  <<< ERROR!
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:717)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603)
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
        at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:573)
        at org.tikv.service.failsafe.CircuitBreakerMetricsImpl.<init>(CircuitBreakerMetricsImpl.java:54)
        at org.tikv.service.failsafe.CircuitBreakerImpl.<init>(CircuitBreakerImpl.java:73)
        at org.tikv.service.failsafe.CircuitBreakerImpl.<init>(CircuitBreakerImpl.java:51)
        at org.tikv.raw.SmartRawKVClient.<init>(SmartRawKVClient.java:70)
        at org.tikv.common.TiSession.createSmartRawClient(TiSession.java:171)
        at org.tikv.raw.SmartRawKVClientTest.testMultiClients(SmartRawKVClientTest.java:79)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
        at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
        at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

Java Client and TiDB/TiKV version info

• Java Client: v3.1.10
• TiKV: v5.0.4
• PD: v5.0.4

[zz-jason]

RCA:

When creating a SmartRawKVClient, a CircuitBreaker is created, and a background thread to record the traffic statistics is created as well. So creating 10241 SmartRawKVClients, there will be 10241 background thread being created as well. It reaches the upper limit that the OS allows JVM to create, so the error appears.

References:

• "java.lang.OutOfMemoryError : unable to create new native Thread"
• How many threads can a Java VM support?