routine load submit task stall

[imay]

Describe the bug

When there are too many concurrent routine load task, all submit_task will wait for a lock when the thread pool is full. following is the stack

This is which hold the _lock, and waiting to enter the thread pool

#0  0x00007f5686e39965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000026cf8ec in __gthread_cond_wait (__mutex=<optimized out>, __cond=__cond@entry=0x56b43c0) at /var/local/gcc/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=this@entry=0x56b43c0, __lock=...) at ../../../.././libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x000000000140f2df in blocking_put (val=..., this=0x56b4388) at /var/local/incubator-doris/be/src/util/blocking_queue.hpp:109
#4  offer (func=..., this=0x56b4388) at /var/local/incubator-doris/be/src/util/thread_pool.hpp:72
#5  doris::RoutineLoadTaskExecutor::submit_task (this=0x56b4380, task=...) at /var/local/incubator-doris/be/src/runtime/routine_load/routine_load_task_executor.cpp:96
#6  0x0000000000ec8dd7 in doris::BackendService::submit_routine_load_task (this=0x5793ce0, t_status=..., tasks=...) at /var/local/incubator-doris/be/src/service/backend_service.cpp:236
#7  0x0000000000fa0372 in doris::BackendServiceProcessor::process_submit_routine_load_task (this=0x5790d80, seqid=1, iprot=<optimized out>, oprot=0x6ee4380, callContext=<optimized out>) at /var/local/incubator-doris/gensrc/build/gen_cpp/BackendService.cpp:6307
#8  0x0000000000f7b324 in doris::BackendServiceProcessor::dispatchCall (this=0x5790d80, iprot=0x6d6e580, oprot=0x6ee4380, fname=..., seqid=1, callContext=0x6d63b80) at /var/local/incubator-doris/gensrc/build/gen_cpp/BackendService.cpp:5200
#9  0x0000000000fa0edc in apache::thrift::TDispatchProcessor::process (this=0x5790d80, in=..., out=..., connectionContext=0x6d63b80) at /var/local/thirdparty/installed/include/thrift/TDispatchProcessor.h:121
#10 0x0000000001fca6c1 in apache::thrift::server::TConnectedClient::run (this=0x6f34d20) at src/thrift/server/TConnectedClient.cpp:62
#11 0x0000000001fc5209 in apache::thrift::concurrency::PthreadThread::threadMain (arg=0x579ad00) at src/thrift/concurrency/PosixThreadFactory.cpp:206
#12 0x00007f5686e35dd5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f5687147ead in clone () from /lib64/libc.so.6

This is others request which is waiting for the _lock

#0  0x00007f5686e3c4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f5686e37dcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f5686e37c98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000021674f2 in pthread_mutex_lock_impl (mutex=0x56b4668) at /var/local/incubator-doris/thirdparty/src/brpc-0.9.0/src/bthread/mutex.cpp:545
#4  pthread_mutex_lock (__mutex=0x56b4668) at /var/local/incubator-doris/thirdparty/src/brpc-0.9.0/src/bthread/mutex.cpp:803
#5  0x000000000140c9d9 in __gthread_mutex_lock (__mutex=0x56b4668) at /usr/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748
#6  lock (this=0x56b4668) at /usr/include/c++/7.3.0/bits/std_mutex.h:103
#7  lock (this=0x7f55ad85f2d0) at /usr/include/c++/7.3.0/bits/std_mutex.h:267
#8  unique_lock (__m=..., this=0x7f55ad85f2d0) at /usr/include/c++/7.3.0/bits/std_mutex.h:197
#9  doris::RoutineLoadTaskExecutor::submit_task (this=0x56b4380, task=...) at /var/local/incubator-doris/be/src/runtime/routine_load/routine_load_task_executor.cpp:37
#10 0x0000000000ec8dd7 in doris::BackendService::submit_routine_load_task (this=0x5793ce0, t_status=..., tasks=...) at /var/local/incubator-doris/be/src/service/backend_service.cpp:236
#11 0x0000000000fa0372 in doris::BackendServiceProcessor::process_submit_routine_load_task (this=0x5790d80, seqid=1, iprot=<optimized out>, oprot=0x12527800, callContext=<optimized out>) at /var/local/incubator-doris/gensrc/build/gen_cpp/BackendService.cpp:6307
#12 0x0000000000f7b324 in doris::BackendServiceProcessor::dispatchCall (this=0x5790d80, iprot=0x6ee43c0, oprot=0x12527800, fname=..., seqid=1, callContext=0x12b7e1c0) at /var/local/incubator-doris/gensrc/build/gen_cpp/BackendService.cpp:5200
#13 0x0000000000fa0edc in apache::thrift::TDispatchProcessor::process (this=0x5790d80, in=..., out=..., connectionContext=0x12b7e1c0) at /var/local/thirdparty/installed/include/thrift/TDispatchProcessor.h:121
#14 0x0000000001fca6c1 in apache::thrift::server::TConnectedClient::run (this=0x11f35dc0) at src/thrift/server/TConnectedClient.cpp:62
#15 0x0000000001fc5209 in apache::thrift::concurrency::PthreadThread::threadMain (arg=0x12b8e000) at src/thrift/concurrency/PosixThreadFactory.cpp:206
#16 0x00007f5686e35dd5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f5687147ead in clone () from /lib64/libc.so.6

[imay]

#1257

[morningman]

Reopen if happen again