glibc malloc deadlock #1311
Description
我们在业务中使用了brpc，现象是一些brpc bthread hung在了glibc的malloc和init_free。导致brpc所有bthread都hung住（业务原因在等其他bthread的资源）
不确定是否和brpc有关系。
Versions (各种版本)
OS: CentOS Linux release 8.1.1911 (Core) 4.18.0-147.el8.x86_64
Compiler: gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)
-DCMAKE_BUILD_TYPE="Debug" -DCMAKE_C_FLAGS="-O0 -g3 -gdwarf-4" -DCMAKE_CXX_FLAGS="-O0 -g3 -gdwarf-4"
brpc: brpc-0.9.8_rc01-1.el8.x86_64（tag 0.9.8-rc01源码build的rpm包）
protobuf: protobuf-3.5.0-7.el8.x86_64
glibc: glibc-2.28-101.el8.x86_64
Additional context/screenshots (更多上下文/截图)
Thread 83 (Thread 0x7f2d597fa700 (LWP 3633599)):
#0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:63
#1 0x00007f2e6ef01e10 in _int_free (av=0x7f2d90000020, p=0x7f2d90204ad0, have_lock=) at malloc.c:4314
#2 0x0000559de3562314 in std::default_deletebrpc::Controller::operator() (this=0x7f2e306f79a0, __ptr=0x7f2d90204ae0) at /usr/include/c++/8/bits/unique_ptr.h:81
#3 0x0000559de3561daf in std::unique_ptr<brpc::Controller, std::default_deletebrpc::Controller >::~unique_ptr (this=0x7f2e306f79a0, __in_chrg=) at /usr/include/c++/8/bits/unique_ptr.h:269
Thread 79 (Thread 0x7f2d5b7fe700 (LWP 3633595)):
#0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:63
#1 0x00007f2e6ef042d3 in __malloc_fork_lock_parent () at arena.c:156
#2 0x00007f2e6ef473ea in __libc_fork () at ../sysdeps/nptl/fork.c:73
#3 0x00007f2e6eef1984 in _IO_new_proc_open (fp=fp@entry=0x7f2d4c06ce40, command=command@entry=0x7f2e7e1760ae "uname -ap", mode=, mode@entry=0x7f2e7e343fae "r") at iopopen.c:122
#4 0x00007f2e6eef1c2c in _IO_new_popen (command=0x7f2e7e1760ae "uname -ap", mode=mode@entry=0x7f2e7e343fae "r") at iopopen.c:203
#5 0x00007f2e7df4ef84 in butil::read_command_output_through_popen (os=..., cmd=) at /usr/src/debug/brpc-0.9.8_rc01-1.el8.x86_64/src/butil/popen.cpp:159
#6 0x00007f2e7df5dcf7 in bvar::ReadVersion::ReadVersion (this=0x7f2d4c05fac0) at /usr/src/debug/brpc-0.9.8_rc01-1.el8.x86_64/src/bvar/default_variables.cpp:608
#7 0x00007f2e7df5dfac in butil::GetLeakySingletonbvar::ReadVersion::create_leaky_singleton () at /usr/src/debug/brpc-0.9.8_rc01-1.el8.x86_64/src/butil/memory/singleton_on_pthread_once.h:41
#8 0x00007f2e7024ae67 in __pthread_once_slow (once_control=0x7f2e7e67b8c0 <butil::GetLeakySingletonbvar::ReadVersion::g_create_leaky_singleton_once>, init_routine=0x7f2e7df5df90 <butil::GetLeakySingletonbvar::ReadVersion::create_leaky_singleton()>) at pthread_once.c:116
Thread 66 (Thread 0x7f2db97fa700 (LWP 3633582)):
#0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:63
#1 0x00007f2e6ef01e10 in _int_free (av=0x7f2d90000020, p=0x7f2d900d39e0, have_lock=) at malloc.c:4314
Thread 63 (Thread 0x7f2dbaffd700 (LWP 3633579)):
#0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:63
#1 0x00007f2e6ef04650 in __GI___libc_malloc (bytes=bytes@entry=7934) at malloc.c:3071
#2 0x00007f2e6ef89381 in __backtrace_symbols (array=0x7f2e405e7ad8, size=) at backtracesyms.c:69
#3 0x0000559de317aeba in BackTrace::BackTrace (this=0x7f2e405e7ad0, s=1) at BackTrace.h:29
#4 0x0000559de3178e7a in handle_fatal_signal (signum=6) at signal_handler.cc:167
#5
#6 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#7 0x00007f2e6eea0b25 in __GI_abort () at abort.c:79
#8 0x00007f2e6eef9897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f2e6f006057 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#9 0x00007f2e6eefffdc in malloc_printerr (str=str@entry=0x7f2e6f0080f8 "malloc(): invalid size (unsorted)") at malloc.c:5366
#10 0x00007f2e6ef02e6c in _int_malloc (av=av@entry=0x7f2d90000020, bytes=bytes@entry=40) at malloc.c:3748
#11 0x00007f2e6ef04662 in __GI___libc_malloc (bytes=40) at malloc.c:3073
#12 0x00007f2e6f871d3c in operator new(unsigned long) () from /lib64/libstdc++.so.6
#13 0x0000559de342482a in __gnu_cxx::new_allocator<std::_Rb_tree_node >::allocate (this=0x7f2d901bf9a0, __n=1) at /usr/include/c++/8/ext/new_allocator.h:111