bthread_id_join->butex_wait死锁问题

[hairet]

Describe the bug (描述bug)
死锁堆栈如下，隐去一些我们业务方的函数，不影响观看源码，因为brpc源码我做过改动，某些行数不一定一致，但是有问题的函数是官方的，原因后面解释：
#0 0x00007f3effb436e9 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x0000000000ba47c2 in bthread::wait_pthread(bthread::ButexPthreadWaiter&, timespec*) () at ./src/bthread/sys_futex.h:37
#2 0x0000000000ba5c7b in bthread::butex_wait(void*, int, timespec const*) () at src/bthread/butex.cpp:588
#3 0x0000000000ba76c8 in bthread_id_join () at src/bthread/id.cpp:526
#4 0x0000000000a60209 in brpc::Join(bthread_id_t) () at ./src/brpc/controller.h:396
#5 0x0000000000a84dcc in brpc::Channel::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*, bool) ()
at src/brpc/channel.cpp:531
#6 0x0000000000a8530f in brpc::Channel::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*) ()
at src/brpc/channel.cpp:298
#7 0x00000000008f8b7e in msg::MsgService_Stub::RpcGetMessageReq(google::protobuf::RpcController*, msg::GetMessageReq const*, msg::GetMessageRes*, google::protobuf::Closure*) ()
at MsgService.pb.cc:6388
#8 0x00000000008e46b9 in msg::SyncClient::RpcGetMessageReq(msg::GetMessageReq const*, msg::GetMessageRes*) () at MsgService.client.cc:26
#9 0x0000000000895186 in RpcGetMessageReq(protocol::message::GetMessageReq const&, protocol::message::PCS_GetMessageRes&) ()
at BrpcClientGw.cpp:42
#10 0x000000000088148e in OnGetMessageReq(protocol::message::GetMessageReq*) () at msg_query.cpp:66
#11 0x00000000009894ca in DispatchRequest(int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >&, int, brpc::Controller*) ()
at handle.cpp:492
#12 0x0000000000988cf7 in LoopParseAndProcessSubHead(brpc::Controller*, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) ()
at handle.cpp:411
#13 0x000000000098b9fb in ProcessFramedRequest(brpc::Controller, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, google::protobuf::Closure) ()
at handle.h:116
#14 0x0000000000b29517 in ProcessMyRequest(brpc::InputMessageBase*) () at src/brpc/policy/my_protocol.cpp:98
#15 0x0000000000a56eb7 in brpc::ProcessInputMessage(void*) () at src/brpc/input_messenger.cpp:133
#16 0x0000000000a47a7c in bthread::TaskGroup::task_runner(long) () at src/bthread/task_group.cpp:305
#17 0x0000000000a48276 in bthread::TaskGroup::run_main_task() () at src/bthread/task_group.cpp:158
#18 0x0000000000ba23f8 in bthread::TaskControl::worker_thread(void*) () at src/bthread/task_control.cpp:73
#19 0x00007f3f00cef182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#20 0x00007f3effb4930d in clone () from /lib/x86_64-linux-gnu/libc.so.6

To Reproduce (复现方法)
高QPS,内存高后，可能出现

Expected behavior (期望行为)
task_runner调度的标准bthread不要切到线程模式

Versions (各种版本)
OS:
Compiler:
brpc:
protobuf:

Additional context/screenshots (更多上下文/截图)

[hairet]

说下我的理解，可能有误差，请指正：
主要问题是https://github.com/brpc/brpc/blob/master/src/bthread/task_group.cpp

在分配内存失败之后，会把bthread的butex_wait切到线程模式，这样看似苟活了一段时间，但是如果QPS高的时候，瞬时相同的butex_wait操作大于总线程数的时候，由于bthread_id_join->butex_wait里面会阻塞在wait_pthread，导致全部可用线程都阻塞了，则没有工作线程可以把callmthod异步回包任务调度回来了，从而导致死锁。

我的观点，如果发现因为内存不足导致的问题，最好可以直接assert,避免这种苟活下死锁引起更复杂问题的查问题成本。

[jamesge]

这个问题治本比较复杂，治标的话可以加条FATAL日志。

[hairet]

这个问题治本比较复杂，治标的话可以加条FATAL日志。

话说如果我们自己的brpc版本，考虑在非pthread 做stack alloc失败，直接 assert，会有啥风险么，请大佬帮忙评估一下

[jamesge]

assert程序直接挂掉了，我理解没什么其他风险。加上-crash_on_fatal_log选项的话，碰到FATAL日志也会挂。

[hairet]

assert程序直接挂掉了，我理解没什么其他风险。加上-crash_on_fatal_log选项的话，碰到FATAL日志也会挂。

好的，多谢解答

[yjhjstz]

瞬时相同的butex_wait操作大于总线程数的时候

@hairet 是什么意思？

[uniqueliulin]

这个卡在了futex_wait，出现死锁。futex_wait的返回值不一定是真，需要判断结果再确定“继续阻塞”或放行，rpc中没有做这个判断操作。http://www.man7.org/linux/man-pages/man2/futex.2.html

[zhangqiang4002]

Thread 1647 (Thread 0x7f62e0316700 (LWP 226)):
#0 0x00007f62e7c66c09 in syscall () from /lib64/libc.so.6
#1 0x0000000000a34092 in futex_wait_private (timeout=0x0, expected=0, addr1=0x7f62e0314990) at ./src/bthread/sys_futex.h:41
#2 bthread::wait_pthread (pw=..., ptimeout=ptimeout@entry=0x0) at src/bthread/butex.cpp:142
#3 0x0000000000a35497 in butex_wait_from_pthread (abstime=0x0, expected_value=7981, b=0x11d06b40, g=) at src/bthread/butex.cpp:589
#4 bthread::butex_wait (arg=arg@entry=0x11d06b40, expected_value=expected_value@entry=7981, abstime=abstime@entry=0x0) at src/bthread/butex.cpp:622
#5 0x0000000000a420e8 in bthread_id_join (id=...) at src/bthread/id.cpp:530
#6 0x0000000000903669 in brpc::Join(bthread_id_t) () at ./src/brpc/controller.h:365
#7 0x000000000091f97f in brpc::Channel::CallMethod (this=0x7f62e0314f60, method=0x4fb7a08, controller_base=0x7f62e0315060, request=0x7f62e0314d00, response=0x7f62e0314be0, done=0x0) at src/brpc/channel.cpp:537
#8 0x0000000000b0d493 in braft::rtb::refresh_leader (group=..., timeout_ms=5000) at /home/svn/joostzhang/elasticfaiss1.5/deps/braft-1.1.1/src/braft/route_table.cpp:194
#9 0x00000000004e3ebe in elasticfaiss::WorkNodeServiceImpl::report_heartbeat (this=this@entry=0x3604780 elasticfaiss::g_work_node) at worker/node_service.cpp:293
#10 0x00000000004e4267 in elasticfaiss::WorkNodeServiceImpl::Run (this=0x3604780 elasticfaiss::g_work_node) at worker/node_service.cpp:83
#11 0x00000000008dc1d9 in butil::SimpleThread::ThreadMain (this=0x3604788 elasticfaiss::g_work_node+8) at src/butil/threading/simple_thread.cc:60
#12 0x0000000000a08990 in butil::(anonymous namespace)::ThreadFunc (params=) at src/butil/threading/platform_thread_posix.cc:79
#13 0x00007f62e94c8dc5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f62e7c6c74d in clone () from /lib64/libc.so.6
遇到同样的问题, 只要抛出这个异常, 必现:
Fail to mmap size=1052672 stack_count=30944, possibly limited by /proc/sys/vm/max_map_count: Cannot allocate memory

[bbbyxxx]

@zhangqiang4002 后面怎么处理的呢