-
Notifications
You must be signed in to change notification settings - Fork 4.1k
Description
Describe the bug (描述bug)
最近在生产环境出了一次故障,现象是流量扛不住的时候,自动扩容,但即使这样,扩容的部分CPU也会升至100%,即使流量后面没有变化,容器数量近乎是原来的6倍
在测试环境压测了下,复现了。现象是开启了3个容器,其中一个容器在一开始CPU逐渐上升至100%,而其他容器的cpu仍然是20%,在开始停止的时刻,其他两个容器cpu也迅速升至100%
#0 0x00007fa34edd1b5d in read () from /lib64/libc.so.6
#1 0x00000000007e824e in google::ReadPersistent(int, void*, unsigned long) ()
#2 0x00000000007e892e in google::FindSymbol(unsigned long, int, char*, int, unsigned long*, unsigned long, Elf64_Shdr const*, Elf64_Shdr const*) ()
#3 0x00000000007e907f in google::SymbolizeAndDemangle(void*, char*, int, unsigned long*) ()
#4 0x00000000006e8cd1 in butil::debug::(anonymous namespace)::ProcessBacktrace(void* const*, unsigned long, butil::debug::(anonymous namespace)::BacktraceOutputHandler*) ()
#5 0x00000000006e8d77 in butil::debug::StackTrace::OutputToStream(std::ostream*) const ()
#6 0x0000000000706f13 in logging::LogStream::FlushWithoutReset() ()
#7 0x0000000000707350 in logging::LogMessage::~LogMessage() ()
#8 0x0000000000610464 in brpc::policy::HttpResponseSender::~HttpResponseSender() ()
#9 0x0000000000613bb4 in brpc::policy::HttpResponseSenderAsDone::~HttpResponseSenderAsDone() ()
#10 0x0000000000498c10 in mtad::ssp::AsyncLoadHandler::SendResponse() ()
#11 0x00000000004e7fae in mtad::ssp::PMPPreLoadSession::OnPreLoadFinish() ()
#12 0x00000000004e6bc7 in brpc::internal::MethodClosure0<mtad::ssp::PMPPreLoadSession, std::shared_ptrmtad::ssp::PMPPreLoadSession >::Run() ()
#13 0x00000000005a1b52 in brpc::Controller::EndRPC(brpc::Controller::CompletionInfo const&) ()
#14 0x00000000005a1eb0 in brpc::Controller::RunEndRPC(void*) ()
#15 0x000000000058f9e7 in bthread::TaskGroup::task_runner(long) ()
#16 0x000000000073f5e1 in bthread_make_fcontext ()
#17 0x0000000000000000 in ?? ()
To Reproduce (复现方法)
Expected behavior (期望行为)
Versions (各种版本)
OS: centos 7.6
Compiler: gcc14.2
brpc:1.11.0
protobuf: 27.3
Additional context/screenshots (更多上下文/截图)
