Skip to content

流量大的时候,引起异常bug,即使扩容也不生效 #2927

@namelij

Description

@namelij

Describe the bug (描述bug)
最近在生产环境出了一次故障,现象是流量扛不住的时候,自动扩容,但即使这样,扩容的部分CPU也会升至100%,即使流量后面没有变化,容器数量近乎是原来的6倍

在测试环境压测了下,复现了。现象是开启了3个容器,其中一个容器在一开始CPU逐渐上升至100%,而其他容器的cpu仍然是20%,在开始停止的时刻,其他两个容器cpu也迅速升至100%

#0 0x00007fa34edd1b5d in read () from /lib64/libc.so.6
#1 0x00000000007e824e in google::ReadPersistent(int, void*, unsigned long) ()
#2 0x00000000007e892e in google::FindSymbol(unsigned long, int, char*, int, unsigned long*, unsigned long, Elf64_Shdr const*, Elf64_Shdr const*) ()
#3 0x00000000007e907f in google::SymbolizeAndDemangle(void*, char*, int, unsigned long*) ()
#4 0x00000000006e8cd1 in butil::debug::(anonymous namespace)::ProcessBacktrace(void* const*, unsigned long, butil::debug::(anonymous namespace)::BacktraceOutputHandler*) ()
#5 0x00000000006e8d77 in butil::debug::StackTrace::OutputToStream(std::ostream*) const ()
#6 0x0000000000706f13 in logging::LogStream::FlushWithoutReset() ()
#7 0x0000000000707350 in logging::LogMessage::~LogMessage() ()
#8 0x0000000000610464 in brpc::policy::HttpResponseSender::~HttpResponseSender() ()
#9 0x0000000000613bb4 in brpc::policy::HttpResponseSenderAsDone::~HttpResponseSenderAsDone() ()
#10 0x0000000000498c10 in mtad::ssp::AsyncLoadHandler::SendResponse() ()
#11 0x00000000004e7fae in mtad::ssp::PMPPreLoadSession::OnPreLoadFinish() ()
#12 0x00000000004e6bc7 in brpc::internal::MethodClosure0<mtad::ssp::PMPPreLoadSession, std::shared_ptrmtad::ssp::PMPPreLoadSession >::Run() ()
#13 0x00000000005a1b52 in brpc::Controller::EndRPC(brpc::Controller::CompletionInfo const&) ()
#14 0x00000000005a1eb0 in brpc::Controller::RunEndRPC(void*) ()
#15 0x000000000058f9e7 in bthread::TaskGroup::task_runner(long) ()
#16 0x000000000073f5e1 in bthread_make_fcontext ()
#17 0x0000000000000000 in ?? ()

To Reproduce (复现方法)

Expected behavior (期望行为)

Versions (各种版本)
OS: centos 7.6
Compiler: gcc14.2
brpc:1.11.0
protobuf: 27.3

Additional context/screenshots (更多上下文/截图)

Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions