Segfault during benchmark client termination

[eric846]

nighthawk/source/client/benchmark_client_impl.cc

Line 117 in e02ea9e

pool()->addDrainedCallback([this]() -> void {

I couldn't tell whether the crash happened when the callback was being added or when it actually ran.

I was using 50-100 concurrency and around 1000 RPS per worker, and an inline RequestSource. These conditions may or may not be necessary. My impression is that it crashed about half the time.

A workaround is to simply comment out the drained callback and fall back on the 5 second timer, at the cost of 5 seconds per benchmark.

[oschaaf]

Ouch; do we have a backtrace by chance?
This change may or may not be related, it affected the shutdown procedure to address very long wait times and hangs in case the target queued up huge amounts of work, and unwinding takes a long (long) time: #628

[eric846]

Tracked down a backtrace:

*** SIGSEGV (@(nil)), XXXXXXXXXXXXXX received by PID XXXXXX (TID XXXXXX) on cpu X; stack trace: ***
XXXXXXXXXXX/0 EXXXXXX xx:xx:xx.xxxxxx XXXXXX process_state.cc:1228 PC:
 @     0xXXXXXXXXXXXX  (unknown)  std::__u::__function::__policy_invoker<>::__call_impl<>()
    @     0xXXXXXXXXXXXX        XXX  FailureSignalHandler()
    @     0xXXXXXXXXXXXX        XXX  (unknown)
    @     0xXXXXXXXXXXXX        XXX  Envoy::ConnectionPool::ConnPoolImplBase::checkForDrained()
    @     0xXXXXXXXXXXXX        XXX  Envoy::Http::HttpConnPoolImplBase::addDrainedCallback()
    @     0xXXXXXXXXXXXX        XXX  Nighthawk::Client::BenchmarkClientHttpImpl::terminate()
    @     0xXXXXXXXXXXXX        XXX  std::__u::__thread_proxy<>()
    @     0xXXXXXXXXXXXX        XXX  start_thread
    @     0xXXXXXXXXXXXX  (unknown)  clone

[eric846]

I haven't seen this happen for many months of heavy use. It was most likely fixed by recent Nighthawk or Envoy changes.