Backtrace support in debug builds

[htuch]

Add something like https://github.com/bombela/backward-cpp to the build (at least for debug builds) and enable dumping a stack trace on crash with a std::terminate_handler.

[htuch]

@mattklein123 Can you reassign to @dnoe?

@dnoe we should make sure this works for exceptions as well. I haven't looked enough at C++ exception implementation to understand what goes on when we get to the top-level (hopefully same case as std::terminate_handler, but it's one of the pain points today, we just get the exception message with no stack trace.

[htuch]

Also, we need to make sure it's still easy for Lyft/Google and others who want to override the handler to do so, as Envoy crash behavior needs to be customized in different environments.

[dnoe]

If I recall correctly this is a pain for exceptions. If it's uncaught all the info about where it was originally thrown is lost by the time you get to the terminate handler. I'll give it though but consider exception backtracing as a phase II feature.

[mattklein123]

I don't have much experience with terminate handlers, but an uncaught exception has the correct stack in a core dump. IIRC terminate is called at the individual throw point after unwind and finding a catch point fails.

[mattklein123]

http://en.cppreference.com/w/cpp/error/terminate

...

"an exception is thrown and not caught (it is implementation-defined whether any stack unwinding is done in this case)"

Please verify but I'm nearly certain that libstdc++ does not unwind in this case, which is how core dumps have the correct stack (via abort).

[dnoe]

I had troubles with this in the past, but maybe I fell into the "implementation-defined" pit. I recall that the core dumps from uncaught exceptions had stack traces that began at std::terminate and ended at the call to abort(). Unfortunately this was a previous job - I can't run the code to check. I'll let you know what I find out.

[dnoe]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55917

This bug starts in 2013 and continues unresolved as of now... but it contains some information on why getting a useful stack trace indicating where the uncaught exception was thrown from std::terminate handler doesn't work right on GCC.

A possible workaround is capturing a backtrace in the EnvoyException constructor (in debug mode) and storing it in the exception object.

[mattklein123]

Yes I know about that issue. We don't use std::thread for that reason. :) I'm positive it will work as you expect, just give it a try.

[htuch]

Does it work with Clang? I reckon developers would be better off using clang in debug builds in any case, way better error messages.

[mattklein123]

Dunno, just give it a try, it's trivial to test. I would imagine it will work fine.