Use `cpptrace` for prettier backtraces from exceptions #3225

ZedThree · 2025-12-05T15:11:35Z

We have historically been reluctant to add too many external dependencies, but with CMake and spack making it easier to pull them in and build them, I thought I'd try this library for generating stack traces.

Two useful things:

it's very portable out of the box, so we don't need to rely on external utilities like addr2line and instead can always enable backtraces
we can print snippets:

#3  in checkData(double)
    at /home/peter/Codes/BOUT++/BOUT-dev/include/bout/utils.hxx:493:65
     491: inline void checkData(BoutReal f) {
     492:   if (!std::isfinite(f)) {
   > 493:     throw BoutException("BoutReal: Operation on non-finite data");
                                                                          ^
     494:   }
     495: }
#4  in operator*(Field3D const&, double)
    at /home/peter/Codes/BOUT++/BOUT-dev/src/field/generated_fieldops.cxx:480:12
     478:   Field3D result{emptyFrom(lhs)};
     479:   checkData(lhs);
   > 480:   checkData(rhs);
                     ^
     481: 
     482:   result.setRegion(lhs.getRegionID());
#5  in Blob2D::rhs(double)
    at /home/peter/Codes/BOUT++/BOUT-dev/examples/blob2d/blob2d.cxx:177:41
     176:     ddt(n) = -bracket(phi, n, BRACKET_ARAKAWA) // ExB term
   > 177:              + 2 * DDZ(n) * (rho_s / R_c)      // Curvature term
                                                  ^
     178:              + D_n * Delp2(n);                 // Diffusion term
     179:     if (compressible) {

and can even enable colour:

It also has its own signal handler we could perhaps use instead of ours (though I've not tested this out).

Lastly, this is maybe a complete enough solution that we could also remove MsgStack/TRACE/AUTO_TRACE?

ZedThree · 2025-12-17T14:21:57Z

Lots of CMake pain here

bendudson · 2025-12-29T18:19:30Z

Thanks @ZedThree ! This does look nice, if the CMake pain can be overcome. I think if it works portably then it could replace all the MsgStack / AUTOTRACE code.

ZedThree · 2026-01-06T11:43:27Z

I've removed AUTO_TRACE completely, and TRACE where it was just essentially repeating the function name (or something like intialising PETSc solver in PetscSolver::init).

That leaves uses like TRACE("PetscMatrix setting values at ({}, {})", petscRow, petscCol) which include function arguments or local variables that the stacktrace won't capture. I think they likely have some value for debugging, but quite limited?

Same goes for uses like TRACE("Creating Outer SOL communicators for Single Null operation") in BoutMesh. I've left them in for now, but I'm not sure how much more information they offer over the exception message + stacktrace.

Here's an example with CMAKE_BUILD_TYPE=RelWithDebInfo:

Error encountered: Stack trace (most recent call first):
#0 (filtered)
#1 (filtered)
#2 (filtered)
#3 in BoutMesh::createCommunicators()
   at /home/peter/Codes/BOUT++/BOUT-dev/src/mesh/impls/bout/boutmesh.cxx:709:64
     707:       }
     708:       // Unconditional exception for demo purposes
   > 709:       throw BoutException("Single null outer SOL not correct\n");      
                                                                         ^
     710:       MPI_Group_free(&group);
     711:     }
#4 in BoutMesh::load()
   at /home/peter/Codes/BOUT++/BOUT-dev/src/mesh/impls/bout/boutmesh.cxx:575:22
     573:   /// Communicator
     574: 
   > 575:   createCommunicators();
                               ^
     576:   output_debug << "Got communicators" << endl;
#5 in BoutInitialise(int&, char**&)
   at /home/peter/Codes/BOUT++/BOUT-dev/src/bout++.cxx:201:28
     199:   bout::globals::mesh = Mesh::create();
     200:   // Load from sources. Required for Field initialisation
   > 201:   bout::globals::mesh->load();
                                     ^
     202: 
     203:   // time_report options are used in BoutFinalise, i.e. after we
#6 in main
   at /home/peter/Codes/BOUT++/BOUT-dev/examples/elm-pb/elm_pb.cxx:2161:1
    2159: };
    2160: 
  > 2161: BOUTMAIN(ELMpb);
          ^
#7 (filtered)
#8 (filtered)
#9 (filtered)

====== Exception thrown ======
Single null outer SOL not correct


=== Additional information ===
 -> Creating Outer SOL communicators for Single Null operation on line 687 of '/home/peter/Codes/BOUT++/BOUT-dev/src/mesh/impls/bout/boutmesh.cxx'

And here's what it looks like with CMAKE_BUILD_TYPE=Release (no debug info):

Error encountered: Stack trace (most recent call first):
#0 (filtered)
#1 (filtered)
#2 (filtered)
#3 in BoutMesh::createCommunicators() [clone .cold]
   at /home/peter/Codes/BOUT++/BOUT-dev/build_release/lib/libbout++.so.5.2.0
#4 in BoutMesh::load()
   at /home/peter/Codes/BOUT++/BOUT-dev/build_release/lib/libbout++.so.5.2.0
#5 in BoutInitialise(int&, char**&)
   at /home/peter/Codes/BOUT++/BOUT-dev/build_release/lib/libbout++.so.5.2.0
#6 in main
   at /home/peter/Codes/BOUT++/BOUT-dev/examples/elm-pb/build/elm_pb
#7 (filtered)
#8 (filtered)
#9 (filtered)

====== Exception thrown ======
Single null outer SOL not correct


=== Additional information ===
 -> Creating Outer SOL communicators for Single Null operation on line 687 of '/home/peter/Codes/BOUT++/BOUT-dev/src/mesh/impls/bout/boutmesh.cxx'

Clearly the TRACE call is more useful when we don't have debug info, but if we recommend always building with it, it's probably not that useful?

@bendudson Thoughts on completely removing MsgStack? Maybe in a separate PR so we can more easily revert it if we change our minds?

CUDA tests are failing due to #3233

github-actions

clang-tidy made some suggestions

include/bout/deriv_store.hxx

include/bout/msg_stack.hxx

include/bout/scorepwrapper.hxx

src/bout++.cxx

github-actions · 2026-01-06T12:04:25Z

src/bout++.cxx

-    bout::globals::mesh = Mesh::create();
-    // Load from sources. Required for Field initialisation
-    bout::globals::mesh->load();
+  bout::globals::mpi = new MpiWrapper();


warning: assigning newly created 'gsl::owner<>' to non-owner 'MpiWrapper *' [cppcoreguidelines-owning-memory]

bout::globals::mpi = new MpiWrapper(); ^

src/solver/impls/euler/euler.cxx

src/sys/derivs.cxx

tests/integrated/test-backtrace/boutexcept.cxx

github-actions

clang-tidy made some suggestions

src/sys/output.cxx

Always generating the backtraces can slow the unit tests down by a factor x2-3 An alternative might be to store the `cpptrace:stacktrace` and only convert to string in `BoutException::what()`

- Add paths to `FetchContent` dirs in config file - Switch to custom versions of cpptrace/libdward-lite - Includes some cmake fixes in both projects

Required because cpptrace prevents in-source builds Also switch to using `subprocess.run` so we can better see errors

Seems to cause MPI problems?

dschwoerer

Looks good to me 👍

cpptrace seems not that tricky to build, but I am wondering whether we should allow to disable it? Would it only require to #ifdef part of one file? For example if you do not have internet while building? Should we try to bundle it into our release tars?

dschwoerer · 2026-01-08T08:58:23Z

manual/sphinx/developer_docs/debugging.rst

+The second utility BOUT++ has to help debugging is the message stack using the
+`TRACE` macro. This is very useful for when a bug only occurs after a long time
+of running, and/or only occasionally. The ``TRACE`` macro can simply be dropped
+in anywhere in the code::


Should we not recommend -g?
Even with TRACE still around, most times -g is more helpful, imho.

Yes, that's a good point. TRACE is now very much for additional info, so I'll try and make sure it's clear that we recommend always using either Debug or RelWithDebInfo CMAKE_BUILD_TYPE, and moving the backtrace section up (your next comment)

dschwoerer · 2026-01-08T08:59:41Z

manual/sphinx/developer_docs/debugging.rst

-The output looks something like this:
+Lastly, BOUT++ can also automatically print a backtrace in the event of a
+crash. This is very useful to include if you ever need to report a bug to the
+developers! The output looks something like this:


Maybe move this up, as I think this is the most helpful information in general?

src/sys/boutexception.cxx

ZedThree · 2026-01-08T10:06:47Z

@dschwoerer Do you have any opinion on completely removing TRACE and MsgStack? Do you find it useful enough to keep?

dschwoerer · 2026-01-08T10:27:33Z

I do not use it. I either use gdb or add print statements 😇

It may add some value in a few cases, but I think in general they should be considered left over debug output, that was once for one person useful, but can go. Probably better to cleanup the code 👍

ZedThree · 2026-01-08T14:27:09Z

cpptrace seems not that tricky to build, but I am wondering whether we should allow to disable it? Would it only require to #ifdef part of one file? For example if you do not have internet while building? Should we try to bundle it into our release tars?

I think it's probably simpler if we just always require it. Setting BOUT_DOWNLOAD_CPPTRACE=off lets you supply a pre-built version.

ZedThree · 2026-01-08T15:17:00Z

I guess this should probably use a git submodule like fmt and mpark.variant, which are also required, rather than FetchContent

dschwoerer · 2026-01-09T07:36:06Z

I guess this should probably use a git submodule like fmt and mpark.variant, which are also required, rather than FetchContent

Yes, that would make it easier to include in the release!

ZedThree requested a review from bendudson December 5, 2025 15:11

ZedThree force-pushed the add-cpptrace branch from a0c128d to 19ae202 Compare December 5, 2025 15:37

ZedThree force-pushed the add-cpptrace branch from a0d6528 to 86dd0c0 Compare January 5, 2026 16:42

github-actions bot reviewed Jan 6, 2026

View reviewed changes

src/sys/output.cxx Show resolved Hide resolved

ZedThree and others added 21 commits January 7, 2026 09:32

Switch to cpptrace for stack traces in exceptions

d9e789c

Add some control over BoutException backtraces

199ba5a

Always generating the backtraces can slow the unit tests down by a factor x2-3 An alternative might be to store the `cpptrace:stacktrace` and only convert to string in `BoutException::what()`

Try to ensure build_defines is included where it's used

dc865d3

Try to make backtrace unit test more portable

e0710f8

cmake: Fix some issues with building with cpptrace

9dd86c5

- Add paths to `FetchContent` dirs in config file - Switch to custom versions of cpptrace/libdward-lite - Includes some cmake fixes in both projects

Prettify symbols in cpptrace output

00995ea

Filter out pre-main functions from stacktrace

3fc530d

Include filtered placeholders

5f77485

Remove AUTO_TRACE macro

cbc4cf2

Remove uses of TRACE which are just the function name

505d5ad

Fix unnecessary includes of msg_stack.hxx

63e78ae

Don't duplicate exception message when thrown from BoutInitialise

c42afa3

Don't print MsgStack dump if it's empty

6d89b6f

Apply clang-format changes

8896f6e

docs: Use build directory for building library

e497fcd

Required because cpptrace prevents in-source builds Also switch to using `subprocess.run` so we can better see errors

Partial revert of d0cd5f0

6ccdca8

Seems to cause MPI problems?

Remove some more unused headers

d8b360f

tests: Add functions to anon namespace

5306125

Don't use char[] for constant

68fab1b

Fix use of reserved identifier

563ab24

docs: Update example backtrace

1a9210c

dschwoerer previously approved these changes Jan 8, 2026

View reviewed changes

docs: Expand on using cpptrace and the backtrace

4ac87c1

ZedThree dismissed dschwoerer’s stale review via 4ac87c1 January 8, 2026 15:18

ZedThree force-pushed the add-cpptrace branch from 93214da to 4ac87c1 Compare January 8, 2026 15:18

ZedThree and others added 3 commits January 8, 2026 15:18

Apply clang-format changes

73e5857

tests: Fix exception test inlining functions

8edb351

Apply black changes

6b03b6c

ZedThree added 4 commits January 9, 2026 13:50

Add cpptrace as git submodule

9855b5f

boutpp: Add cpptrace to sdist

7fa1c5f

docs: Init all submodules

68904f9

cmake: Update cpptrace include dir for install

4a09719

Use cpptrace for prettier backtraces from exceptions #3225

Are you sure you want to change the base?

Use cpptrace for prettier backtraces from exceptions #3225

Conversation

ZedThree commented Dec 5, 2025

Uh oh!

ZedThree commented Dec 17, 2025

Uh oh!

bendudson commented Dec 29, 2025

Uh oh!

ZedThree commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dschwoerer left a comment

Choose a reason for hiding this comment

Uh oh!

dschwoerer Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

ZedThree Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

dschwoerer Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ZedThree commented Jan 8, 2026

Uh oh!

dschwoerer commented Jan 8, 2026

Uh oh!

ZedThree commented Jan 8, 2026

Uh oh!

ZedThree commented Jan 8, 2026

Uh oh!

dschwoerer commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Use `cpptrace` for prettier backtraces from exceptions #3225

Use `cpptrace` for prettier backtraces from exceptions #3225

ZedThree commented Jan 6, 2026 •

edited

Loading