Force non random order of execution for C++ tests

[schnellerhase]

Catch2 >=3.9.0 changes execution order to random by default. This needs to be deactivated for MPI parallel testing.

[garth-wells]

I don't think this will work due to the hdf5-mpi package depending on OpenMPI.

Which function call is leading to the hang with OpenMPI ?

[schnellerhase]

MPI_Dist_graph_create_adjacent causes an MPI error - without any further output.

[schnellerhase]

But the problem appears to be open-mpi 5.0.7 -> 5.0.8. After a local update I can reproduce it.

[garth-wells]

MPI_Dist_graph_create_adjacent causes an MPI error - without any further output.

Any particular line in DOLFINx, or is it any call to MPI_Dist_graph_create_adjacent? OpenMPI and MPICH have had bugs in the past with MPI_Dist_graph_create_adjacent - we have some odd looking std::vector::reserve calls in places to work around these.

[schnellerhase]

Any call.

[garth-wells]

I can't reproduce failure locally on macOS and with OpenMPI 5.0.8 from Homebrew.

[schnellerhase]

MPI_ERROR code 15 is MPI_ERR_TRUNCATE and means 'Message truncated on receive'.

[schnellerhase]

Maybe race condition. Cycling mpiexec -np 3 ./unittests [la_vector] crashes most times, but passes occasionally.

[schnellerhase]

Found it - was not the open-mpi bump, but the catch2 update. New version changes default behaviour of execution order to random https://github.com/catchorg/Catch2/releases/tag/v3.9.0. This means, on every process a different execution order of the test cases is generated. So MPI communications of different test cases, in particular with different data types line up, causing all kinds of incorrect data or unmatched communications.