BUG: VTK corrupted for 1D runs with MPI

[vdbma]

Describe the issue:

When using MPI for a 1D simulation, chuncks of the output VTK files appear to be missing or have a wrong value (I'm guessing some uninitialised array value).
The missing data region does not align with MPI domains, but the regions affected do not look random.

Above: Illustration of the RHO fields of vtk 0000 (initial condition).
MPI: compiled with mpi, run with mpi
NOMPI: compiled with mpi, run without mpi
MPI_1C: compiled with mpi, run with mpi, with COMPONENTS set to 1 (other runs have 2)
Dotted lines show the MPI domains.

Above: Time evolution: always the same regions are affected.

Additional remarks:

• The code does not crash
• using idefix debug does not change the behaviour
• Kokkos_bound check does not detect any problem
• the code evolution appears to be ok, to me this points to a bug in the output routines

Error message:

runtime information:

Reproduced with:

• the HD/KHI test set to 1 DIMENSION
• Compiled both with 1 and 2 COMPONENTS (see below)
• compiled with MPI (all cases) : Open MPI 4.1.6
• CPU: AMD Ryzen 7 PRO 7840U
• OS: Linux Mint 22 with kernel 6.8.0-59-generic
• Idefix info: Idefix version 2.2.01-ede2e1f5 Built against Kokkos 40301

[glesur]

Can't reproduce your issue with openmpi 5, but I've seen similar things every now and then with openmpi 4, likely related to openmpi's buggy ompio component.

Can you try to turn it off (this forces openmpi to use MPICH's mpi I/O, called romio in openmpi, which is more stable)?
mpirun --mca io ^ompio ...

[vdbma]

Yes, using mpirun --mca io ^ompio ... appears to fix the issue, thanks !

[glesur]

added this problem in the FAQs for future reference