BUG: NaNs with NIMHD resistivities with coarsening in X3-dir on GPU

[aborderies]

Describe the issue:

When I tried to switch my setup to 3D, I found that when I activate coarsening in the X3 direction (phi coordinate in spherical) and non-ideal MHD effects (ambipolar diffusion or ohmic effect, with rkl or explicit), I get NaNs that appear directly on the first cycle when I run idefix on GPU (tested on A100 and RTX 2080 SUPER), but not when I run it on CPU where everything is fine.

Here is a setup that reproduce that: Collapse_Setup.zip

Thanks in advance for the help.

Error message:

runtime information:

Idefix version 2.2.00-54482d58
Kokkos 40301

[glesur]

I suspect your coarsening levels are incorrect beyond the axis. You should use
coarseningLevel(j,i) = 1 + log2(1/fabs(sin(th(j))));
to define your coarsening levels, as the code do use coarsening levels in the ghost zones for the electrical currents.

[aborderies]

Hi @glesur, thanks for your answere, that does seem to be the source of the problem! Out of curiosity, why do NaNs only appear on GPUs and not on CPUs in this situation?

[glesur]

the result of int(log2(-1)) is implementation-dependent, and it seems that nvcc's math library is more prone to make Nans than gcc's or llvm's (llvm answer is -2147483648, which is not Nan...!)