Intermittent FPE "erroneous arithmetic operation" in gqttq tmad test (for cpp512z with FPTYPE=f only: fix it with 'volatile')

[valassi]

While rerunning tests in PR #841 I came across a new FPE "Floating-point exception - erroneous arithmetic operation" in gqttq tmad tests.

This is very surprising because I think that there is actually no change in the code (just some makefile changes leading to file name changes). I will try to rerun the test.

Anyway, for reference the issue is here in tmad/logs_gqttq_mad/log_gqttq_mad_f_inl0_hrd0.txt

...
*** (2-512z) EXECUTE MADEVENT_CPP x10 (create events.lhe) ***
--------------------
CUDACPP_RUNTIME_FBRIDGEMODE = (not set)
CUDACPP_RUNTIME_VECSIZEUSED = 8192
--------------------
81920 1 1 ! Number of events and max and min iterations
0.000001 ! Accuracy (ignored because max iterations = min iterations)
0 ! Grid Adjustment 0=none, 2=adjust (NB if = 0, ftn26 will still be used if present)
1 ! Suppress Amplitude 1=yes (i.e. use MadEvent single-diagram enhancement)
0 ! Helicity Sum/event 0=exact
1 ! Channel number (1-N) for single-diagram enhancement multi-channel (NB used even if suppress amplitude is 0!)
--------------------
Executing ' ./build.512z_f_inl0_hrd0/madevent_cpp < /tmp/avalassi/input_gqttq_x10_cudacpp > /tmp/avalassi/output_gqttq_x10_cudacpp'

Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation.

Backtrace for this error:
#0  0x7f2a1a623860 in ???
#1  0x7f2a1a622a05 in ???
#2  0x7f2a1a254def in ???
#3  0x7f2a1ae20acc in ???
#4  0x7f2a1acc4575 in ???
#5  0x7f2a1ae1d4c9 in ???
#6  0x7f2a1ae2570d in ???
#7  0x7f2a1ae2afa1 in ???
#8  0x43008b in ???
#9  0x431c10 in ???
#10  0x432d47 in ???
#11  0x433b1e in ???
#12  0x44a921 in ???
#13  0x42ebbf in ???
#14  0x40371e in ???
#15  0x7f2a1a23feaf in ???
#16  0x7f2a1a23ff5f in ???
#17  0x403844 in ???
#18  0xffffffffffffffff in ???
./madX.sh: line 379: 3004240 Floating point exception(core dumped) $timecmd $cmd < ${tmpin} > ${tmp}
ERROR! ' ./build.512z_f_inl0_hrd0/madevent_cpp < /tmp/avalassi/input_gqttq_x10_cudacpp > /tmp/avalassi/output_gqttq_x10_cudacpp' failed
 PDF set = nn23lo1
 alpha_s(Mz)= 0.1300 running at 2 loops.
 alpha_s(Mz)= 0.1300 running at 2 loops.
 Renormalization scale set on event-by-event basis
 Factorization   scale set on event-by-event basis


 getting user params
Enter number of events and max and min iterations: 
 Number of events and iterations        81920           1           1

[valassi]

Very strange. I have rerun the test and the FPE has disappeared. Closing as not reproducible.

[valassi]

Note: similar issues have resurfaced in susy_gg_t1t1, being debugged in #826

[valassi]

I have found again this SIGFPE in gqttq for FPTYPE=f, while running with the code using Olivier's patch #850 for the susy xsec mismatch #825.

My impression (or hope) is that this is the same issue as #855, i.e. a SIGFPE in fortran aloha_functions.f that can be fixed with volatile (see PR #857). I will test that too.

[valassi]

I have debugged this further.

First point, this is intermittent. Sometimes the code succeeds, sometimes the code fails (rerunning the same executable multiple times). Maybe half half, maybe less.

Second, this is NOT RELATED to the other SIGFPE #855 in rotxxx. So it will not be fixed by #857.

This crash happens deep inside cudacpp, within the color. I managed to create a gdb trace after rebuilding all with -g. This is in 19a2e0c from PR #860

cd gq_ttq.mad/SubProcesses/P1_gu_ttxu
make cleanall
make -j FPTYPE=f BACKEND=cpp512z 
./madevent_cpp < /tmp/avalassi/input_gqttq_x1_cudacpp

and

[avalassi@itscrd90 gcc11/usr] /data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu> gdb ./madevent_cpp 
...
(gdb) run < /tmp/avalassi/input_gqttq_x1_cudacpp
...
Program received signal SIGFPE, Arithmetic exception.
0x00007ffff7f98db1 in mg5amcCpu::sigmaKin (allmomenta=0x7ffff76bf040, allcouplings=0x7ffff7b57040, allrndhel=<optimized out>, 
    allrndcol=0x6300d00, allMEs=0x6310d80, channelId=channelId@entry=1, allNumerators=0x6341000, allDenominators=0x6351080, 
    allselhel=0x6320e00, allselcol=0x6330e80, nevt=16384) at CPPProcess.cc:1193
1193                if( okcol )
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.34-60.el9.x86_64 libgcc-11.3.1-4.3.el9.alma.x86_64 libgfortran-11.3.1-4.3.el9.alma.x86_64 libgomp-11.3.1-4.3.el9.alma.x86_64 libquadmath-11.3.1-4.3.el9.alma.x86_64 libstdc++-11.3.1-4.3.el9.alma.x86_64
(gdb) p okcol
$1 = <optimized out>
(gdb) p allrndcol
$3 = (const mgOnGpu::fptype *) 0x6300d00
(gdb) p ievt
$4 = <optimized out>
(gdb) p ieppV
$5 = <optimized out>
(gdb) p targetamp
$6 = {{3.6287187e-05, 0.00301690097, 9.26938374e-05, 0.000283826666, 0.000752002583, 0.000340251776, 1.23037262e-06, 0.00068957475, 
    0.000313476485, 5.8289319e-05, 0.00402065413, 0.00385404564, 3.3489403e-05, 0.0149808694, 0.000554771861, 0.00240404671}, {
    3.6287187e-05, 0.00301690097, 9.26938374e-05, 0.000283826666, 0.000752002583, 0.000340251776, 1.23037262e-06, 0.00068957475, 
    0.000313476485, 5.8289319e-05, 0.00402065413, 0.00385404564, 3.3489403e-05, 0.0149808694, 0.000554771861, 0.00240404671}, {
    3.6287187e-05, 0.00301690097, 9.26938374e-05, 0.000283826666, 0.000752002583, 0.000340251776, 1.23037262e-06, 0.00068957475, 
    0.000313476485, 5.8289319e-05, 0.00402065413, 0.00385404564, 3.3489403e-05, 0.0149808694, 0.000554771861, 0.00240404671}, {
    3.6287187e-05, 0.00301690097, 9.26938374e-05, 0.000283826666, 0.000752002583, 0.000340251776, 1.23037262e-06, 0.00068957475, 
    0.000313476485, 5.8289319e-05, 0.00402065413, 0.00385404564, 3.3489403e-05, 0.0149808694, 0.000554771861, 0.00240404671}}
(gdb) p neppV
$7 = 16
(gdb) p icolC
$8 = <optimized out>
(gdb) p ncolor
$9 = 4
(gdb) w
Missing arguments.
(gdb) l
1188    #if defined MGONGPU_CPPSIMD
1189                const bool okcol = allrndcol[ievt] < ( targetamp[icolC][ieppV] / targetamp[ncolor - 1][ieppV] );
1190    #else
1191                const bool okcol = allrndcol[ievt] < ( targetamp[icolC] / targetamp[ncolor - 1] );
1192    #endif
1193                if( okcol )
1194                {
1195                  allselcol[ievt] = icolC + 1; // NB Fortran [1,ncolor], cudacpp [0,ncolor-1]
1196                  break;
1197                }

I suspect that this is related instead to the iconfig-channel mapping issues that @oliviermattelaer investigated in #852 ?

Anyway, keep this open.

To use the debugger, I added these patches

[avalassi@itscrd90 gcc11/usr] /data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu> git diff  --no-ext-diff
diff --git a/epochX/cudacpp/gq_ttq.mad/Source/make_opts b/epochX/cudacpp/gq_ttq.mad/Source/make_opts
index e4b87ee6a..6ccc273c1 100644
--- a/epochX/cudacpp/gq_ttq.mad/Source/make_opts
+++ b/epochX/cudacpp/gq_ttq.mad/Source/make_opts
@@ -1,7 +1,7 @@
 DEFAULT_CPP_COMPILER=g++
 DEFAULT_F2PY_COMPILER=f2py3
 DEFAULT_F_COMPILER=gfortran
-GLOBAL_FLAG=-O3 -ffast-math -fbounds-check
+GLOBAL_FLAG=-g -O3 -ffast-math -fbounds-check
 MACFLAG=
 MG5AMC_VERSION=SpecifiedByMG5aMCAtRunTime
 PYTHIA8_PATH=NotInstalled
diff --git a/epochX/cudacpp/gq_ttq.mad/SubProcesses/cudacpp.mk b/epochX/cudacpp/gq_ttq.mad/SubProcesses/cudacpp.mk
index 89da34009..b8fa4e131 100644
--- a/epochX/cudacpp/gq_ttq.mad/SubProcesses/cudacpp.mk
+++ b/epochX/cudacpp/gq_ttq.mad/SubProcesses/cudacpp.mk
@@ -387,6 +387,10 @@ else
   ###override OMPFLAGS = # disable OpenMP MT on all other platforms (default before #575)
 endif
 
+# Debug SIGFPE crash #845
+override OMPFLAGS=
+override OPTFLAGS=-g -O3
+
 #-------------------------------------------------------------------------------
 
 #=== Configure defaults and check if user-defined choices exist for RNDGEN (legacy!), HASCURAND, HASHIPRAND