OpenBLAS test suite failures on ARM neoverse_v1

[casparvl]

We're seeing somewhat higher number of OpenBLAS test suite failures on ARM neoverse_v1 in:

• {2023.06} foss/2022b #309

For now, we've increased the number of accepted failures, but we'll need to investigate this issue in more detail. Are these test just out of numerical bounds? Or way off?

See also:

• {2023.06}[2021b] FlexiBLAS v3.0.4 (+ OpenBLAS v0.3.18) #291

It did not seem to be a problem however in:

• add EasyBuild 4.7.2 + foss/2021a to EESSI pilot 2023.06 #260 however

which build OpenBLAS 0.3.15 on ARM neoverse_v1 (see here)

[boegel]

@casparvl Can you include the high-level summary that shows how many tests fail, and maybe also dig up more details on the failing tests (should probably go in a gist), which we can use when reporting this upstream (or getting some input on this from @bartoldeman)?

[casparvl]

SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    1315683         107     (0.008%)        0       (0.000%)
DOUBLE PRECISION        1314777         66      (0.005%)        0       (0.000%)
COMPLEX                 773609          97      (0.013%)        0       (0.000%)
COMPLEX16               776246          74      (0.010%)        0       (0.000%)

--> ALL PRECISIONS      4180315         344     (0.008%)        0       (0.000%)


== 2023-07-19 12:16:15,506 openblas.py:113 INFO 4180315 LAPACK tests run - 344 failed due to numerical errors - 0 failed due to other errors

More detailed (though not complete) test log from bot/easybuild/build/OpenBLAS/0.3.21/GCC-12.2.0/OpenBLAS-0.3.21/lapack-netlib/TESTING/testing_results.txt:

...
Matrix order=   15, type=10, seed= 798,1691,2423, 745, result  5 is 8.389E+06
 Matrix order=   15, type=11, seed= 931,1787, 557,2429, result  5 is 8.389E+06
 Matrix order=   15, type=17, seed= 529,3615,1764,1221, result  5 is 8.389E+06
 Matrix order=   15, type=18, seed=3991, 625,3539,2581, result  5 is 8.389E+06
 Matrix order=   15, type=19, seed=2400,1821, 218,1365, result  5 is 8.389E+06
 Matrix order=   15, type=20, seed=2155,2073,3686, 149, result  5 is 8.389E+06
 Matrix order=   15, type=21, seed=3823,2194,3510,3029, result  5 is 8.389E+06
 Matrix order=   15, type=22, seed=2951, 608,1256,3481, result  5 is 8.389E+06
 Matrix order=   15, type=23, seed=1243,2927, 263, 357, result  5 is 8.389E+06
 Matrix order=   15, type=24, seed=1487,3956,2976,3649, result  5 is 8.389E+06
 Matrix order=   15, type=25, seed=2605,2211,3982,2721, result  5 is 8.389E+06
 Matrix order=   15, type=26, seed=   8,1600,1726,2817, result  5 is 8.389E+06
 Matrix order=   20, type= 8, seed= 624, 693,1681,   9, result  5 is 8.389E+06
 Matrix order=   20, type=10, seed= 563,2705, 428, 633, result  5 is 8.389E+06
 Matrix order=   20, type=12, seed= 112,1059,2403,2121, result  5 is 8.389E+06
 Matrix order=   20, type=13, seed=3448,3524, 410,2697, result  5 is 8.389E+06
 Matrix order=   20, type=14, seed= 248,   6,3354,3273, result  5 is 8.389E+06
 Matrix order=   20, type=17, seed= 723,4024,2193,2265, result  5 is 8.389E+06
 Matrix order=   20, type=18, seed= 828, 842,1787,3733, result  5 is 8.389E+06
 Matrix order=   20, type=19, seed= 252,3327,3300,1389, result  5 is 8.389E+06
 Matrix order=   20, type=20, seed=1904,2346,3199,2437, result  5 is 8.389E+06
 Matrix order=   20, type=21, seed=4066,  26,1253, 221, result  5 is 8.389E+06
 Matrix order=   20, type=22, seed=3234,3664,3237,3473, result  5 is 8.389E+06
 Matrix order=   20, type=23, seed= 481, 957,3230,3689, result  5 is 8.389E+06
 Matrix order=   20, type=24, seed=2871,2618,1987,2949, result  5 is 8.389E+06
 Matrix order=   20, type=25, seed=2604,3233, 644,3529, result  5 is 8.389E+06
 Matrix order=   20, type=26, seed=1465,2266,3976,3833, result  5 is 8.389E+06
 SGV drivers:     64 out of   1092 tests failed to pass the threshold
...
 Matrix order=   20, type=10, seed= 563,2705, 428, 633, result  5 is 4.504D+15
 Matrix order=   20, type=12, seed= 112,1059,2403,2121, result  5 is 4.504D+15
 Matrix order=   20, type=13, seed=3448,3524, 410,2697, result  5 is 4.504D+15
 Matrix order=   20, type=17, seed= 723,4024,2193,2265, result  5 is 4.504D+15
 Matrix order=   20, type=18, seed= 828, 842,1787,3733, result  5 is 4.504D+15
 Matrix order=   20, type=19, seed= 252,3327,3300,1389, result  5 is 4.504D+15
 Matrix order=   20, type=20, seed=1904,2346,3199,2437, result  5 is 4.504D+15
 Matrix order=   20, type=21, seed=4066,  26,1253, 221, result  5 is 4.504D+15
 Matrix order=   20, type=22, seed=3234,3664,3237,3473, result  5 is 4.504D+15
 Matrix order=   20, type=23, seed= 481, 957,3230,3689, result  5 is 4.504D+15
 Matrix order=   20, type=24, seed=2871,2618,1987,2949, result  5 is 4.504D+15
 Matrix order=   20, type=25, seed=2604,3233, 644,3529, result  5 is 4.504D+15
 Matrix order=   20, type=26, seed=1465,2266,3976,3833, result  5 is 4.504D+15
 DGV drivers:     34 out of   1092 tests failed to pass the threshold
...
 Matrix order=    6, type=16, seed= 852,1170,4018,2529, result  5 is 4.504D+15
 Matrix order=    8, type=20, seed= 202,3818,2830,2373, result  5 is 4.504D+15
 Matrix order=    8, type=25, seed= 582,1680, 289,1745, result  5 is 4.504D+15
 Matrix order=   10, type= 8, seed= 573,2931,1057, 529, result  5 is 4.504D+15
 Matrix order=   10, type=10, seed= 308, 212,3333,  73, result  5 is 4.504D+15
 Matrix order=   10, type=13, seed=3036,2372,2403,2421, result  5 is 4.504D+15
 Matrix order=   15, type=12, seed=2288,1863,3505, 805, result  5 is 4.504D+15
 Matrix order=   15, type=13, seed=3415,2880,2965,2889, result  5 is 4.504D+15
 Matrix order=   15, type=16, seed= 681,2778,2914,2225, result  5 is 4.504D+15
 Matrix order=   15, type=17, seed= 304,3863, 350,1657, result  5 is 4.504D+15
 Matrix order=   15, type=21, seed=2362, 980, 534,3365, result  5 is 4.504D+15
 Matrix order=   15, type=25, seed=2888,1855, 900,2225, result  5 is 4.504D+15
 Matrix order=   20, type= 8, seed= 636, 119,  47,2269, result  5 is 4.504D+15
 Matrix order=   20, type= 9, seed=1007,2268,2913,  29, result  5 is 4.504D+15
 Matrix order=   20, type=10, seed= 104,1378,2159,1485, result  5 is 4.504D+15
 Matrix order=   20, type=11, seed= 927, 874,3720,2349, result  5 is 4.504D+15
 Matrix order=   20, type=14, seed=2659,2147, 552,1661, result  5 is 4.504D+15
 Matrix order=   20, type=21, seed=2012,3064,3728,2789, result  5 is 4.504D+15
 DGV drivers:     18 out of   1092 tests failed to pass the threshold
...
 N=  30 M=   0, P=   5, type  8, test  2, ratio=  555331.
 N=   3 M=   0, P=  20, type  8, test  2, ratio= 0.889025E+07
 N=  30 M=   0, P=  20, type  8, test  2, ratio= 0.219820E+07
 M=   3 P=   0, N=  30, type  7, test  1, ratio= 0.800042E+07
 N=  30 M=   3, P=   0, type  7, test  1, ratio= 0.772279E+07
 M=   3 P=   5, N=  30, type  7, test  1, ratio= 0.852669E+07
 N=  30 M=   3, P=   5, type  8, test  2, ratio= 0.119398E+07
 N=   3 M=   3, P=  20, type  8, test  2, ratio= 0.998240E+07
 N=  30 M=   3, P=  20, type  8, test  2, ratio=  976876.
 N=   3 M=  10, P=   5, type  8, test  2, ratio= 0.121475E+08
 N=  30 M=  10, P=   5, type  7, test  1, ratio= 0.127652E+08
 N=  30 M=  10, P=   5, type  8, test  2, ratio=  714258.
 M=  10 P=  20, N=  30, type  7, test  1, ratio= 0.510405E+07
 N=  30 M=  10, P=  20, type  8, test  2, ratio= 0.208493E+07
 GQR:     14 out of   1728 tests failed to pass the threshold
...
 Matrix order=   20, type=16, seed= 647,2328,1944, 557, result  5 is 8.389E+06
 CGV drivers:      1 out of   1092 tests failed to pass the threshold
...
 CHESV , UPLO='U', N =    3, type  2, test  1, ratio = 0.10874E+07
 CHESVX, FACT='N', UPLO='U', N =    3, type  2, test  1, ratio = 0.10874E+07
 CHESV , UPLO='U', N =    5, type  9, test  1, ratio = 0.32042E+07
 CHESVX, FACT='N', UPLO='U', N =    5, type  9, test  1, ratio = 0.32042E+07
 CHESV , UPLO='U', N =    5, type 10, test  1, ratio = 0.35394E+07
 CHESVX, FACT='N', UPLO='U', N =    5, type 10, test  1, ratio = 0.35394E+07
 CHESV , UPLO='U', N =   10, type  2, test  1, ratio = 0.16458E+07
 CHESVX, FACT='N', UPLO='U', N =   10, type  2, test  1, ratio = 0.16458E+07
 CHESV , UPLO='U', N =   10, type  7, test  1, ratio = 0.14266E+06
 CHESVX, FACT='N', UPLO='U', N =   10, type  7, test  1, ratio = 0.14266E+06
 CHESV , UPLO='U', N =   10, type  9, test  1, ratio = 0.31372E+07
 CHESVX, FACT='N', UPLO='U', N =   10, type  9, test  1, ratio = 0.31372E+07
 CHESV , UPLO='U', N =   10, type 10, test  1, ratio = 0.16814E+07
 CHESVX, FACT='N', UPLO='U', N =   10, type 10, test  1, ratio = 0.16814E+07
 CHESV , UPLO='U', N =   50, type  2, test  1, ratio = 0.86168E+07
 CHESVX, FACT='N', UPLO='U', N =   50, type  2, test  1, ratio = 0.86168E+07
 CHESV , UPLO='U', N =   50, type  7, test  1, ratio = 0.80869E+06
 CHESVX, FACT='N', UPLO='U', N =   50, type  7, test  1, ratio = 0.80869E+06
 CHESV , UPLO='U', N =   50, type  8, test  1, ratio = 0.52818E+06
 CHESVX, FACT='N', UPLO='U', N =   50, type  8, test  1, ratio = 0.52818E+06
 CHESV , UPLO='U', N =   50, type  9, test  1, ratio = 0.85655E+07
 CHESVX, FACT='N', UPLO='U', N =   50, type  9, test  1, ratio = 0.85655E+07
 CHESV , UPLO='U', N =   50, type 10, test  1, ratio = 0.32095E+07
 CHESVX, FACT='N', UPLO='U', N =   50, type 10, test  1, ratio = 0.32095E+07
 CHE drivers:     24 out of   1072 tests failed to pass the threshold
...
 Matrix order=   12, type=24, seed=2693,2404,3046,2957, result  5 is 4.504D+15
 Matrix order=   12, type=25, seed= 578, 723, 929,1637, result  5 is 4.504D+15
 Matrix order=   12, type=26, seed=2061,1512,1968, 125, result  5 is 4.504D+15
 Matrix order=   20, type=16, seed= 647,2328,1944, 557, result  5 is 4.504D+15
 Matrix order=   20, type=17, seed=1585,3902,3906,1293, result  5 is 4.504D+15
 Matrix order=   20, type=18, seed=1063,2113,3640,2685, result  5 is 4.504D+15
 Matrix order=   20, type=19, seed=2305,3879, 305, 381, result  5 is 4.504D+15
 Matrix order=   20, type=20, seed= 897,3616, 121,2173, result  5 is 4.504D+15
 Matrix order=   20, type=21, seed= 676,3851,3089,3965, result  5 is 4.504D+15
 Matrix order=   20, type=22, seed=1763,2855,1954,1469, result  5 is 4.504D+15
 Matrix order=   20, type=23, seed=1338,3437,3180,3285, result  5 is 4.504D+15
 Matrix order=   20, type=24, seed=3522,1685,3785, 813, result  5 is 4.504D+15
 Matrix order=   20, type=25, seed=2931,3978,3195,3781, result  5 is 4.504D+15
 Matrix order=   20, type=26, seed= 449,1038,1670,1437, result  5 is 4.504D+15
 ZGV drivers:     53 out of   1092 tests failed to pass the threshold

[casparvl]

Just for my future self: that file is also available from the host on

/mnt/shared/home/bot/eessi-bot-software-layer/jobs/2023.07/pr_309/event_8d3f0620-2624-11ee-870f-e48080cd3472/run_005/linux_aarch64_neoverse_v1/eessi-2023.06-software/previous_tmp/build_step/bot/easybuild/build/OpenBLAS/0.3.21/GCC-12.2.0/OpenBLAS-0.3.21/lapack-netlib/TESTING/testing_results.txt

No need to go into the container first...

[casparvl]

Gist with the full log is here

[casparvl]

Created upstream issue OpenMathLib/OpenBLAS#4187

[casparvl]

Docs about how to interpret LAPACK test results: https://netlib.org/lapack/faq.html#_how_do_i_interpret_lapack_testing_failures

[casparvl]

For Reference, on Neoverse_n1 we get some failuers as well, but way less. A manual build with

source /cvmfs/pilot.eessi-hpc.org/versions/2023.06/init/bash
module load EasyBuild
eb OpenBLAS-0.3.21-GCC-12.2.0.eb --disable-cleanup-builddir --disable-cleanup-tmpdir -r --rebuild

showed:

                        -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    1315683         21      (0.002%)        0       (0.000%)
DOUBLE PRECISION        1314777         30      (0.002%)        0       (0.000%)
COMPLEX                 776527          3       (0.000%)        0       (0.000%)
COMPLEX16               777338          1       (0.000%)        0       (0.000%)

--> ALL PRECISIONS      4184325         55      (0.001%)        0       (0.000%)

Detailed logs (lapack_tests.txt):

Matrix order=    8, type=25, seed= 889,2561,1319, 841, result  5 is 8.389E+06
 Matrix order=   10, type=17, seed=1420,1879,1628,  85, result  5 is 8.389E+06
 Matrix order=   10, type=24, seed=1210,3887,1474,2237, result  5 is 8.389E+06
 Matrix order=   10, type=26, seed=3592,3268,3086,1181, result  5 is 8.389E+06
 Matrix order=   15, type=18, seed=3991, 625,3539,2581, result  5 is 8.389E+06
 Matrix order=   15, type=20, seed=2155,2073,3686, 149, result  5 is 8.389E+06
 Matrix order=   15, type=21, seed=3823,2194,3510,3029, result  5 is 8.389E+06
 Matrix order=   15, type=24, seed=1487,3956,2976,3649, result  5 is 8.389E+06
 Matrix order=   15, type=25, seed=2605,2211,3982,2721, result  5 is 8.389E+06
 Matrix order=   15, type=26, seed=   8,1600,1726,2817, result  5 is 8.389E+06
 Matrix order=   20, type=16, seed= 904,3323, 291,1585, result  5 is 8.389E+06
 Matrix order=   20, type=17, seed= 723,4024,2193,2265, result  5 is 8.389E+06
 Matrix order=   20, type=18, seed= 828, 842,1787,3733, result  5 is 8.389E+06
 Matrix order=   20, type=20, seed=1904,2346,3199,2437, result  5 is 8.389E+06
 Matrix order=   20, type=21, seed=4066,  26,1253, 221, result  5 is 8.389E+06
 Matrix order=   20, type=22, seed=3234,3664,3237,3473, result  5 is 8.389E+06
 Matrix order=   20, type=23, seed= 481, 957,3230,3689, result  5 is 8.389E+06
 Matrix order=   20, type=24, seed=2871,2618,1987,2949, result  5 is 8.389E+06
 Matrix order=   20, type=25, seed=2604,3233, 644,3529, result  5 is 8.389E+06
 SGV drivers:     19 out of   1092 tests failed to pass the threshold
...
Matrix order=    8, type=25, seed= 582,1680, 289,1745, result  5 is 8.389E+06
 Matrix order=   15, type=17, seed= 304,3863, 350,1657, result  5 is 8.389E+06
 SGV drivers:      2 out of   1092 tests failed to pass the threshold
...
 Matrix order=   10, type=24, seed=1210,3887,1474,2237, result  5 is 4.504D+15
 Matrix order=   15, type=20, seed=2155,2073,3686, 149, result  5 is 4.504D+15
 Matrix order=   15, type=21, seed=3823,2194,3510,3029, result  5 is 4.504D+15
 Matrix order=   15, type=24, seed=1487,3956,2976,3649, result  5 is 4.504D+15
 Matrix order=   15, type=25, seed=2605,2211,3982,2721, result  5 is 4.504D+15
 Matrix order=   20, type=12, seed= 112,1059,2403,2121, result  5 is 4.504D+15
 Matrix order=   20, type=17, seed= 723,4024,2193,2265, result  5 is 4.504D+15
 Matrix order=   20, type=18, seed= 828, 842,1787,3733, result  5 is 4.504D+15
 Matrix order=   20, type=20, seed=1904,2346,3199,2437, result  5 is 4.504D+15
 Matrix order=   20, type=24, seed=2871,2618,1987,2949, result  5 is 4.504D+15
 DGV drivers:     10 out of   1092 tests failed to pass the threshold
...
 Matrix order=   15, type=17, seed= 304,3863, 350,1657, result  5 is 4.504D+15
 Matrix order=   15, type=21, seed=2362, 980, 534,3365, result  5 is 4.504D+15
 Matrix order=   15, type=25, seed=2888,1855, 900,2225, result  5 is 4.504D+15
 Matrix order=   20, type=16, seed=3366,1285,3695, 789, result  5 is 4.504D+15
 Matrix order=   20, type=19, seed=1811,3312,1916,3573, result  5 is 4.504D+15
 Matrix order=   20, type=21, seed=2012,3064,3728,2789, result  5 is 4.504D+15
 DGV drivers:      6 out of   1092 tests failed to pass the threshold
...
N=  30 M=   0, P=   5, type  8, test  2, ratio=  555331.
 N=   3 M=   0, P=  20, type  8, test  2, ratio= 0.889025E+07
 N=  30 M=   0, P=  20, type  8, test  2, ratio= 0.219820E+07
 M=   3 P=   0, N=  30, type  7, test  1, ratio= 0.800042E+07
 N=  30 M=   3, P=   0, type  7, test  1, ratio= 0.772279E+07
 M=   3 P=   5, N=  30, type  7, test  1, ratio= 0.852669E+07
 N=  30 M=   3, P=   5, type  8, test  2, ratio= 0.119398E+07
 N=   3 M=   3, P=  20, type  8, test  2, ratio= 0.998240E+07
 N=  30 M=   3, P=  20, type  8, test  2, ratio=  976876.
 N=   3 M=  10, P=   5, type  8, test  2, ratio= 0.121475E+08
 N=  30 M=  10, P=   5, type  7, test  1, ratio= 0.127652E+08
 N=  30 M=  10, P=   5, type  8, test  2, ratio=  714258.
 M=  10 P=  20, N=  30, type  7, test  1, ratio= 0.510405E+07
 N=  30 M=  10, P=  20, type  8, test  2, ratio= 0.208493E+07
 GQR:     14 out of   1728 tests failed to pass the threshold
...
Matrix order=   10, type=23, seed=2605, 629,2321,1285, result  5 is 8.389E+06
 Matrix order=   12, type=19, seed=1401, 178,4093, 253, result  5 is 8.389E+06
 Matrix order=   20, type=19, seed=2305,3879, 305, 381, result  5 is 8.389E+06
 CGV drivers:      3 out of   1092 tests failed to pass the threshold
...
 Matrix order=   20, type=16, seed= 647,2328,1944, 557, result  5 is 4.504D+15
 ZGV drivers:      1 out of   1092 tests failed to pass the threshold

 All tests for ZGV drivers  passed the threshold (   1092 tests run)
...

[juntangc]

How much performance benefit does -ftree-vectorize offer? If you take it out for neoversev1, the number of errors will be 110 instead of 344.

			-->   LAPACK TESTING SUMMARY  <--
SUMMARY             	nb test run 	numerical error   	other error  
================   	===========	=================	================  
REAL             	1317867		0	(0.000%)	0	(0.000%)	
DOUBLE PRECISION	1316961		14	(0.001%)	0	(0.000%)	
COMPLEX          	774701		96	(0.012%)	0	(0.000%)	
COMPLEX16         	778430		0	(0.000%)	0	(0.000%)	

--> ALL PRECISIONS	4187959		110	(0.003%)	0	(0.000%)