LAPACK test failures with NVHPC 23.7

[Crivella]

Hi,

I am trying to build OpenBLAS using

• NVHPC 23.7
• CUDA 12.1.1

Environment variables:

• CFLAGS -O2 -tp=host -Mflushz
• CXXFLAGS -O2 -tp=host -Mflushz
• FCFLAGS -O2 -tp=host -Mflushz
• FFLAGS -O2 -tp=host -Mflushz
• F90LAGS -O2 -tp=host -Mflushz

while running the LAPACK test suite with

make lapack-test  BINARY='64'  CC='nvc'  FC='nvfortran'  MAKE_NB_JOBS='-1'  USE_OPENMP='1'  USE_THREAD='1'

after a minor patch to the Makefile inside lapack-netlib/INSTALL/Makefile by adding -Mnomain to the testlsame and testsecond build recipes (to solve a multiple definition of `main' error) (Not sure if this is worth a separate issue, if the test there is wrongly compiling C sources for Fortran exes or if it is supposed to be like that and the compiler would normally automatically disable the fortran main when needed)
i was able to get it to complete but i get several numerical (and also other) failures tested with the following versions of OpenBLAS.

• 0.3.23

			-->   LAPACK TESTING SUMMARY  <--
SUMMARY             	nb test run 	numerical error   	other error  
================   	===========	=================	================  
REAL             	1108743		286	(0.026%)	0	(0.000%)	
DOUBLE PRECISION	1255005		280	(0.022%)	0	(0.000%)	
COMPLEX          	762259		153	(0.020%)	0	(0.000%)	
COMPLEX16         	699834		350	(0.050%)	0	(0.000%)	

--> ALL PRECISIONS	3825841		1069	(0.028%)	0	(0.000%)

• 0.3.24

			-->   LAPACK TESTING SUMMARY  <--
SUMMARY             	nb test run 	numerical error   	other error  
================   	===========	=================	================  
REAL             	1108743		286	(0.026%)	0	(0.000%)	
DOUBLE PRECISION	1255005		280	(0.022%)	0	(0.000%)	
COMPLEX          	746859		189	(0.025%)	0	(0.000%)	
COMPLEX16         	684434		386	(0.056%)	0	(0.000%)	

--> ALL PRECISIONS	3795041		1141	(0.030%)	0	(0.000%)	

• 0.3.27

			-->   LAPACK TESTING SUMMARY  <--
SUMMARY             	nb test run 	numerical error   	other error  
================   	===========	=================	================  
REAL             	1108743		37171	(3.353%)	0	(0.000%)	
DOUBLE PRECISION	1255005		37165	(2.961%)	0	(0.000%)	
COMPLEX          	762259		1448	(0.190%)	0	(0.000%)	
COMPLEX16         	671348		37391	(5.570%)	14	(0.002%)	

--> ALL PRECISIONS	3797355		113175	(2.980%)	14	(0.000%)

[martin-frbg]

What is your cpu here (and does it help if you leave out the -tp host and/or -Mflushz) ? The LAPACK testsuite is a bit fragile with respect to optimizations, but seeing this many deviations is unusual.

[Crivella]

I am running the tests on a system with a

• AMD EPYC 7452 (Zen2)
• NVIDIA A100

Indeed testing the various combination of optimization flags:

• vanilla = -O2
• tphost = -O2 -tp-host
• flushz= -O2 -Mflushz

Atleast for v0.3.23 I am getting significantly less failures (with all recipes, so I guess in that case it was the combination of tp=host and -Mflushz doing it)

Still for v0.3.27 I am still getting a significant amount of failures also with default -O2.

I've also launched a test compilation specifically with tp=zen to see if it makes a difference

./0.3.23/vanilla/OpenBLAS-0.3.23/build_lapack-test.log                                                                                                                                                                                           [0/2114]
                        -->   LAPACK TESTING SUMMARY  <--                                                                
SUMMARY                 nb test run     numerical error         other error                                              
================        ===========     =================       ================                                         
REAL                    1326099         12      (0.001%)        0       (0.000%)                                         
DOUBLE PRECISION        1326921         36      (0.003%)        0       (0.000%)                                         
COMPLEX                 778063          6       (0.001%)        0       (0.000%)                                         
COMPLEX16               786918          22      (0.003%)        0       (0.000%)                                         
                                                                                                                         
--> ALL PRECISIONS      4218001         76      (0.002%)        0       (0.000%)                

./0.3.23/flushz/OpenBLAS-0.3.23/build_lapack-test.log                                                                    
                        -->   LAPACK TESTING SUMMARY  <--                                                                
SUMMARY                 nb test run     numerical error         other error                                              
================        ===========     =================       ================                                         
REAL                    1326099         12      (0.001%)        0       (0.000%)                                         
DOUBLE PRECISION        1326921         36      (0.003%)        0       (0.000%)                                         
COMPLEX                 778063          6       (0.001%)        0       (0.000%)                                         
COMPLEX16               786918          22      (0.003%)        0       (0.000%)                                         
                                                                                                                         
--> ALL PRECISIONS      4218001         76      (0.002%)        0       (0.000%)         

./0.3.23/tphost/OpenBLAS-0.3.23/build_lapack-test.log                                                                    
                        -->   LAPACK TESTING SUMMARY  <--                                                                
SUMMARY                 nb test run     numerical error         other error                                              
================        ===========     =================       ================                                         
REAL                    1328283         0       (0.000%)        0       (0.000%)                                         
DOUBLE PRECISION        1329105         0       (0.000%)        0       (0.000%)                                         
COMPLEX                 779587          24      (0.003%)        0       (0.000%)                                         
COMPLEX16               780654          94      (0.012%)        0       (0.000%)                                         

--> ALL PRECISIONS      4217629         118     (0.003%)        0       (0.000%)            

./0.3.27/vanilla/OpenBLAS-0.3.27/build_lapack-test.log        
                        -->   LAPACK TESTING SUMMARY  <--     
SUMMARY                 nb test run     numerical error         other error                                                 
================        ===========     =================       ================                                            
REAL                    1326099         36897   (2.782%)        0       (0.000%)                                            
DOUBLE PRECISION        1326921         36921   (2.782%)        0       (0.000%)                                            
COMPLEX                 778063          26      (0.003%)        0       (0.000%)                                            
COMPLEX16               753260          37090   (4.924%)        18      (0.002%)                                            

--> ALL PRECISIONS      4184343         110934  (2.651%)        18      (0.000%)                       

./0.3.27/flushz/OpenBLAS-0.3.27/build_lapack-test.log         
                        -->   LAPACK TESTING SUMMARY  <--     
SUMMARY                 nb test run     numerical error         other error                                                 
================        ===========     =================       ================                                            
REAL                    1326099         36897   (2.782%)        0       (0.000%)                                            
DOUBLE PRECISION        1326921         36921   (2.782%)        0       (0.000%)                                            
COMPLEX                 778063          36891   (4.741%)        0       (0.000%)                                            
COMPLEX16               753260          37090   (4.924%)        18      (0.002%)                                            

--> ALL PRECISIONS      4184343         147799  (3.532%)        18      (0.000%)                                      

./0.3.27/tphost/OpenBLAS-0.3.27/build_lapack-test.log         
                        -->   LAPACK TESTING SUMMARY  <--     
SUMMARY                 nb test run     numerical error         other error                                                 
================        ===========     =================       ================                                            
REAL                    1328283         36885   (2.777%)        0       (0.000%)                                            
DOUBLE PRECISION        1329105         36885   (2.775%)        0       (0.000%)                                            
COMPLEX                 779587          16929   (2.172%)        0       (0.000%)                                            
COMPLEX16               752168          37135   (4.937%)        14      (0.002%)                                            

--> ALL PRECISIONS      4189143         127834  (3.052%)        14      (0.000%) 

[Crivella]

As an updated this is with -O2 -tp=zen, so i guess -tp=host is picking up the correct architecture as it gives the same result

                        -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error         other error  
================        ===========     =================       ================  
REAL                    1328283         36885   (2.777%)        0       (0.000%)
DOUBLE PRECISION        1329105         36885   (2.775%)        0       (0.000%)
COMPLEX                 779587          16929   (2.172%)        0       (0.000%)
COMPLEX16               752168          37135   (4.937%)        14      (0.002%)

--> ALL PRECISIONS      4189143         127834  (3.052%)        14      (0.000%)

[martin-frbg]

Test results are basically the same when one builds https://github.com/Reference-LAPACK/LAPACK (from which almost all LAPACK in OpenBLAS is imported) with NVHPC - the big jump in numerical errors comes about from the addition of the ?GEQP3RK functions ( truncated QR with column pivoting, Reference-LAPACK/lapack#891 ) in LAPACK-3.12 that was picked up by OpenBLAS 0.3.26

[martin-frbg]

?GEQP3RK failures can probably be fixed by applying https://patch-diff.githubusercontent.com/raw/Reference-LAPACK/lapack/pull/961.diff (which I thought I had merged already, but apparently hadn't - note that the PR is still open as there appear to be more problems with these tests)

[martin-frbg]

Testing results with NVHPC 24.3 (and the default -tp=px) on Zen2 after applying the current version of PR 961

                        -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error         other error  
================        ===========     =================       ================  
REAL                    1347924         298     (0.022%)        0       (0.000%)
DOUBLE PRECISION        1252821         58584   (4.676%)        0       (0.000%)
COMPLEX                 1002430         159     (0.016%)        0       (0.000%)
COMPLEX16               914135          425     (0.046%)        18      (0.002%)

--> ALL PRECISIONS      4517310         59466   (1.316%)        18      (0.000%)

and

                        -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error         other error  
================        ===========     =================       ================  
REAL                    1347924         298     (0.022%)        0       (0.000%)
DOUBLE PRECISION        1494186         316     (0.021%)        0       (0.000%)
COMPLEX                 959365          159     (0.017%)        0       (0.000%)
COMPLEX16               914135          425     (0.046%)        18      (0.002%)

--> ALL PRECISIONS      4715610         1198    (0.025%)        18      (0.000%)

after fixing the obviously unintentional commenting of the fix for the double precision case, as mentioned in the discussion thread for this PR)

[martin-frbg]

Corresponding results with GCC 13.2

                        -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error         other error  
================        ===========     =================       ================  
REAL                    1569648         0       (0.000%)        0       (0.000%)
DOUBLE PRECISION        1570470         0       (0.000%)        0       (0.000%)
COMPLEX                 979363          5       (0.001%)        0       (0.000%)
COMPLEX16               994955          91      (0.009%)        62      (0.006%)

--> ALL PRECISIONS      5114436         96      (0.002%)        62      (0.001%)

and some of these failures may be related to further inconsistencies in the test that are addressed by Reference-LAPACK PR 1005

[Crivella]

I just tested with the mentioned patch (both as is and without the 3 commented lines and i can confirm i can reproduce the lower number of errors.
Thanks, fell free to close the issue or keep it open to track the merging of the LAPACK PR

Patch as is

                        -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error         other error  
================        ===========     =================       ================  
REAL                    1569648         0       (0.000%)        0       (0.000%)
DOUBLE PRECISION        1329105         58268   (4.384%)        0       (0.000%)
COMPLEX                 1021282         24      (0.002%)        0       (0.000%)
COMPLEX16               993863          250     (0.025%)        14      (0.001%)

--> ALL PRECISIONS      4913898         58542   (1.191%)        14      (0.000%)

uncommented lines

                        -->   LAPACK TESTING SUMMARY  <--
SUMMARY                 nb test run     numerical error         other error  
================        ===========     =================       ================  
REAL                    1569648         0       (0.000%)        0       (0.000%)
DOUBLE PRECISION        1570470         0       (0.000%)        0       (0.000%)
COMPLEX                 1021282         24      (0.002%)        0       (0.000%)
COMPLEX16               993863          250     (0.025%)        14      (0.001%)

--> ALL PRECISIONS      5155263         274     (0.005%)        14      (0.000%)