Circular link dependency error on Windows in 5.3.0

[alexchandel]

Building with VS 2019 as following fails, with CBLAS complaining of unresolved external symbols from SuperLU

mkdir build ; cd build
cmake .. -Denable_internal_blaslib=YES -DXSDK_ENABLE_Fortran=FALSE
msbuild SuperLU.sln

Build Error

  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\link.exe /ER
  RORREPORT:QUEUE /OUT:"C:\Users\Dev\Git\superlu\build\CBLAS\Debug\blas.dll" /INCREMENTAL /ILK:"blas.dir\Debug\blas.ilk
  " /NOLOGO kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib adv
  api32.lib /MANIFEST /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /DEBUG /PDB:"C:/Users/Dev/Git/s
  uperlu/build/CBLAS/Debug/blas.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:/Users/Dev/Git/super
  lu/build/CBLAS/Debug/blas.lib" /MACHINE:X64  /machine:x64 /DLL blas.dir\Debug\isamax.obj
  blas.dir\Debug\sasum.obj
  blas.dir\Debug\saxpy.obj
  blas.dir\Debug\scopy.obj
  blas.dir\Debug\sdot.obj
  blas.dir\Debug\snrm2.obj
  blas.dir\Debug\srot.obj
  blas.dir\Debug\sscal.obj
  blas.dir\Debug\sswap.obj
  blas.dir\Debug\sgemv.obj
  blas.dir\Debug\ssymv.obj
  blas.dir\Debug\strsv.obj
  blas.dir\Debug\sger.obj
  blas.dir\Debug\ssyr2.obj
  blas.dir\Debug\idamax.obj
  blas.dir\Debug\dasum.obj
  blas.dir\Debug\daxpy.obj
  blas.dir\Debug\dcopy.obj
  blas.dir\Debug\ddot.obj
  blas.dir\Debug\dnrm2.obj
  blas.dir\Debug\drot.obj
  blas.dir\Debug\dscal.obj
  blas.dir\Debug\dswap.obj
  blas.dir\Debug\dgemv.obj
  blas.dir\Debug\dsymv.obj
  blas.dir\Debug\dtrsv.obj
  blas.dir\Debug\dger.obj
  blas.dir\Debug\dsyr2.obj
  blas.dir\Debug\icamax.obj
  blas.dir\Debug\scasum.obj
  blas.dir\Debug\caxpy.obj
  blas.dir\Debug\ccopy.obj
  blas.dir\Debug\scnrm2.obj
  blas.dir\Debug\cscal.obj
  blas.dir\Debug\cswap.obj
  blas.dir\Debug\cdotc.obj
  blas.dir\Debug\cgemv.obj
  blas.dir\Debug\chemv.obj
  blas.dir\Debug\ctrsv.obj
  blas.dir\Debug\cgerc.obj
  blas.dir\Debug\cher2.obj
  blas.dir\Debug\izamax.obj
  blas.dir\Debug\dzasum.obj
  blas.dir\Debug\zaxpy.obj
  blas.dir\Debug\zcopy.obj
  blas.dir\Debug\dznrm2.obj
  blas.dir\Debug\zscal.obj
  blas.dir\Debug\dcabs1.obj
  blas.dir\Debug\zswap.obj
  blas.dir\Debug\zdotc.obj
  blas.dir\Debug\zgemv.obj
  blas.dir\Debug\zhemv.obj
  blas.dir\Debug\ztrsv.obj
  blas.dir\Debug\zgerc.obj
  blas.dir\Debug\zher2.obj
zhemv.obj : error LNK2019: unresolved external symbol input_error referenced in function zhemv_ [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
ztrsv.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
zgerc.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
zher2.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
ctrsv.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
cgerc.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
cher2.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
zgemv.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
dger.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
dsyr2.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
cgemv.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
chemv.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
ssyr2.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
dgemv.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
dsymv.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
dtrsv.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
sgemv.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
ssymv.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
strsv.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
sger.obj : error LNK2001: unresolved external symbol input_error [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
icamax.obj : error LNK2019: unresolved external symbol r_imag referenced in function icamax_ [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
scasum.obj : error LNK2001: unresolved external symbol r_imag [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
caxpy.obj : error LNK2001: unresolved external symbol r_imag [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
scnrm2.obj : error LNK2001: unresolved external symbol r_imag [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
cgerc.obj : error LNK2019: unresolved external symbol r_cnjg referenced in function cgerc_ [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
cher2.obj : error LNK2001: unresolved external symbol r_cnjg [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
cdotc.obj : error LNK2001: unresolved external symbol r_cnjg [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
cgemv.obj : error LNK2001: unresolved external symbol r_cnjg [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
chemv.obj : error LNK2001: unresolved external symbol r_cnjg [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
ctrsv.obj : error LNK2001: unresolved external symbol r_cnjg [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
ctrsv.obj : error LNK2019: unresolved external symbol c_div referenced in function ctrsv_ [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
dznrm2.obj : error LNK2019: unresolved external symbol d_imag referenced in function dznrm2_ [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
zgerc.obj : error LNK2019: unresolved external symbol d_cnjg referenced in function zgerc_ [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
zher2.obj : error LNK2001: unresolved external symbol d_cnjg [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
zdotc.obj : error LNK2001: unresolved external symbol d_cnjg [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
zgemv.obj : error LNK2001: unresolved external symbol d_cnjg [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
zhemv.obj : error LNK2001: unresolved external symbol d_cnjg [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
ztrsv.obj : error LNK2001: unresolved external symbol d_cnjg [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
ztrsv.obj : error LNK2019: unresolved external symbol z_div referenced in function ztrsv_ [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
C:\Users\Dev\Git\superlu\build\CBLAS\Debug\blas.dll : fatal error LNK1120: 7 unresolved externals [C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj]
Done Building Project "C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj" (default targets) -- FAILED.
Done Building Project "C:\Users\Dev\Git\superlu\build\CBLAS\blas.vcxproj.metaproj" (default targets) -- FAILED.
Done Building Project "C:\Users\Dev\Git\superlu\build\ALL_BUILD.vcxproj.metaproj" (default targets) -- FAILED.
Done Building Project "C:\Users\Dev\Git\superlu\build\SuperLU.sln" (default targets) -- FAILED.
Build FAILED.

I tried msbuild CBLAS\blas.vcxproj and msbuild -p:BuildProjectReferences=false CBLAS\blas.vcxproj, but the same error happens.

I also tried msbuild -p:BuildProjectReferences=false SRC\superlu.vcxproj, which almost compiled but failed to link blas.lib (as it wasn't built):

"C:\Users\Dev\Git\superlu\build\SRC\superlu.vcxproj" (default target) (1) ->
(Link target) ->
  LINK : fatal error LNK1104: cannot open file '..\CBLAS\Debug\blas.lib' [C:\Users\Dev\Git\superlu\build\SRC\superlu.vc
xproj]

Note that the last two failures ONLY happen on 5.3.0. On 5.2.1, the following commands succeed:

cmake .. -Denable_internal_blaslib=YES
msbuild -p:BuildProjectReferences=false SRC\superlu.vcxproj
msbuild -p:BuildProjectReferences=false CBLAS\blas.vcxproj

[tcaduser]

My guess is that input_error is supposed to be the equivalent of the famous BLAS function, xerbla, and is provided in many BLAS implementations . Windows linking is very difficult with circular references. I build SuperLU 5.3 on windows as a static libraries without any blas, by tricking it with -DBLAS_FOUND=ON on the cmake command line.

cmake  -DCMAKE_C_FLAGS="/fp:strict" -Denable_complex=OFF -Denable_single=OFF -Denable_doc=OFF -Denable_tests=OFF -DXSDK_ENABLE_Fortran=OFF -DBLAS_FOUND=ON ..

Then when I link the application, I choose my favorite blas implementation, which hopefully has its own self contained error handling.

If you must have dynamic libraries and want the CBLAS, I suggest copying SRC/input_error.c into CBLAS and adding it into the CBLAS/CMakeLists.txt.

[xiaoyeli]

I think it's a good suggestion from @tcaduser .
I updated the master branch. This will be in the next release.

[alexchandel]

Is that sufficient? There are several other unresolved external symbols beyond input_error, like r_imag, r_cnjg, d_imag, d_cnjg, c_div, z_div.

Also, why did this only begin in 5.3.0? Building in 5.2.1 succeeds. Was the code significantly rearchitected?

[alexchandel]

@xiaoyeli adding input_error.c to CBLAS/CMakeLists.txt seems to break the build.

[alexchandel]

Working through the build, this isn't limited to input_error(), or to Windows. CBLAS references a half-dozen symbols defined in superlu, leading to link errors:

$ VERBOSE=1 cmake .. -Denable_internal_blaslib=YES && cmake --build .                                   
Re-run cmake no build system arguments

Process XSDK defaults ...
USE_XSDK_DEFAULTS = 'TRUE'
-- XSDK: Setting default BUILD_SHARED_LIBS=TRUE
-- XSDK: Setting default CMAKE_BUILD_TYPE=DEBUG
-- SuperLU will be built as a shared library.
-- The C compiler identification is AppleClang 14.0.0.14000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- The Fortran compiler identification is GNU 12.2.0
-- Checking whether Fortran compiler has -isysroot
-- Checking whether Fortran compiler has -isysroot - yes
-- Checking whether Fortran compiler supports OSX deployment target flag
-- Checking whether Fortran compiler supports OSX deployment target flag - yes
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /usr/local/bin/gfortran - skipped
-- Did not find or specify BLAS so configure to build internal CBLAS ...
-- Configuring done
CMake Warning (dev):
  Policy CMP0042 is not set: MACOSX_RPATH is enabled by default.  Run "cmake
  --help-policy CMP0042" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  MACOSX_RPATH is not specified for the following targets:

   blas
   matgen
   superlu

This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /Users/dev/Git/superlu/build
[  0%] Building C object CBLAS/CMakeFiles/blas.dir/input_error.c.o
[  0%] Building C object CBLAS/CMakeFiles/blas.dir/isamax.c.o
[  0%] Building C object CBLAS/CMakeFiles/blas.dir/sasum.c.o
[  1%] Building C object CBLAS/CMakeFiles/blas.dir/saxpy.c.o
[  1%] Building C object CBLAS/CMakeFiles/blas.dir/scopy.c.o
[  1%] Building C object CBLAS/CMakeFiles/blas.dir/sdot.c.o
[  2%] Building C object CBLAS/CMakeFiles/blas.dir/snrm2.c.o
[  2%] Building C object CBLAS/CMakeFiles/blas.dir/srot.c.o
[  2%] Building C object CBLAS/CMakeFiles/blas.dir/sscal.c.o
[  2%] Building C object CBLAS/CMakeFiles/blas.dir/sswap.c.o
[  3%] Building C object CBLAS/CMakeFiles/blas.dir/sgemv.c.o
[  3%] Building C object CBLAS/CMakeFiles/blas.dir/ssymv.c.o
[  3%] Building C object CBLAS/CMakeFiles/blas.dir/strsv.c.o
[  4%] Building C object CBLAS/CMakeFiles/blas.dir/sger.c.o
[  4%] Building C object CBLAS/CMakeFiles/blas.dir/ssyr2.c.o
[  4%] Building C object CBLAS/CMakeFiles/blas.dir/idamax.c.o
[  4%] Building C object CBLAS/CMakeFiles/blas.dir/dasum.c.o
[  5%] Building C object CBLAS/CMakeFiles/blas.dir/daxpy.c.o
[  5%] Building C object CBLAS/CMakeFiles/blas.dir/dcopy.c.o
[  5%] Building C object CBLAS/CMakeFiles/blas.dir/ddot.c.o
[  6%] Building C object CBLAS/CMakeFiles/blas.dir/dnrm2.c.o
[  6%] Building C object CBLAS/CMakeFiles/blas.dir/drot.c.o
[  6%] Building C object CBLAS/CMakeFiles/blas.dir/dscal.c.o
[  6%] Building C object CBLAS/CMakeFiles/blas.dir/dswap.c.o
[  7%] Building C object CBLAS/CMakeFiles/blas.dir/dgemv.c.o
[  7%] Building C object CBLAS/CMakeFiles/blas.dir/dsymv.c.o
[  7%] Building C object CBLAS/CMakeFiles/blas.dir/dtrsv.c.o
[  8%] Building C object CBLAS/CMakeFiles/blas.dir/dger.c.o
[  8%] Building C object CBLAS/CMakeFiles/blas.dir/dsyr2.c.o
[  8%] Building C object CBLAS/CMakeFiles/blas.dir/icamax.c.o
[  8%] Building C object CBLAS/CMakeFiles/blas.dir/scasum.c.o
[  9%] Building C object CBLAS/CMakeFiles/blas.dir/caxpy.c.o
[  9%] Building C object CBLAS/CMakeFiles/blas.dir/ccopy.c.o
[  9%] Building C object CBLAS/CMakeFiles/blas.dir/scnrm2.c.o
[ 10%] Building C object CBLAS/CMakeFiles/blas.dir/cscal.c.o
[ 10%] Building C object CBLAS/CMakeFiles/blas.dir/cswap.c.o
[ 10%] Building C object CBLAS/CMakeFiles/blas.dir/cdotc.c.o
[ 10%] Building C object CBLAS/CMakeFiles/blas.dir/cgemv.c.o
[ 11%] Building C object CBLAS/CMakeFiles/blas.dir/chemv.c.o
[ 11%] Building C object CBLAS/CMakeFiles/blas.dir/ctrsv.c.o
[ 11%] Building C object CBLAS/CMakeFiles/blas.dir/cgerc.c.o
[ 12%] Building C object CBLAS/CMakeFiles/blas.dir/cher2.c.o
[ 12%] Building C object CBLAS/CMakeFiles/blas.dir/izamax.c.o
[ 12%] Building C object CBLAS/CMakeFiles/blas.dir/dzasum.c.o
[ 13%] Building C object CBLAS/CMakeFiles/blas.dir/zaxpy.c.o
[ 13%] Building C object CBLAS/CMakeFiles/blas.dir/zcopy.c.o
[ 13%] Building C object CBLAS/CMakeFiles/blas.dir/dznrm2.c.o
[ 13%] Building C object CBLAS/CMakeFiles/blas.dir/zscal.c.o
[ 14%] Building C object CBLAS/CMakeFiles/blas.dir/dcabs1.c.o
[ 14%] Building C object CBLAS/CMakeFiles/blas.dir/zswap.c.o
[ 14%] Building C object CBLAS/CMakeFiles/blas.dir/zdotc.c.o
[ 15%] Building C object CBLAS/CMakeFiles/blas.dir/zgemv.c.o
[ 15%] Building C object CBLAS/CMakeFiles/blas.dir/zhemv.c.o
[ 15%] Building C object CBLAS/CMakeFiles/blas.dir/ztrsv.c.o
[ 15%] Building C object CBLAS/CMakeFiles/blas.dir/zgerc.c.o
[ 16%] Building C object CBLAS/CMakeFiles/blas.dir/zher2.c.o
[ 16%] Linking C shared library libblas.dylib
Undefined symbols for architecture x86_64:
  "_c_div", referenced from:
      _ctrsv_ in ctrsv.c.o
  "_d_cnjg", referenced from:
      _zdotc_ in zdotc.c.o
      _zgemv_ in zgemv.c.o
      _zhemv_ in zhemv.c.o
      _ztrsv_ in ztrsv.c.o
      _zgerc_ in zgerc.c.o
      _zher2_ in zher2.c.o
  "_d_imag", referenced from:
      _dznrm2_ in dznrm2.c.o
  "_r_cnjg", referenced from:
      _cdotc_ in cdotc.c.o
      _cgemv_ in cgemv.c.o
      _chemv_ in chemv.c.o
      _ctrsv_ in ctrsv.c.o
      _cgerc_ in cgerc.c.o
      _cher2_ in cher2.c.o
  "_r_imag", referenced from:
      _icamax_ in icamax.c.o
      _scasum_ in scasum.c.o
      _caxpy_ in caxpy.c.o
      _scnrm2_ in scnrm2.c.o
  "_z_div", referenced from:
      _ztrsv_ in ztrsv.c.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CBLAS/libblas.dylib] Error 1
make[1]: *** [CBLAS/CMakeFiles/blas.dir/all] Error 2
make: *** [all] Error 2