binaries to output as libblas.dll and liblapack.dll

[fommil]

Can you please output separate dynamic libraries for your project:

• libblas3.dll containing BLAS and CBLAS
• liblapack3.dll containing LAPACK (and no LAPACKE, in keeping with industry norms)

instead of bundling it all into a single DLL. This allows upstream developers to swap the BLAS/LAPACK implementation at runtime to OpenBLAS (if compiled against the reference implementation, for example) or to change to another implementation for performance reasons (e.g. to cuBLAS for the BLAS portion).

This is directly impacting me in fommil/netlib-java#27 on Windows, but it would appear that Linux distribution managers already take care of this.

[xianyi]

Hi @fommil ,

Thank you for the suggestion.

For BLAS, it's no problem. However, OpenBLAS optimized some functions in LAPACK, which called OpenBLAS internal functions. When this separate liblapack3.dll links other BLAS library (e.g. cuBLAS or ATLAS), it will cause the error.

Xianyi

[fommil]

so create libblas and liblapack which dynamically load libopenblas (or inline your functions). the important thing is that you stick to the industry normed api :-)

[fommil]

@xianyi it also seems that you require a few more libraries to be available at runtime that most people wouldn't have:

86_64-w64-mingw32-objdump -p /opt/win-x86_64/openblas/lib/libopenblas.dll | grep DLL
    DLL
 vma:            Hint    Time      Forward  DLL       First
    DLL Name: ADVAPI32.dll
    DLL Name: libgfortran-3.dll
    DLL Name: KERNEL32.dll
    DLL Name: msvcrt.dll

libgfortran-3.dll can be included statically... have a look at how I do this with MinGW and I'm not 100% sure if modern machines actually have ADVAPI32.dll. I'm certainly getting link problems on Windows 8 64 bit.

[fommil]

AND, libgfortran-1-3.dll depends on libquadmath-0.dll and libgcc_s_seh-1.dll :-P

I use these MinGW command linker parameters on 64 bit

https://github.com/fommil/netlib-java/blob/master/native_system/win-x86_64/pom.xml#L90

and these on 32 bit

https://github.com/fommil/netlib-java/blob/master/native_system/win-i686/pom.xml#L90

in my project.

NOTE the bizarre -Wl,--kill-at on 32bit Windows! Essential to ensure your methods are named as you think they are.

[kkofler]

FYI, unfortunately, it looks like ATLAS is also using different library names these days. See this discussion:
https://lists.fedoraproject.org/pipermail/devel/2013-September/189467.html

For GNU/Linux distributions, different library names mean that the BLAS/LAPACK implementation has to be picked at compile time, and that build systems of dependent packages may need patching to support the new names.

[susilehtola]

fommil: libgfortran-1-3.dll, libquadmath-0.dll and libgcc_s_seh-1.dll are gcc runtime libraries. There's no sense in linking them statically, since otherwise the binary size will explode. You'll still have them on any MinGW system.

[fommil]

@kkofler that isn't true for debian and I doubt it ever will be.

@susilehtola the benefit is to the end user. these are small libs.

[susilehtola]

That'd be 1.5MB extra for each binary / library. I don't think it's that small. There's no sense in duplicating system libraries, because you'll have to recompile whenever bug fixes are made and the system libraries are always available.

I'm all for maintaining the current system of having a single library, since only then can you get consistent behavior. As a package maintainer I want to be able to select the library to be used, and not some frankensystem where the output is different for different users.

[susilehtola]

Case in point: parallel software. This would be totally broken if the undelying library can change, e.g. serial <-> pthreads <-> openmp.

[kkofler]

Well, I'm also a Fedora packager, and I think that, as long as we want to support multiple implementations of BLAS/LAPACK, the system which has been used for ATLAS in Fedora until now (i.e. the implementations all have the same sonames, and ld.so.conf.d is used to tell the linker to pick a specific implementation at runtime) is much better. For us packagers, it means that we only have to BuildRequire lapack-devel and not have to worry about what the most optimized implementation available for the given (possibly secondary) architecture is. For the end users, it means that they can install the most optimized implementation we provide for their CPU, or even build their own (tuned for the exact properties of their machine), without having to recompile all the packages using BLAS and/or LAPACK.

In addition, as I have already written on the Fedora devel list, shipping multiple implementations in a way where different packages can (easily) link a different implementation (technically, this was also possible with the old solution by abusing rpath, but then the rpath is your problem, rpaths are evil) is a surefire recipe for symbol conflicts. The situation I'm thinking of is an application linking one BLAS implementation, but the libraries it uses linking one or more other one(s). In the best case, you'd end up with an unexpected implementation (and so you're no better off than before if you were relying on getting a specific one). In the worst case, you end up with symbols from multiple implementations which are incompatible enough to crash the program or cause it to fail with an unresolved symbol. (What happens if the linker loads the reference libblas.so.3 first and resolves the BLAS symbols against that, then loads libopenblas.so.* and tries to resolve the LAPACK symbols against that?) So the only realistic solution I see for Fedora if libopenblas.so is what we're going to be stuck with is to build everything against OpenBLAS and support only OpenBLAS. Of course, you as OpenBLAS maintainers probably like this solution. ;-) But I'm not convinced that this is realistically doable in Fedora due to the politics involved. We haven't even managed to prevent MySQL-Community and Apache OpenOffice from being (re)introduced next to the default MariaDB and LibreOffice. So the symbol conflicts would likely become a serious practical problem.

But yes, incompatibilities with parallelization are a problem. The default implementation (shipped as libblas.so.3 and liblapack.so.3) can only realistically be serial. But shipping the libraries with different names doesn't help there, unless you're also renaming all the symbols. The previous paragraph on symbol conflicts applies here too! To properly support parallel LAPACK/BLAS, you have to build separate serial and parallel versions of ALL the libraries that use LAPACK and/or BLAS, even indirectly! That would probably be many more packages than you expect.

[fommil]

@susilehtola for the avoidance of doubt, I'm only referring to the convenience Windows builds when talking about static linking. For all other platforms, which tend to have sane package managers, of course the dynamic linking is the preference.

[fommil]

+1 for what @kkofler is saying. If you ship a single file then it is next to impossible for users to use your library at runtime using standard conventions. You may as well change all your symbol names and demand that users code against your API instead of BLAS/LAPACK.

[susilehtola]

But as I stated above, none of this really is relevant because as nowadays all cpus are multicore, shortly every program will link to a parallel library which are not interchangeable.

[fommil]

@susilehtola if your LAPACK impl is dependent on your BLAS impl, then document it. If anybody does mix and match implementations, it is perfectly acceptable to have undefined runtime behaviour. Can you give an example of how this works: I struggle to see how you can introduce incompatibilties when implementing the standard BLAS API. If you have your own routines that are BLAS-like, then it is acceptable to put them into a separate library and your BLAS and LAPACK can both access them.

You're missing the fact that there are decades of binaries that link to the libblas and liblapack libraries and you will break binary link compatibility for no good reason, and also force vendor dependence. For linear algebra developers, it is ideal to be able to build against the reference API and then users swap the implementation at runtime. If it is not possible to do that with your library, then you are not implementing BLAS/LAPACK, period: you are implementing something BLAS-like and something LAPACK-like. Just like cuBLAS and clBLAS.