OpenBLAS threadsafety issues for downstream libraries (NumPy)

[seberg]

Since OpenBLAS is one of the most commonly used BLAS implementations with NumPy, there are currently two points being discussed within NumPy. One thing is the spawning of too many threads when using multi processing/python threads (although is probably something we need to find a solution for, but pointers are welcome!).

The main issue is the thread safety of OpenBLAS when multiple threads are used. This may not be problematic for many projects explicitly using OpenBLAS, but NumPy is a bit different:

• NumPy does not actually know which BLAS is being used.
• Thus, NumPy has no control on the number of threads (the system default is used).

Together this means NumPy will often use multi threaded OpenBLAS. Now if someone uses a numpy function within threads they might get silently incorrect results (numpy/numpy#11046) and this seems true for OpenBLAS versions 0.2.18, 0.2.20, 0.3.0 (probably generally).

The possibility of silently wrong results deeply troubles me. Downstream users of NumPy are unlikely to be aware of such issues, or might not even notice that they are using BLAS at all. The problem is that there seems to be no easy solution.

We could use locking to disable threading, but that probably is incorrect/unnecessary often as well.
I have been talking with @mattip and others at NumPy, and hoped we can start a small discussion here. Maybe there are fixes/workarounds we can implement in NumPy, but maybe we can also provide help/resources to make OpenBLAS thread safe?

[brada4]

Spawning too many threads is not exactly thread safety issue, it is a configuration choice made by user.

The accuracy issue indeed reproduces even with OpenBLAS threading disabled (but not with ATLAS or reference BLAS)

btw numpy perfectly knows which blas is used, and can set threads as needed (at first sight - could be only problem is no support for OMP nesting in OpenBLAS, or that MKL gets some special treatment, or both)

python -c "import numpy; print(numpy.__config__.show())"
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/lib64']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/lib64']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/lib64']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
blis_info:
  NOT AVAILABLE
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/lib64']
    define_macros = [('HAVE_CBLAS', None)]
    language = c
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
None

[seberg]

No, unless I am very very confused, that is incorrect. The config does not tell you what BLAS is being used. Systems such as debian will link to a library somewhere and which library this is actually can be changed at any time. That configuration gives you an idea about which numpy would be compiling against, but it does not know whether this is actually used.

The only way to find out what is actually being picked up at run time is to use ldd. In fact, we (or rather Anaconda) had fun bugs already where the import of things such as Qt, changed which BLAS was picked up from MKL to Accelerate.

Spawning too many threads is not (EDIT: A thread safety issue). Personally I am more worried about the other issue. I do not see the incorrect results if I use OMP_NUM_THREADS=1 but maybe it is just less likely or depends on versions.

EDIT2: To be clear, this is just as much about whether we may be able to help OpenBLAS here as the other way around. If it is important enough, we might be able to find some resources for these issues.

[martin-frbg]

There have been several efforts since 0.2.18 to remove races, but there may be some still lurking in the level 3 BLAS code. One option to check this is to compile OpenBLAS with make USE_SIMPLE_THREADED_LEVEL3=1 - having a reproducer for the issue should help track this down.

[brada4]

Yup, alternatives did not change config output, I agree it is build config.

I see incorrect results with pthreads openblas, single thread via OPENBLAS_NUM_THREADS=1 or not. They go away with OMP_NUM_THREADS=1 which has no impact on pthread openblas, but probably serializes python multiprocessing?

[brada4]

@martin-frbg reproducer is in linked issue. No problem with netlib or atlas (ubuntu14) , Reproduces well with 0.3.3, incl setting archaic coretypes, have not tried develop version yet.

[brada4]

I am wondering what is shared gemm path that may pick or write something random - could be *_copy ?
The Repeater mentions small matrix sizes...

[martin-frbg]

Does the problem go away with SIMPLE_THREADED_LEVEL3 ? (though if this keeps occuring even with OpenBLAS limited to a single thread, it could be multiple numpy threads each invoking OpenBLAS in parallel in a scenario similar to #1536)

[seberg]

@martin-frbg so this issue should be a specific one and not a general issue, at least with newer OpenBLAS versions? That would be good news! The FAQ read a bit like it might be a deeper/bigger issue.

About OMP/NUM_THREADs, I am not actually sure on the details (or on how the debian/arch Openblas is compiled). On my Arch system OPENBLAS_NUM_THREADS=1 seems also sufficient.

[bbbbbbbbba]

Yup, alternatives did not change config output, I agree it is build config.

I see incorrect results with pthreads openblas, single thread via OPENBLAS_NUM_THREADS=1 or not. They go away with OMP_NUM_THREADS=1 which has no impact on pthread openblas, but probably serializes python multiprocessing?

If I'm understanding this correctly, the issue just got even more troublesome. On my computer (1 CPU, 2 cores), OPENBLAS_NUM_THREADS=1 works perfectly as a workaround, and that is why we thought it's a conflict between python threading and OpenBLAS threading. But if the issue reproduces on some system even with OPENBLAS_NUM_THREADS=1, then there may be another bug hiding somewhere in the stack...

[martin-frbg]

I should hope that the current situation is a lot better than it was at 0.2.18 but there may still be unfixed bugs in the code, and some of them may show up only on fast systems with many cores. If it is true that the problem persists at OPENBLAS_NUM_THREADS=1 that would be quite worrying indeed. @brada4 are you sure of that ?

[brada4]

Yes, absolutely sure with opensuse pthread:
AMD A10-5750M APU/PILEDRIVER or SANDYBRIDGE or ATOM
Absolutely no problem with ubuntu 14 OMP version:
Atom(TM) CPU N450/ATOM
I will try to narrow down the issue tomorrow.

[bbbbbbbbba]

@brada4 I don't know whether this is relevant, but I'm not sure about this part:

OMP_NUM_THREADS=1 which has no impact on pthread openblas

It seems from this part of code that it's the other way round:

• pthread openblas looks at OMP_NUM_THREADS (as a fallback if it doesn't see OPENBLAS_NUM_THREADS or GOTO_NUM_THREADS), but
• OpenMP openblas disregards OPENBLAS_NUM_THREADS completely.

[brada4]

OPENBLAS_NUM_THREADS=1 alone does not address issue for me.

[seberg]

@martin-frbg good, that makes me slightly less nervous :). Would like to still say that if there is somehow we may be able to help, please ask. Not that we have much resources, but if something is important enough maybe we can help out.

@bbbbbbbbba, @brada4 yeah, then my arch at least is just the pthread version, so it does not matter which one I use. I agree that OPENBLAS_NUM_THREADS=1 persistence makes it a lot worse, but frankly it is well above my pain threshold even without it. I would be surprised if half the numpy users even knows this switch exists and that will include those using threading, or using a library that uses threading... (EDIT: OK, I am a bit confused, but it doesn't matter much ;))