cmake: DYNAMIC_ARCH build broken on OS X

[leezu]

With latest develop, DYNAMIC_ARCH build on OS X fails with symbol(s) not found for architecture x86_64 errors for symbols like _xtrsm_outucopy_ZEN", referenced from: _gotoblas_ZEN in libopenblas.a(setparam_ZEN.c.o).

See for example this Github Actions log: https://github.com/leezu/OpenBLAS/runs/719019580?check_suite_focus=true
The Makefile build works https://github.com/leezu/OpenBLAS/runs/719019582?check_suite_focus=true

Besides this bug, would it be sensible to enable Github Actions CI for OpenBLAS repo? I'm happy to open a PR with an slightly extended version of leezu@560c39d used to produce above log.

[martin-frbg]

Please pull from develop, not master - the latter is almost three years out of date. There is a github action for homebrew builds in the develop tree, though it currently fails in an xcode version check after the successful build.

[leezu]

This issue is about the develop version. Sorry for using the wrong word in the description.

[martin-frbg]

Oh, I see. From your log the problem seems to be not so much ZEN architecture but the essentially unimplemented "quadruple precision support, not sure why a CMAKE build would suddenly attempt this on OSX. Would you happen to have set EXPRECISION=1 or QUAD_PRECISION=1 in an environment variable like CFLAGS, perhaps in the context of a different project ? (Alos not sure what to make of the cmake... -DANDROID=1 either - was this an attempt to cross-compile ?)

[martin-frbg]

Can you make this go away with -DNO_EXPRECISION=1 ? (Can't replicate this on Linux, possibly osx should be treated like the BSDs where this option gets set automatically since #2076)

[leezu]

I'm trying the cmake -DDYNAMIC_ARCH=1 -DNOFORTRAN=0 -DBUILD_WITHOUT_LAPACK=0 -DUSE_THREAD=1 -DUSE_LOCKING=1 .. build at https://github.com/leezu/OpenBLAS/runs/723909998?check_suite_focus=true and cmake -DDYNAMIC_ARCH=1 -DNOFORTRAN=0 -DBUILD_WITHOUT_LAPACK=0 -DUSE_THREAD=1 -DUSE_LOCKING=1 -DNO_EXPRECISION=1 .. build at https://github.com/leezu/OpenBLAS/runs/723910520?check_suite_focus=true

The DANDROID was taken from the Conan package file (where it is only set on Linux / Android): https://github.com/conan-io/conan-center-index/blob/375275ef84db5afd7aa22f618b774180c829958c/recipes/openblas/all/conanfile.py#L63-L65

[leezu]

Thanks @martin-frbg. According to above Github Actions runs, -DNO_EXPRECISION=1 fixes the DYNAMIC_ARCH cmake build on macOS.

In general, could you recommend "how experimental" you would consider the cmake build to be? Is there a plan to consider it "stable" at some point?

[martin-frbg]

The problem with the cmake build is that it is a comparatively late addition where the gmake build already used some trickery (recursions and kernel configuration files that use gmake conditional syntax), and not all combinations of options will have been tested on all platforms. Your build logs seem to suggest that USE_LOCKING may not have been passed correctly, I'll check that later today. (~~Though this should be¨¨ Actually this is redundant with the USE_THREAD option you set - if you leave that out the USE_LOCKING gets applied as it should be) Not sure I understand what the "libs dependency" is that Conan is working around with their use of -DANDROID - if it is necessary in an OSX context, it could be just the "-lm" that gets added on both Linux and BSD together with the NO_EXPRECISION.

[leezu]

@martin-frbg there is also another issue when building with cmake -DDYNAMIC_ARCH=1 -DNOFORTRAN=0 -DBUILD_WITHOUT_LAPACK=0 .. on OSX:

2020-06-01T19:07:57.5971040Z In file included from /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/build/kernel/CMakeFiles/sgemm_kernel_SKYLAKEX.c:7:
2020-06-01T19:07:57.5971320Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:504:34: error: inline assembly requires more registers than available
2020-06-01T19:07:57.5974160Z     for(;n_count>23;n_count-=24) COMPUTE_n24
2020-06-01T19:07:57.5974290Z                                  ^
2020-06-01T19:07:57.5974860Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:454:5: note: expanded from macro 'COMPUTE_n24'
2020-06-01T19:07:57.5975260Z     "vbroadcastss %8,%%zmm0; vmovups %10,%%zmm1; vmovups %11,%%zmm2;"\
2020-06-01T19:07:57.5975390Z     ^
2020-06-01T19:07:57.5975500Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:504:34: error: inline assembly requires more registers than available
2020-06-01T19:07:57.5976110Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:454:5: note: expanded from macro 'COMPUTE_n24'
2020-06-01T19:07:57.5976240Z     "vbroadcastss %8,%%zmm0; vmovups %10,%%zmm1; vmovups %11,%%zmm2;"\
2020-06-01T19:07:57.5976330Z     ^
2020-06-01T19:07:57.5976440Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:504:34: error: inline assembly requires more registers than available
2020-06-01T19:07:57.5976980Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:454:5: note: expanded from macro 'COMPUTE_n24'
2020-06-01T19:07:57.5977160Z     "vbroadcastss %8,%%zmm0; vmovups %10,%%zmm1; vmovups %11,%%zmm2;"\
2020-06-01T19:07:57.5977260Z     ^
2020-06-01T19:07:57.5977360Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:505:34: error: inline assembly requires more registers than available
2020-06-01T19:07:57.5977810Z     for(;n_count>19;n_count-=20) COMPUTE(20)
2020-06-01T19:07:57.5977910Z                                  ^
2020-06-01T19:07:57.5978400Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:421:5: note: expanded from macro 'COMPUTE'
2020-06-01T19:07:57.5978520Z     "vbroadcastss %7,%%zmm0; vmovups %9,%%zmm1; vmovups %10,%%zmm2;"\
2020-06-01T19:07:57.5978630Z     ^
2020-06-01T19:07:57.5978730Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:506:34: error: inline assembly requires more registers than available
2020-06-01T19:07:57.5979170Z     for(;n_count>15;n_count-=16) COMPUTE(16)
2020-06-01T19:07:57.5979270Z                                  ^
2020-06-01T19:07:57.5979760Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:421:5: note: expanded from macro 'COMPUTE'
2020-06-01T19:07:57.5979880Z     "vbroadcastss %7,%%zmm0; vmovups %9,%%zmm1; vmovups %10,%%zmm2;"\
2020-06-01T19:07:57.5979970Z     ^
2020-06-01T19:07:57.5980080Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:507:34: error: inline assembly requires more registers than available
2020-06-01T19:07:57.5980500Z     for(;n_count>11;n_count-=12) COMPUTE(12)
2020-06-01T19:07:57.5980610Z                                  ^
2020-06-01T19:07:57.5981100Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:421:5: note: expanded from macro 'COMPUTE'
2020-06-01T19:07:57.5981200Z     "vbroadcastss %7,%%zmm0; vmovups %9,%%zmm1; vmovups %10,%%zmm2;"\
2020-06-01T19:07:57.5981300Z     ^
2020-06-01T19:07:57.5981410Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:508:32: error: inline assembly requires more registers than available
2020-06-01T19:07:57.5981830Z     for(;n_count>7;n_count-=8) COMPUTE(8)
2020-06-01T19:07:57.5981930Z                                ^
2020-06-01T19:07:57.5982420Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:421:5: note: expanded from macro 'COMPUTE'
2020-06-01T19:07:57.5982530Z     "vbroadcastss %7,%%zmm0; vmovups %9,%%zmm1; vmovups %10,%%zmm2;"\
2020-06-01T19:07:57.5982620Z     ^
2020-06-01T19:07:57.5982710Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:509:32: error: inline assembly requires more registers than available
2020-06-01T19:07:57.5983710Z     for(;n_count>3;n_count-=4) COMPUTE(4)
2020-06-01T19:07:57.5983820Z                                ^
2020-06-01T19:07:57.5984500Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:421:5: note: expanded from macro 'COMPUTE'
2020-06-01T19:07:57.5984650Z     "vbroadcastss %7,%%zmm0; vmovups %9,%%zmm1; vmovups %10,%%zmm2;"\
2020-06-01T19:07:57.5984740Z     ^
2020-06-01T19:07:57.5984840Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:510:32: error: inline assembly requires more registers than available
2020-06-01T19:07:57.5985320Z     for(;n_count>1;n_count-=2) COMPUTE(2)
2020-06-01T19:07:57.5985420Z                                ^
2020-06-01T19:07:57.5985900Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:421:5: note: expanded from macro 'COMPUTE'
2020-06-01T19:07:57.5986030Z     "vbroadcastss %7,%%zmm0; vmovups %9,%%zmm1; vmovups %10,%%zmm2;"\
2020-06-01T19:07:57.5986120Z     ^
2020-06-01T19:07:57.5986220Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:511:19: error: inline assembly requires more registers than available
2020-06-01T19:07:57.5986330Z     if(n_count>0) COMPUTE(1)
2020-06-01T19:07:57.5986410Z                   ^
2020-06-01T19:07:57.5986910Z /Users/runner/runners/2.263.0/work/OpenBLAS/OpenBLAS/kernel/x86_64/sgemm_kernel_16x4_skylakex_3.c:421:5: note: expanded from macro 'COMPUTE'
2020-06-01T19:07:57.5987030Z     "vbroadcastss %7,%%zmm0; vmovups %9,%%zmm1; vmovups %10,%%zmm2;"\
2020-06-01T19:07:57.5987110Z     ^
2020-06-01T19:07:57.5987190Z 10 errors generated.

reference: https://github.com/leezu/OpenBLAS/runs/728422254?check_suite_focus=true

[martin-frbg]

Hmm. Wonder why that is - when the build gets there it must have confirmed that AVX512 support is available. Possibly an OSX LLVM bug ?