py-scipy fails to build oneapi@2025.3.0 on ubuntu

[eap]

Proposed fix

Running spack config add "packages:py-scipy:require:['%gcc']" fixed the build. Maybe the generic config oneapi should specify "%gcc"

Issue description

Attempting to build this spec with oneapi 2025.3.0 fails on unbuntu:

package spec: py-scipy@1.16.0 build_system=python_pip arch=linux-ubuntu24.04-icelake %c,cxx,fortran=oneapi@2025.3.0

Full output of `spack spec`

intel-oneapi-mkl@2025.3~cluster+envmods~gfortran~ilp64+shared build_system=generic mpi_family=none threads=none arch=linux-ubuntu24.04-icelake

[+] ^pkg-config@0.29.2+internal_glib build_system=autotools arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[e] ^gmake@4.3~guile build_system=generic patches:=599f134 arch=linux-ubuntu24.04-icelake

[e] ^intel-oneapi-compilers@2025.3.0amd+envmodsnvidia build_system=generic arch=linux-ubuntu24.04-icelake

[e] ^intel-oneapi-runtime@2025.3.0 build_system=generic arch=linux-ubuntu24.04-icelake

[+] ^py-cython@3.0.12 build_system=python_pip arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^py-setuptools@69.2.0 build_system=generic arch=linux-ubuntu24.04-icelake

[+] ^py-meson-python@0.16.0 build_system=python_pip arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[+] ^meson@1.7.0 build_system=python_pip patches:=0f0b1bd arch=linux-ubuntu24.04-icelake

[+] ^ninja@1.12.1+re2c build_system=generic patches:=93f4bb3 arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^re2c@3.1 build_system=autotools arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^py-packaging@25.0 build_system=python_pip arch=linux-ubuntu24.04-icelake

[+] ^py-flit-core@3.8.0 build_system=python_pip arch=linux-ubuntu24.04-icelake

[+] ^py-pyproject-metadata@0.9.1 build_system=python_pip arch=linux-ubuntu24.04-icelake

[+] ^py-numpy@1.26.4 build_system=python_pip patches:=873745d arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^py-pip@25.1.1 build_system=generic arch=linux-ubuntu24.04-icelake

[+] ^py-pybind11@2.13.6+ipo build_system=cmake build_type=Release generator=ninja arch=linux-ubuntu24.04-icelake %cxx=oneapi@2025.3.0

[+] ^cmake@3.31.8doc+ncurses+ownlibsqtgui build_system=generic build_type=Release arch=linux-ubuntu24.04-icelake %c,cxx=gcc@13.3.0

[+] ^curl@8.11.1gssapildaplibidn2librtmplibsshlibssh2+nghttp2 build_system=autotools libs:=shared,static tls:=openssl arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^nghttp2@1.65.0 build_system=autotools arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^py-pythran@0.16.1 build_system=python_pip arch=linux-ubuntu24.04-icelake %cxx=oneapi@2025.3.0

[+] ^py-beniget@0.4.1 build_system=python_pip arch=linux-ubuntu24.04-icelake

[+] ^py-gast@0.5.4 build_system=python_pip arch=linux-ubuntu24.04-icelake

[+] ^py-ply@3.11 build_system=python_pip arch=linux-ubuntu24.04-icelake

[+] ^py-wheel@0.45.1 build_system=generic arch=linux-ubuntu24.04-icelake

[+] ^python@3.11.11+bz2crypt+ctypes+dbmdebug+libxml2+lzmaoptimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssltkinter+uuid+zlib build_system=generic patches:=13fa8bf,b0615b2,ebdca64,f2fd060 arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^bzip2@1.0.8debugpic+shared build_system=generic arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[+] ^diffutils@3.10 build_system=autotools arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[+] ^expat@2.7.1~libbsd build_system=autotools arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^gdbm@1.23 build_system=autotools arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[+] ^gettext@0.23.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[+] ^libxml2@2.13.5http+picpython+shared build_system=autotools arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[+] ^tar@1.35 build_system=autotools zip=pigz arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[+] ^pigz@2.8 build_system=makefile arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[+] ^zstd@1.5.7+pic+programs build_system=makefile compression:=none libs:=shared,static arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^libffi@3.4.8 build_system=autotools arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^ncurses@6.5~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^openssl@3.4.1~docs+shared build_system=generic certs=system arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^perl@5.40.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[+] ^berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^readline@8.2 build_system=autotools patches:=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[+] ^sqlite@3.46.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[+] ^util-linux-uuid@2.41 build_system=autotools arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[+] ^xz@5.6.3~pic build_system=autotools libs:=shared,static arch=linux-ubuntu24.04-icelake %c=oneapi@2025.3.0

[+] ^zlib@1.3.1+optimize+pic+shared build_system=makefile arch=linux-ubuntu24.04-icelake %c,cxx=oneapi@2025.3.0

[+] ^python-venv@1.0 build_system=generic arch=linux-ubuntu24.04-icelake

==> Error: ProcessError: Command exited with status 1:
    '/opt/spack-stack/envs/unified-oneapi/install/none/none/python-venv-1.0-lkn2xni/bin/python3' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-ve
rsion-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/opt/spack-stack/envs/unified-on
eapi/install/intel-oneapi-compilers/2025.3.0/py-scipy-1.16.0-sohqi6l' '--config-settings=builddir=build' '--config-settings=compile-args=-j12' '--config-settings=
setup-args=-Dfortran_std=legacy' '--config-settings=setup-args=-Dblas=mkl-dynamic-lp64-seq' '--config-settings=setup-args=-Dlapack=mkl-dynamic-lp64-seq' '.'
==> Installing py-scipy-1.16.0-sohqi6lqqihnoxgtlzvd5qwu2zjfqa3f [260/292]

4 errors found in build log:
     3668      [1437/1443] Linking target scipy/ndimage/_ni_label.cpython-311-x86_64-linux-gnu.so
     3669      [1438/1443] Compiling C++ object scipy/optimize/_highspy/_core.cpython-311-x86_64-linux-gnu.so.p/.._.._.._subprojects_highs_src_highs_bindings.cpp
             .o
     3670      [1439/1443] Linking target scipy/optimize/_highspy/_core.cpython-311-x86_64-linux-gnu.so
     3671      [1440/1443] Compiling C++ object scipy/interpolate/_rbfinterp_pythran.cpython-311-x86_64-linux-gnu.so.p/meson-generated__rbfinterp_pythran.cpp.o
     3672      [1441/1443] Linking target scipy/interpolate/_rbfinterp_pythran.cpython-311-x86_64-linux-gnu.so
     3673      [1442/1443] Compiling C++ object scipy/sparse/sparsetools/_sparsetools.cpython-311-x86_64-linux-gnu.so.p/bsr.cxx.o
  >> 3674    FAILED: scipy/sparse/sparsetools/_sparsetools.cpython-311-x86_64-linux-gnu.so.p/bsr.cxx.o
     3675      /opt/spack-stack/envs/unified-oneapi/install/none/none/compiler-wrapper-1.0-bg47rrw/libexec/spack/oneapi/icpx -Iscipy/sparse/sparsetools/_sparseto
             ols.cpython-311-x86_64-linux-gnu.so.p -Iscipy/sparse/sparsetools -I../scipy/sparse/sparsetools -I../scipy/_build_utils/src -I../../../../../../envs/
             unified-oneapi/install/intel-oneapi-compilers/2025.3.0/py-numpy-1.26.4-3c5o3d7/lib/python3.11/site-packages/numpy/core/include -I/opt/spack-stack/en
             vs/unified-oneapi/install/intel-oneapi-compilers/2025.3.0/python-3.11.11-pt7uag5/include/python3.11 -fvisibility=hidden -fvisibility-inlines-hidden
             -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c++17 -O3 -fp-model=strict -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_9
             _API_VERSION -MD -MQ scipy/sparse/sparsetools/_sparsetools.cpython-311-x86_64-linux-gnu.so.p/bsr.cxx.o -MF scipy/sparse/sparsetools/_sparsetools.cpy
             thon-311-x86_64-linux-gnu.so.p/bsr.cxx.o.d -o scipy/sparse/sparsetools/_sparsetools.cpython-311-x86_64-linux-gnu.so.p/bsr.cxx.o -c ../scipy/sparse/s
             parsetools/bsr.cxx
  >> 3676      fatal error: error in backend: Do not know how to scalarize this operator's operand!
     3677
     3678      PLEASE append the compiler options "-save-temps -v", rebuild the application to get the full command which is failing and submit a bug report to h
             ttps://software.intel.com/en-us/support/priority-support which includes the failing command, input files for the command and the crash backtrace (if
              any).
     3679      Stack dump:
     3680      0.    Program arguments: /opt/intel/oneapi/compiler/2025.3/bin/compiler/clang++ @/tmp/icpx1027574457NUF1Jl/icpxargqFYSG0
     3681      1.    <eof> parser at end of file
     3682      2.    Code generation

[climbfuji]

py-scipy@1.16 builds successfully on my laptop (arch=almalinux9-skylake), on Atlantis and on Nautilus with oneapi@2025.3.0.

I am concerned that compiling scipy with gcc causes problems with other Python and non-Python (e.g. MET) packages.

How is your gcc compiler defined on Ubuntu? Intel still uses the GCC compiler for libstdc etc I think.

[climbfuji]

Another possibility is that optimizing for icelake is causing the issue.

[eap]

After quite a bit of work on this I have come to the following conclusions.

1. This issue seems to only affect some platform-compiler combinations, even within intel.
2. Adding py-scipy:require: %gcc can create a split-compiler issue even with constraints or toolchain requirements.
3. I was able to fix py-scipy in %oneapi by adding require:'cxxflags="-O2"' to my py-scipy package config.

[climbfuji]

After quite a bit of work on this I have come to the following conclusions.

1. This issue seems to only affect some platform-compiler combinations, even within intel.
2. Adding py-scipy:require: %gcc can create a split-compiler issue even with constraints or toolchain requirements.
3. I was able to fix py-scipy in %oneapi by adding require:'cxxflags="-O2"' to my py-scipy package config.

I think we can make this a universal setting for oneapi in configs/common/packages_oneapi.yaml. I am pretty sure this doesn't hurt any of the sites where it currently builds w/o -O2.