cbtfsummary mpi portion of the collector will not build on Cori at NERSC due to issues with cc? #6
Description
When trying to get MPI information on the Cray with cbtfsummarry we are getting an error when using the PrgEnv-gnu cc wrapper to build the summary collector mpi wrappers.
I switched the /usr/bin/cc to finding the PrgEnv-gnu version of cc, which is apparently in /opt/cray/pe/craype/default on the Cray, cori at NERSC.
When I do:
module swap PrgEnv-intel PrgEnv-gnu
module load gcc
spack install openspeedshop@develop%gcc@7.3.0 +mpich
I can get through building all the other required packages - no issues. That is totally different from the last time I tried it. Although, that might have been on Trinity at LANL - this is Cori at NERSC, which has been the easiest to build on, in general.
I am including the files I'm changing and the diff of the two files.
galaro@cori11:~/cbtf-krell> module list
Currently Loaded Modulefiles:
1) modules/3.2.10.6 8) cray-libsci/18.03.1 15) job/2.2.3-6.0.7.0_44.1__g6c4e934.ari
2) nsg/1.2.0 9) udreg/2.3.2-6.0.7.0_33.18__g5196236.ari 16) dvs/2.7_2.2.113-6.0.7.1_7.1__g1bbc03e
3) gcc/7.3.0 10) ugni/6.0.14.0-6.0.7.0_23.1__gea11d3d.ari 17) alps/6.6.43-6.0.7.0_26.4__ga796da3.ari
4) craype-haswell 11) pmi/5.0.13 18) rca/2.2.18-6.0.7.0_33.3__g2aa4f39.ari
5) craype-network-aries 12) dmapp/7.1.1-6.0.7.0_34.3__g5a674e0.ari 19) atp/2.1.1
6) craype/2.5.14 13) gni-headers/5.0.12.0-6.0.7.0_24.1__g3b1768f.ari 20) PrgEnv-gnu/6.0.4
7) cray-mpich/7.7.0 14) xpmem/2.2.15-6.0.7.1_5.8__g7549d06.ari
jgalaro@cori11:~/cbtf-krell> gcc -v
Using built-in specs.
COLLECT_GCC=/opt/gcc/7.3.0/bin/../snos/bin/gcc
COLLECT_LTO_WRAPPER=/opt/gcc/7.3.0/snos/libexec/gcc/x86_64-suse-linux/7.3.0/lto-wrapper
Target: x86_64-suse-linux
Configured with: ../cray-gcc-7.3.0-201801270210.d61239fc6000b/configure --prefix=/opt/gcc/7.3.0/snos --disable-nls --libdir=/opt/gcc/7.3.0/snos/lib --enable-languages=c,c++,fortran --with-gxx-include-dir=/opt/gcc/7.3.0/snos/include/g++ --with-slibdir=/opt/gcc/7.3.0/snos/lib --with-system-zlib --enable-shared --enable-__cxa_atexit --build=x86_64-suse-linux --with-ppl --with-cloog --disable-multilib
Thread model: posix
gcc version 7.3.0 20180125 (Cray Inc.) (GCC)
jgalaro@cori11:/cbtf-krell> cc -v/cbtf-krell>
Using built-in specs.
COLLECT_GCC=/opt/gcc/7.3.0/bin/../snos/bin/gcc
COLLECT_LTO_WRAPPER=/opt/gcc/7.3.0/snos/libexec/gcc/x86_64-suse-linux/7.3.0/lto-wrapper
Target: x86_64-suse-linux
Configured with: ../cray-gcc-7.3.0-201801270210.d61239fc6000b/configure --prefix=/opt/gcc/7.3.0/snos --disable-nls --libdir=/opt/gcc/7.3.0/snos/lib --enable-languages=c,c++,fortran --with-gxx-include-dir=/opt/gcc/7.3.0/snos/include/g++ --with-slibdir=/opt/gcc/7.3.0/snos/lib --with-system-zlib --enable-shared --enable-__cxa_atexit --build=x86_64-suse-linux --with-ppl --with-cloog --disable-multilib
Thread model: posix
gcc version 7.3.0 20180125 (Cray Inc.) (GCC)
COMPILER_PATH=/opt/gcc/7.3.0/snos/libexec/gcc/x86_64-suse-linux/7.3.0/:/opt/gcc/7.3.0/snos/libexec/gcc/x86_64-suse-linux/7.3.0/:/opt/gcc/7.3.0/snos/libexec/gcc/x86_64-suse-linux/:/opt/gcc/7.3.0/snos/lib/gcc/x86_64-suse-linux/7.3.0/:/opt/gcc/7.3.0/snos/lib/gcc/x86_64-suse-linux/
LIBRARY_PATH=/opt/gcc/7.3.0/snos/lib/gcc/x86_64-suse-linux/7.3.0/:/opt/gcc/7.3.0/snos/lib/gcc/x86_64-suse-linux/7.3.0/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/opt/gcc/7.3.0/snos/lib/gcc/x86_64-suse-linux/7.3.0/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-march=core-avx2' '-static' '-D' '__CRAYXC' '-D' '__CRAY_HASWELL' '-D' '__CRAYXT_COMPUTE_LINUX_TARGET' '-u' 'pthread_mutex_destroy' '-D' 'TARGET_LINUX' '-v' '-I' '/opt/cray/pe/libsci/18.03.1/GNU/7.1/x86_64/include' '-I' '/opt/cray/pe/mpt/7.7.0/gni/mpich-gnu/5.1/include' '-I' '/opt/cray/rca/2.2.18-6.0.7.0_33.3__g2aa4f39.ari/include' '-I' '/opt/cray/alps/6.6.43-6.0.7.0_26.4__ga796da3.ari/include' '-I' '/opt/cray/xpmem/2.2.15-6.0.7.1_5.8__g7549d06.ari/include' '-I' '/opt/cray/gni-headers/5.0.12.0-6.0.7.0_24.1__g3b1768f.ari/include' '-I' '/opt/cray/pe/pmi/5.0.13/include' '-I' '/opt/cray/ugni/6.0.14.0-6.0.7.0_23.1__gea11d3d.ari/include' '-I' '/opt/cray/udreg/2.3.2-6.0.7.0_33.18__g5196236.ari/include' '-I' '/opt/cray/wlm_detect/1.3.3-6.0.7.0_47.2__g7109084.ari/include' '-I' '/opt/cray/krca/2.2.4-6.0.7.1_5.23__g8505b97.ari/include' '-I' '/opt/cray-hss-devel/8.0.0/include' '-L/opt/cray/pe/libsci/18.03.1/GNU/7.1/x86_64/lib' '-L/opt/cray/dmapp/default/lib64' '-L/opt/cray/pe/mpt/7.7.0/gni/mpich-gnu/5.1/lib' '-L/opt/cray/rca/2.2.18-6.0.7.0_33.3__g2aa4f39.ari/lib64' '-L/opt/cray/alps/6.6.43-6.0.7.0_26.4__ga796da3.ari/lib64' '-L/opt/cray/xpmem/2.2.15-6.0.7.1_5.8__g7549d06.ari/lib64' '-L/opt/cray/pe/pmi/5.0.13/lib64' '-L/opt/cray/ugni/6.0.14.0-6.0.7.0_23.1__gea11d3d.ari/lib64' '-L/opt/cray/udreg/2.3.2-6.0.7.0_33.18__g5196236.ari/lib64' '-L/opt/cray/pe/atp/2.1.1/libApp' '-L/opt/cray/wlm_detect/1.3.3-6.0.7.0_47.2__g7109084.ari/lib64'
/opt/gcc/7.3.0/snos/libexec/gcc/x86_64-suse-linux/7.3.0/collect2 -plugin /opt/gcc/7.3.0/snos/libexec/gcc/x86_64-suse-linux/7.3.0/liblto_plugin.so -plugin-opt=/opt/gcc/7.3.0/snos/libexec/gcc/x86_64-suse-linux/7.3.0/lto-wrapper -plugin-opt=-fresolution=/tmp/cc13MW2q.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lc -m elf_x86_64 -static -u pthread_mutex_destroy /usr/lib/../lib64/crt1.o /usr/lib/../lib64/crti.o /opt/gcc/7.3.0/snos/lib/gcc/x86_64-suse-linux/7.3.0/crtbeginT.o -L/opt/cray/pe/libsci/18.03.1/GNU/7.1/x86_64/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.0/gni/mpich-gnu/5.1/lib -L/opt/cray/rca/2.2.18-6.0.7.0_33.3__g2aa4f39.ari/lib64 -L/opt/cray/alps/6.6.43-6.0.7.0_26.4__ga796da3.ari/lib64 -L/opt/cray/xpmem/2.2.15-6.0.7.1_5.8__g7549d06.ari/lib64 -L/opt/cray/pe/pmi/5.0.13/lib64 -L/opt/cray/ugni/6.0.14.0-6.0.7.0_23.1__gea11d3d.ari/lib64 -L/opt/cray/udreg/2.3.2-6.0.7.0_33.18__g5196236.ari/lib64 -L/opt/cray/pe/atp/2.1.1/libApp -L/opt/cray/wlm_detect/1.3.3-6.0.7.0_47.2__g7109084.ari/lib64 -L/opt/gcc/7.3.0/snos/lib/gcc/x86_64-suse-linux/7.3.0 -L/opt/gcc/7.3.0/snos/lib/gcc/x86_64-suse-linux/7.3.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/opt/gcc/7.3.0/snos/lib/gcc/x86_64-suse-linux/7.3.0/../../.. -rpath=/opt/cray/pe/gcc-libs --no-as-needed -lAtpSigHandler -lAtpSigHCommData --undefined=_ATP_Data_Globals --undefined=__atpHandlerInstall -lpthread -lsci_gnu_71_mpi -lsci_gnu_71 -lpthread -lhugetlbfs -lmpich_gnu_51 -lrt -lugni -lpthread -lpmi -lpthread -lalpslli -lpthread -lwlm_detect -lalpsutil -lpthread -lrca -lxpmem -lugni -lpthread -ludreg --as-needed -lgfortran -lquadmath --no-as-needed --as-needed -lmvec --no-as-needed --as-needed -lm --no-as-needed --as-needed -lpthread --no-as-needed --start-group -lgcc -lgcc_eh -lc --end-group /opt/gcc/7.3.0/snos/lib/gcc/x86_64-suse-linux/7.3.0/crtend.o /usr/lib/../lib64/crtn.o
/usr/lib/../lib64/crt1.o: In function _start': /home/abuild/rpmbuild/BUILD/glibc-2.22/csu/../sysdeps/x86_64/start.S:114: undefined reference to main'
collect2: error: ld returned 1 exit status
jgalaro@cori11:
The errors I'm seeing:
spack install openspeedshop@develop%gcc@7.3.0 +mpich
==> diffutils is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/diffutils-3.6-rkoi2gv6hb6u2llgznjfsetxtliotsqi
==> bzip2 is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/bzip2-1.0.6-o4wpe27mb42ohakqxnmxf2trych7ij4l
==> pkg-config is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/pkg-config-0.29.2-u2gomdsvq67q55wjxrssfna7jkxckhka
==> xz is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/xz-5.2.4-62czkwibst3iww6vgchn7antsetpmhcx
==> zlib is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/zlib-1.2.11-uw4fzi4m6feyszysqptjqj42z5z3ekg6
==> libxml2 is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/libxml2-2.9.8-xfskjry5leltvol2wsvqcfi6p2blwt3b
==> ncurses is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/ncurses-6.1-3qx2xoey36v53wznrkw5ocjtbzdu2p5m
==> tar is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/tar-1.30-rauwacuuevb6r37kfc7eddfnggshnq5z
==> gettext is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/gettext-0.19.8.1-mlvm6lzsjyvzpjpr73ke5bat73b3piqr
==> binutils is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/binutils-2.31.1-t2buzwali7nnepsqvqzg5goxy4rmkt3r
==> readline is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/readline-7.0-ztwkzaj6s2jjczfbel2rpbvkqpwqmfkj
==> gdbm is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/gdbm-1.14.1-eqk77qd6kiv4hzyfgxb5wwi7eyfmztmg
==> perl is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/perl-5.26.2-lzuxybjocjj7kdhh3pw5gjgdjqwmtd2w
==> help2man is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/help2man-1.47.4-n2unm4q5n5bainczgt6ib5ciq3azevha
==> libsigsegv is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/libsigsegv-2.11-xvt6ng6xyjj4yjrmljxwua6tnxol3kmt
==> m4 is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/m4-1.4.18-atz6g674azjixpkdau6fkkcxufnfw5xe
==> bison is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/bison-3.0.5-zupd6jdujhpb2kfb6x32mpp2bvepakyy
==> boost is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/boost-1.68.0-55pcrpyax7a4xorz6u6yrxc5dchbdoy4
==> openssl is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/openssl-1.0.2o-7phfslyi6azswiplgvr2wvzxbve22zkm
==> cmake is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/cmake-3.12.2-corqorwue655lm66ltpkwzrrqossm42m
==> mrnet is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/mrnet-5.0.1-3-62plvcgc5pmzw2pbdw43aebaabapcjtq
==> libiconv is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/libiconv-1.15-c3kh54uga2ptrssjpcedctyjrrr47p4v
==> xerces-c is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/xerces-c-3.2.2-fjnqasur4v2ajsf2qqjey4zquuk3eavb
==> cbtf is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/cbtf-develop-nocqvzgf43drlgr3iom6fhmpbqa2sqde
==> elfutils is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/elfutils-0.173-nhajs276s6x3cw2vl2d77pnqrz7ycnmx
==> libiberty is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/libiberty-2.31.1-eehra6eqdx743nu2rfwoo36oh6g7evpo
==> dyninst is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/dyninst-develop-4u2zvgc6rvxlrjznlytdjfzc52n6tx4h
==> gotcha is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/gotcha-1.0.2-osuvufu3bk7hbz44cbvtyjzzkrb7dypz
==> libmonitor is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/libmonitor-2013.02.18-ake66dgj7qf4wcgocrgbmh2ac3jh2iad
==> libunwind is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/libunwind-1.2.1-246x5s7wbdb26aoto7f6hjw3kxckkzjc
==> sqlite is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/sqlite-3.23.1-vbm5vvim32yoe5wvpg53ment6lqvwtm2
==> python is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/python-2.7.15-u2igd43lmosejrmm7zgrm6lt6keq6fa3
==> ninja is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/ninja-1.8.2-qloefczn3fg3iohyvuvomlf2e4vnzlpo
==> llvm-openmp-ompt is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/llvm-openmp-ompt-tr6_forwards-ztk3beqx3m2iju7jvwljx6mtm2bfst4t
==> mpich@7.7.0 : externally installed in /opt/cray/pe/mpt/7.7.0/gni/mpich-gnu/5.1
==> mpich@7.7.0 : already registered in DB
==> papi is already installed in /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc-7.3.0/papi-5.6.0-7qh3mzppbwoab46uc72pn3s7bxwb57vw
==> Installing cbtf-krell
==> Fetching from /global/u2/j/jgalaro/clean_spack/spack/var/spack/cache/cbtf-krell/cbtf-krell-develop.tar.gz failed.
==> Fetching file:/global/homes/j/jgalaro/cbtf-krell-develop.tar.gz
######################################################################## 100.0%
==> Staging archive: /global/u2/j/jgalaro/clean_spack/spack/var/spack/stage/cbtf-krell-develop-n6sxsozvqmazrmtyzhaalf6ssyyjybbg/cbtf-krell-develop.tar.gz
==> Created stage in /global/u2/j/jgalaro/clean_spack/spack/var/spack/stage/cbtf-krell-develop-n6sxsozvqmazrmtyzhaalf6ssyyjybbg
==> No patches needed for cbtf-krell
==> Building cbtf-krell [CMakePackage]
==> Executing phase: 'cmake'
==> Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
'make'
4 errors found in build log:
19333 make[2]: Leaving directory '/tmp/jgalaro/spack-stage/spack-stage-lCtE0B/cbtf-krell/build_cbtf_krell'
19334 [ 97%] Built target overview-monitor-mrnet-mpi
19335 make -f core/collectors/overview/mpi/CMakeFiles/overview-mpich-wrappers.dir/build.make core/collectors/overview/mpi/CMakeFiles/overview-mpich-wrappers.dir/depend
19336 make[2]: Entering directory '/tmp/jgalaro/spack-stage/spack-stage-lCtE0B/cbtf-krell/build_cbtf_krell'
19337 [ 97%] Generating C code for mpich-wrappers.c from wrappers.w
19338 cd /tmp/jgalaro/spack-stage/spack-stage-lCtE0B/cbtf-krell/build_cbtf_krell/core/collectors/overview/mpi && /global/u2/j/jgalaro/clean_spack/spack/opt/spack/cray-cnl6-haswell/gcc
-7.3.0/python-2.7.15-u2igd43lmosejrmm7zgrm6lt6keq6fa3/bin/python /global/u2/j/jgalaro/clean_spack/spack/var/spack/stage/cbtf-krell-develop-n6sxsozvqmazrmtyzhaalf6ssyyjybbg/cbtf-
krell/core/collectors/overview/mpi/wrap.py -c /opt/cray/pe/craype/default/bin/cc -f /global/u2/j/jgalaro/clean_spack/spack/var/spack/stage/cbtf-krell-develop-n6sxsozvqmazrmtyzha
alf6ssyyjybbg/cbtf-krell/core/collectors/overview/mpi/wrappers.w -o /tmp/jgalaro/spack-stage/spack-stage-lCtE0B/cbtf-krell/build_cbtf_krell/core/collectors/overview/mpi/mpich-wr
appers.c
19339 Error: Couldn't run '/opt/cray/pe/craype/default/bin/cc -E ' for parsing mpi.h.
19340 Process exited with code 1.
19341 core/collectors/overview/mpi/CMakeFiles/overview-mpich-wrappers.dir/build.make:64: recipe for target 'core/collectors/overview/mpi/mpich-wrappers.c' failed
19342 make[2]: *** [core/collectors/overview/mpi/mpich-wrappers.c] Error 1
19343 make[2]: Leaving directory '/tmp/jgalaro/spack-stage/spack-stage-lCtE0B/cbtf-krell/build_cbtf_krell'
19344 CMakeFiles/Makefile2:7106: recipe for target 'core/collectors/overview/mpi/CMakeFiles/overview-mpich-wrappers.dir/all' failed
19345 make[1]: *** [core/collectors/overview/mpi/CMakeFiles/overview-mpich-wrappers.dir/all] Error 2
19346 make[1]: Leaving directory '/tmp/jgalaro/spack-stage/spack-stage-lCtE0B/cbtf-krell/build_cbtf_krell'
19347 Makefile:132: recipe for target 'all' failed
19348 make: *** [all] Error 2
See build log for details:
/global/u2/j/jgalaro/clean_spack/spack/var/spack/stage/cbtf-krell-develop-n6sxsozvqmazrmtyzhaalf6ssyyjybbg/cbtf-krell/spack-build.out
jgalaro@cori11:~/clean_spack/spack/var/spack/repos/builtin/packages>