DEBUG {2023.06}[foss/2023a] TensorFlow v2.15.1 w/ CUDA 12.1.1 by trz42 · Pull Request #808 · EESSI/software-layer

trz42 · 2024-11-12T11:31:35Z

PR to debug issues building TensorFlow v2.15.1 with CUDA v12.1.1

Uses an updated tensorflow.py easyblock that solves an ImportError issue with libnccl.so.2. See tweak libpaths in TensorFlow easyblock by adding directory containing libnccl.so.2 easybuilders/easybuild-easyblocks#3497

Notes:

we should build Bazel, ml_dtypes and tensorboard first and install them in the directory for CPU-only software (double-check if and why there are not there yet)
- Bazel/6.3.1 is installed but not Bazel/6.1.0 which is a dependency for this PR
- ml_dtypes is not installed ... not sure if it should be (see comment/question for tensorboard below) ... OR it's a new dependency for TensorFlow (check easyconfig for CPU-only version)
- tensorboard/2.13.0 is available as an extension of the CPU-only installation of TensorFlow/2.13.0-foss-2023a ... we might want to install the extension under the GPU directory?
we should check why cuDNN is installed again (in directory for CPU-only software) ... maybe related to switching to EESSI-extend/2023.06-easybuild and the installation path not being configured correctly

riscv-eessi-io-bot · 2024-11-12T11:31:39Z

Instance eessi-bot-riscv is configured to build for:

architectures: riscv64/generic
repositories: riscv.eessi.io-20240402

eessi-bot · 2024-11-12T11:31:40Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-compat, eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software

eessi-bot · 2024-11-12T11:31:40Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi-hpc.org-2023.06-software, eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software, eessi.io-2023.06-compat

trz42 · 2024-11-12T11:32:30Z

Just build for a single CPU architecture...

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

riscv-eessi-io-bot · 2024-11-12T11:32:34Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-12T11:32:34Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 28458, for details & status see DEBUG {2023.06}[foss/2023a] TensorFlow v2.15.1 w/ CUDA 12.1.1 #808 (comment)

eessi-bot · 2024-11-12T11:32:34Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-12T11:32:38Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_808/28458

date	job status	comment
Nov 12 11:32:37 UTC 2024	submitted	job id `28458` awaits release by job manager
Nov 12 11:32:49 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 12 11:37:52 UTC 2024	running	job `28458` is running
Nov 12 11:52:06 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-28458.out` ❌ found message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1731411645.tar.gz` size: 698 MiB (732482400 bytes) entries: 71 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software no software packages in tarball other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 `2023.06/software/linux/x86_64/amd/zen2/modules/all/cuDNN/8.9.2.26-CUDA-12.1.1.lua` `2023.06/software/linux/x86_64/amd/zen2/modules/numlib/cuDNN/8.9.2.26-CUDA-12.1.1.lua` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/easybuild/` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/easybuild/cuDNN-8.9.2.26-CUDA-12.1.1-easybuild-devel` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/easybuild/cuDNN-8.9.2.26-CUDA-12.1.1.eb` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/easybuild/easybuild-cuDNN-8.9.2.26-20241112.113857.log.bz2` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/easybuild/easybuild-cuDNN-8.9.2.26-20241112.113857_test_report.md` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/easybuild/reprod/` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/easybuild/reprod/cuDNN-8.9.2.26-CUDA-12.1.1.eb` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/easybuild/reprod/cuDNN-8.9.2.26-CUDA-12.1.1.env` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/easybuild/reprod/easyblocks/` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/easybuild/reprod/easyblocks/cudnn.py` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/easybuild/reprod/easyblocks/tarball.py` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/easybuild/reprod/hooks/` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/easybuild/reprod/hooks/eb_hooks.py` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_adv_infer.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_adv_infer_v8.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_adv_train.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_adv_train_v8.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_backend.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_backend_v8.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_cnn_infer.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_cnn_infer_v8.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_cnn_train.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_cnn_train_v8.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_ops_infer.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_ops_infer_v8.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_ops_train.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_ops_train_v8.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_v8.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_version.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/include/cudnn_version_v8.h` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib64` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_adv_infer.so` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_adv_infer.so.8` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_adv_infer.so.8.9.2` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_adv_infer_static.a` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_adv_infer_static_v8.a` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_adv_train.so` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_adv_train.so.8` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_adv_train.so.8.9.2` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_adv_train_static.a` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_adv_train_static_v8.a` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_cnn_infer.so` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_cnn_infer.so.8` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_cnn_infer.so.8.9.2` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_cnn_infer_static.a` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_cnn_infer_static_v8.a` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_cnn_train.so` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_cnn_train.so.8` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_cnn_train.so.8.9.2` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_cnn_train_static.a` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_cnn_train_static_v8.a` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_ops_infer.so` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_ops_infer.so.8` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_ops_infer.so.8.9.2` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_ops_infer_static.a` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_ops_infer_static_v8.a` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_ops_train.so` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_ops_train.so.8` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_ops_train.so.8.9.2` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_ops_train_static.a` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn_ops_train_static_v8.a` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn.so` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn.so.8` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/lib/libcudnn.so.8.9.2` `2023.06/software/linux/x86_64/amd/zen2/software/cuDNN/8.9.2.26-CUDA-12.1.1/LICENSE`
Nov 12 11:52:06 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 440.373 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 433.835 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.69 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.52 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.96 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.28 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.33 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.31 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7853.4 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7703.42 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-28458.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

The building failed with

Usage: eb [options] easyconfig [...]

eb: error: no such option: --include-easyblock-from-commit

NOTE, it seems odd that cuDNN is "built" again. Something must be odd with the build settings.

trz42 · 2024-11-12T13:10:40Z

Rebuilding after arg typo got fixed...

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

riscv-eessi-io-bot · 2024-11-12T13:10:43Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-12T13:10:43Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 28459, for details & status see DEBUG {2023.06}[foss/2023a] TensorFlow v2.15.1 w/ CUDA 12.1.1 #808 (comment)

eessi-bot · 2024-11-12T13:10:44Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-12T13:10:48Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_808/28459

date	job status	comment
Nov 12 13:10:47 UTC 2024	submitted	job id `28459` awaits release by job manager
Nov 12 13:11:15 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 12 13:17:18 UTC 2024	running	job `28459` is running

job failed and job manager crashed when trying to update the above table with a too large update (~ 300 KB) ... might be related to that the install path is wrong (CPU vs GPU directory)

…-layer into 2023.06-software.eessi.io-TensorFlow-2.15.1-2023a-CUDA-12.1.1-debug

trz42 · 2024-11-21T15:59:47Z

Rebuilding after EESSI-extend module got updated...

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

riscv-eessi-io-bot · 2024-11-21T15:59:50Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

riscv-eessi-io-bot · 2024-11-21T15:59:50Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-21T15:59:50Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 30160, for details & status see DEBUG {2023.06}[foss/2023a] TensorFlow v2.15.1 w/ CUDA 12.1.1 #808 (comment)

eessi-bot · 2024-11-21T15:59:51Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-21T15:59:54Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_808/30160

date	job status	comment
Nov 21 15:59:53 UTC 2024	submitted	job id `30160` awaits release by job manager
Nov 21 15:59:58 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 21 16:06:01 UTC 2024	running	job `30160` is running
Nov 22 00:00:08 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-30160.out` ✅ no message matching `FATAL:` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1732228724.tar.gz` size: 61 MiB (64328002 bytes) entries: 2139 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `Bazel/6.1.0-GCCcore-12.3.0.lua` `ml_dtypes/0.3.2-gfbf-2023a.lua` `tensorboard/2.15.1-gfbf-2023a.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `Bazel/6.1.0-GCCcore-12.3.0` `ml_dtypes/0.3.2-gfbf-2023a` `tensorboard/2.15.1-gfbf-2023a` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 no other files in tarball
Nov 22 00:00:08 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 430.067 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 427.848 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.48 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.59 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.68 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.47 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.33 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.33 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7532.0 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7526.99 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30160.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

trz42 · 2024-11-21T17:36:57Z

Try building with different Bazel version (6.3.1 instead of 6.1.0)...

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

riscv-eessi-io-bot · 2024-11-21T17:37:01Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

riscv-eessi-io-bot · 2024-11-21T17:37:01Z

Updates by the bot instance eessi-bot-riscv (click for details)

account trz42 has NO permission to send commands to the bot

eessi-bot · 2024-11-21T17:37:01Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 30161, for details & status see DEBUG {2023.06}[foss/2023a] TensorFlow v2.15.1 w/ CUDA 12.1.1 #808 (comment)

eessi-bot · 2024-11-21T17:37:01Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from trz42
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2024-11-21T17:37:05Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2024.11/pr_808/30161

date	job status	comment
Nov 21 17:37:04 UTC 2024	submitted	job id `30161` awaits release by job manager
Nov 21 17:37:51 UTC 2024	released	job awaits launch by Slurm scheduler
Nov 21 17:43:00 UTC 2024	running	job `30161` is running
Nov 21 19:34:08 UTC 2024	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-30161.out` ✅ no message matching `FATAL:` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1732215789.tar.gz` size: 16 MiB (17300141 bytes) entries: 2122 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `ml_dtypes/0.3.2-gfbf-2023a.lua` `tensorboard/2.15.1-gfbf-2023a.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `ml_dtypes/0.3.2-gfbf-2023a` `tensorboard/2.15.1-gfbf-2023a` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 `2023.06/init/easybuild/eb_hooks.py`
Nov 21 19:34:08 UTC 2024	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 440.802 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %scale=1_node %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default P: perf: 440.058 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /31ac6ab9 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.77 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_allreduce %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /f3be40a2 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 4.82 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /10e66fba @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 9.03 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_Micro_Benchmarks_coll %benchmark_info=mpi.collective.osu_alltoall %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /5be57ae7 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 8.37 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /c8c9aff5 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.32 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_latency %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /9795e491 @BotBuildTests:x86-64-amd-zen2-node+default P: latency: 0.33 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %device_type=cpu /48da21c5 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7752.93 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_Micro_Benchmarks_pt2pt %benchmark_info=mpi.pt2pt.osu_bw %scale=1_node %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %device_type=cpu /1b8c1ca2 @BotBuildTests:x86-64-amd-zen2-node+default P: bandwidth: 7780.14 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-30161.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

laraPPr · 2025-06-27T14:20:46Z

@trz42 Can you retarget this pr?

trz42 · 2025-08-13T07:32:48Z

Closing this. I think, @TopRichard has worked on this (or another TensorFlow version) and found a solution.

DEBUG {2023.06}[foss/2023a] TensorFlow v2.15.1 with CUDA 12.1.1

25c9812

trz42 added 2023.06-software.eessi.io 2023.06 version of software.eessi.io accel:nvidia labels Nov 12, 2024

fix typo in arg name

b935155

Merge branch '2023.06-software.eessi.io' of github-trz:EESSI/software…

4001d36

…-layer into 2023.06-software.eessi.io-TensorFlow-2.15.1-2023a-CUDA-12.1.1-debug

This was referenced Nov 13, 2024

{2023.06} adds a script that loads EESSI-extend and installs it if it is not available yet #812

Closed

{2023.06}[2023b] CUDA 12.4.0 and UCX-CUDA 1.15.0 #817

Closed

Merge branch '2023.06-software.eessi.io' of github-trz:EESSI/software…

4cfd111

…-layer into 2023.06-software.eessi.io-TensorFlow-2.15.1-2023a-CUDA-12.1.1-debug

parse hook to change Bazel version of build dependency

c3353c1

trz42 closed this Aug 13, 2025

Conversation

trz42 commented Nov 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

riscv-eessi-io-bot Bot commented Nov 12, 2024

Uh oh!

eessi-bot Bot commented Nov 12, 2024

Uh oh!

eessi-bot Bot commented Nov 12, 2024

Uh oh!

trz42 commented Nov 12, 2024

Uh oh!

riscv-eessi-io-bot Bot commented Nov 12, 2024

Uh oh!

eessi-bot Bot commented Nov 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot Bot commented Nov 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot Bot commented Nov 12, 2024 • edited by trz42 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trz42 commented Nov 12, 2024

Uh oh!

riscv-eessi-io-bot Bot commented Nov 12, 2024

Uh oh!

eessi-bot Bot commented Nov 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot Bot commented Nov 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot Bot commented Nov 12, 2024 • edited by trz42 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trz42 commented Nov 21, 2024

Uh oh!

riscv-eessi-io-bot Bot commented Nov 21, 2024

Uh oh!

riscv-eessi-io-bot Bot commented Nov 21, 2024

Uh oh!

eessi-bot Bot commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot Bot commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot Bot commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trz42 commented Nov 21, 2024

Uh oh!

riscv-eessi-io-bot Bot commented Nov 21, 2024

Uh oh!

riscv-eessi-io-bot Bot commented Nov 21, 2024

Uh oh!

eessi-bot Bot commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot Bot commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot Bot commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laraPPr commented Jun 27, 2025

Uh oh!

trz42 commented Aug 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

trz42 commented Nov 12, 2024 •

edited

Loading

eessi-bot Bot commented Nov 12, 2024 •

edited

Loading

eessi-bot Bot commented Nov 12, 2024 •

edited

Loading

eessi-bot Bot commented Nov 12, 2024 •

edited by trz42

Loading

eessi-bot Bot commented Nov 12, 2024 •

edited

Loading

eessi-bot Bot commented Nov 12, 2024 •

edited

Loading

eessi-bot Bot commented Nov 12, 2024 •

edited by trz42

Loading

eessi-bot Bot commented Nov 21, 2024 •

edited

Loading

eessi-bot Bot commented Nov 21, 2024 •

edited

Loading

eessi-bot Bot commented Nov 21, 2024 •

edited

Loading

eessi-bot Bot commented Nov 21, 2024 •

edited

Loading

eessi-bot Bot commented Nov 21, 2024 •

edited

Loading

eessi-bot Bot commented Nov 21, 2024 •

edited

Loading