CUDA support

[ocaisa]

I was experimenting with CUDA support within EESSI and ran into the issue that, when using CUDA compiled with the EESSI stack, the CUDA libraries from the host are not seen by the executables created by nvcc. This is because it looks for the CUDA driver libraries in the prefix, where they do not exist. There are a few viable solutions:

• Use LD_PRELOAD...but this requires that you know exactly which libraries to preload.
• Use LD_LIBRARY_PATH...but in the system that I was on, the CUDA libraries are in /usr/lib64 so this would drag in a lot of unwanted libraries. This could be worked around by symlinking the libraries to a more unique location in the host (Compute Canada uses /usr/lib64/nvidia and has a script that creates the necessary symlinks).
• Modify the loader to (also) look in an appropriate directory - this is what Compute Canada does, see https://github.com/ComputeCanada/gentoo-overlay/blob/master/sys-libs/glibc/glibc-2.30-r8.ebuild#L1111

[peterstol]

For systems with glibc in the prefix using /usr/lib64/nvidia as location may be a good choice.

On my brightcomputing system the CUDA libraries can be simply linked from /cm/local/apps/cuda/libs/current/lib64
Creating the symlinks will require admin privileges though and can vary between systems.
Could EESSI provide different CUDA versions and use the matching one with the kernel driver in a similar way as the architecture is matched with Archspec?

[bedroge]

Bart reported on Slack yesterday that the libcuda.so of new CUDA versions no longer link against this versioned libnvidia-fatbinaryload.so.XXX.YY:

Removed libnvidia-fatbinaryloader.so from the driver package. This functionality is now built into other driver libraries.

This means that, with these newer version, we could also make symlinks to the host's libcuda. The only annoying thing is that the location can, apparently, differ between distros, so I guess we would need something like a variant symlink that allows site to override the location of libcuda.so if necessary.

[ocaisa]

I don't understand this stuff well enough but I think that a symlink might only get us out of jail with a pure CUDA code? Comparing the lib64 to the stubs directory inside a CUDA toolkit installation, it looks like you also need libnvidia-ml.so.

I took a look at the OpenGL configuration for JSC and if we want to use visualisation capabilities on the available GPU, we would also need some of the other driver libraries.

[bedroge]

Yes, you probably need more, but I guess the same thing could be done for those? The advantage of that approach could be that it would for instance work out of the box on all systems that have these libraries in /usr/lib64. On other systems, the variable for the variant symlink would have to be overridden in the cvmfs configuration.

[ocaisa]

We could probably learn a lot from what happens with containers and GPUs, they would have the same issues with passing through driver libraries.

[ocaisa]

See https://sylabs.io/guides/3.5/user-guide/gpu.html#library-search-options

They maintain two lists, one for NVIDIA, one for AMD

[ocaisa]

Thanks to #91, the 2021.03 release of EESSI can successfully compile and run CUDA code if symlinks to the driver libraries are placed in /opt/eessi/lib