[FEA]: Trigger `add_dll_directory(found_path)` logic even if `was_already_loaded_from_elsewhere` is true

[gmarkall]

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Runtime Error

Component

cuda.bindings

Describe the bug

Attempting to use NVRTC on Windows with the NVRTC wheels from cuda-python fails with:

nvrtc: error: failed to open nvrtc-builtins64_129.dll.
  Make sure that nvrtc-builtins64_129.dll is installed correctly.

I can work around this by copying nvrtc-builtins64_129.dll to a folder where other loaded DLLs exist, e.g.

copy "$SYS_PREFIX\Lib\site-packages\nvidia\cuda_nvrtc\bin\nvrtc-builtins64_*.dll" "$SYS_PREFIX"

How to Reproduce

I will have to create a smaller reproducer, but I observed this by pip installing Numba-CUDA in CI and running its testsuite: https://github.com/NVIDIA/numba-cuda/pull/353/files#diff-9753d897094baa6cf320a53ee98e7059f8477dbf3c6d467078ce35bcf3354f57R31-R57

Example run: https://github.com/NVIDIA/numba-cuda/actions/runs/16526296179/job/46740959886#step:8:8351

I did similar on my local machine and observed the same issue. I will update this issue once I have a shorter reproducer.

Expected behavior

Using NVRTC should not emit an NVRTC error about loading the builtins.

Operating System

Windows 11, amd64

nvidia-smi output

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 572.13                 Driver Version: 572.13         CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                     TCC   |   00000001:00:00.0 Off |                  Off |
| N/A   38C    P8              9W /   70W |       9MiB /  16384MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

[gmarkall]

As discussed in Slack, this is probably because Numba-CUDA has a couple of routes by which it opens NVRTC itself, but then attempts to use the CUDA bindings to actually compile programs.

[leofang]

This is how the NVRTC wheel is supported in cuda-bindings/cuda-core:

• in cuda-bindings 12.8 and earlier, we preload nvrtc-builtins when loading nvrtc here (I applied the same logic from nvmath)
• in 12.9+, @rwgk added add_dll_directory support to generalize this logic without hard-coding the dll names

Now to expand what Graham noted above:

• numba-cuda loads NVRTC (but not nvrtc-builtins)
• pathfinder sees that NVRTC is already loaded, so it does not attempt to search/load
• but the add_dll_directory logic is part of the search logic, so in this case the nvrtc-builtins dll is not loaded or placed in the search paths

This seems like a very sharp edge in pathfinder, for which I am not sure if it's fixable at all. In an offline discussion, @rwgk suggested:

We could make it so that the add_dll_directory(found_path) logic is triggered even if was_already_loaded_from_elsewhere is true. It'll still (and unavoidably) be a side effect, but a more consistent one.

[gmarkall]

Numba-cuda should not be loading nvrtc at all when using the cuda bindings, so if I fix that problem then I think we should avoid this sharp edge?

[leofang]

Yes that'd be ideal.

[leofang]

I keep this issue open for now (until Ralf makes a decision) just because I have a feeling that numba-cuda might not be the only one that gets trapped by this.

[carterbox]

I think nvmath-python is also catching this issue in our attempted migration to pathfinder. We tried to resolve this by calling pathfinder.load_nvidia_dynamic_lib('nvrtc-builtins'), but I believe this doesn't work because nvrtc-builtins is not in SUPPORTED_NVIDIA_LIBNAMES.

Would supporting loading nvrtc-builtins separately from nvrtc solve the issue?