RuntimeError: The currently running cuFFT version 11203 does not support LTO callbacks

[samaid]

I received this error report via LI message:

"Hello,
I was playing around with nvmath and ran into some issues that I was hoping you could help me with.
I don't have a gpu on my laptop so I am using google colab which has CUDA toolkit version 12.2. When I try to run an example with an epilog, it gives an error saying CUDA toolkit version 12.6U2 or higher is required. I tried installing the latest CUDA toolkit (version 12.8) in the notebook but the error still occurs. When I run the command 'nvcc --version' it shows I have CUDA 12.8. I am attaching a screenshot of the error.
Any help would be appreciated!"

[samaid]

Some responses from nvmath-python engineers:

1. No enough information. nvcc --version just shows the compiler version. nvidia-smi --version is the most useful because it shows the system's driver version. If the driver is too old, then the system doesn't support CTK 12.8.
2. Also, even if CTK 12.8 installed, it might not be picked as the default version when linking nvmath (due to some setup issues on user end). Try running
   import nvmath; print(nvmath.bindings.cufft.get_version())
   to see what cufft version is linked to nvmath python and
   import cupy; print(cupy.cuda.get_local_runtime_version())
   to see what CUDA version is picked up by default.
3. Need to understand how the environment has been set up. The easiest way to get what's needed would be to install nvmath with [cu12,dx] extras.
   If for whatever reason there are different libcuffts available, the LD_PRELOAD with the right version solves that problem, but then comes nvjitlink, nvrtc, numba, and nvvm. Resolving those manually could be quite discouraging task for the user.

[Yash-Deodhar]

Hello, thank you very much for your suggestions. After trying the suggestions, these were the outputs:

To install the library is used: pip install nvmath-python[cu12,dx]. Is there any way I could update the driver on google colab? If not, is there any platform where I could access GPUs and have CTK 12.8?

[carterbox]

It looks like pytorch (and other packages) are pre-installed into the collab environment. pytorch constrains the cufft wheel to an old version which does not support LTO callbacks.

!pip install nvmath-python[cu12,dx] nvidia-cufft-cu12==11.3.*

Collecting nvidia-cufft-cu12==11.3.*
  Downloading nvidia_cufft_cu12-11.3.3.83-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.7 kB)
Requirement already satisfied: nvmath-python[cu12,dx] in /usr/local/lib/python3.11/dist-packages (0.3.0)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.11/dist-packages (from nvidia-cufft-cu12==11.3.*) (12.5.82)
Requirement already satisfied: numpy<3,>=1.24 in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (2.0.2)
Requirement already satisfied: cupy-cuda12x in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (13.3.0)
Requirement already satisfied: nvidia-cublas-cu12 in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (12.5.3.2)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12 in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (12.5.82)
Requirement already satisfied: nvidia-cuda-runtime-cu12 in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (12.5.82)
Requirement already satisfied: nvidia-curand-cu12 in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (10.3.6.82)
Requirement already satisfied: nvidia-cusolver-cu12 in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (11.6.3.83)
Requirement already satisfied: nvidia-cusparse-cu12 in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (12.5.1.3)
Requirement already satisfied: cuda-python>=12 in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (12.6.2.post1)
Requirement already satisfied: numba==0.60 in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (0.60.0)
Requirement already satisfied: nvidia-cuda-cccl-cu12>12.4.127 in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (12.8.90)
Requirement already satisfied: nvidia-cuda-nvcc-cu12 in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (12.5.82)
Requirement already satisfied: nvidia-mathdx~=24.4.0 in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (24.4.0)
Requirement already satisfied: pynvjitlink-cu12>=0.2 in /usr/local/lib/python3.11/dist-packages (from nvmath-python[cu12,dx]) (0.5.2)
Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in /usr/local/lib/python3.11/dist-packages (from numba==0.60->nvmath-python[cu12,dx]) (0.43.0)
Requirement already satisfied: fastrlock>=0.5 in /usr/local/lib/python3.11/dist-packages (from cupy-cuda12x->nvmath-python[cu12,dx]) (0.8.3)
Downloading nvidia_cufft_cu12-11.3.3.83-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (193.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 193.1/193.1 MB 6.2 MB/s eta 0:00:00
Installing collected packages: nvidia-cufft-cu12
  Attempting uninstall: nvidia-cufft-cu12
    Found existing installation: nvidia-cufft-cu12 11.2.3.61
    Uninstalling nvidia-cufft-cu12-11.2.3.61:
      Successfully uninstalled nvidia-cufft-cu12-11.2.3.61
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torch 2.6.0+cu124 requires nvidia-cublas-cu12==12.4.5.8; platform_system == "Linux" and platform_machine == "x86_64", but you have nvidia-cublas-cu12 12.5.3.2 which is incompatible.
torch 2.6.0+cu124 requires nvidia-cuda-cupti-cu12==12.4.127; platform_system == "Linux" and platform_machine == "x86_64", but you have nvidia-cuda-cupti-cu12 12.5.82 which is incompatible.
torch 2.6.0+cu124 requires nvidia-cuda-nvrtc-cu12==12.4.127; platform_system == "Linux" and platform_machine == "x86_64", but you have nvidia-cuda-nvrtc-cu12 12.5.82 which is incompatible.
torch 2.6.0+cu124 requires nvidia-cuda-runtime-cu12==12.4.127; platform_system == "Linux" and platform_machine == "x86_64", but you have nvidia-cuda-runtime-cu12 12.5.82 which is incompatible.
torch 2.6.0+cu124 requires nvidia-cudnn-cu12==9.1.0.70; platform_system == "Linux" and platform_machine == "x86_64", but you have nvidia-cudnn-cu12 9.3.0.75 which is incompatible.
torch 2.6.0+cu124 requires nvidia-cufft-cu12==11.2.1.3; platform_system == "Linux" and platform_machine == "x86_64", but you have nvidia-cufft-cu12 11.3.3.83 which is incompatible.
torch 2.6.0+cu124 requires nvidia-curand-cu12==10.3.5.147; platform_system == "Linux" and platform_machine == "x86_64", but you have nvidia-curand-cu12 10.3.6.82 which is incompatible.
torch 2.6.0+cu124 requires nvidia-cusolver-cu12==11.6.1.9; platform_system == "Linux" and platform_machine == "x86_64", but you have nvidia-cusolver-cu12 11.6.3.83 which is incompatible.
torch 2.6.0+cu124 requires nvidia-cusparse-cu12==12.3.1.170; platform_system == "Linux" and platform_machine == "x86_64", but you have nvidia-cusparse-cu12 12.5.1.3 which is incompatible.
torch 2.6.0+cu124 requires nvidia-nvjitlink-cu12==12.4.127; platform_system == "Linux" and platform_machine == "x86_64", but you have nvidia-nvjitlink-cu12 12.5.82 which is incompatible.
Successfully installed nvidia-cufft-cu12-11.3.3.83

https://colab.research.google.com/drive/1dObZ-erWE_y-x2biz8sZIt0b3WdVZFBC?usp=sharing

As someone who does not use Collab, my suggestion is to create a new environment with the packages you want instead of modifying the existing environment (However that is done on collab).

[Yash-Deodhar]

This works, thank you very much!