No cuBLAS

[Priestru]

AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |

Is it possible to add option to enable cuBLAS support like in an original Llama.cpp?

[jmtatsch]

Yes see docker PR for how to

[Priestru]

It seems to be about openblas, not cublas.

[jmtatsch]

Well you asked about openblas originally. Cublas should work exactly the same way.

[Priestru]

Yeah i wrote BLAS without making it clear what i speak about, my bad. Yet i struggle to realize how to force micromamba that used in ooba's webui to accept -DLLAMA_CUBLAS=ON parameters

[snxraven]

ggml-org/llama.cpp#1044

It needs to be compiled with an ENV Var set:
make clean && LLAMA_CUBLAS=1 make

[Priestru]

Well i can compile original llama.cpp with such but when it comes to llama-cpp-python within micromamba env things become too complicated for my limited abilities. If i can read anything that could bring me closer to solution i would deeply appreciate it.

[abetlen]

cuBLAS definitely works, I've tested installing and using cuBLAS by installing with the LLAMA_CUBLAS=1 flag and then python setup.py develop installing. It doesn't show up in that list because the function that prints the flags hasn't been updated yet in llama.cpp. It should work though (check nvidia-smi and you'll see some usage) and there's a good 25-30% speedup to eval times that should be pretty noticeable too.

[Priestru]

I'm sorry but i struggle to understand how to make llama_cpp_python-0.1.36-cp310-cp310-win_amd64.whl that has n_batch of 512 instead of 8 and cuBLAS enabled.

[Priestru]

Hard to believe but with help of GPT it seems like i managed to learn how to and make it.

[gjmulder]

Hard to believe but with help of GPT it seems like i managed to learn how to and make it.

Care to share? 😄

[Priestru]

Hard to believe but with help of GPT it seems like i managed to learn how to and make it.

Care to share? 😄

Thanks god it doesn't work. It would be too much of a cultural shock if i dared to pull it off from the first attempt for real.

[Priestru]

-- cuBLAS found
-- The CUDA compiler identification is NVIDIA 12.1.105
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/bin/nvcc.exe - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- CMAKE_SYSTEM_PROCESSOR: AMD64
-- x86 detected
-- GGML CUDA sources found, configuring CUDA architecture
-- Configuring done (8.2s)
-- Generating done (0.0s)
-- Build files have been written to: E:/LLaMA/llama-cpp-python/_skbuild/win-amd64-3.10/cmake-build
[3/5] Building CUDA object vendor\llama.cpp\CMakeFiles\ggml.dir\ggml-cuda.cu.obj
ggml-cuda.cu
tmpxft_00002550_00000000-10_ggml-cuda.cudafe1.cpp
[4/5] Install the project...-- Install configuration: "Release"
-- Installing: E:/LLaMA/llama-cpp-python/_skbuild/win-amd64-3.10/cmake-install/llama_cpp/llama.dll

Tho looks just fine, no idea why my dll isn't working

creating 'dist\llama_cpp_python-0.1.36-cp310-cp310-win_amd64.whl' and adding '_skbuild\win-amd64-3.10\setuptools\bdist.win-amd64\wheel' to it
adding 'llama_cpp/__init__.py'
adding 'llama_cpp/llama.dll'
adding 'llama_cpp/llama.py'
adding 'llama_cpp/llama_cpp.py'
adding 'llama_cpp/llama_types.py'
adding 'llama_cpp/server/__main__.py'
adding 'llama_cpp_python-0.1.36.dist-info/LICENSE.md'
adding 'llama_cpp_python-0.1.36.dist-info/METADATA'
adding 'llama_cpp_python-0.1.36.dist-info/WHEEL'
adding 'llama_cpp_python-0.1.36.dist-info/top_level.txt'
adding 'llama_cpp_python-0.1.36.dist-info/RECORD'

WHL looks perfect setting aside it doesn't work at all

[Priestru]

Traceback (most recent call last):
File “E:\LLaMA\oobabooga-windows\installer_files\env\lib\site-packages\llama_cpp\llama_cpp.py”, line 54, in _load_shared_library
return ctypes.CDLL(str(lib_path))
File "E:\LLaMA\oobabooga-windows\installer_files\env\lib\ctypes_init.py", line 374, in init
self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module ‘E:\LLaMA\oobabooga-windows\installer_files\env\lib\site-packages\llama_cpp\llama.dll’ (or one of its dependencies). Try using the full path with constructor syntax.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “E:\LLaMA\oobabooga-windows\text-generation-webui\server.py”, line 101, in load_model_wrapper
shared.model, shared.tokenizer = load_model(shared.model_name)
File “E:\LLaMA\oobabooga-windows\text-generation-webui\modules\models.py”, line 104, in load_model
from modules.llamacpp_model_alternative import LlamaCppModel
File “E:\LLaMA\oobabooga-windows\text-generation-webui\modules\llamacpp_model_alternative.py”, line 9, in
from llama_cpp import Llama, LlamaCache
File “E:\LLaMA\oobabooga-windows\installer_files\env\lib\site-packages\llama_cpp_init_.py”, line 1, in
from .llama_cpp import *
File “E:\LLaMA\oobabooga-windows\installer_files\env\lib\site-packages\llama_cpp\llama_cpp.py”, line 67, in
_lib = _load_shared_library(_lib_base_name)
File “E:\LLaMA\oobabooga-windows\installer_files\env\lib\site-packages\llama_cpp\llama_cpp.py”, line 56, in _load_shared_library
raise RuntimeError(f"Failed to load shared library ‘{_lib_path}’: {e}")
RuntimeError: Failed to load shared library ‘E:\LLaMA\oobabooga-windows\installer_files\env\lib\site-packages\llama_cpp\llama.dll’: Could not find module ‘E:\LLaMA\oobabooga-windows\installer_files\env\lib\site-packages\llama_cpp\llama.dll’ (or one of its dependencies). Try using the full path with constructor syntax.

When i use my wheel that made with cublas enabled it creates 720KB dll file that somehow fail to work. Could anyone help with some ideas?