Crashes on Windows when importing model

[MKwareContributions]

I run koboldcpp.exe, wait till it asks to import model and after selecting model it just crashes with these logs:

I am running Windows 8.1 with 8 GB of RAM and 6014 MB of VRAM (according to dxdiag).
What do I do?

[LostRuins]

Hmm this looks like you may be out of RAM during the load attempt. It may be a struggle to run with only 8GB of RAM. I have released a new version which should use slightly less RAM, can you please try it again?

[MKwareContributions]

Still crashes with the same error. Should I get more RAM?

[LostRuins]

@MKware00 you could, or how about trying a smaller model first? Here is a tiny one that should definitely work on most devices. https://huggingface.co/ggerganov/ggml/resolve/main/ggml-model-gpt-2-117M.bin

You need at least version 1.0.9beta. Download the newest version of KoboldCPP here: https://github.com/LostRuins/koboldcpp/releases/latest

and let me know if this one also crashes, then it might be a separate issue...

[MKwareContributions]

Still crashes and the same error.

[LostRuins]

Then I'm not so sure, could be some sort of problem when reading the file directly. Unfortunately, it will be quite difficult to debug this remotely. All I can suggest for now is to either try copy the file and program to a different drive, or a different device, and try again.

[MKwareContributions]

Tried redownloading both koboldcpp.exe and the model you provided to another drive, no luck.

[wattsinaname]

I am also running into this issue:
Traceback (most recent call last):
File "koboldcpp.py", line 369, in
File "koboldcpp.py", line 326, in main
File "koboldcpp.py", line 64, in load_model
OSError: [WinError -1073741795] Windows Error 0xc000001d
[8000] Failed to execute script 'koboldcpp' due to unhandled exception!

but with 32GB of quad channel RAM, 8GB of VRAM running on 7 threads. Ive tried with --noblas, requantizing the original weights I'm running the most recent version. I've also been able to run the the opt 2.7B, GPT4all and AutoGPT models locally, I just really love your idea of ease of use for multiple ggml models and would honestly prefer to use kobold.

[TheWandering514]

I also have the very same OSError (no models working here on the latest koboldcpp.exe v1.1, all throwing the same error), and I have a suspicion as to what might be causing the issue, although I don't know for sure for the other people running into this problem in this thread.

I have a very dated CPU that doesn't support all the instruction sets mentioned in the Makefile, such as AVX2 and FMA - maybe the illegal instructions are being caused by the missing instruction sets. The comments in the Makefile did suggest that the flags might need to be tweaked on some architectures, and I'm getting the feeling my architecture might be one of those that requires such tweaks. Unfortunately I don't have the knowhow to try rebuilding the exe to better fit my hardware to test whether my suspicions were correct.

[LostRuins]

Ah yes, so it does seem like very old CPUs without avx2 fail to work, and it doesn't do runtime checks like I thought. I will see if I can somehow get it to work automatically.

[LostRuins]

Hi @TheWandering514 @wattsinaname @MKware00 I have made a test compatibility build that does not use any AVX2 build flags. Can you try it?

https://github.com/LostRuins/koboldcpp/releases/download/v1.1/koboldcpp_noavx2.exe

[wattsinaname]

Ah yes, so it does seem like very old CPUs without avx2 fail to work, and it doesn't do runtime checks like I thought. I will see if I can somehow get it to work automatically.

hmmmm, wierd because my cpu has AVX2 "System Info: AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |".

I also tried the new exe:
C:\Users\Tom\Desktop\koboldcpp-1.0.10\models>koboldcpp_noavx2.exe ggml-vicuna-7b-4bit-rev1.bin 5001 --noblas
Initializing dynamic library: koboldcpp.dll
Loading model: C:\Users\Tom\Desktop\koboldcpp-1.0.10\models\ggml-vicuna-7b-4bit-rev1.bin
[Parts: 1, Threads: 7]

---

Identified as LLAMA model: (ver 3)
Attempting to Load...

System Info: AVX = 1 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
llama_model_load: loading model from 'C:\Users\Tom\Desktop\koboldcpp-1.0.10\models\ggml-vicuna-7b-4bit-rev1.bin' - please wait ...
llama_model_load: n_vocab = 32001
llama_model_load: n_ctx = 2048
llama_model_load: n_embd = 4096
llama_model_load: n_mult = 256
llama_model_load: n_head = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot = 128
llama_model_load: f16 = 2
llama_model_load: n_ff = 11008
llama_model_load: n_parts = 1
llama_model_load: type = 1
llama_model_load: ggml map size = 4017.70 MB
llama_model_load: ggml ctx size = 81.25 KB
llama_model_load: mem required = 5809.78 MB (+ 1026.00 MB per state)
Traceback (most recent call last):
File "koboldcpp.py", line 379, in
File "koboldcpp.py", line 333, in main
File "koboldcpp.py", line 64, in load_model
OSError: [WinError -1073741795] Windows Error 0xc000001d
[9020] Failed to execute script 'koboldcpp' due to unhandled exception!

I'm using an older XEON server cpu, e5-2689, that could be the issue? I really appreciate you trying though! The AI dev community is awesome.

[TheWandering514]

Hi @TheWandering514 @wattsinaname @MKware00 I have made a test compatibility build that does not use any AVX2 build flags. Can you try it?

https://github.com/LostRuins/koboldcpp/releases/download/v1.1/koboldcpp_noavx2.exe

This build works on my architecture, thank you.

[LostRuins]

@wattsinaname That is very strange indeed - if you are already able to run llama.cpp and gpt4all.cpp locally then this should not be an issue - after all they are using the same libraries!