Substantially slower than llama.cpp

[horenbergerb]

Running on Ubuntu, Intel Core i5-12400F, 32GB RAM.

Built according to README. Running the program with
python llamacpp_for_kobold.py ../llama.cpp/models/30Bnew/ggml-model-q4_0-ggjt.bin --threads 6

Generation seems to take ~5 seconds per token. This is substantially slower than llama.cpp, where I'm averaging around 900ms/token.

At first I thought it was an issue with the threading, but now I'm not so sure... Has anyone else observed similar performance discrepancies? Am I missing something?

[LostRuins]

That is strange, especially if you're using the same parameters. This repo contains a standalone main.cpp file too which is the unmodified llama.cpp main.cpp example - can you try building that make main and see if you achieve the same speed as the main repo? Try running both with the same short prompt, same thread count and batch size = 8, for best comparison

[Belarrius1]

I'm on Ubuntu Server 22.04 LTS with Intel Core i7-10750H (12threads) and, with OpenBLAS for the first inference.

65B LLaMA model with "--threads 2" and OpenBLAS active

And now with 65B LLaMA model with "--threads 11" and OpenBLAS active

I don't see any difference in response speed between 2 and 11 threads

Slowness is not related to the number of threads,
It seems rather that the limitation comes from the "processor time" side such as the processing speed per cycle, threading has no impact on the speed on my side

Or maybe something else limits it, I don't know

[LostRuins]

It's generally memory bound - but it depends. Using too many threads can actually be counter productive if RAM is slow as the threads will spin-lock and busy wait when idle, wasting CPU. In the next build I plan to decrease the default number of threads.

However, @horenbergerb issue seems to be something else since 5 seconds per token is way too slow. (I am getting like 10x that speed). I suspect some compiler flag is wrong, which is why I request them to build the main.exe/main file which is powered by the mostly unchanged llama.cpp within my repo, and see if the bottleneck is there, or due to a bug in my own custom code.

[dogjamboree]

I'm seeing the same speed issue with a mac M2 processor. I don't have the details but all i know is it's way slower than llama.cpp. Of course I'm having issues with linking OPENBLAS so I don't know if that's a red herring or not...

[LostRuins]

@dogjamboree @horenbergerb if possible could you:

1. provide me the compile flags used to build the official llama.cpp (just copy the output from console when building & linking)
2. compare timings against the llama.cpp in my own repo by triggering make main and running the executable with the exact same parameters you use for the llama.cpp repo.

Thanks!

[dogjamboree]

Sure, I'll give it a shot...I'm not super tech savvy and just following orders on most of this stuff, haha.

…

On Sun, Apr 2, 2023 at 8:25 AM LostRuins ***@***.***> wrote: @dogjamboree <https://github.com/dogjamboree> @horenbergerb <https://github.com/horenbergerb> if possible could you: 1. provide me the compile flags used to build the official llama.cpp (just copy the output from console when building & linking) 2. compare timings against the llama.cpp in my own repo by triggering make main and running the executable with the exact same parameters you use for the llama.cpp repo. Thanks! — Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A4WD5ABCML44SYVM5F2HPEDW7GK7HANCNFSM6AAAAAAWP6XJ4Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[horenbergerb]

@LostRuins Here's the output after running make clean and then make:

(textgen) captdishwasher@captainofthedishwasher-MS-7D43:~/horenbergerb/llama/llamacpp-for-kobold$ make
I llama.cpp build info: 
I UNAME_S:  Linux
I UNAME_P:  x86_64
I UNAME_M:  x86_64
I CFLAGS:   -I.              -O3 -DNDEBUG -std=c11   -fPIC -pthread -mf16c -mfma -mavx2 -mavx -msse3  -pthread -mavx -mavx2 -mfma -mf16c -msse3
I CXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread -pthread
I LDFLAGS:  
I CC:       cc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
I CXX:      g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0

cc  -I.              -O3 -DNDEBUG -std=c11   -fPIC -pthread -mf16c -mfma -mavx2 -mavx -msse3  -pthread -mavx -mavx2 -mfma -mf16c -msse3   -c ggml.c -o ggml.o
g++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread -pthread -c llama.cpp -o llama.o
g++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread -pthread -c examples/common.cpp -o common.o
g++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread -pthread examples/main/main.cpp ggml.o llama.o common.o -o main 

====  Run ./main -h for help.  ====

g++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread -pthread -c expose.cpp -o expose.o
g++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread -pthread -c llama_adapter.cpp -o llama_adapter.o
g++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread -pthread -c llamaextra.cpp -o llamaextra.o
g++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread -pthread expose.o ggml.o llama_adapter.o llamaextra.o common.o -shared -o llamacpp.dll 
g++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread -pthread examples/quantize/quantize.cpp ggml.o llama.o -o quantize 
cc  -I.              -O3 -DNDEBUG -std=c11   -fPIC -pthread -mf16c -mfma -mavx2 -mavx -msse3  -pthread -mavx -mavx2 -mfma -mf16c -msse3 -DGGML_USE_OPENBLAS -c ggml.c -o ggml_blas.o
Your OS is  and does not appear to be Windows. If you want to use openblas, please link it manually with LLAMA_OPENBLAS=1

Note: I have also built with LLAMA_OPENBLAS=1 and didn't see a huge change in performance. Even on llama.cpp, though, openblas only marginally changed my performance.

[horenbergerb]

Here's outputs from running a small generation with main:

(textgen) captdishwasher@captainofthedishwasher-MS-7D43:~/horenbergerb/llama/llamacpp-for-kobold$ ./main -m ../llama.cpp/models/30Bnew/ggml-model-q4_0-ggjt.bin -c 2048 --temp 0.6 --top_k 100 --top_p 0.9 --repeat_last_n 500 --repeat_penalty 1.2 -t 6 -n 32 -p "They"
main: seed = 1680467725
llama_model_load: loading model from '../llama.cpp/models/30Bnew/ggml-model-q4_0-ggjt.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 2048
llama_model_load: n_embd  = 6656
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 52
llama_model_load: n_layer = 60
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 17920
llama_model_load: n_parts = 4
llama_model_load: type    = 3
llama_model_load: ggml map size = 19391.80 MB
llama_model_load: ggml ctx size = 151.25 KB
llama_model_load: mem required  = 21695.95 MB (+ 3124.00 MB per state)
llama_model_load: loading tensors from '../llama.cpp/models/30Bnew/ggml-model-q4_0-ggjt.bin'
llama_model_load: model size = 19391.35 MB / num tensors = 543
llama_init_from_file: kv self size  = 3120.00 MB

system_info: n_threads = 6 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
sampling: temp = 0.600000, top_k = 100, top_p = 0.900000, repeat_last_n = 500, repeat_penalty = 1.200000
generate: n_ctx = 2048, n_batch = 8, n_predict = 32, n_keep = 0


 They say a picture is worth 1000 words, so I’m going to let these images tell the story of our day in Siena.
llama_print_timings:        load time =  2695.55 ms
llama_print_timings:      sample time =    87.34 ms /    32 runs   (    2.73 ms per run)
llama_print_timings: prompt eval time =  1182.88 ms /     2 tokens (  591.44 ms per token)
llama_print_timings:        eval time = 28091.74 ms /    31 runs   (  906.19 ms per run)
llama_print_timings:       total time = 30874.97 ms
(textgen) captdishwasher@captainofthedishwasher-MS-7D43:~/horenbergerb/llama/llamacpp-for-kobold$

Comparatively, the same script running the baseline llama.cpp was about 867.69ms per run, so they're actually pretty close... This is making me feel crazy. I'll try running llamacpp-for-kobold again and make sure it's still slow.

[Belarrius1]

LLAMA_OPENBLAS for me change A LOT for the first inference, it's like 5x time faster

[horenbergerb]

Here are some logs from generation to show where I'm observing slowness:

Input: {"n": 1, "max_context_length": 1024, "max_length": 80, "rep_pen": 1.08, "temperature": 0.62, "top_p": 0.9, "top_k": 0, "top_a": 0, "typical": 1, "tfs": 1, "rep_pen_range": 1024, "rep_pen_slope": 0.7, "sampler_order": [0, 1, 2, 3, 4, 5, 6], "prompt": "[The following is an interesting chat message log between You and \nKoboldAI.]\n\nYou: Hi.\n\nKoboldAI: Hello.\nYou: Hey Miku, what's up?\nKoboldAI: Not much. I just got back from a long trip.\nYou: Oh that's dope. Where did you travel to?\nKoboldAI:", "quiet": true}
Time Taken - Processing:134.0s, Generation:763.6s, Total:897.6s
Processing Prompt (74 / 74 tokens)
Generating (80 / 80 tokens)

Output:  I was in the UK.
You: Wow, did you go for the World Cup?
KoboldAI: No, it was for work.
You: Oh ok. How was your experience there?
KoboldAI: It was great!
You: Cool. Have you ever been there before?
KoboldAI: Yes, I went there when
127.0.0.1 - - [02/Apr/2023 16:50:07] "POST /api/latest/generate/ HTTP/1.1" 200 -

Input: {"n": 1, "max_context_length": 1024, "max_length": 80, "rep_pen": 1.08, "temperature": 0.62, "top_p": 0.9, "top_k": 0, "top_a": 0, "typical": 1, "tfs": 1, "rep_pen_range": 1024, "rep_pen_slope": 0.7, "sampler_order": [0, 1, 2, 3, 4, 5, 6], "prompt": "[The following is an interesting chat message log between You and \nKoboldAI.]\n\nYou: Hi.\n\nKoboldAI: Hello.\nYou: Hey Miku, what's up?\nKoboldAI: Not much. I just got back from a long trip.\nYou: Oh that's dope. Where did you travel to?\nKoboldAI: I was in the UK.\nYou: Woah, that's pretty far from home. Don't you live in Georgia?\nKoboldAI:", "quiet": true}
Time Taken - Processing:264.4s, Generation:769.4s, Total:1033.8s

Seems to me that generation is taking ~750s for ~80 tokens?
BTW, let me know if this data isn't useful; if it's just cluttering the issue I'll delete it.

[LostRuins]

@horenbergerb data is useful, that's fine. You are running the same model in both cases? In your last example I cannot see the the load parameters, so I can't tell how many threads were used or which model/flags were set.

Can you try again with the latest version v1.0.8beta https://github.com/LostRuins/koboldcpp/releases/tag/v1.0.8beta , and use the same prompt (one word "There") and set amount to generate from 80 to 32 tokens to match your main example?

Oops forgot you're on linux, can you pull and build? thanks.

[Belarrius1]

@LostRuins Wow!

I "git clone" and "make LLAMA_OPENBLAS=1" the project then lunch with the exact same parameters, I change nothing and here the results

Before:
Time Taken - Processing:6.1s, Generation:42.4s, Total:48.5s
Time Taken - Processing:6.4s, Generation:43.8s, Total:50.2s
Time Taken - Processing:6.8s, Generation:42.1s, Total:48.9s
Time Taken - Processing:6.6s, Generation:46.1s, Total:52.7s

Now:
Time Taken - Processing:2.4s, Generation:16.0s, Total:18.4s
Time Taken - Processing:2.3s, Generation:16.1s, Total:18.5s
Time Taken - Processing:2.4s, Generation:15.4s, Total:17.8s
Time Taken - Processing:2.6s, Generation:19.0s, Total:21.6s

(On Ubuntu Server 22.04 with i7 10750H)

[LostRuins]

Progress! For anyone else running older versions, do update and compare!