[Enhancement] Add support for Metal inference

[beebopkim]

Please make enabled Metal inference for GGML weight models.

Now, llama.cpp can generate texts using Metal inference on Apple Silicon computers. It is very good news for M1/M2 users, and I can run LLaMA 65B GGML q4_0 model on my M1 Max computer at the speed of 4 ~ 5 tokens/s. It is awesome and really fantastic!

koboldcpp repository already has related source codes from llama.cpp like ggml-metal.h, ggml-metal.m, and ggml-metal.metal. So please make them available during inference for text generation. It would be a very special present for Apple Silicon computer users.

[LostRuins]

Sure, I can try that, but I will need someone who can test it for me as I don't have a mac to test myself.

I have added the flags to the makefile. Can you please try building with make LLAMA_METAL=1 and tell me if it works? If not, please detail what errors there are. Thanks

[MeeTee55]

Does kobold.cpp use metal by default when you compile it with LLAMA_METAL=1? In llama.cpp you need to start main with "-ngl 1" to use it.

[LostRuins]

You should use the koboldcpp command line arg --gpulayers which sets the same value.

[Dusterwald]

I've got an M2 Max MacBook that I've just tried it on. Ran make clean followed by make LLAMA_METAL=1.
It throws the following error for me:

Undefined symbols for architecture arm64:
  "_ggml_metal_add_buffer", referenced from:
      _llama_init_from_file in gpttype_adapter_failsafe.o
  "_ggml_metal_get_tensor", referenced from:
      llama_eval_internal(llama_context&, int const*, int, int, int, char const*) in gpttype_adapter_failsafe.o
  "_ggml_metal_graph_compute", referenced from:
      llama_eval_internal(llama_context&, int const*, int, int, int, char const*) in gpttype_adapter_failsafe.o
  "_ggml_metal_init", referenced from:
      _llama_init_from_file in gpttype_adapter_failsafe.o
ld: symbol(s) not found for architecture arm64

Not sure if I'm missing something or what. This is with the latest commit to the concedo_experimental branch (commit 6635f7e).
Hope that helps @LostRuins. Happy to do more testing if needed.

[LostRuins]

I've made a new commit to that branch to add the flags, can you see if it works now?

[Dusterwald]

That fixed the build problem, thanks. Will need to evaluate the performance a bit later.

[Dusterwald]

Tried running it with Metal enabled (on a 65B model) using this python3 koboldcpp.py --gpulayers 80 models/VicUnlocked-Alpaca-65B.ggmlv3.q4_0.bin

But that throws an error during initialization:

llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 8192
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 64
llama_model_load_internal: n_layer    = 80
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 22016
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 65B
llama_model_load_internal: ggml ctx size =    0.18 MB
llama_model_load_internal: mem required  = 38610.46 MB (+ 5120.00 MB per state)
.
llama_init_from_file: kv self size  = 5120.00 MB
ggml_metal_init: allocating
ggml_metal_init: using MPS
ggml_metal_init: loading '(null)'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=258 "The file name is invalid."

Unfortunately looks a bit cryptic to me, no idea what it means.

[beebopkim]

ggml_metal_init: using MPS
ggml_metal_init: loading '(null)'
ggml_metal_init: error: Error Domain=NSCocoaErrorDomain Code=258 "The file name is invalid."

Unfortunately looks a bit cryptic to me, no idea what it means.

This error is happened when codes in ggml-metal.m tried to load 'ggml-metal.metal' through line 102 and 115, though ggml-metal.metal file was existed in same directory, it threw an error. I have no idea how to deal with this. Please suggest any idea to solve this problem.

101:        //NSString * path = [[NSBundle mainBundle] pathForResource:@"../../examples/metal/metal" ofType:@"metal"];
102:        NSString * path = [[NSBundle mainBundle] pathForResource:@"ggml-metal" ofType:@"metal"];
103:        fprintf(stderr, "%s: loading '%s'\n", __func__, [path UTF8String]);
104:
105:        NSString * src  = [NSString stringWithContentsOfFile:path encoding:NSUTF8StringEncoding error:&error];
106:        if (error) {
107:            fprintf(stderr, "%s: error: %s\n", __func__, [[error description] UTF8String]);
108:            exit(1);
109:        }  
110:    
111:        ctx->library = [ctx->device newLibraryWithSource:src options:nil error:&error];
112:        if (error) {
113:            fprintf(stderr, "%s: error: %s\n", __func__, [[error description] UTF8String]);
114:            exit(1);
115:        }

[kmilner]

I'm not familiar enough with objective c to tell why the NSBundle pathForResource works in the llama.cpp repo but not here, however if you want a work around you could change those lines to use the relative path directly:
e.g. comment out line 102 and 103, and change 105 to be

NSString * src  = [NSString stringWithContentsOfFile:@"./ggml-metal.metal" encoding:NSUTF8StringEncoding error:&error];

I imagine this will break if you try to embed kobold in an NSBundle (like a .app folder), but it works for the normal local invocation.

[LostRuins]

Hmm I have not tried the metal implementation at all, nor have I modified any of the files. If you have recommendations to change them do let me know or send a PR, as I am not familiar with it at all.

[beebopkim]

Finally, I have made a pull request #221 using a patch file to change the path of ggml-metal.metal file in ggml-metal.m

Below clip is an example with LLaMA 65B GGML q4_0.

koboldcpp.metal.mp4

[Dusterwald]

I applied the fix manually and can confirm that it works. Getting ~3.5 T/sec on the 65B model with my M2 Max. Pretty damn good, and fast enough for chat. Prompt processing is the biggest bottleneck now, I hope that can be sped up with Metal too at some point.