Feature Request: Support Kimi K2

[yeahdongcn]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Kimi K2 is a new open-source model available.

I tried to load this converted gguf but failed due to two issues:

1. Number of experts exceeds 256

llama_model_loader: - kv  24:                     deepseek2.expert_count u32              = 384

2. Missing tokenizer.ggml.model

llama_model_load: error loading model: error loading model vocabulary: key not found in model: tokenizer.ggml.model
llama_model_load_from_file_impl: failed to load model

Motivation

From the benchmark data, Kimi K2 turns to be the most powerful oss model now.

Possible Implementation

No response

[jukofyork]

1. Number of experts exceeds 256

llama_model_loader: - kv 24: deepseek2.expert_count u32 = 384

This should be an easy fix:

llama.cpp/src/llama-hparams.h

Line 9 in f5e96b3

#define LLAMA_MAX_EXPERTS 256 // DeepSeekV3

2. Missing `tokenizer.ggml.model`

llama_model_load: error loading model: error loading model vocabulary: key not found in model: tokenizer.ggml.model
llama_model_load_from_file_impl: failed to load model

https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/tokenization_kimi.py

As far as I can see, everything else seems to be exactly the same as deepseek-v3 architecture (it has only 1 dense layer at the start instead of 3 though).

I'm downloading it now - very slowly lol.

[jukofyork]

It might also be worth trying the hack suggested here: #12610

[yeahdongcn]

Yeah, I got the first one working. However, I’m using a cloud instance with limited storage, so I’m unable to perform the conversion on it.

I was able to extend the storage. Here is the new issue in converting from HF:

root@rocm-jupyter-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp# python3 convert_hf_to_gguf.py /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e
INFO:hf-to-gguf:Loading model: 2f7e011530aa89b56ab4277c26fc4aed3c52822e
WARNING:hf-to-gguf:Failed to load model config from /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e: The repository /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e contains custom code which must be executed to correctly load the model. You can inspect the repository content at /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e .
 You can inspect the repository content at https://hf.co//scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
WARNING:hf-to-gguf:Trying to load config.json instead
INFO:hf-to-gguf:Model architecture: DeepseekV3ForCausalLM
WARNING:hf-to-gguf:Failed to load model config from /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e: The repository /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e contains custom code which must be executed to correctly load the model. You can inspect the repository content at /scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e .
 You can inspect the repository content at https://hf.co//scratch/hub/models--moonshotai--Kimi-K2-Instruct/snapshots/2f7e011530aa89b56ab4277c26fc4aed3c52822e.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
WARNING:hf-to-gguf:Trying to load config.json instead
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:gguf: loading model weight map from 'model.safetensors.index.json'
INFO:hf-to-gguf:gguf: loading model part 'model-1-of-61.safetensors'
INFO:hf-to-gguf:token_embd.weight,            torch.bfloat16 --> F16, shape = {7168, 163840}
INFO:hf-to-gguf:blk.0.attn_norm.weight,       torch.bfloat16 --> F32, shape = {7168}
INFO:hf-to-gguf:blk.0.ffn_down.weight,        torch.float8_e4m3fn --> F16, shape = {18432, 7168}
Traceback (most recent call last):
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 7411, in <module>
    main()
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 7405, in main
    model_instance.write()
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 410, in write
    self.prepare_tensors()
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 5679, in prepare_tensors
    super().prepare_tensors()
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 277, in prepare_tensors
    for new_name, data_torch in (self.modify_tensors(data_torch, name, bid)):
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 5676, in modify_tensors
    return [(self.map_tensor_name(name), data_torch)]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/llama.cpp/convert_hf_to_gguf.py", line 236, in map_tensor_name
    raise ValueError(f"Can not map tensor {name!r}")
ValueError: Can not map tensor 'model.layers.0.mlp.down_proj.weight_scale_inv'
root@rocm-jupyter-gpu-mi300x1-192gb-devcloud-atl1:~/llama.cpp#

[danielhanchen]

The tiktoken tokenizer does look like an issue - similar to Hunyuan's changes - I think it's mostly similar to GPT4o's tokenizer maybe?

[CISC]

I was able to extend the storage. Here is the new issue in converting from HF:
ValueError: Can not map tensor 'model.layers.0.mlp.down_proj.weight_scale_inv'

They have uploaded FP8 tensors, you have to convert them to BF16 first, using DeepSeek's convert.pyfp8_cast_bf16.py.

[jukofyork]

I was able to extend the storage. Here is the new issue in converting from HF:
ValueError: Can not map tensor 'model.layers.0.mlp.down_proj.weight_scale_inv'

They have uploaded FP8 tensors, you have to convert them to BF16 first, using DeepSeek's convert.py.

This won't work on Ampere or lower cards sadly. I hacked out the Triton stuff and created this version:

#11446 (comment)

that should work just using CPU if needed.

[jukofyork]

Actually, I'm not sure what convert.py is for. It's this code that is used to convert form fp8 to bf16:

https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/inference/fp8_cast_bf16.py

that sadly relies on this Triton kerel:

https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py

which won't work unless your GPU has native fp8 (ie: Ada or newer).

[CISC]

Actually, I'm not sure what convert.py is for. It's this code that is used to convert form fp8 to bf16:

https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/inference/fp8_cast_bf16.py

Ah, sorry, I just remembered it was in that repo and picked the first reasonably looking file. :)

[danielhanchen]

By the way if it helps, I made BF16 via the DeepSeek script conversion - https://huggingface.co/unsloth/Kimi-K2-Instruct-BF16

The tokenizer needs a new regex it seems, but I'm assuming it's similar to Hunyuan's conversion process, but more similar to GPT4o's tokenizer. Also they seem to tokenize Chinese characters more differently

[artraze]

After to converting to bf16 (and trusting remote) the tokenizer seems to be a problem.

convert_hf_to_gguf.py fails on vocab_size = self.hparams.get("vocab_size", len(tokenizer.vocab)) due to vocab not existing on tokenizer. If you bypass that you get

WARNING:hf-to-gguf:** WARNING: The BPE pre-tokenizer was not recognized!
WARNING:hf-to-gguf:**          There are 2 possible reasons for this:
WARNING:hf-to-gguf:**          - the model has not been added to convert_hf_to_gguf_update.py yet
WARNING:hf-to-gguf:**          - the pre-tokenization config has changed upstream
WARNING:hf-to-gguf:**          Check your model files and convert_hf_to_gguf_update.py and update them accordingly.
WARNING:hf-to-gguf:** ref:     https://github.com/ggml-org/llama.cpp/pull/6920
WARNING:hf-to-gguf:**
WARNING:hf-to-gguf:** chkhsh:  81212dc7cdb7e0c1074ca62c5aeab0d43c9f52b8a737be7b12a777c953027890

Hacking kimi-k2 into ./convert_hf_to_gguf_update.py and running yields:

Traceback (most recent call last):
  File "/home/aicli/llama/llama.cpp/./convert_hf_to_gguf_update.py", line 268, in <module>
    tokenizer = AutoTokenizer.from_pretrained(f"models/tokenizers/{name}", trust_remote_code=True)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aicli/llama/llama.cpp/venv/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py", line 949, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers_modules.kimi-k2.configuration_deepseek.DeepseekV3Config'> to build an AutoTokenizer.
Model type should be one of AlbertConfig, AlignConfig, ...

I would assume that this is not finding the configuration_deepseek.py file or that isn't properly registering its contents but I don't know any of these libraries so I tapped out at that point.

[nicoboss]

Here my patches to get convert_hf_to_gguf .py to work for Kimi-K2-Instruct. I have absolutely no idea if this is correct but it converts. Please correct me if I did anything wrong.

I downloaded the original model and used the following command using the script from original DeepSeek V3 to conveart it to BF16: CUDA_VISIBLE_DEVICES=1 venv/bin/python fp8_cast_bf16.py --input-fp8-hf-path /cpool/Kimi-K2-Instruct --output-bf16-hf-path /bpool/Kimi-K2-Instruct_BF16

diff --git a/convert_hf_to_gguf.py b/convert_hf_to_gguf.py
index 8afb425b..80046c2a 100755
--- a/convert_hf_to_gguf.py
+++ b/convert_hf_to_gguf.py
@@ -608,12 +608,12 @@ class TextModel(ModelBase):

         from transformers import AutoTokenizer
         tokenizer = AutoTokenizer.from_pretrained(self.dir_model)
-        vocab_size = self.hparams.get("vocab_size", len(tokenizer.vocab))
-        assert max(tokenizer.vocab.values()) < vocab_size
+        vocab_size = self.hparams.get("vocab_size", tokenizer.vocab_size)
+        #assert max(tokenizer.vocab.values()) < vocab_size

         tokpre = self.get_vocab_base_pre(tokenizer)

-        reverse_vocab = {id_: encoded_tok for encoded_tok, id_ in tokenizer.vocab.items()}
+        reverse_vocab = {id_: encoded_tok for encoded_tok, id_ in tokenizer.get_vocab().items()}
         added_vocab = tokenizer.get_added_vocab()

         added_tokens_decoder = tokenizer.added_tokens_decoder
@@ -780,6 +780,9 @@ class TextModel(ModelBase):
         if chkhsh == "877081d19cf6996e2c4ff0e1236341e9b7bde288f5311a56a937f0afbbb3aeb5":
             # ref: https://huggingface.co/deepseek-ai/DeepSeek-V3
             res = "deepseek-v3"
+        if chkhsh == "81212dc7cdb7e0c1074ca62c5aeab0d43c9f52b8a737be7b12a777c953027890":
+            # ref: https://huggingface.co/moonshotai/Kimi-K2-Instruct
+            res = "deepseek-v3"
         if chkhsh == "b3f499bb4255f8ca19fccd664443283318f2fd2414d5e0b040fbdd0cc195d6c5":
             # ref: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
             res = "deepseek-r1-qwen"

[nicoboss]

Simply converting the model as described above and increasing LLAMA_MAX_EXPERTS is not enough to get the model to load as it fails with lama_model_load: error loading model: error loading model vocabulary: cannot find tokenizer merges in model file.

root@AI:/apool/llama.cpp/build/bin# ./llama-cli -m /cpool/Kimi-K2-Instruct.gguf -c 2048
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
  Device 1: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
build: 5962 (46c3d25b) with cc (Debian 12.2.0-14+deb12u1) 12.2.0 for x86_64-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4090) - 23686 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 4090) - 2572 MiB free
llama_model_loader: loaded meta data with 43 key-value pairs and 1096 tensors from /cpool/Kimi-K2-Instruct.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Kimi K2 Instruct_BF16
llama_model_loader: - kv   3:                         general.size_label str              = 384x14B
llama_model_loader: - kv   4:                            general.license str              = other
llama_model_loader: - kv   5:                       general.license.name str              = modified-mit
llama_model_loader: - kv   6:                      deepseek2.block_count u32              = 61
llama_model_loader: - kv   7:                   deepseek2.context_length u32              = 131072
llama_model_loader: - kv   8:                 deepseek2.embedding_length u32              = 7168
llama_model_loader: - kv   9:              deepseek2.feed_forward_length u32              = 18432

llama_model_loader: - kv  22:         deepseek2.attention.key_length_mla u32              = 192
llama_model_loader: - kv  23:       deepseek2.attention.value_length_mla u32              = 128
llama_model_loader: - kv  24:       deepseek2.expert_feed_forward_length u32              = 2048
llama_model_loader: - kv  25:                     deepseek2.expert_count u32              = 384
llama_model_loader: - kv  26:              deepseek2.expert_shared_count u32              = 1
llama_model_loader: - kv  27:             deepseek2.expert_weights_scale f32              = 2.827000
llama_model_loader: - kv  28:              deepseek2.expert_weights_norm bool             = true
llama_model_loader: - kv  29:               deepseek2.expert_gating_func u32              = 2
llama_model_loader: - kv  30:             deepseek2.rope.dimension_count u32              = 64
llama_model_loader: - kv  31:                deepseek2.rope.scaling.type str              = yarn
llama_model_loader: - kv  32:              deepseek2.rope.scaling.factor f32              = 32.000000
llama_model_loader: - kv  33: deepseek2.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  34: deepseek2.rope.scaling.yarn_log_multiplier f32              = 0.100000
llama_model_loader: - kv  35:               general.quantization_version u32              = 2
llama_model_loader: - kv  36:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  37:                         tokenizer.ggml.pre str              = deepseek-v3
llama_model_loader: - kv  38:                      tokenizer.ggml.tokens arr[str,163840]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  39:                  tokenizer.ggml.token_type arr[i32,163840]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  40:                tokenizer.ggml.bos_token_id u32              = 163584
llama_model_loader: - kv  41:                tokenizer.ggml.eos_token_id u32              = 163585
llama_model_loader: - kv  42:                    tokenizer.chat_template str              = {% if tools -%}\n    {{ '<|im_system|>...
llama_model_loader: - type  f32:  365 tensors
llama_model_loader: - type  f16:  731 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = F16
print_info: file size   = 1912.14 GiB (16.00 BPW)
llama_model_load: error loading model: error loading model vocabulary: cannot find tokenizer merges in model file

llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model '/cpool/Kimi-K2-Instruct.gguf'
main: error: unable to load model

[nyuuzyou]

This won't work on Ampere or lower cards sadly. I hacked out the Triton stuff and created this version:

#11446 (comment)

that should work just using CPU if needed.

I used triton-cpu for the conversion without GPU and it worked well with minimal modifications to the original script.