Support Llama 3 conversion#6745
Merged
ggerganov merged 8 commits intoggml-org:masterfrom Apr 21, 2024
Merged
Conversation
The tokenizer is BPE.
|
What a 🐐 |
|
What a champion lol. PR open within 30 minutes of model release. |
Contributor
|
Doesn't seem that the |
|
I can't convert 70b on this EDIT: run with "--vocab-type bpe" |
|
This is what we did to get the model out -- it doesn't seem like the special tokens are added properly. We are looking deeper for further improvements / fixes. Edit by JG: made collapsible |
Contributor
Author
@USBhost did you try with |
Contributor
|
The instruct models need the |
When I add https://huggingface.co/meta-llama/Meta-Llama-3-70B/blob/main/original/tokenizer.model I get the same error as on convert.py |
Contributor
Author
@m18coppola the instruct models use two different EOS tokens: the standard one ( I'm not sure how to replicate this behaviour yet. The best solution would be to use a list of eos/stop tokens, but I don't know how to do it, any suggestions on where to look? Another idea would be to use |
|
@pcuenca for the changes: "special": false on <|start_header_id|> <|end_header_id|> <|eot_id|> |
Contributor
Author
@jxy Our comments were sent at the same time :) Yes, that's one of the solutions I mentioned, but I'm not sure it will work consistently, I've seen models that use various terminators depending on context. We can try it out though, I'll take a look. |
|
Sorry lads I had to run with --vocab-type bpe |
Closed
Contributor
From the model card on HF: terminators = [
tokenizer.eos_token_id,
tokenizer.convert_tokens_to_ids("<|eot_id|>")
]Not sure if this is helpful or not 😅 but thought I might as well mention it. |
Contributor
|
It seems the model generates |
Contributor
|
It's always the tokenizer. The tokenizers are always a mess. Special tokens apply to the instruct tuned model. The ChatFormat class in the source code shows how they implemented it. The They're using Lots of new special tokens. special_tokens = [
"<|begin_of_text|>",
"<|end_of_text|>",
"<|reserved_special_token_0|>",
"<|reserved_special_token_1|>",
"<|reserved_special_token_2|>",
"<|reserved_special_token_3|>",
"<|start_header_id|>",
"<|end_header_id|>",
"<|reserved_special_token_4|>",
"<|eot_id|>", # end of turn
] + [
f"<|reserved_special_token_{i}|>"
for i in range(5, self.num_reserved_special_tokens - 5)
]This should be interesting (and not in a fun way either). This is gonna create another level of complexity. |
Contributor
Instead of remapping which creates more confusion, just update the generation code to stop on eot_id. At least from my cursory tests, all special texts are tokenized properly out of the box. I did a bit of testing and chat works. |
Contributor
|
Okay, it's in there. # BOS / EOS token IDs
self.bos_id: int = self.special_tokens["<|begin_of_text|>"]
self.eos_id: int = self.special_tokens["<|end_of_text|>"]
self.pad_id: int = -1
self.stop_tokens = {
self.special_tokens["<|end_of_text|>"],
self.special_tokens["<|eot_id|>"],
}@pcuenca The list of stop tokens are usually added during inference. The chat templates have been embedded lately into I think I get it now. Completions: Instructions: That's how I'm interpreting it at the moment. Feel free to correct me. |
Contributor
|
@teleprint-me Yep, you just have to stop on eot_id instead which is: You can use the tokenization tool to test: https://github.com/ggerganov/llama.cpp/blob/master/examples/tokenize/tokenize.cpp
|
Contributor
|
This appears to work for chatting with the model (instruct): |
Contributor
|
In the original Now it's assuming the huggingface BPE format instead of BPE in a general implementation as it was originally. These changes continue to break the The current implementation for 17:46:36 | /mnt/valerie/remote/ggerganov/llama.cpp
(.venv) git:(llama3-conversion | θ) λ python convert.py --vocab-type bpe /mnt/valerie/models/meta-llama/Meta-Llama-3-8B-Instruct
Loading model file /mnt/valerie/models/meta-llama/Meta-Llama-3-8B-Instruct/consolidated.00.pth
params = Params(n_vocab=128256, n_embd=4096, n_layer=32, n_ctx=4096, n_ff=14336, n_head=32, n_head_kv=8, n_experts=None, n_experts_used=None, f_norm_eps=1e-05, rope_scaling_type=None, f_rope_freq_base=500000.0, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=None, path_model=PosixPath('/mnt/valerie/models/meta-llama/Meta-Llama-3-8B-Instruct'))
Traceback (most recent call last):
File "/mnt/valerie/remote/ggerganov/llama.cpp/convert.py", line 1555, in <module>
main()
File "/mnt/valerie/remote/ggerganov/llama.cpp/convert.py", line 1522, in main
vocab, special_vocab = vocab_factory.load_vocab(vocab_types, model_parent_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/valerie/remote/ggerganov/llama.cpp/convert.py", line 1424, in load_vocab
vocab = self._create_vocab_by_path(vocab_types)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/valerie/remote/ggerganov/llama.cpp/convert.py", line 1414, in _create_vocab_by_path
raise FileNotFoundError(f"Could not find a tokenizer matching any of {vocab_types}")
FileNotFoundError: Could not find a tokenizer matching any of ['bpe']The issue propogates from the
These issues are not related to this PR but are now affecting it.
BPE tokenizer implementations keep me up at night. Rant aside, the It should be noted that Llama 1 and Llama 2 used 17:51:49 | /mnt/valerie/models/meta-llama
λ file Meta-Llama-3-8B-Instruct/tokenizer.model
Meta-Llama-3-8B-Instruct/tokenizer.model: ASCII text
17:52:05 | /mnt/valerie/models/meta-llama
λ file /mnt/scsm/models/facebook/llama-2/llama-2-7b/tokenizer.model
/mnt/scsm/models/facebook/llama-2/llama-2-7b/tokenizer.model: data |
Contributor
|
cc @dranger003 - I really appreciated your ppl chart visual + measured ppl gap table for different quantization types for CommandR+. Do you think you would be willing to recreate those comparisons on L3 70b (base or Instruct, preferably base?) Thanks |
Contributor
|
@teleprint-me Are you saying that it's a happy coincidence that the current llama.cpp implementation happens to tokenize correctly or there exists character sequences out there that will be tokenized incorrectly? |
Member
|
Does anyone have convert instructions that work - I'm trying both Meta and HF models using this PR and none of the convert scripts work: $ python3.11 convert.py ~/Data/llama3/Meta-Llama-3-8B/ --outfile ./models/llama-8b-v3/ggml-model-f16.gguf --outtype f16 --vocab-type bpe
FileNotFoundError: Could not find a tokenizer matching any of ['bpe']$ python3.11 convert-hf-to-gguf.py ~/Data/huggingface/Meta-Llama-3-8B/ --outfile ./models/llama-8b-v3/ggml-model-f16.gguf --outtype f16
FileNotFoundError: File not found: /Users/ggerganov/Data/huggingface/Meta-Llama-3-8B/tokenizer.modelI see a few other people reporting the same problems. Those who succeeded - what were the necessary changes? |
Contributor
|
@ggerganov Try |
Member
|
@ddh0 The $ python3.11 convert-hf-to-gguf.py ~/Data/huggingface/Meta-Llama-3-8B/ --outfile ./models/llama-8b-v3/ggml-model-f16.gguf --outtype f16 --vocab-type bpe
usage: convert-hf-to-gguf.py [-h] [--vocab-only] [--awq-path AWQ_PATH] [--outfile OUTFILE] [--outtype {f32,f16}] [--bigendian] [--use-temp-file] model
convert-hf-to-gguf.py: error: unrecognized arguments: --vocab-type bpe |
Contributor
|
@ggerganov These are working on my end.
|
|
Sorry for bothering you guys. It's just a lack of slash in the path... Thank you for your prompt reply! |
Contributor
|
Trust me, you're not alone 😅. I don't know how many times I've been stymied by a '/' or a '\'. |
The documentation really needs to be better. How many resources are being wasted because the documentation is telling people to use convert.py? I know it just cost me about a day. Why isn't there just one interface anyway? Very confusing. I've been making my own quants for months now and I still don't know which one to use when. There should be only one interface and the documentation should be up-to-date and accurate. Crazy ideas, I know |
Collaborator
|
Actually I've been wondering, what's the purpose of the convert.py script? If the hf one does everything needed, should convert.py be removed? |
Well, according to the main readme, that's the only one you should even know about and use. I feel like I'm taking crazy pills |
Contributor
Thanks, sorry for the late response, just saw this one. |
Contributor
|
The purpose of the convert.py script is to partially load tensors dynamically as the conversion occurs. This reduces memory usage during the conversion process. Normally, the entire models weights are loaded which is very RAM intensive. So thats why it exists. It should be easy to understand why this is valuable to have. |
Contributor
Is this true..? I need the full amount of RAM to load models when using convert-hf-to-gguf, multiple hundred GB for the biggest ones |
Contributor
|
It depends on the model. Load a raw torch model (like a 7B one) and watch it as begins to consume about 40gb of RAM. Thats a lot of RAM. Just to load the model! I think clraifying the scripts name would probably help with the confusion. Perhaps convert-torch.py would be more appropriate. |
Contributor
|
I load raw 7B models with only like 20gb of VRAM :S are you loading in FP32? |
Contributor
|
Well, the other option would be bfloat, or half, right? Quants weren't as popular and as widely available when I originally tested it. This was about 2 years ago... wow, time flies. 💀 |
A step in the right direction. Why not merge the scripts into a unified convert.py? |
Contributor
|
@kalomaze Here it is, this is using 400 chunks imatrix on wiki.train for quants below Q6_K.
|
Did anyone find a solution to this?
|
Contributor
|
@mapleroyal You should use |
Even though I'm using the original meta (i.e. non-hf) model? |
Contributor
|
By original you mean the |
Yes, exactly. Ok, got it. Thank you. |
mofosyne
added
Review Complexity : High
Seunghhon
pushed a commit
to Seunghhon/llama.cpp
that referenced
this pull request
Apr 26, 2026
* Support Llama 3 conversion The tokenizer is BPE. * style * Accept suggestion Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com> * llama : add llama_token_is_eog() ggml-ci * llama : auto-detect more EOT tokens when missing in KV data * convert : replacing EOS token is a hack * llama : fix codegemma EOT token + add TODOs * llama : fix model type string for 8B model --------- Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Seunghhon
pushed a commit
to Seunghhon/llama.cpp
that referenced
this pull request
Apr 26, 2026
phuongncn
pushed a commit
to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4
that referenced
this pull request
Apr 28, 2026
* Support Llama 3 conversion The tokenizer is BPE. * style * Accept suggestion Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com> * llama : add llama_token_is_eog() ggml-ci * llama : auto-detect more EOT tokens when missing in KV data * convert : replacing EOS token is a hack * llama : fix codegemma EOT token + add TODOs * llama : fix model type string for 8B model --------- Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
phuongncn
pushed a commit
to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4
that referenced
this pull request
Apr 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The tokenizer is BPE.