Allow conversion of Llama / Mistral HF models

[pcuenca]

This allows to convert fine-tuned models with convert-hf-to-gguf.py. The base architecture is set to llama, as in the models converted by @TheBloke. If necessary, we can add a new entry to constants.py.

[cebtenzzre]

Why not just use:

@Model.register("LlamaForCausalLM", "MistralForCausalLM", "MixtralForCausalLM")

on the existing MixtralModel (call it LlamaModel)? I don't see a point in supporting Mistral in this script without supporting Llama, and these classes are identical so they can be merged.

[pcuenca]

Sure, we can do that. I tried to mimic what I saw in the existing entries.

In my opinion, it could be interesting to use different architectures for Mistral (and Mixtral) for informative purposes. If you click on the GGUF information for any Mistral file on the Hugging Face Hub, there's nothing that refers to mistral except the filename. But that would also require additional changes, so happy to use the same entry for the three of them!

[cebtenzzre]

I tried to convert this model: https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca

I got this exception:

$ TMPDIR=/var/tmp ./convert-hf-to-gguf.py ~/dirs/text-ai-models/dl/Mistral-7B-OpenOrca --outfile /dev/null
Loading model: Mistral-7B-OpenOrca
gguf: This GGUF file is for Little Endian only
Set model parameters
Set model tokenizer
Traceback (most recent call last):
  File "/home/jared/src/forks/llama.cpp/./convert-hf-to-gguf.py", line 2073, in <module>
    main()
  File "/home/jared/src/forks/llama.cpp/./convert-hf-to-gguf.py", line 2060, in main
    model_instance.set_vocab()
  File "/home/jared/src/forks/llama.cpp/./convert-hf-to-gguf.py", line 1051, in set_vocab
    self._set_vocab_sentencepiece()
  File "/home/jared/src/forks/llama.cpp/./convert-hf-to-gguf.py", line 324, in _set_vocab_sentencepiece
    piece = tokenizer.id_to_piece(token_id)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jared/.venv/lib/python3.11/site-packages/sentencepiece/__init__.py", line 1179, in _batched_func
    return _func(self, arg)
           ^^^^^^^^^^^^^^^^
  File "/home/jared/.venv/lib/python3.11/site-packages/sentencepiece/__init__.py", line 1172, in _func
    raise IndexError('piece id is out of range.')
IndexError: piece id is out of range.

This model converts fine with convert.py.

[pcuenca]

Testing conversion with fine-tuned models I experienced this problem: #6320. If that PR is merged, then we can also use _set_vocab_sentencepiece() here.

[pcuenca]

Sorry for the delay, @cebtenzzre, I could only return to this today. Testing the model you mentioned (https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca) I experienced an inconsistency in the sentencepiece vocab method as reported in #6320. In addition, I had not taken care of tensor permutation in my initial PR.

I tested conversion of https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca and verified that generation matches conversion with convert.py using a temperature of 0.

[pcuenca]

It looks like the CI timed out in 6 hours.

[cebtenzzre]

This PR conflicts with #6355 which renames HfVocab to LlamaHfVocab and makes it specific to models with tokenizer.json - Mistral-7B-OpenOrca only has a tokenizer.model.

To be consistent with convert.py with the default --vocab-type, after that PR is merged you would want to do something like:

try:
    self. _set_vocab_sentencepiece()
except FileNotFoundError:
    self._set_vocab_llama_hf()

This benefits from a conditional dependency on transformers (sentencepiece is a required dependency atm) and accurate token scores when tokenizer.model is available. Does that seem reasonable?

[pcuenca]

@cebtenzzre yes, makes total sense! I merged and applied those changes, then tested with Mistral-7B-OpenOrca and Mistral-7B-v0.1.

[cebtenzzre]

With the changes I added, the set of metadata keys is exactly the same as those written by convert.py (checked with gguf-dump), with the only difference that general.name is "dl" with convert.py (it uses the parent directory name, which supposedly has meaning for the original llama checkpoints) and "Mistral-7B-OpenOrca" with convert-hf-to-gguf.py, which is much more useful.

[pcuenca]

Thank you! Much appreciated! 🙌