add hf2gguf conv format of q4_0 q4_1 q5_0 q5_1

[chentyjpm]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[compilade]

The main reason I'm hesitant to add this is that llama-quantize uses Q4_K and Q6_K for the token embeddings when quantizing to Q4_0, Q4_1, Q5_0, or Q5_1, and so unlike --outtype q8_0, this is not equivalent to using llama-quantize.

Although I did make an exception for this in #8151 for TQ1_0 and TQ2_0.

Maybe a temporary workaround could be a clear warning in the help text of --outtype, and/or at the end of conversion with these types.

[ggerganov]

Yes, it will cause confusion having different mixtures called the same way. Better to not add this functionality in the python scripts

[chentyjpm]

The main reason I'm hesitant to add this is that llama-quantize uses Q4_K and Q6_K for the token embeddings when quantizing to Q4_0, Q4_1, Q5_0, or Q5_1, and so unlike --outtype q8_0, this is not equivalent to using llama-quantize.

Although I did make an exception for this in #8151 for TQ1_0 and TQ2_0.

Maybe a temporary workaround could be a clear warning in the help text of --outtype, and/or at the end of conversion with these types.

thanks for review !

I read for cpp code in "static void llama_model_quantize_internal" function . er... but i did not find the place where token embeddings is insterted to quantized model.
can I make python code same as llama-quantize by to add Q4_K and Q6_K for the token embeddings for converted model