quantize: fix F16/F32 downcast to q6_K

[JohannesGaessler]

While converting the official Gemma FP32 GGUF to FP16 with quantize I noticed that token_embd.weight was being converted to q6_k. This seems to be a simple oversight in the code where the only data type checked against is q8_0 but not f16/f32. This PR adds the missing checks.

[ggerganov]

Wouldn't it make more sense to not quantize at all if the requested type is F16 or F32:

diff --git a/llama.cpp b/llama.cpp
index 24944216..a08d3874 100644
--- a/llama.cpp
+++ b/llama.cpp
@@ -12106,8 +12106,10 @@ static void llama_model_quantize_internal(const std::string & fname_inp, const s
                llama_format_tensor_shape(tensor).c_str(),
                ggml_type_name(tensor->type));
 
+        bool quantize = ggml_is_quantized(quantized_type);
+
         // This used to be a regex, but <regex> has an extreme cost to compile times.
-        bool quantize = name.rfind("weight") == name.size() - 6; // ends with 'weight'?
+        quantize &= name.rfind("weight") == name.size() - 6; // ends with 'weight'?
 
         // quantize only 2D tensors
         quantize &= (ggml_n_dims(tensor) == 2);

note: I haven't tested this

[JohannesGaessler]

With the proposed code change it would not be possible to do f16 <-> f32 conversions because the tensors would simply be copied.

[ggerganov]

I pushed an alternative fix + some minor code cleanup