ggml : remove bit shuffling

[ggerganov]

Implementation of #1241

Avoid unnecessary bit shuffling by packing the quants in a better way.
Requires model re-quantization

• Q4_0

  • quantize
  • dequantize
  • dot ARM NEON
  • dot AVX

• Q4_1

  • quantize
  • dequantize
  • dot ARM NEON
  • dot AVX

• Q5_0

  • quantize
  • dequantize
  • dot ARM NEON
  • dot AVX
  • dot WASM SIMD

• Q5_1

  • quantize
  • dequantize
  • dot ARM NEON
  • dot AVX
  • dot WASM SIMD

---

New timings:

Model	Measure	F16	Q4_0	Q4_1	Q5_0	Q5_1	Q8_0
7B	ms/tok @ 4th	127	49	56	89	92	74
7B	ms/tok @ 8th	120	44	52	49	52	70
13B	ms/tok @ 4th	261*	91	103	173	177	139
13B	ms/tok @ 8th	316*	81	95	103	113	134

• these numbers vary a lot since the model is on the 32GB limit of my MacBook

Old timings:

Model	Measure	F16	Q4_0	Q4_1	Q5_0	Q5_1	Q8_0
7B	ms/tok @ 4th	128	56	61	91	95	75
7B	ms/tok @ 8th	128	47	55	53	59	75
13B	ms/tok @ 4th	239	104	113	176	185	141
13B	ms/tok @ 8th	240	85	99	108	117	147

overall, all these numbers seem to have about +/- 10% variablility from run to run. not ideal benchmark, but not sure what else to do

[ggerganov]

Unfortunately, Q4_2 does not fit into this pattern, unless we introduce a Q8_2 block with size of 16 instead of 32 elements.
Any suggestions how to proceed? Maybe drop Q4_2 support?

[sw]

A few remarks:

• quantize_row_q4_1_reference is broken, it's missing the i*qk in x[...]
• Q5 interleaves the nibbles but not the single MSBs, that could be confusing. The scalar implementation seems to be broken, maybe it's because of that?
• I still think this could be done in a non-breaking manner for Q4, by simply changing the order of Q8

[ggerganov]

@sw

I still think this could be done in a non-breaking manner for Q4, by simply changing the order of Q8

Yes, I'm still hesitating. But I think Q8 quantization will be sub-optimal this way.
Not much, but still.

[ggerganov]

Q5 interleaves the nibbles but not the single MSBs, that could be confusing. The scalar implementation seems to be broken, maybe it's because of that?

Somehow perplexity computation with Q5_0 + cuBLAS is currently broken and I don't see the error.
It works on M1.

Edit: fixed

[Green-Sky]

https://github.com/ggerganov/llama.cpp/blob/0e48eb6f6b3588d78656267b3b8029b7711f6cdf/ggml.h#L191

can we increment this value by 1 ?

edit: oh, it was all in the llama.h/.cpp

[sw]

can we increment this value by 1 ?

That would make the unaffected formats incompatible - F16, Q8. The clean way would be to define new formats Q4_4, Q4_5, etc. But that gets unwieldy quickly.

[LostRuins]

@sw doesn't have to be though, during loading exceptions can be added in llama.cpp to treat old f16 and q8 format with either file versions 1 or 2 as forwards compatible.

[ggerganov]

Close in favor of #1405

[ProfessorSparrs]

great, I finally compiled it on my pc(no avx2 support) AND with cuda support. And this change makes none of my models load :(. I dont know how to quantize things, Ive read a lot about it and I doubt I even have the PC-resources to do it.

[Green-Sky]

@ProfessorSparrs if you have the f16 files, qnantizing is very easy and WAY less recource intensive than running the model. :) (check the quantize executable)