Vulkan IQ4_NL Support

[0cc4m]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

---

• Add IQ4_NL support to Vulkan to resolve the issue with iq4_nl fallbacks in k-quants (llama : change fallback type IQ4_NL -> Q4_0 #8489 and Bug: QWEN2 quantization GGML_ASSERT #7805 (comment)).
• Increase the mat_mul_id matrix multiplication row_ids buffer size to allow larger MoEs (like DeepSeek-Coder-V2-Lite which had the iq4_nl issue) to work with Vulkan.
• Fix Vulkan test code that was broken after the last rework

[sorasoras]

How much effort is needed to support Iq4xs in additional to iq4nl？
vulkan backend would be a lot more useful if iq4xs is supported.

[0cc4m]

How much effort is needed to support Iq4xs in additional to iq4nl？ vulkan backend would be a lot more useful if iq4xs is supported.

Can you elaborate what specific cases that would enable?

[sorasoras]

How much effort is needed to support Iq4xs in additional to iq4nl？ vulkan backend would be a lot more useful if iq4xs is supported.

Can you elaborate what specific cases that would enable?

IQ4XS is common used among community due to its small size and better PPL than Q4KM. It s a sweet spot in GGUF quant series.

[0cc4m]

How much effort is needed to support Iq4xs in additional to iq4nl？ vulkan backend would be a lot more useful if iq4xs is supported.

Can you elaborate what specific cases that would enable?

IQ4XS is common used among community due to its small size and better PPL than Q4KM. It s a sweet spot in GGUF quant series.

It's quite a bit of effort, but at least it's easier than the other i-quants. I can't do it now, but should be able to at some point in the not-too-distant future.

[oldgithubman]

• I have read the contributing guidelines

• Self-reported review complexity:

  • Low
  • Medium
  • High

• Add IQ4_NL support to Vulkan to resolve the issue with iq4_nl fallbacks in k-quants (llama : change fallback type IQ4_NL -> Q4_0 #8489 and Bug: QWEN2 quantization GGML_ASSERT #7805 (comment)).

• Increase the mat_mul_id matrix multiplication row_ids buffer size to allow larger MoEs (like DeepSeek-Coder-V2-Lite which had the iq4_nl issue) to work with Vulkan.

• Fix Vulkan test code that was broken after the last rework

Where there's a will, there's a way =;-)

[slaren]

While testing this I got tests failures with fp16/fp32 mul mat, but it also happens on master.

Vulkan0: NVIDIA GeForce RTX 3090 Ti (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32
  MUL_MAT(type_a=f32,type_b=f32,m=71,n=82,k=367,bs=[1,1],nr=[1,1]): [MUL_MAT] NMSE = 1.791590151 > 0.000500000 FAIL
  MUL_MAT(type_a=f32,type_b=f32,m=73,n=31,k=10,bs=[1,1],nr=[1,1]): [MUL_MAT] NMSE = 1.618188214 > 0.000500000 FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=42,n=85,k=77,bs=[1,1],nr=[1,1]): [MUL_MAT] NMSE = 1.844454779 > 0.000500000 FAIL
  MUL_MAT(type_a=f32,type_b=f32,m=106,n=17,k=50,bs=[1,1],nr=[1,1]): [MUL_MAT] NMSE = 1.509752347 > 0.000500000 FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=80,n=110,k=345,bs=[1,1],nr=[1,1]): [MUL_MAT] NMSE = 1.739823578 > 0.000500000 FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=73,n=46,k=361,bs=[1,1],nr=[1,1]): [MUL_MAT] NMSE = 1.642865213 > 0.000500000 FAIL
  MUL_MAT(type_a=f32,type_b=f32,m=18,n=27,k=153,bs=[1,1],nr=[1,1]): [MUL_MAT] NMSE = 1.802821297 > 0.000500000 FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=12,n=80,k=182,bs=[1,1],nr=[1,1]): [MUL_MAT] NMSE = 1.590408512 > 0.000500000 FAIL
  MUL_MAT(type_a=f32,type_b=f32,m=110,n=42,k=6,bs=[1,1],nr=[1,1]): [MUL_MAT] NMSE = 1.487417955 > 0.000500000 FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=98,n=56,k=484,bs=[1,1],nr=[1,1]): [MUL_MAT] NMSE = 0.988843480 > 0.000500000 FAIL
  MUL_MAT(type_a=f32,type_b=f32,m=8,n=22,k=223,bs=[1,1],nr=[1,1]): [MUL_MAT] NMSE = 1.419249893 > 0.000500000 FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=63,n=51,k=452,bs=[1,1],nr=[1,1]): [MUL_MAT] NMSE = 1.033080833 > 0.000500000 FAIL

[LostRuins]

Hi @0cc4m can I check, what exactly is this assert testing for?

ref: LostRuins#1337

[0cc4m]

https://github.com/ggerganov/llama.cpp/blob/3252afb32363230a6e0a3356c75c526049913a89/ggml/src/vulkan-shaders/mul_mm.comp#L74

For the maximum number of row_ids the mat_mul_id shader can handle.

[LostRuins]

Deepseek 16B MoE (6/64 experts)

nei0 = 6
nei1 = 1024

nei0 x nei1 = 6144