Misc. bug: GLM-4.7-Flash (Vulkan on 7900 XTX) inference faster @d8192 than @d4096 or @d0

[Nindaleth]

Name and Version

version: 7901 (8a98ba4)
built with GNU 15.2.1 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-bench

Command line

./build/bin/llama-bench --model ./models/GLM-4.7-Flash-IQ4_XS.gguf -ngl 99 -fa 0,1 -d 0,4096,8192

Problem description & steps to reproduce

With FA enabled, I get generally worse performance with Vulkan than with ROCm, but I'm not here about PP this time. What is interesting is that smaller pre-filled context window results in worse TG performance (see the logs below for full table including fa=0 and ROCm).

This is on Fedora 43, 7900 XTX, ROCm 6.4.2, Vulkan 1.4.342, Mesa RADV 26.1.0-0.5.gita8fac76:

model	size	params	backend	ngl	fa	test	t/s
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128	102.97 ± 0.08
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4096	41.81 ± 0.01
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d8192	117.59 ± 0.07

First Bad Commit

No response

Relevant log output

Logs

# device info on ROCm
  Device 0: Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32
# device info on Vulkan
ggml_vulkan: 0 = Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
# reproducible results
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | ROCm       |  99 |  0 |           pp512 |      2590.64 ± 22.66 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | Vulkan     |  99 |  0 |           pp512 |        916.86 ± 2.69 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | ROCm       |  99 |  0 |           tg128 |         83.17 ± 0.03 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | Vulkan     |  99 |  0 |           tg128 |         74.51 ± 0.06 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | ROCm       |  99 |  0 |   pp512 @ d4096 |      1146.33 ± 20.77 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | Vulkan     |  99 |  0 |   pp512 @ d4096 |        272.53 ± 2.52 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | ROCm       |  99 |  0 |   tg128 @ d4096 |         44.94 ± 0.02 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | Vulkan     |  99 |  0 |   tg128 @ d4096 |         23.02 ± 0.12 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | ROCm       |  99 |  0 |   pp512 @ d8192 |        734.88 ± 5.89 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | Vulkan     |  99 |  0 |   pp512 @ d8192 |        341.09 ± 1.12 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | ROCm       |  99 |  0 |   tg128 @ d8192 |         27.81 ± 0.02 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | Vulkan     |  99 |  0 |   tg128 @ d8192 |         73.74 ± 0.04 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | ROCm       |  99 |  1 |           pp512 |      2684.54 ± 19.55 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | Vulkan     |  99 |  1 |           pp512 |        786.92 ± 1.40 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | ROCm       |  99 |  1 |           tg128 |         89.74 ± 0.08 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | Vulkan     |  99 |  1 |           tg128 |        102.97 ± 0.08 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | ROCm       |  99 |  1 |   pp512 @ d4096 |       1202.67 ± 3.16 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | Vulkan     |  99 |  1 |   pp512 @ d4096 |        748.20 ± 2.73 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | ROCm       |  99 |  1 |   tg128 @ d4096 |         82.65 ± 0.07 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | Vulkan     |  99 |  1 |   tg128 @ d4096 |         41.81 ± 0.01 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | ROCm       |  99 |  1 |   pp512 @ d8192 |        775.44 ± 2.65 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | Vulkan     |  99 |  1 |   pp512 @ d8192 |        637.03 ± 3.20 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | ROCm       |  99 |  1 |   tg128 @ d8192 |         76.71 ± 0.03 |
| deepseek2 30B.A3B IQ4_XS - 4.25 bpw |  15.15 GiB |    29.94 B | Vulkan     |  99 |  1 |   tg128 @ d8192 |        117.59 ± 0.07 |

[0cc4m]

I think the pp512 discrepancy with ROCm might be IQ4_XS. We don't support that as well as non-iq quants on Vulkan yet.

But yeah, something odd is going on with tg, can you check if that also happens with a different quant like q4_k or q4_0?

[jeffbolznv]

I wasn't able to reproduce this on my system with either the coopmat2 or coopmat1 path. I don't think the quant type should matter that much since the context size primarily affects attention and FA doesn't depend on the type. The small_cache optimization might explain an unexpected change from d=0 to d=4096, but I don't know why it would get faster at d=8192.

[Nindaleth]

I think the pp512 discrepancy with ROCm might be IQ4_XS. We don't support that as well as non-iq quants on Vulkan yet.

OK, I chedked with Q4_K_M and Q4_0 and the tests agree (I didn't check specifically against ROCm, but the PP is already 2x faster compared to IQ4_XS).

But yeah, something odd is going on with tg, can you check if that also happens with a different quant like q4_k or q4_0?

I don't think the quant type should matter that much since the context size primarily affects attention and FA doesn't depend on the type.

I half-hoped that a fresh start of the machine today would show it was an ephemeral issue, but the trend is the same across IQ4_XS, Q4_K_M and Q4_0. All following still tested at b7901 (8a98ba4):

./build/bin/llama-bench --model ./models/GLM-4.7-Flash-IQ4_XS.gguf,./models/GLM-4.7-Flash-Q4_K_M.gguf,./models/GLM-4.7-Flash-Q4_0.gguf -ngl 99 -fa 1 -d 4736,4737

model	size	params	backend	ngl	fa	test	t/s
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4736	1248.34 ± 19.79
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4737	1251.24 ± 20.14
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4736	36.98 ± 1.06
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4737	127.14 ± 0.10
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	1	pp512 @ d4736	1294.74 ± 14.34
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	1	pp512 @ d4737	1294.89 ± 15.92
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	1	tg128 @ d4736	35.02 ± 0.16
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	1	tg128 @ d4737	102.62 ± 0.92
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512 @ d4736	1370.91 ± 19.51
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512 @ d4737	1370.37 ± 16.15
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128 @ d4736	36.64 ± 0.44
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128 @ d4737	119.76 ± 0.13

Full logs

./build/bin/llama-bench --model ./models/GLM-4.7-Flash-IQ4_XS.gguf -ngl 99 -fa 0,1 -d 0,4096,8192

model	size	params	backend	ngl	fa	test	t/s
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	ROCm	99	0	pp512	2602.22 ± 3.08
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	ROCm	99	0	pp512 @ d4096	1152.34 ± 19.69
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	ROCm	99	0	pp512 @ d8192	737.74 ± 5.31
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	ROCm	99	0	tg128	83.21 ± 0.02
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	ROCm	99	0	tg128 @ d4096	44.83 ± 0.15
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	ROCm	99	0	tg128 @ d8192	27.77 ± 0.05
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	ROCm	99	1	pp512	2685.84 ± 17.46
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	ROCm	99	1	pp512 @ d4096	1196.70 ± 16.45
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	ROCm	99	1	pp512 @ d8192	772.77 ± 2.87
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	ROCm	99	1	tg128	89.48 ± 0.32
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	ROCm	99	1	tg128 @ d4096	82.41 ± 0.46
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	ROCm	99	1	tg128 @ d8192	76.56 ± 0.32
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	0	pp512	1144.69 ± 10.22
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	0	pp512 @ d4096	277.32 ± 0.65
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	0	pp512 @ d8192	342.30 ± 0.80
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	0	tg128	86.56 ± 0.81
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	0	tg128 @ d4096	22.61 ± 0.13
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	0	tg128 @ d8192	73.60 ± 0.07
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512	902.04 ± 2.50
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4096	1174.08 ± 9.37
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d8192	906.69 ± 5.21
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128	106.50 ± 0.05
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4096	41.67 ± 0.17
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d8192	116.85 ± 1.18

./build/bin/llama-bench --model ./models/GLM-4.7-Flash-Q4_K_M.gguf -ngl 99 -fa 0,1 -d 0,4096,8192

model	size	params	backend	ngl	fa	test	t/s
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	ROCm	99	0	pp512	2226.66 ± 49.93
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	ROCm	99	0	pp512 @ d4096	1076.55 ± 15.98
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	ROCm	99	0	pp512 @ d8192	700.00 ± 5.67
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	ROCm	99	0	tg128	79.35 ± 0.04
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	ROCm	99	0	tg128 @ d4096	43.45 ± 0.24
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	ROCm	99	0	tg128 @ d8192	27.29 ± 0.08
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	ROCm	99	1	pp512	2328.66 ± 4.57
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	ROCm	99	1	pp512 @ d4096	1131.24 ± 2.65
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	ROCm	99	1	pp512 @ d8192	739.06 ± 2.40
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	ROCm	99	1	tg128	85.48 ± 0.19
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	ROCm	99	1	tg128 @ d4096	79.11 ± 0.23
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	ROCm	99	1	tg128 @ d8192	73.23 ± 0.18
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	0	pp512	1174.17 ± 1.65
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	0	pp512 @ d4096	280.37 ± 0.43
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	0	pp512 @ d8192	346.11 ± 1.22
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	0	tg128	75.25 ± 0.05
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	0	tg128 @ d4096	21.91 ± 0.46
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	0	tg128 @ d8192	65.89 ± 0.05
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	1	pp512	925.81 ± 2.12
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	1	pp512 @ d4096	1197.23 ± 22.62
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	1	pp512 @ d8192	939.42 ± 7.80
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	1	tg128	90.03 ± 0.62
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	1	tg128 @ d4096	38.25 ± 0.79
deepseek2 30B.A3B Q4_K - Medium	17.05 GiB	29.94 B	Vulkan	99	1	tg128 @ d8192	95.53 ± 1.18

./build/bin/llama-bench --model ./models/GLM-4.7-Flash-Q4_0.gguf -ngl 99 -fa 0,1 -d 0,4096,8192

model	size	params	backend	ngl	fa	test	t/s
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	0	pp512	2511.30 ± 61.53
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	0	pp512 @ d4096	1138.63 ± 17.58
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	0	pp512 @ d8192	726.29 ± 17.54
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	0	tg128	82.59 ± 1.60
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	0	tg128 @ d4096	44.89 ± 0.02
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	0	tg128 @ d8192	27.79 ± 0.01
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	1	pp512	2610.66 ± 15.68
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	1	pp512 @ d4096	1193.57 ± 3.04
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	1	pp512 @ d8192	768.34 ± 2.29
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	1	tg128	89.94 ± 1.04
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	1	tg128 @ d4096	83.20 ± 0.02
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	1	tg128 @ d8192	77.20 ± 0.04
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	pp512	1229.43 ± 2.67
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	pp512 @ d4096	279.62 ± 1.58
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	pp512 @ d8192	347.47 ± 0.97
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	tg128	85.67 ± 0.21
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	tg128 @ d4096	22.15 ± 0.10
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	tg128 @ d8192	71.66 ± 0.52
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512	957.57 ± 1.19
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512 @ d4096	1274.68 ± 7.77
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512 @ d8192	969.85 ± 14.63
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128	105.24 ± 0.07
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128 @ d4096	40.81 ± 0.03
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128 @ d8192	111.79 ± 0.16

searching for the specific point

model	size	params	backend	ngl	fa	test	t/s
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512	1137.09 ± 2.74
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d512	737.67 ± 17.04
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d1024	1867.61 ± 114.76
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d1536	1764.23 ± 41.98
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d2048	1591.56 ± 29.16
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d2560	1440.59 ± 22.69
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d3072	1341.32 ± 12.01
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d3584	1239.44 ± 7.15
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4096	614.98 ± 17.02
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4608	1272.82 ± 18.51
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4640	1263.51 ± 21.06
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4672	1260.46 ± 19.68
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4704	1250.00 ± 12.46
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4736	1252.84 ± 20.58
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4737	1226.82 ± 21.82
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4738	1247.96 ± 22.36
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4740	1245.80 ± 18.13
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4742	1246.77 ± 16.68
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4744	1250.62 ± 15.24
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4768	1235.40 ± 24.39
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d5000	1204.68 ± 17.77
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d5032	1201.97 ± 15.87
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d5064	1226.46 ± 14.82
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d5096	1198.53 ± 14.63
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d5120	1197.31 ± 17.43
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d5632	1135.32 ± 14.12
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d6144	1065.76 ± 22.74
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d6656	1018.92 ± 19.10
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d7168	989.31 ± 9.19
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d7680	949.37 ± 7.99
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d8192	895.65 ± 24.19
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128	105.55 ± 1.10
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d512	105.00 ± 0.58
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d1024	86.26 ± 0.86
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d1536	73.25 ± 0.93
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d2048	64.28 ± 0.46
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d2560	54.90 ± 0.87
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d3072	50.44 ± 0.05
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d3584	44.40 ± 0.71
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4096	41.35 ± 0.63
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4608	38.46 ± 0.08
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4640	38.26 ± 0.12
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4672	38.03 ± 0.20
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4704	37.33 ± 0.74
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4736	37.72 ± 0.23
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4737	127.10 ± 0.22
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4738	127.22 ± 0.06
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4740	127.17 ± 0.14
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4742	127.28 ± 0.27
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4744	127.16 ± 0.05
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4768	126.99 ± 0.12
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d5000	126.78 ± 0.09
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d5032	126.44 ± 0.13
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d5064	126.15 ± 0.12
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d5096	125.78 ± 0.10
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d5120	125.85 ± 0.28
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d5632	123.76 ± 0.35
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d6144	122.07 ± 0.27
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d6656	121.09 ± 0.13
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d7168	120.59 ± 0.15
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d7680	118.84 ± 0.32
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d8192	117.10 ± 0.58

[0cc4m]

Does this still happen after #19625?

[Nindaleth]

Yes, still does:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	fa	test	t/s
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	0	pp512	915.46 ± 3.39
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	0	pp512 @ d4096	277.01 ± 0.48
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	0	pp512 @ d8192	341.71 ± 0.76
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	0	tg128	86.99 ± 0.08
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	0	tg128 @ d4096	22.50 ± 1.47
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	0	tg128 @ d8192	76.35 ± 0.08
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512	902.16 ± 1.52
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d4096	759.53 ± 1.38
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	pp512 @ d8192	630.11 ± 3.26
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128	106.83 ± 0.17
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d4096	41.04 ± 0.08
deepseek2 30B.A3B IQ4_XS - 4.25 bpw	15.15 GiB	29.94 B	Vulkan	99	1	tg128 @ d8192	119.17 ± 0.07

build: 43e1cbd (8255)

[0cc4m]

Something is definitely wrong with your setup, pp is way lower than expected, and it doesn't make sense for tg to be as fast or faster at 8k context than it is at 0.

Strix Halo for reference:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV STRIX_HALO) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	fa	test	t/s
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	pp512	1037.03 ± 13.90
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	tg128	66.07 ± 0.20
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	pp512 @ d4096	543.97 ± 1.68
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	tg128 @ d4096	38.51 ± 0.15
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	pp512 @ d8192	356.18 ± 0.40
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	tg128 @ d8192	24.71 ± 0.03
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512	981.31 ± 5.19
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128	68.16 ± 0.40
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512 @ d4096	498.39 ± 1.17
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128 @ d4096	55.52 ± 0.11
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512 @ d8192	316.59 ± 1.00
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128 @ d8192	47.33 ± 0.09

[Nindaleth]

Something is definitely wrong with your setup

Yes, something is definitely wrong with my Vulkan setup, or more specifically, something's wrong with current Mesa 26.1.0-git. ROCm is fine and compared to last summer even a bit faster.

Sorry for the false alarm.

[0cc4m]

If you find problems with upstream Mesa, can you open an issue on their end? That way they can resolve it before it hits releases.

[Nindaleth]

Sure, I will. I made a stupid mistake and forgot an env var in one of my files which caused loading of AMDVLK instead of RADV... the git mesa still has an issue too, proper results this time. Great Vulkan showing 👍:

https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF/blob/main/GLM-4.7-Flash-Q4_0.gguf
./build/bin/llama-bench --model ./models/GLM-4.7-Flash-Q4_0.gguf -ngl 99 -fa 0,1 -d 0,4096,8192

mesa-vulkan-drivers-0:25.3.6-2.fc43.x86_64

model	size	params	backend	ngl	fa	test	t/s
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	pp512	3101.43 ± 11.22
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	tg128	138.36 ± 0.20
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	pp512 @ d4096	280.69 ± 0.38
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	tg128 @ d4096	97.20 ± 0.19
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	pp512 @ d8192	348.72 ± 1.21
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	tg128 @ d8192	73.04 ± 0.03
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512	3069.61 ± 15.38
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128	140.03 ± 0.27
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512 @ d4096	1255.64 ± 5.39
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128 @ d4096	114.09 ± 0.20
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512 @ d8192	965.68 ± 8.09
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128 @ d8192	109.42 ± 0.08

build: 43e1cbd (8255)

mesa-vulkan-drivers-0:26.1.0-0.11.git9775621.fc43.i686

model	size	params	backend	ngl	fa	test	t/s
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	pp512	3176.83 ± 4.53
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	tg128	137.90 ± 0.14
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	pp512 @ d4096	282.02 ± 0.66
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	tg128 @ d4096	23.19 ± 0.03
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	pp512 @ d8192	348.87 ± 0.76
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	0	tg128 @ d8192	74.73 ± 0.04
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512	3155.29 ± 6.56
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128	140.30 ± 0.29
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512 @ d4096	1259.16 ± 8.70
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128 @ d4096	40.35 ± 0.03
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	pp512 @ d8192	956.69 ± 6.04
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	Vulkan	99	1	tg128 @ d8192	112.53 ± 0.16

build: 43e1cbd (8255)

rocm-hip-runtime-6.4.1.60401-83.el9.x86_64

model	size	params	backend	ngl	fa	test	t/s
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	0	pp512	2536.74 ± 7.99
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	0	tg128	85.50 ± 0.14
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	0	pp512 @ d4096	1140.05 ± 13.97
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	0	tg128 @ d4096	45.99 ± 0.03
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	0	pp512 @ d8192	734.79 ± 0.97
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	0	tg128 @ d8192	28.36 ± 0.02
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	1	pp512	2619.56 ± 11.84
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	1	tg128	91.35 ± 0.05
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	1	pp512 @ d4096	1198.61 ± 2.41
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	1	tg128 @ d4096	84.25 ± 0.03
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	1	pp512 @ d8192	772.44 ± 1.24
deepseek2 30B.A3B Q4_0	16.03 GiB	29.94 B	ROCm	99	1	tg128 @ d8192	78.02 ± 0.03

build: 43e1cbd (8255)

[Nindaleth]

Mesa issue logged: https://gitlab.freedesktop.org/mesa/mesa/-/issues/15038

[Nindaleth]

As recommended in the Mesa issue, setting env var RADV_PERFTEST=nogttspill fixes this. This env var is actually documented in the OP of #10879, but either I never noticed it when reading that discussion, or I didn't observe any change then and thus ignored the var.

I appreciate getting another piece of the performance puzzle. I believe this issue can be closed.

[0cc4m]

It's still odd behaviour. Were you that close to the VRAM limit or why did it start paging out (and then stop again once context increased)?