Misc. bug: Vulkan's performance degradation(TG) on A770 from b7194 and FA problem

[savvadesogle]

Name and Version

llama-cli -v
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b7209\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b7209\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b7209\ggml-cpu-haswell.dll
build: 7209 (7f8ef50) with clang version 19.1.5 for x86_64-pc-windows-msvc

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

llama-bench
llama-server

Command line

llama-bench -m T:\models\lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf  -ngl 100 -fa 0,1


llama-bench -m T:\models\lmstudio-community\Meta-Llama-3.1-8B-Instruct-GGUF\Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 100 -fa 0,1

Problem description & steps to reproduce

Hello.

Drop in token generation (TG) performance compared to the B7189 version on the Intel Arc A770.
Between b7189 and b7209.

Driver: 8250
cpu: xeon 2699v3 x2
GPU: 1x A770

Models:
53 -> 42 t/s lmstudio-community\Meta-Llama-3.1-8B-Instruct-GGUF\Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
60 -> 54 t/s lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf

B7209

| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     | 100 |  0 |           pp512 |        921.00 ± 3.22 |
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     | 100 |  0 |           tg128 |         42.39 ± 0.07 |
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     | 100 |  1 |           pp512 |        280.12 ± 0.44 |
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     | 100 |  1 |           tg128 |         29.39 ± 0.02 |

build: 7f8ef50cc (7209)

sometimes flash attention crashes (with no error, The bench just stops)

| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan     | 100 |  0 |           pp512 |        885.48 ± 5.59 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan     | 100 |  0 |           tg128 |         54.20 ± 0.07 |

B7189

| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     | 100 |  0 |           pp512 |        918.06 ± 4.28 |
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     | 100 |  0 |           tg128 |         53.63 ± 0.10 |
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     | 100 |  1 |           pp512 |        280.26 ± 0.70 |
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     | 100 |  1 |           tg128 |         34.50 ± 0.04 |

| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan     | 100 |  0 |           pp512 |        884.45 ± 5.87 |
| gpt-oss 20B MXFP4 MoE          |  11.27 GiB |    20.91 B | Vulkan     | 100 |  0 |           tg128 |         60.95 ± 0.06 |

First Bad Commit

Between b7189 and b7209

[savvadesogle]

Logs

gpt-oss-20b-b7209.txt
llama3.1-8b-q4km-b7209.txt

[pwilkin]

@savvadesogle Can you please git bisect the exact issue that causes the performance drop?

[0cc4m]

Most likely culprit is #16900, I guess. I did test with A770, but only on Linux.

Edit: The flash attention issue is likely #17389 (comment)

[savvadesogle]

the exact issue that causes the performance drop

The last worked is b7194

The problem starts from B7195 (build: 47a268e (7195))

LOGS:

Details

Microsoft Windows [Version 10.0.22631.5189]

C:\llm\llama-cpp\VULKAN>cd C:\llm\llama-cpp\VULKAN\b7194

C:\llm\llama-cpp\VULKAN\b7194>llama-bench -m T:\models\lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf -ngl 100 -fa 0,1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b7194\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b7194\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b7194\ggml-cpu-haswell.dll

model	size	params	backend	ngl	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	pp512	892.69 + 4.36
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	tg128	60.65 + 0.27

C:\llm\llama-cpp\VULKAN\b7194>cd ..\b71208

C:\llm\llama-cpp\VULKAN\b71208>llama-bench -m T:\models\lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf -ngl 100 -fa 0,1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b71208\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b71208\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b71208\ggml-cpu-haswell.dll

model	size	params	backend	ngl	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	pp512	890.67 + 7.08
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	tg128	53.73 + 0.17

C:\llm\llama-cpp\VULKAN\b71208>cd ..\b71205

C:\llm\llama-cpp\VULKAN\b71205>llama-bench -m T:\models\lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf -ngl 100 -fa 0,1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b71205\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b71205\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b71205\ggml-cpu-haswell.dll

model	size	params	backend	ngl	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	pp512	888.28 + 4.72
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	tg128	54.32 + 0.07

C:\llm\llama-cpp\VULKAN\b71205>cd ..\b71200
Системе не удается найти указанный путь.

C:\llm\llama-cpp\VULKAN\b71205>cd ..\b7200

C:\llm\llama-cpp\VULKAN\b7200>llama-bench -m T:\models\lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf -ngl 100 -fa 0,1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b7200\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b7200\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b7200\ggml-cpu-haswell.dll

model	size	params	backend	ngl	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	pp512	886.25 + 6.23
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	tg128	53.74 + 0.11

C:\llm\llama-cpp\VULKAN\b7200>cd ..\b7198

C:\llm\llama-cpp\VULKAN\b7198>llama-bench -m T:\models\lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf -ngl 100 -fa 0,1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b7198\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b7198\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b7198\ggml-cpu-haswell.dll

model	size	params	backend	ngl	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	pp512	889.31 + 5.80
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	tg128	54.13 + 0.20

C:\llm\llama-cpp\VULKAN\b7198>cd ..\b7196

C:\llm\llama-cpp\VULKAN\b7196>llama-bench -m T:\models\lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf -ngl 100 -fa 0,1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b7196\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b7196\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b7196\ggml-cpu-haswell.dll

model	size	params	backend	ngl	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	pp512	890.18 + 6.04
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	tg128	54.04 + 0.22

C:\llm\llama-cpp\VULKAN\b7196>cd ..\b7195

C:\llm\llama-cpp\VULKAN\b7195>llama-bench -m T:\models\lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf -ngl 100 -fa 0,1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b7195\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b7195\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b7195\ggml-cpu-haswell.dll

model	size	params	backend	ngl	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	pp512	888.57 + 4.68
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	tg128	53.88 + 0.16

C:\llm\llama-cpp\VULKAN\b7195>cd ..\b7194

C:\llm\llama-cpp\VULKAN\b7194>llama-bench -m T:\models\lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf -ngl 100 -fa 0,1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b7194\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b7194\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b7194\ggml-cpu-haswell.dll

model	size	params	backend	ngl	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	pp512	887.46 + 5.80
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	tg128	60.82 + 0.15

C:\llm\llama-cpp\VULKAN\b7194>cd ..\b7195

C:\llm\llama-cpp\VULKAN\b7195>llama-bench -m T:\models\lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf -ngl 100 -fa 0
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b7195\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b7195\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b7195\ggml-cpu-haswell.dll

model	size	params	backend	ngl	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	pp512	888.37 + 7.81
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	tg128	54.21 + 0.09

build: 47a268e (7195)

C:\llm\llama-cpp\VULKAN\b7195>

[savvadesogle]

Edit: The flash attention issue is likely #17389 (comment)

and yes, a build 7063 with FA is working.

C:\llm\llama-cpp\VULKAN\b7063>llama-bench -m T:\models\lmstudio-community\gpt-oss-20b-GGUF\gpt-oss-20b-MXFP4.gguf  -ngl 100 -fa 0,1
load_backend: loaded RPC backend from C:\llm\llama-cpp\VULKAN\b7063\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from C:\llm\llama-cpp\VULKAN\b7063\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\llm\llama-cpp\VULKAN\b7063\ggml-cpu-haswell.dll

model	size	params	backend	ngl	fa	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	pp512	888.70 + 5.89
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	0	tg128	59.99 + 0.05
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	1	pp512	849.76 + 3.32
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	Vulkan	100	1	tg128	56.17 + 0.08

build: 38eaf32 (7063)

[savvadesogle]

Hello @rillomas and @0cc4m 👋

Is there any chance to restore performance for TG on Intel cards (DG2, A770)?

for GPT-OSS 20B 63 t/s (build: c6f7a42 (7189)) vs 55 t/s (latest build: 091a46c (7815))
for llama3.1 8b (q4_k_m) 55 t/s (build: c6f7a42 (7189)) vs 43 t/s (latest build: 091a46c (7815))

ps Power limits changed from 190W to 228W for this test vs #17628 (comment)