vulkan: improve partial offloading performance on AMD

[0cc4m]

I saw a big difference between Vulkan and ROCm performance in partial offloads. I narrowed it down to transfer speeds for weight transfer from CPU to GPU with offloaded ops. One possible explanation is that using the dedicated transfer queue on AMD may be faster than using a compute queue, so I implemented using a transfer queue for async transfers as well and synchronizing transfers using a timeline semaphore. This does improve performance.

Then I checked and found that the dedicated transfer queue on AMD is not exposed by the Linux driver by default, so it's not actually being used. The difference comes from using a second queue (the graphics queue) for transfers, so I assume the issue was the compute queue being congested with other work.

This helps on AMD RDNA4, but not on GCN and not on Nvidia. I couldn't test Intel because the Linux driver only exposes a single queue.

I also changed the Vulkan backend offload_op function to work like the CUDA backend's, to cover some more cases. And I moved some redundant context code into a function to reduce code duplication.

[inforithmics]

Would this help with --cpu-moe (weight offloading to cpu) ?

[0cc4m]

It should, yes. Any partial offload.

[inforithmics]

I checked out the pull request and indeed it was faster too with ---n-cpu-moe (on an 890M Integrated GPU).
Even the TG Performance went up from 13 to 15 tg/s with ---n-cpu-moe 20 and gpt-oss:20B.

[characharm]

common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
S:\LLM\Vulkan\llama.cpp\ggml\src\ggml-vulkan\ggml-vulkan.cpp:6816: GGML_ASSERT(src->device == dst->device) failed

disabling intel gpu - no crash

[0cc4m]

@characharm Thanks for testing it, there was a check missing. Can you try again?

[characharm]

@characharm Thanks for testing it, there was a check missing. Can you try again?

Yeah, looks like it's not crashing anymore.

[rhjdvsgsgks]

Then I checked and found that the dedicated transfer queue on AMD is not exposed by the Linux driver by default, so it's not actually being used.

transfer queue is supported since mesa 26. but hidden under a flag RADV_PERFTEST=transfer_queue

I couldn't test Intel because the Linux driver only exposes a single queue.

it is also supported on intel. but only avaliable when using xe drm driver (rather than i915) and gen > 12.5

[0cc4m]

Then I checked and found that the dedicated transfer queue on AMD is not exposed by the Linux driver by default, so it's not actually being used.

transfer queue is supported since mesa 26. but hidden under a flag RADV_PERFTEST=transfer_queue

I know, I've tested it. It's slightly faster than using the graphics queue. The majority of the improvement comes from any second queue.

I couldn't test Intel because the Linux driver only exposes a single queue.

it is also supported on intel. but only avaliable when using xe drm driver (rather than i915) and gen > 12.5

That's good to know, thank you! I'll give it a try with Xe.

[0cc4m]

I get mixed results with Xe, slightly improved pp, slightly worse tg. I'll leave it off there, for now.

[acbits]

Then I checked and found that the dedicated transfer queue on AMD is not exposed by the Linux driver by default, so it's not actually being used.

transfer queue is supported since mesa 26. but hidden under a flag RADV_PERFTEST=transfer_queue

I couldn't test Intel because the Linux driver only exposes a single queue.

it is also supported on intel. but only avaliable when using xe drm driver (rather than i915) and gen > 12.5

So do we have to upgrade to Mesa 26 or is the RADV_PERFTEST flag is also available on older versions?

[0cc4m]

It's available on older versions as well. I'm not sure why it was reported again for 26.0

[HumerousGorgon]

This merged PR seems to be causing issues on the vulkan backend regarding gibberish output with -fa on on Intel GPUs.

[tvall43]

This merged PR seems to be causing issues on the vulkan backend regarding gibberish output with -fa on on Intel GPUs.

im also just getting gibberish on my setup with this. qwen3.5-35b on 2 radeon pro v340l's. reverting this fixed it, havent done any other debugging

[0cc4m]

Please see if #20233 helped.

[tvall43]

Please see if #20233 helped.

seems like that fixed it!
edit: uh nevermind. also needed this #20097 (comment) but i think its working now