Models loaded to AMD Radeon with Vulkan generate a garbage

[demoleh]

Describe the Issue
Tried new release 1.101.1 and checked several models loading to AMD Radeon GPU using nocuda release with Vulkan and it produces the garbage in responses when loaded. Tried changing parameters, nothing helps. 1.100 release and before never had this issue with the same parameters and same models.

Additional Information:
GPU: AMD Radeon Pro RX 6800 32GB
KoboldCpp: 1.101.1

[dombach]

I'm experiencing the same issue with Vulkan and an AMD Mi50 32GB (Windows 11).
With version 1.101.1, only garbage is being produced. I've tested multiple models.
When using the RTX 3090 in the same system with Vulkan, everything works fine.
However, downgrading to version 1.100.1 restores full functionality with the Mi50.

[LostRuins]

@demoleh : On windows or Linux? it happens on any model?

@dombach Are you using the same model?

[dombach]

I'm running Windows 11 (AMD drivers from rdn-id).
I've tested various models: QWEN 30BA3B, Mistral Small 3.2, and others.
The same models are used on both GPUs (Mi50 and RTX 3090), and across both versions 1.101.1 and 1.100.1.
With the current version (1.101.1), the Mi50 produces only garbage output—regardless of Flash Attention being on or off, whether using the NoCUDA or CUDA version, or which model is selected.
All previous versions work perfectly with the same settings on the Mi50.

Example:
Qwen3-30B-A3B-Instruct-2507 - same config files loaded with both versions on Mi50.

Give me a concise answer about what YAML is.

Version 1.100.1
YAML (YAML Ain't Markup Language) is a human-readable data serialization format used for configuration files and data exchange. It’s designed to be simple and intuitive, supporting lists, maps, and basic data types, with indentation defining structure rather than braces or brackets.

Version 1.101.1
.uml.uml.uml.uml.ul/el/el/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele/ele

[demoleh]

@demoleh : On windows or Linux? it happens on any model?

Windows. Tried a several different models, always the same garbage on AMD GPU. Switched to Nvidia, normal output.

[demoleh]

Using Vulcan (old GPU) on AMD GPU works okay, by the way.
Tried current llamacpp b6940 - it generates the same garbage with select AMD GPU, not Nvidia GPU
Then loaded old build b6490 - no issues, so the bug with Vulcan+AMD (not sure if all GPUs or not) appeared somewhere in llamacpp

[LostRuins]

Indeed, the bug would have come from upstream llama.cpp since there was no downstream vulkan change between these versions.

@0cc4m mentioned it could be due to your use of custom drivers. Have you tried using different drivers? edit: sorry wrong person

Also since you are willing to test llama.cpp directly, could you find the first llama.cpp release that has this issue so we can pinpoint the triggering commit? if possible share the command line args you used to launch llama.cpp and the model name.

[LostRuins]

also try updating your drivers (which ones are you using?)

[demoleh]

That was it. Updated the drivers to PRO Edition 25.Q3 (with WHQL 25.10.25.01/ Windows Store 32.0.21025.1024) and that completely resolved the issue - tested with a couple of models and the garbage output is gone. I assume that was due to the support of Vulkan 1.4 which was missing in the old drivers. Thank you a lot!

[dombach]

Since I rely on the rdn-id graphics drivers to get the Mi50 running under Windows, I apparently have no way of using the recent koboldcpp versions.

[demoleh]

Since I rely on the rdn-id graphics drivers to get the Mi50 running under Windows, I apparently have no way of using the recent koboldcpp versions.

Did it work for you with Vulcan (Old CPU) selection?

[LostRuins]

I still need help identifying the specific llamacpp commit that triggered this

[dombach]

I just tested the "Use Vulkan (Old CPU)" option, and it indeed works with the Mi50 and version 1.101.1.
By the way, the CPU is a relatively new Ryzen 8700G.

[LostRuins]

The reason why it works with old CPU mode is because all vulkan extensions e.g. coopmat are disabled in this mode. I would still like to find where in llama.cpp it broke