Feature Request: Transition to the Windows HIP (ROCm) SDK

[neilopet]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Optimization Brief: AMD Strix Halo (RDNA 3.5) Hardware Enhancements

Now that the Vulkan Sparse Allocator is stable and the upstream Qwen3.5 caching bug is resolved, we have an unconstrained dual-model engine. The next phase is optimizing llama.cpp to fully exploit the RDNA 3.5 Unified Memory Architecture (UMA) for our agent harness.

Please review these four hardware-level enhancements, ranked by implementation effort, and prepare to integrate them.

---

4. Transition to the Windows HIP (ROCm) SDK

The Target: Vulkan is a graphics API running translated SPIR-V compute shaders. AMD’s native compute language is HIP. Compiling llama.cpp on Windows with the HIP SDK links the engine directly to hipBLAS/rocBLAS, which contain hand-tuned, assembly-level microkernels written specifically for RDNA 3.5 architecture, yielding mathematically perfect register allocation.

Motivation

Effort: Very High (9/10)
Expected Improvement: 15% - 25% overall token generation throughput uplift (target: ~46-50 t/s).

Possible Implementation

The Implementation:
This requires completely swapping the build toolchain. You must bypass the Vulkan CMake parameters and compile exclusively for HIP.

Prerequisites for the environment:

1. Install AMD HIP SDK for Windows.
2. Install Strawberry Perl (required for the hipcc compiler wrapper).
3. Ensure MSVC and Ninja are in the PATH.

The Build Script:

# Set compiler paths explicitly to the HIP SDK LLVM toolchain
$env:CC = "C:\Program Files\AMD\ROCm\5.7\bin\clang.exe"
$env:CXX = "C:\Program Files\AMD\ROCm\5.7\bin\clang++.exe"
$env:HIP_PATH = "C:\Program Files\AMD\ROCm\5.7"

# Note: gfx1151 is the specific microarchitecture target for Strix Halo / RDNA 3.5
cmake -G "Ninja Multi-Config" -S . -B build_hip `
    -DCMAKE_BUILD_TYPE=Release `
    -DLLAMA_HIPBLAS=ON `
    -DAMDGPU_TARGETS="gfx1151" `
    -DCMAKE_C_COMPILER=$env:CC `
    -DCMAKE_CXX_COMPILER=$env:CXX

cmake --build build_hip --config Release

[neilopet]

Status: Blocked by AMD Driver/Runtime Issues — But Situation is Evolving

This is a thorough assessment of HIP viability on gfx1151 (Strix Halo) Windows. The short answer is: HIP is currently 16x slower than Vulkan on this hardware, and the root causes are in AMD's driver/runtime stack, not in llama.cpp code we can fix. However, AMD is actively fixing some of these issues.

Our Benchmark Data (Same Hardware, Same Model)

Backend	pp512 (t/s)	tg128 (t/s)	Notes
Vulkan (current)	520	33.8	CoopMat + Wave32 + FA
Vulkan + spec decode	—	12.1	With Qwen3-1.7B draft
HIP (b8119)	25-32	2.1	Stock ROCm
HIP + spec decode	—	11.2	Draft model masks bad tg
KoboldCpp nocuda	126.7	4.58	28% slower than Vulkan

HIP pp is 16-20x slower and tg is 16x slower than Vulkan. Speculative decoding masks the tg gap (11.2 vs 12.1 t/s) but at the cost of draft model memory overhead.

Why "Elbow Grease" Won't Work Here (Unlike Vulkan)

When Vulkan was slow, the bottlenecks were in llama.cpp code we could patch:

• PR vulkan: fix push constant size for quantize_q8_1 ggml-org/llama.cpp#18687: Push constant fix — recovered 75% tg regression in quantize_q8_1 shader
• PR Vulkan Scalar Flash Attention Refactor ggml-org/llama.cpp#19625: Vulkan scalar FA refactor — +7% pp improvement, RDNA wave32 tuning
• Our sparse binding patch — bypassed AMDVLK per-allocation limit without 3x perf regression
• Our chunked staging patch — prevented OOM during checkpoint saves

The HIP bottlenecks are in AMD's driver and runtime stack — code we cannot modify:

1. VMM: no — hipMalloc Falls to Shared Memory Past ~32GB

ROCm issue: ROCm/ROCm#5940

The device reports VMM: no (no Virtual Memory Management). When total hipMalloc allocation exceeds ~32GB, subsequent allocations silently fall to shared system memory instead of VRAM. For a 46GB model, this means 14GB+ ends up in slow system RAM.

UPDATE: AMD engineer closed this on Feb 12 as "Fixed in upcoming driver release." This is the single biggest blocker — if the fix lands, HIP memory allocation would work correctly up to 96GB.

2. rocBLAS gfx1151 Kernel Performance Regression — 2x Slower Than gfx1100

ROCm issue: ROCm/ROCm#4748

rocBLAS ships with pre-tuned assembly kernels per GPU architecture. The gfx1151 kernels are 2x slower than the gfx1100 code path for matrix multiplication. Since GEMM dominates LLM inference, this halves throughput at the rocBLAS layer before llama.cpp even gets involved.

This is an AMD tuning problem — they need to ship optimized microkernels for gfx1151. Still open.

3. GPU Page Faults on Basic Tensor Operations

ROCm issue: ROCm/ROCm#5991

Basic PyTorch GPU operations cause GFX Hub page faults from the Command Processor Frontend. The kernel reports walker, mapping, and permission errors. This affects all ROCm compute, not just llama.cpp. Still open.

4. Segfaults on VRAM Access

ROCm issue: ROCm/ROCm#5853

Any VRAM allocation throws segfault with official ROCm packages. Only TheRock (AMD's experimental build system) works. Pre-built binaries are all broken. Still open.

5. Official Support Status is Unclear

ROCm issue: ROCm/ROCm#4499, ROCm/ROCm#5339

gfx1151 is not in the official ROCm compatibility matrix. Community members have been asking since mid-2025 whether it's supported at all. AMD's documentation is contradictory.

The Issue's Proposed Implementation Has Additional Problems

• ROCm version 5.7 is cited — current is 6.x/7.x, and 5.7 predates gfx1151 support entirely
• LLAMA_HIPBLAS=ON is the legacy CMake flag — current llama.cpp uses GGML_HIP=ON
• gfx1151 is correct but requires ROCm 6.3+ minimum
• Strawberry Perl dependency is real but adds significant setup complexity on Windows

Assessment: Not Dead, But Blocked

Unlike Vulkan where the fixes were in our code, HIP's problems are in AMD's stack:

Issue	Status	In Our Control?
VMM: no (hipMalloc ~32GB wall)	FIXED (unreleased driver)	No — AMD driver fix
rocBLAS 2x regression	Open	No — AMD kernel tuning
GPU page faults	Open	No — AMD driver bug
VRAM segfaults	Open	No — AMD runtime bug

Recommendation

Keep this issue open but deprioritize. When AMD releases the driver with the VMM fix (ggml-org#5940), we should:

1. Install the new driver and re-benchmark HIP
2. Check if rocBLAS The completion stopped but the output is not complete? ggml-org/llama.cpp#4748 was also addressed
3. If pp and tg are within 2-3x of Vulkan, investigate llama.cpp-level HIP optimizations
4. If still 10x+ slower, the rocBLAS kernel gap is the binding constraint and there's nothing we can do in llama.cpp

The Vulkan backend is delivering 520 t/s pp / 33.8 t/s tg today with our patches. Even in an optimistic HIP scenario, matching that would require AMD to ship fully tuned gfx1151 rocBLAS kernels.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.