Eval bug: FLASH_ATTN_EXT GLM 4.7 Flash tensor schema not supported on CUDA

[pwilkin]

Name and Version

version: 7779 (6df686b)

Operating systems

Linux

GGML backends

CUDA

Hardware

4x 4090, 1x 3090

Models

GLM 4.7 Flash Q4_K_M

Problem description & steps to reproduce

This model uses a custom variant of DeepSeek's MLA and the tensor sizes are unsupported. For n_tokens = 2

Q = [576, 2, 20, 1]
V = [576, 256, 1, 1]

First Bad Commit

No response

Relevant log output

common_debug_cb_eval: fattn-0 = (f32) FLASH_ATTN_EXT(CPU#Qcur-0 (view) (permuted)#0{576, 2, 20, 1}, CPU#cache_k_l0 (view) (permuted)#0{576, 256, 1, 1}}) = {512, 20, 2, 1}

[pwilkin]

@am17an @JohannesGaessler I tried to see if I can add this but the 20 looks so weird here and I wasn't able to reasonably add a case for it :/

[jukofyork]

It could be the v_dim being 256, as deekseek and Kimi used 128.

[pwilkin]

gqa_ratio is 2 here and 16 in case of DeepSeek.

[JohannesGaessler]

gqa_ratio is 2

Do you mean 20? In any case, the ncols2 value is only used for optimization and can in principle be set to arbitrary powers of 2 that cleanly divide the GQA ratio. So I would recommend adding a case with DKQ == 576 && DV == 512 && ncols2 == 4 which should work out-of-the-box. However, specifically for batch size 1 the tensor core utilization in the MMA kernel will only be 50% so you should also check whether the tile kernel is faster in that case.

[am17an]

I'm taking a look at this