mtmd: qwen3 audio support (qwen3-omni and qwen3-asr)

[ngxson]

Status:

• qwen3-omni-moe working (vision + audio input)
• qwen3-asr working

[samshipengs]

Any updates on this?

[yimlin]

any updates?

[michoecho]

I wrote a working Qwen3-ASR support for my own use at https://github.com/michoecho/llama.cpp/commits/qwen3_asr_support. (I successfully used it to transcribe some lectures in Chinese). I don't know if it's good enough for upstreaming, because I wasn't thinking about qwen3-omni at all. (I have no idea what "deepstack" is). But you could use it as a working base if you are getting wrong results.

At a glance, what mainly seems to be missing from this PR is:

• To make my changes work properly, I had to fix a preexisting bug in whisper preprocessing which was causing the last audio chunk to be lost during: michoecho@63b3c1e#diff-a027f93a5e0a3fe643975f0ae176db52a3330a9422857b4f6fd9bfbac134c863R384. (I haven't reported it because I'm not 100% sure it's a bug — maybe I'm not seeing something — but I'm 99% sure).
• The ggml_permute seems to have channels and frames swapped around.
• Qwen3-ASR uses <|audio_start|> and <|audio_end|> instead of <|audio_bos|> and <|audio_eos|>.
• The audio encoder expects windowed (/chunked) attention, with window size between 1s and 8s. If you run the encoder with full attention on a 30s chunk, you will get bogus results. I didn't want to implement windowed attention (because I would have to learn how to do that), so I just solved it in the preprocessing layer by splitting audio into 8s chunks instead of 30s chunks.
• As a comment in this PR acknowledges, the reference implementation runs the conv2d layers on chunks of length 100. I followed the reference implementation. I don't know either if this chunking is necessary. I didn't test the non-chunked variant.
• The default chat template of Qwen3-ASR doesn't work with llama.cpp. (It expects the audio to be passed via some special params). (My fork doesn't care about chat templates at all either, because my application constructs the prompt directly anyway. But if you want to implement a chat template, the prompt expected by the model isn't anything fancy, it's basically just chatml with audio used as the user message).

By the way, note that Qwen3-ForcedAligner (the timestamp predictor model) has the same architecture as Qwen3-ASR, so if you implement support for the latter, you almost get support for the former too. "Almost" because the ForcedAligner is a non-autoregressive classification model. (You put in the encoded audio and the transcribed text with some <timestamp> tokens mixed in, then you run a single prediction on it, and the logits on the <timestamp> tokens will describe the timestamp at those points in the text). I'm not sure how to integrate something like that with llama.cpp's abstractions. For my private use case (generating subtitles for the Chinese lectures) I added "support" for Qwen3-ForcedAligner too, but it's too hacky to post.

[ngxson]

both qwen3-omni and qwen3-asr are working with this PR, GGUF will be uploaded shortly

[ngxson]

* To make my changes work properly, I had to fix a preexisting bug in whisper preprocessing which was causing the last audio chunk to be lost during: [michoecho@63b3c1e#diff-a027f93a5e0a3fe643975f0ae176db52a3330a9422857b4f6fd9bfbac134c863R384](https://github.com/michoecho/llama.cpp/commit/63b3c1ec0cb1f73f4cb3a7056ae7356b413452f2#diff-a027f93a5e0a3fe643975f0ae176db52a3330a9422857b4f6fd9bfbac134c863R384). (I haven't reported it because I'm not 100% sure it's a bug — maybe I'm not seeing something — but I'm 99% sure).

Chunking can be implemented via a follow-up PR, this PR processes the input as 30s chunk for simplicity

* Qwen3-ASR uses `<|audio_start|>` and `<|audio_end|>` instead of `<|audio_bos|>` and `<|audio_eos|>`.

Thanks for pointing out, that need to be fixed in this PR

* The default chat template of Qwen3-ASR doesn't work with llama.cpp. (It expects the audio to be passed via some special params). (My fork doesn't care about chat templates at all either, because my application constructs the prompt directly anyway. But if you want to implement a chat template, the prompt expected by the model isn't anything fancy, it's basically just `chatml` with audio used as the user message).

That was fixed by simply push a chatml jinja template to GGUF upon conversion

By the way, note that Qwen3-ForcedAligner (the timestamp predictor model) has the same architecture as Qwen3-ASR, so if you implement support for the latter, you almost get support for the former too. "Almost" because the ForcedAligner is a non-autoregressive classification model. (You put in the encoded audio and the transcribed text with some <timestamp> tokens mixed in, then you run a single prediction on it, and the logits on the <timestamp> tokens will describe the timestamp at those points in the text). I'm not sure how to integrate something like that with llama.cpp's abstractions. For my private use case (generating subtitles for the Chinese lectures) I added "support" for Qwen3-ForcedAligner too, but it's too hacky to post.

Hmm yeah that sounds complicated, will see if it worth implementing knowing that another model (voxtral from mistral) having somewhat same logic

[CISC]

Instead of handling this everywhere, can't we just merge in all sub-configs in thinker_config here:

llama.cpp/convert_hf_to_gguf.py

Lines 974 to 976 in eefcfee

	if "thinker_config" in config:
	# rename for Qwen2.5-Omni
	config["text_config"] = config["thinker_config"]["text_config"]

[ngxson]

hmm that can be quite dangerous because the sub config may have conflict keys with the thinker_config

I think it's fine to keep this as-is (a bit lazy to re-test this). plus, we only have one single place in the whole file that does this.

for ref, normally a text model never have to read the vision config, but this is the specific case for qwen3 to support "deep stack". from qwen3.5, they removed the deep stack

[CISC]

Uhoh, I think \r\n strikes again...

[ngxson]

hmm seems like it mostly happens when you add a new line in the suggestion, right?

[CISC]

hmm seems like it mostly happens when you add a new line in the suggestion, right?

Yes, but only when you (and one other person so far) commit them. :)

[ngxson]

hmm, the conversion is broken somehow, the omni mmproj now only contains audio tensors, not vision tensors

digging into this...

[ngxson]

OK it got a bit messy with multi inheritance, but I fixed in the last commit

CC @pwilkin if you can give the 2nd approval, thanks!

[ngxson]

pinging @ggml-org/maintainers if someone can give an approval, thanks!

[bennmann]

no audio output for the nice voices?

[pwilkin]

@ngxson sorry, somehow missed this one.

[erazortt]

no audio output for the nice voices?

Yes, that would also be of interest for me. Is there audio output planned?

[pablopla]

Can you show example how to use it? I'm getting short transcription in low quality when transcribing a 1 minute audio file.
The qwen-asr online demo works fine.
Should I use a prompt and set specific parameters?
Should I send the request to "/v1/audio/transcriptions" or ""/v1/chat/completions"