feat[vLLM × v5]: Add vLLM compatibility for audio models

[harshaljanjani]

What does this PR do?

→ This PR introduces compat fixes across several audio models to ensure they can be loaded and used by a companion vLLM PR. These changes are deliberate and are blocking this vLLM PR which adds audio backend compatibility to vLLM. Once this PR is merged, the other PR will be marked ready for review!
→ Outlining the design choices of one PR without context from the other didn't make much sense to me, so I wrote a doc that outlines both sets of changes together and explains their deliberate nature, amongst other valuable things!
→ The v5 tracker doesn’t mention the audio backend, but it is certainly a significant gap that needs to be addressed. After this is merged, I'll open an issue tracker for the Transformers audio backend work in vLLM so the efforts can stay organized.

Please refer to the document for the reasoning behind these changes in context with the vLLM PR!
Document: v5 x vLLM Audio Backend Support Document

Related Issues:

→ Current v5 tracker: vllm-project/vllm#38379
→ vllm-project/vllm#38902

@vasqu @ArthurZucker

Code Agent Policy

• I confirm that this is not a pure code agent PR.

Before submitting

• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.

[harshaljanjani]

The CI failures are unrelated to this PR (the GraniteSpeech failure is likely a pre-existing issue as I documented).

[Rocketknight1]

cc @hmellor as well

[ArthurZucker]

LGTM but can you make sure it's tested ! ?

[eustlb]

Amazing to see it's working out of the box! 🔥 How did you test?
Edit: Okay I see it's there

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, glmasr, granite_speech, musicflamingo, vibevoice_acoustic_tokenizer, vibevoice_asr

[harshaljanjani]

LGTM but can you make sure it's tested ! ?

@ArthurZucker So I tested Granite Speech, Audio Flamingo 3, GLM-ASR and VibeVoice-ASR (the changed models) again just to verify the tests pass, and they do, except the one I mentioned in the document previously which isn't related to this change. It's missing a skip like AudioFlamingo3 and Voxtral. Guess it doesn't hurt to add it so I've added it within the scope of this PR itself, and after that the CI is green (not sure about the local issue on my side when fetching the URL since it's actually valid, didn't happen in the last run).

Before the necessary skip (GraniteSpeechForConditionalGenerationModelTest::test_inputs_embeds_matches_input_ids):

After the necessary skip (all model tests):

RUN_SLOW=1 pytest -q -o log_cli=false tests/models/granite_speech/test_modeling_granite_speech.py tests/models/audioflamingo3/test_modeling_audioflamingo3.py tests/models/glmasr/test_modeling_glmasr.py tests/models/vibevoice_asr/test_modeling_vibevoice_asr.py

Amazing to see it's working out of the box! 🔥 How did you test?
Edit: Okay I see it's there

@eustlb Yupp it's tested :)
Also I think you'd find this gist for how I benchmarked the models, and the documentation of the quirks I encountered valuable as well. I'll mark the vLLM PR ready for review once this is out of the way.

[harshaljanjani]

Good day @ArthurZucker @eustlb, just checking in to see if there have been any updates so that the vLLM PR can be unblocked :)

[ArthurZucker]

Let's go, I'd be happy if we can see some stuff than can be taken from the vllm PR to here to help standardize! 🤗

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[harshaljanjani]

Got kicked out of the merge queue 😓