Paligemma: fix generation with Gemma2

[zucchini-nlp]

What does this PR do?

Fixes #36029 and adds tests for the model, imo we need tests with different LM backbone because Gemma-2 is special

This is a quick fix but I think we should make this kind of fix on LM work out-of-the-box, by adding it as kwargs for example. Most LMs accept loss_kwargs thus we can make all multimodal models also accept kwargs that are simply passed further to the LM. WDYT?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

we can just use kwargs no?

[zucchini-nlp]

I think making it explicit that kwargs will be used by an only an LM was better

[Cyrilvallez]

Fine with me! Thanks a lot!

[ArthurZucker]

let's say that an integration test is most welcome as well!

[zucchini-nlp]

yeah, was quite low in priority for the patch so I decided to skip it for now :)

[ArthurZucker]

For transparency, this commit needs to be modified for the patch, only applying changes for PaliGemnma2

[hiyouga]

Hi @zucchini-nlp @ArthurZucker , I find this PR can lead to an abnormal loss value if gradient accumulation is enabled. Initially reported in hiyouga/LlamaFactory#7443 , the trainer assumes these model accept loss kwargs because of the existence of lm_kwargs (1), but actually they do not (2), resulting in an unexpected loss value. This is also related to the gradient accumulation fix PR #34511 , cc @muellerzr

1. https://github.com/huggingface/transformers/blob/v4.50.0/src/transformers/trainer.py#L633-L640
2. https://github.com/huggingface/transformers/blob/v4.50.0/src/transformers/models/llava/modeling_llava.py#L460-L464

[hiyouga]

This is a quick fix but I think we should make this kind of fix on LM work out-of-the-box, by adding it as kwargs for example. Most LMs accept loss_kwargs thus we can make all multimodal models also accept kwargs that are simply passed further to the LM. WDYT?

Yeah, it is true that most LMs accept such kwargs, but multimodal models compute the loss themselves, unless we reuse the loss computed by the language model part.

[hiyouga]

A similar bug report regarding the gemma3 model: hiyouga/LlamaFactory#7416

[zucchini-nlp]

Yeah, it is true that most LMs accept such kwargs, but multimodal models compute the loss themselves, unless we reuse the loss computed by the language model part.

Yep, I will look into that and find a better solution

[zucchini-nlp]

@hiyouga sorry, just got to look in detail at the issue. Indeed this causes problems when accumulating grads. I think the best solution would be to make all VLMs (touched by this PR first and then others not touched also) compute loss with self.loss_function

Would you like to submit a PR for that?

[hiyouga]

@zucchini-nlp Hi, I'm currently deeply occupied with writing my thesis and really don't have much spare time at the moment. Unfortunately, I won't be able to submit the PR. Thanks for understanding!

[zucchini-nlp]

@hiyouga yeah, no problem :)