Gemma4 resizing per layer inputs

[zucchini-nlp]

What does this PR do?

Fixes #45276 and #45335

In gemma4 per-layer inputs have to be resized as long as they aren't part of soft multimodal tokens

Repro for T5 gemma:

from transformers import T5GemmaForConditionalGeneration

model = T5GemmaForConditionalGeneration.from_pretrained("harshaljanjani/tiny-t5gemma-test")
encoder = model.resize_token_embeddings(model.vocab_size + 10)
decoder = model.model.decoder.embed_tokens
head = model.get_output_embeddings()
print(encoder.weight.shape, decoder.weight.shape, head.weight.shape)
# LM head resize is reverted back because of tying. Decoder is never resized
>>> torch.Size([256010, 512]) torch.Size([256000, 512]) torch.Size([256000, 512])

Gemma3n has soft mm tokens, and the current state of 3N is not good. I see unused vocab entries in mm_projection 😢 and if when apply simply resizing to per-layer-input, we'll get even more unused entries

Could be done in the correct way if we filter out mm-tokens, but I'd prefer to leave it for now

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

same as base class, so no need to override

[zucchini-nlp]

not a fan of it tbh, but ig it's better than overriding resize

[zucchini-nlp]

if we had this, we'd know that gemma4 resize doesn't work well. Added now

[zucchini-nlp]

@bot /repo

[github-actions]

Repo. Consistency bot fixed some files and pushed the changes.

[ArthurZucker]

SGTM, its a bit dirty to have to call add hook to module but model specific so fine b y me!

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: blip, colmodernvbert, gemma3, gemma3n, gemma4, lfm2_vl, paligemma, qwen3_vl, qwen3_vl_moe, t5gemma