Look for the pad_token_id in the right place for Llama4

[Rocketknight1]

Llama4 look for pad_token_id on self.config in some cases, but I think it actually lives on self.config.text_config. This PR should fix things! There was a similar issue with Qwen3, but thankfully I couldn't find any other affected models.

Fixes #43525

[Rocketknight1]

cc @zucchini-nlp

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

Thanks for a quick fix!

Can you also check if models reported in #43334 (comment) actually need a fix or not? Qwen3-VL-MoE for sure is faulty and has no PAD in its text config. I am not sure about other models

I want us to fix those pad issues all at once if possible

[zucchini-nlp]

i think in case of llama4, we need to fix the modeling code to obtain it from pad_token_id = self.config.text_config.pad_token_id. Usually the special tokens live inside a text config

[zucchini-nlp]

I ran a tiny test and got 16 models failing, might be worth checking these ones? 👀

FAILED tests/models/esm/test_modeling_esm.py::EsmModelTest::test_attention_outputs - TypeError: ne() received an invalid combination of arguments - got (NoneType), but expected one of:
FAILED tests/models/exaone4/test_modeling_exaone4.py::Exaone4ModelTest::test_attention_outputs - AttributeError: 'Exaone4Config' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/glm46v/test_modeling_glm46v.py::Glm46VModelTest::test_attention_outputs - AttributeError: 'Glm4vTextConfig' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/glm4v/test_modeling_glm4v.py::Glm4vModelTest::test_attention_outputs - AttributeError: 'Glm4vTextConfig' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/glm_image/test_modeling_glm_image.py::GlmImageModelTest::test_attention_outputs - AttributeError: 'GlmImageTextConfig' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/glm_ocr/test_modeling_glm_ocr.py::GlmOcrModelTest::test_attention_outputs - AttributeError: 'GlmOcrTextConfig' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/gpt_bigcode/test_modeling_gpt_bigcode.py::GPTBigCodeModelTest::test_attention_outputs - AttributeError: 'GPTBigCodeConfig' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/gpt_bigcode/test_modeling_gpt_bigcode.py::GPTBigCodeMHAModelTest::test_attention_outputs - AttributeError: 'GPTBigCodeConfig' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/gpt_neox/test_modeling_gpt_neox.py::GPTNeoXModelTest::test_attention_outputs - AttributeError: 'GPTNeoXConfig' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/gptj/test_modeling_gptj.py::GPTJModelTest::test_attention_outputs - AttributeError: 'GPTJConfig' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/jetmoe/test_modeling_jetmoe.py::JetMoeModelTest::test_attention_outputs - AttributeError: 'JetMoeConfig' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/mpt/test_modeling_mpt.py::MptModelTest::test_attention_outputs - AttributeError: 'MptConfig' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/phi/test_modeling_phi.py::PhiModelTest::test_attention_outputs - AttributeError: 'PhiConfig' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/qwen3_vl_moe/test_modeling_qwen3_vl_moe.py::Qwen3VLMoeModelTest::test_attention_outputs - AttributeError: 'Qwen3VLMoeTextConfig' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/stablelm/test_modeling_stablelm.py::StableLmModelTest::test_attention_outputs - AttributeError: 'StableLmConfig' object has no attribute 'pad_token_id'. Did you mean: 'bos_token_id'?
FAILED tests/models/tvp/test_modeling_tvp.py::TVPModelTest::test_attention_outputs - AttributeError: 'TvpConfig' object has no attribute 'pad_token_id'

[Rocketknight1]

Hey @zucchini-nlp, sorry for the delay while I chased CI issues! I think this is actually okay, and we don't need to fix other models. This only applies to VLMs where pad_token_id may be on the root config or the text_config, but the other cases of that were fixed here and here. In the other cases in your list, I think those are raw text LMs which probably don't have this issue, since they don't have text_config, right?

[zucchini-nlp]

Oh yeah, we also have llama4! Sure, let's merge it, I wonder why it was fixed before with the other batch of models haha

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: llama4