🚨 Generic Sequence Classifier works for multimodal models

[zucchini-nlp]

Fixes #44625 and #44406 (comment)

After long discussion, ig this is the easiest and most straightforward way for us to use Generic classes with mulitmodals, and keep their signature. Overloading didn't work as I expected 😓

Ans I am quite reluctant to use TypedDicts, as we will have a lot of unrelated args even after filtering by modality.

Working example:

from transformers import AutoConfig, AutoModelForSequenceClassification, Qwen3_5TextForSequenceClassification

model_name = "onnx-internal-testing/tiny-random-Qwen3_5ForConditionalGeneration"
config = AutoConfig.from_pretrained(model_name, num_labels=1)
model = AutoModelForSequenceClassification.from_pretrained(model_name, config=config)
print(model.config.num_labels == 1)

# suppose a text-only classifier is saved on the hub
text_model = Qwen3_5TextForSequenceClassification(config.text_config)
text_model.save_pretrained("tmp_dir")
text_config = AutoConfig.from_pretrained("tmp_dir", num_labels=5)
text_model = AutoModelForSequenceClassification.from_pretrained("tmp_dir", config=text_config)
print(text_model.config.num_labels == 5)

This should work for ST

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

The main change: to allow a single class to be loaded as LLM (text_config) or as multimodal lm (vlm config)

Not sure if it aligns well with transformers tho

[vasqu]

Some initial comments, I have a bit of my doubts because the signature is not properly reflecting VLMs. Not sure if it would rather make sense to have a dedicated one instead

Also, are these really the only ones that use it? A bit surprising but oh well

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, gemma3, qwen3_5

[github-actions]

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=44664&sha=277261

[zucchini-nlp]

Failing test is unrelated and due to bad regex, same as in #45313. The previous fix didn't fix it for all cases 😢

[vasqu]

Let's add 🚨 to the title, I hope not too many relied on the exact class name for qwen 3.5 text here

But I think this is the best solution for now then, sad that the override didnt work out 😢