Add sequence classification support for small Gemma 3 models

[adamreichold]

Feature request

While #39465 added support for larger Gemma 3 (4b parameters or more) via Gemma3ForSequenceClassification, it did not add a Gemma3TextForSequenceClassification as for example proposed in #36755 (comment). However this is necessary to use the small model variants with 1b and 270m parameters.

Motivation

Especially the smallest model with 270m parameters might be an interesting replacement for older model architectures like DistilBERT in resource-constrained environments. We have seen reasonable performance in full-parameter conversation fine-tuning, but being able to use a task-specific head for sequence classification would improve things considerably.

Your contribution

We could test proposed changes by adapting our existing training scripts which currently only work with 4b parameters or larger model variants.

[abdokaseb]

That's totally make sense to add specially after having 270m model. I will start to make a PR for this issue.

[abdokaseb]

@ArthurZucker can we add the Gemma3TextForSequenceClassification class? To handle the small models?

Knowing that Gemma3 already has two model configs normal one Gemma3Config and text one Gemma3TextConfig

[umweltinfo]

For posterity and because we were somewhat surprised: We tried to replace the DistilBERT distilbert-base-german-cased checkpoint with google/gemma-3-270m for a text classification task dear to us and Gemma 3 failed to outpeform DistilBERT in terms of accuracy and F-score. We were able to outperform DistilBERT, but only using the much larger google/gemma-3-1b-pt and a larger context window.

Of course, this is comparing apples to oranges as Gemma 3 is both multimodal and multilingual, but I think there is a need for a more modern model to replace the varied but by now 6 years old DistilBERT checkpoints which still appear to hold their own for narrowly scoped tasks.