Avoid T5GemmaModelTest::test_eager_matches_sdpa_inference being flaky

[ydshieh]

What does this PR do?

See https://app.circleci.com/pipelines/github/huggingface/transformers/144943/workflows/f4bc060c-b21f-4528-9393-9fadc27fe49e/jobs/1916024

This is specific to T5GemmaForTokenClassification (maybe T5GemmaForSequenceClassification too), as they specify

self.model(use_cache=False, ...)

and goes through

        if attention_mask is None and past_key_values is None:
            attention_mask = make_default_2d_attention_mask(input_ids, inputs_embeds, self.config.pad_token_id)

The logic is correct, but a sequence of no place to attend is problematic in terms of numerical equivalence.

[ydshieh]

we probably better to force this for all model tester classes

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: t5gemma

[manueldeprada]

😮 nice detective work!! ofc, argmax over -inf mask is very unstable, I remember chasing something similar.

I agree we should search similar cases. Maybe adding a check in PretrainedModel.forward() so that sum of key projections >>> -inf? and then locally run the testsuite and see if that conditions triggers?

[ydshieh]

Leave as a TODO, got to sleep now 😴

Thanks a lot for approving.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.