Skip M4T test_retain_grad_hidden_states_attentions

[ylacombe]

What does this PR do?

After investigating the reasons for the test_retain_grad_hidden_states_attentions flaky failure, I realized the speech encoder attentions can be None with a non-zero probability when training=True. Skipping the test is the fastest fix.

Fixes #28036

cc @gante @amyeroberts @ydshieh

[ArthurZucker]

Alright, thanks 🤗

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

Thanks for fixing!

If training is allowed to happen on the model but it can fail e.g. with attentions being None, could you open an issue to track this? Training should either be prevented with an exception or made possible (probably 1 then the other)

[ylacombe]

Hey @amyeroberts, in theory, training is supported for the tasks that translate inputs (text or audio) into texts, since it's a classic LLM with classic objective.
To improve training, the model randomly skip layers in the speech encoder block (thus having None as attention weights), but it doesn't break training when it happens.