Add _loss_is_scaled_for_ga to allow custom trainers to control gradient accumulation loss scaling

[abigailtech]

Added a _loss_is_scaled_for_ga property that custom trainers can override to explicitly control gradient accumulation loss scaling. The default implementation preserves backward compatibility. Custom trainers can now simply override this property to return False instead of manipulating model_accepts_loss_kwargs.

Fixes #43604

[Rocketknight1]

cc @qgallouedec

[qgallouedec]

If I understand correctly model_accepts_loss_kwargs checks if the model forward has kwargs, but is actually used to decide if the loss should be scaled or not, am I right? The underlying assumption if that if the model accepts kwargs, then it takes num_items_in_batch as argument, which means that it scales the loss by itself.
TBH, I'm wondering if we should instead aim for a more ambitious refactor here.

[abigailtech]

If I understand correctly model_accepts_loss_kwargs checks if the model forward has kwargs, but is actually used to decide if the loss should be scaled or not, am I right? The underlying assumption if that if the model accepts kwargs, then it takes num_items_in_batch as argument, which means that it scales the loss by itself. TBH, I'm wondering if we should instead aim for a more ambitious refactor here.

yees, thats right. I'd be open to a more ambitious refactor, do you have a specific direction in mind?