Training text_to_img with multi gpu setup

[flixmk]

Hi,

I really like the text_to_image training script and had really good results with it. I want to train a model now with multiple GPUs.
My current understanding:
I want to use a data paralell setup thus each GPU gets its batches and at the end they sync the gradients. This should not result in a significant increase of training time (right?).
So far, when training with 2 GPUs the training time is doubled compared to a single GPU setup. The loss converges faster because of the higher batch size but the overall throughput / h would not increase that way.

Please correct me if i am wrong.

If i am correct, how do i enable the DP setup in accelerate or should I revert to torch and its DataParalell wrapper?

[patrickvonplaten]

Hey @flixmk,

You can make use of accelerate for multi-gpu. You just need to configure your accelerate to use multiple GPUs. So here: https://github.com/huggingface/diffusers/tree/main/examples/text_to_image#running-locally-with-pytorch just make sure to correctly configure:

accelerate config

@sayakpaul it could make sense to maybe write a short section about multi-GPU training since it's the 2nd or third issues about multi-gpu training now

[sayakpaul]

Definitely makes sense.

#2997 to keep better track of it :)

[sayakpaul]

Hi @flixmk, with #3126 being merged, we should have better coverage of multi-GPU support for our training examples. Please check them out.

For now, I am closing this issue, but feel free to re-open :)