Dreambooth class sampling to use xformers if enabled

[mu94-csl]

Currently, the training can use xformers, but not the inference for class sampling before the training.
The prior preservation class sampling of ~200 images is a major bottleneck right now (~6x time over actual training).
This PR allows xformers to be used for both training/inference, for both full and LoRA dreambooth.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[patrickvonplaten]

Hmm given that much higher speed-ups can be gained with PT 2.0 now, should we maybe just advertise PT 2.0 ? cc @sayakpaul

[sayakpaul]

Hmm given that much higher speed-ups can be gained with PT 2.0 now, should we maybe just advertise PT 2.0

I think the community still uses xFormers a lot. Maybe it's better to make it clear from the docs that if someone is using PyTorch 2.0, the efficient attention processor will be used by default and they shouldn't have to enable xFormers for that.

diffusers/src/diffusers/models/attention_processor.py

Line 143 in 4bae76e

AttnProcessor2_0() if hasattr(F, "scaled_dot_product_attention") and scale_qk else AttnProcessor()

WDYT?

[mu94-csl]

Hmm given that much higher speed-ups can be gained with PT 2.0 now, should we maybe just advertise PT 2.0 ?

Actually yes, I found no difference for Turing and Ampere GPUs, between PT2.0 and xformers.

My PR was 'misled' due to my experiments on a Pascal-gen GPU (still usable :D), for which PT 2.0 perhaps does not launch the correct kernels, but xformers help tremendously.

Also the current DreamBooth script fails on PT 2.0 due to some CUDA errors but works with xformers, suggesting that the PT backend is still quirky - see #3325.

[patrickvonplaten]

Hmm given that much higher speed-ups can be gained with PT 2.0 now, should we maybe just advertise PT 2.0

I think the community still uses xFormers a lot. Maybe it's better to make it clear from the docs that if someone is using PyTorch 2.0, the efficient attention processor will be used by default and they shouldn't have to enable xFormers for that.

diffusers/src/diffusers/models/attention_processor.py

Line 143 in 4bae76e

AttnProcessor2_0() if hasattr(F, "scaled_dot_product_attention") and scale_qk else AttnProcessor()

WDYT?

Yes good idea, we could throw a warning if we detect both PT and xformers to be installed maybe at init?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.