Similar to https://github.com/huggingface/diffusers/pull/3594, the text encoder LoRA should also use PT 2.0 SPDA. Sorry, this slipped my mind in https://github.com/huggingface/diffusers/pull/3594.
