[Pipeline] Extending Stable Diffusion for generating videos

[sayakpaul]

Since Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation has been there for sometime now, it'd be cool to officially have it supported from Diffusers 🧨

The best part is the official repository (https://github.com/showlab/Tune-A-Video) itself builds on top of Diffusers.

Architecture-wise, the main change is to inflate the UNet to operate spatiotemporally. This is implemented in the UNet3DConditionModel.

@zhangjiewu will it be possible to publish a few weights on the Hugging Face Hub for the community to try out quickly? Happy to help with the process :)

We're more than happy to help if a community member wants to pick this up. As it will be the first end-to-end video pipeline in Diffusers, I'm very excited about it.

[Abhinay1997]

@sayakpaul I’d like to work on this.

[sayakpaul]

Please feel free to proceed right away! We are more than happy to help.

[sayakpaul]

Seems like there are a few pre-trained models here already: https://huggingface.co/Tune-A-Video-library

[Abhinay1997]

@sayakpaul what is the expected outcome ?

My understanding is:

1. We make the TuneAVideoPipeline and it's dependency, UNet3DConditionModel available via diffusers.
2. We provide some trained TuneAVideoPipeline compatible checkpoints for users to play around with.
3. We add tests

Let me know if I'm missing something

[sayakpaul]

Yes, that is correct. However, for the pipeline to work, we first need to add the inflated classes of UNet and its related components, such as attention.

[Abhinay1997]

Ohh Right. Well, I'll start and open a draft PR.

[jorgemcgomes]

@Abhinay1997 FYI, some findings based on my own experiments with Tune-A-Video:

1. using prior preservation loss (as implemented in https://github.com/bryandlee/Tune-A-Video/blob/main/train.py) helps a lot with the relevance of the output videos and in avoiding overfitting.
   This is not part of the official Tune-A-Video implementation, but it's something you might want to consider when building the training script.

2. Using more training frames (I tried up to 32) resulted in a more stable loss and higher quality output. So, definitely don't hardcode it to the 8 frames found in the official implementation.

[sayakpaul]

Thank you for sharing your experience, definitely very helpful to keep in mind. However, for the first PR, we won't be considering the training script.

But when we do consider the training script, would you be up for contributing a PR?

[jorgemcgomes]

@sayakpaul Yes, definitely. I'll keep an eye on the PR that @Abhinay1997 will open 👍

[Abhinay1997]

@jorgemcgomes thanks for the inputs. Will keep this in mind. I'm sure we'll need these details down the line.

[Abhinay1997]

Hi @sayakpaul added a draft PR #2455.

So, a few follow up questions:-

1. We'll be replacing einops.rearrange, einops.repeat with equivalent torch operations where applicable, correct ? I see einops being commented out in other files too. Just wanted to confirm.

2. The BasicTransformerBlock in TuneAVideo's attention.py uses SparseCausalAttention. But diffusers already has a BasicTransformerBlock with CrossAttention. Can we make the BasicTransfomerBlock configurable then ? For now I am calling it BasicSparseTransformerBlock to differentiate.

[sayakpaul]

I replied on your PR :)

[zhangjiewu]

Hi folks, thank you for your great efforts in integrating Tune-A-Video to diffusers. We have made some updates to our implementation, resulting in improved consistency. We hope that these changes would be helpful.

Please let me know if there is anything I can assist you with. :)

[Abhinay1997]

Thanks for letting me know @zhangjiewu. I'll cross check to see if I have missed anything. :)