New Discrete Diffusion Model: Wuerstchen

[patrickvonplaten]

Model/Pipeline/Scheduler description

The amazing @dome272 published a new diffusion model, called Wuerstchen: https://github.com/dome272/Wuerstchen

I think it would be a very nice addition to the library given it's superior speed.

Anybody interested in picking this one up?

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Paper: https://arxiv.org/abs/2306.00637
Code: https://github.com/dome272/Wuerstchen
Model: https://github.com/dome272/Wuerstchen

@dome272

[alexblattner]

this is nice, but have you considered implementing this:
https://github.com/jannerm/ddpo

It's training through human input but improved
here's a paper:
https://rl-diffusion.github.io/

[Abhinay1997]

Wuerstchen is a fascinating read!

[kadirnar]

this is nice, but have you considered implementing this: https://github.com/jannerm/ddpo

It's training through human input but improved here's a paper: https://rl-diffusion.github.io/

Are you considering adding it to the Diffusers library?

[adit299]

I am new to this repository, but I wouldn't mind taking a crack at it!

[dg845]

Not sure which model @adit299 is referring to but I'd be interested in trying to implement the DDPO model if no one else is planning to work on it :).

[adit299]

Not sure which model @adit299 is referring to but I'd be interested in trying to implement the DDPO model if no one else is planning to work on it :).

I was referring to the Wuerstchen model.

[dome272]

Would love to if you would give it a shot.

[dg845]

Might be a little off-topic but my understanding is that the DDPO models are already supported in diffusers from a pipeline perspective. The models released with the original code are all finetuned from Stable Diffusion v1.4 (and so are compatible with StableDiffusionPipeline) and already have diffusers-compatible checkpoints (1, 2, 3, 4).

Something that could still be added is an example for finetuning a model using DDPO, not sure if the original authors are planning to work on that or not.

[rudolphyo]

@dome272 Hi!

I was eager to dive in implementation of Würstchen pipeline to examples/community, but noticed that you are one of the original authors and already making some progress.
My question is, would you mind if I'll take my chances in implementing, or you'd like to do it yourself?
Maybe you see a way we can collaborate on this ?

By the way, paper have gorgeous generation examples!

[dome272]

Hey @rudolphyo !
I would to if you would give it a shot. We can also talk more about this on discord. Feel free to add me discord=dome1

[rudolphyo]

@dome272, thanks for such fast reply!
I'll definitely circle back to you on discord in case of stumbling upon peculiar issues

[rudolphyo]

@patrickvonplaten hi!
I'd like to share a brief plan here and also have some guidance, cause it appeared more complex than I thought at first.

• I was hoping to reuse model's code from diffusers/models (e.g. vq_model.py, resnet.py) as it stated in contribution guidelines but it seems like it would be more explicit and beneficial to add new module for VQ-GAN[0] ? Same applies for Paella[1] which samples from predicted noise ("token predictor")
• As I can see from other PRs models should be introduced in diffusers/models so its ok to exceed scope of examples/community? I was able to find some info bit by bit on this matter in various issues and discussions, but maybe I missed some resource and you can point me in the right direction ?

here you can find links to author's original code
[0] https://github.com/dome272/Wuerstchen/blob/main/vqgan.py
[1] https://github.com/dome272/Wuerstchen/blob/main/modules.py

thanks in advance !

[patrickvonplaten]

Hey @rudolphyo,

Thanks for the nice summary! Fully agree with you think a vq_gan.py file in diffusers/models is warranted here as wuerstchen works with tokens. More than happy to guide you in a PR. Also cc @patil-suraj here