Kandinsky Prior doesn't work with SD.Next

[patrickvonplaten]

How to configure prior to work with kandinsky 2.2?
Seems to not work in SD.Next

[patrickvonplaten]

Having issues with kandinsky 2.2, simply loading a prior model When loading kandinsky-2.2-prior using diffusers.DiffusionPipeline.from_pretrained it fails:

ValueError: Pipeline <class 'diffusers.pipelines.kandinsky.pipeline_kandinsky.KandinskyPipeline'> expected {'text_encoder', 'movq', 'tokenizer', 'unet', 'scheduler'}, but only {'text_encoder', 'tokenizer', 'scheduler'} were passed

To me it looks like its trying to load using kansinsky-2-2 pipeline instead of kandinsky-2-2-prior pipeline, like model config is incorrect?

[yiyixuxu]

do we still have this issue??

the _class_name was incorrect but updated before this issue was created
https://huggingface.co/kandinsky-community/kandinsky-2-2-prior/discussions/1

[vladmandic]

i confim that changes to model_index.json solve that particular problem,
but i still cannot use kandinsky 2.2 (no such problems with kandinsky 2.1)

if i pass prompt, i get error:

TypeError: KandinskyV22Pipeline.call() got an unexpected keyword argument 'prompt'

but if i don't pass prompt, i get error:

/home/vlado/.local/lib/python3.10/site-packages/diffusers/pipelines/kandinsky2_2/pipeline_kandinsky2_2_prior.py:412 in call
ValueError: prompt has to be of type str or list but is <class 'NoneType'>

[yiyixuxu]

@vladmandic
yeah that's expected actually. 2.2 no longer accept prompt, see more here:
https://huggingface.co/docs/diffusers/api/pipelines/kandinsky#kandinsky-22

[vladmandic]

well, that's a deal-breaker for me (meaning sd.next) as i cannot have massive if/then blocks for every different pipeline.
sd 1.5, sd 2.1, sd-xl, kandinsky 2.1, unidiffusers all work. kandinsky 2.2 doesn't. as interesting of a model it may be, it will get left behind - and the whole point of diffusers was to have uniform usage.

[yiyixuxu]

@patrickvonplaten

we can add back prompt argument and deprecate it?
let me know what you think

[patrickvonplaten]

Kandinsky 2.2 works as follows:

Prior:

prompt -> prior pipeline ->x image_embeds 

Decoder

image_emb -> decoder pipeline -> image

The decoder pipeline which we call KandinskyV22Pipeline simply doesn't take a prompt as an input (most decoder-only pipelines don't), so it really wouldn't make sense for us to add it that'd just be wrong and a dummy variable.

Kandinsky 2.1 is different because the prompt is passed to both the pior and decoder (which is unusual):

Prior:

prompt -> prior pipeline -> image_emb

Decoder

image_emb, prompt -> decoder pipeline -> image

[patrickvonplaten]

That being said, I fully agree with

the whole point of diffusers was to have uniform usage.

and we should make sure better functionality is provided here. The downstream user should indeed not have to worry about how to best stitch together diffusers pipelines and we should provide better automated behavior here.

[patrickvonplaten]

Working on an improved design now so that all pipelines can be called with a single pipeline instance:
#4207

This will allow:

pipe = KandinskyV22CombinedPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-decoder")

to download everything.

This way we can also delete all this boilerplate code: https://github.com/vladmandic/automatic/blob/511a8cbb61f2c9c09e64a96df0b801e7e73a4236/modules/sd_models.py#L522 cc @vladmandic

for both Kandinsky 2.1 and 2.2.

Would this work for you?

[vladmandic]

for hf models, right now i'm using DiffusionPipeline.from_pretrained and it selects which pipeline to instantiate.
if its done behind the scene, great.
if not, then i need a way to determine model type to use programatically.

[patrickvonplaten]

We're working on a better mechanism to load all models for a specific task with AutoPipelineForTextToImage.from_pretrained(...)