Would a new "AutoPipeline" API abstraction be helpful?

[patrickvonplaten]

Many checkpoints can be used for different tasks, e.g. img2img, text2img, but in order to do this at the moment, one has to know what pipelines are linked to each other, e.g. one has to know that StableDiffusionPipeline is linked to StableDiffusionImg2ImgPipeline when switching from text2img to img2img. Would a new API such as:

AutoForText2Image, AutoForImage2Image, and AutoForInpainting, so that the following case is possible:

from diffusers import AutoForText2Image

text2img = AutoForText2Image.from_pretrained("runwayml/stable-diffusion-v1-5")

img2img = AutoForImage2Image(**text2img.components)

[alexblattner]

I know you guys don't like my idea, but I think that there's too many pipelines and that there should be a way to use parts of one with another without recreating the pipeline from scratch or using another available. why should there be so many inpainting pipelines when there's just one part that's changed from the original to fit the inpainting part?

I understand wanting to have easily distinguishable pipelines, but this will needlessly increase the workload in my opinion. It's harder in the short term but amazing for the long term

[qnlbnsl]

Hi, This would be incredibly helpful for backend servers where we need to support multiple pipelines. currently i tried using

if existing_pipeline is not None:
        components = existing_pipeline.pipeline.components  # type: ignore
        logger.debug("Retrieved components from existing pipeline")
        if queueName == "txt2img":
            pipeline = cast(
                StableDiffusionPipeline,
                StableDiffusionPipeline.from_pretrained(**components),
            )
        elif queueName == "img2img":
            pipeline = cast(
                StableDiffusionImg2ImgPipeline,
                StableDiffusionImg2ImgPipeline.from_pretrained(**components),
            )
        elif queueName == "inpaint":
            pipeline = cast(
                StableDiffusionInpaintPipeline,
                StableDiffusionInpaintPipeline.from_pretrained(**components),
            )
        else:
            raise ValueError(f"Invalid queueName: {queueName}")

Where the existing pipeline was an array of pipelines wrapped in a custom object. However i noticed that reusing cross queue pipelines was causing the new pipeline to be of the same type as the original one. Maybe i just have something done wrong.
For Example: If existing pipeline was a StableDiffusionPipeline and i am trying to load StableDiffusionImg2ImgPipeline it will load the StableDiffusionPipeline again.

[patrickvonplaten]

Ok let's go ahead with a Auto<Task>Pipeline so that we can easily switch between tasks for the same checkpoint.
I'd say we implement the following API:

from diffusers import AutoPipelineForTextToImage, AutoPipelineForImageToImage, AutoPipelineForInpainting

text2image = AutoPipelineForTextToImage.from_pretrained("runwayml/stable-diffusion-v1-5")
image2image = AutoPipelineForImageToImage.from_pipe(text2image)  # zero cost -> we just transfer the components
...

@yiyixuxu would you like to give this a try?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.