[Draft] AutoPipeline

[yiyixuxu]

Draft for AutoPipeline proposed here #3855

I followed similar patterns for handling compatible schedulers: we link the pipelines that share model checkpoints with a _linked_pipelines attribute and specify the task each pipeline can do with a task attribute.

In this PR so far, I only linked the StableDiffusion Pipelines (text2img, img2img, inpaint) - if the overall design is ok, I will link all other pipelines as well.

I followed the API proposed, but I think it will be easier if we only have one AutoDiffusion pipeline and use a task argument.

from diffusers import AutoPipelineForImageToImage, AutoPipelineForTextToImage, AutoPipelineForInpainting

pipe_img2img = AutoPipelineForImageToImage.from_pretrained("runwayml/stable-diffusion-v1-5")
pipe_txt2img = AutoPipelineForTextToImage.from_pretrained("runwayml/stable-diffusion-v1-5")
pipe_inpaint = AutoPipelineForInpainting.from_pretrained("runwayml/stable-diffusion-v1-5")

print(type(pipe_img2img), type(pipe_txt2img), type(pipe_inpaint))

<class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion_img2img.StableDiffusionImg2ImgPipeline'> 
<class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> 
<class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion_inpaint.StableDiffusionInpaintPipeline'>

pipe_txt2img = AutoPipelineForTextToImage.from_pipe(pipe_img2img)
pipe_inpaint = AutoPipelineForInpainting.from_pipe(pipe_txt2img)
pipe_img2img = AutoPipelineForImageToImage.from_pipe(pipe_inpaint)
print(type(pipe_img2img), type(pipe_txt2img), type(pipe_inpaint))

<class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion_img2img.StableDiffusionImg2ImgPipeline'> 
<class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> 
<class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion_inpaint.StableDiffusionInpaintPipeline'>

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Hey Yiyi,

The idea was rather something like the following:

class AutoPipelineForText2Image(ConfigMixin):

     config_name = "model_index.json"

     @classmethod
    def from_pretrained(cls, pretrained_model_or_path, **kwargs):
        config = cls.load_config(pretrained_model_or_path)

        text_2_image_cls = _get_task_class(AUTO_TEXT2IMAGE_PIPELINES_MAPPING, config["_class_name"])

        return text_2_image_cls.from_pretrained(pretrained_model_or_path, **kwargs)
        
    @classmethod
    def from_pipe(pipeline: DiffusionPipeline, **kwargs):
         text_2_image_cls = get_task_class(AUTO_TEXT2IMAGE_PIPELINES_MAPPING, pipeline.config["_class_name"])

         return text_2_image_cls(**pipeline.components)

Note that TEXT2IMAGE_PIPELINES should look a bit like the following:

AUTO_TEXT2IMAGE_PIPELINES_MAPPING = OrderedDict(
    ("stable-diffusion", StableDiffusionPipeline),
    ...
)

and the function _get_task_class should do the following:

• 1.) go over all auto mappings (t2i, i2i, inpaint) to find "stable-diffusion"
• 2.) Then retrieve StableDiffusionPipeline from AUTO_TEXT2IMAGE_PIPELINES_MAPPING

Similarly we should imlpement img2img and inpaint.

The important thing here is that AutoPipelineForText2Image should not inherit from DiffusionPipeline.

Also could we move all this into a auto_pipeline.py file?

[patrickvonplaten]

Hmm, I think it's nicer to have AutoPipelineForText2Image because otherwise AutoDiffusionPipeline is essentially the same as DiffusionPipeline

[yiyixuxu]

@patrickvonplaten

ohh so similar to Transformers
sure! could you maybe explain a little bit why it shouldn't inherit from DiffusionPipeline?

[patrickvonplaten]

ohh so similar to Transformers
sure! could you maybe explain a little bit why it shouldn't inherit from DiffusionPipeline?

Sure! The reason it should not inherit from DiffusionPipeline is because it should instead dispatch to DiffusionPipeline and because it's not the same thing as DiffusionPipeline. DiffusionPipeline is mostly used as a "base" class for all "real" pipelines that are used to run code. AutoPipelineForText2Image is however not a "real" pipeline:

• There is no paper called AutoPipelineForText2Image
• We will never save a pipeline under the AutoPipelineForText2Image class name - therefore it shouldn't inherit the save_pretrained function

We want AutoPipelineForText2Image just be responsible for finding out the correct pipeline class, e.g. StableDiffusionPipeline and then just use StableDiffusionPipeline.from_pretrained(...) => so in that sense AutoPipelineForText2Image will never be created as its own class (there is no such AutoPipelineForText2Image) and always return an actual / existing class. If it inherits from DiffusionPipeline that would mean that it can be its own class which we don't want

[yiyixuxu]

to be continued here #4138