Add StableDiffusionXLControlNetInpaintPipeline

[viiika]

What does this PR do?

Fixes # (issue)

Add a new class StableDiffusionXLControlNetInpaintPipeline() for using SDXL Inpaint Model with ControlNets.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[patrickvonplaten]

Can we also add some tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[viiika]

Can we also add some tests?

Sure.

[yiyixuxu]

is this same as #4694 ?

[viiika]

is this same as #4694 ?

They are similar but not the same. This PR mainly adapted from src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py, so it has the same coding style with pipeline_stable_diffusion_xl series. #4694 mainly adapted from controlnet, so it may have more similar class structure with controlnet instead of stable_diffusion_xl.

[deepconv]

Thanks for the work. I have tried to use the newly introduced class in this PR, but in my experiments, the inpainting result is often not compatible with the context. I'm using depth condition though.
For SD1.5, the inpainting model is a separate model trained in inpainting mode, instead of directly using the base SD1.5 model.
Can SDXL base model without special training work well for inpainting?

[yiyixuxu]

cc @harutatsuakiyama here

We only need one inpaiting pipeline for SDXL so it would be great if you guys could work together! We can make sure you are both the author of the same PR:)

I think since we have already reviewed this PR #4694, maybe we will continue to work on this PR? @viiika would you be willing to review it too and point out things that you implemented in your PR but are missing there?

or, you guys decide to go with this PR that would be totally fine too!

Thanks

YiYi

[kfzyqin]

cc @harutatsuakiyama here

We only need one inpaiting pipeline for SDXL so it would be great if you guys could work together! We can make sure you are both the author of the same PR:)

I think since we have already reviewed this PR #4694, maybe we will continue to work on this PR? @viiika would you be willing to review it too and point out things that you implemented in your PR but are missing there?

or, you guys decide to go with this PR that would be totally fine too!

Thanks

YiYi

Thank you @yiyixuxu. I am more than happy to work in conjunction with @viiika.