Differential Diffusion with SD3

[exx8]

Is your feature request related to a problem? Please describe.
One might wish to soft inpaint or edit pictures with differential diffusion with SD3.

Describe the solution you'd like.
I believe we can port diff-diff to SD3, by adding a new pipeline for it.

Describe alternatives you've considered.
Diff-diff is already supported with XL.
Traditionally, diffusers supported inpainting via a custom checkpoint.
I am not sure if SAI plans to release such a checkpoint for SD3.
A port of differential diffusion for SD3 might fill this gap as well.

Additional context.
If I understand correctly, ComfyUI might already support diff-diff with SD3.

[asomoza]

I did a quick test to see if it worked in comfyui with diff-diff, I can't really tell if it works or not because I don't see the really good results I got with SDXL and diffusers. Maybe I'm doing something wrong with comfyui.

source	without	with

I agree that diff-diff will fill the current gap of a lack of an inpainting model. I did the pipeline for SDXL but currently I won't have the bandwidth to do the SD3 one anytime soon. Are you willing to open a PR with it?

[exx8]

Hi,

I need to see the workflow to understand what you have tried so far. Currently, I do not have the capacity to open a PR according to the standards of diffusers.

I am considering adding a new release for my repo, which will hopefully support some or all of the new diffusion models, including SC, PixArt-Σ, SD3, and Hunyuan-DiT. I also need to consider how to support this research in the long run, as it is becoming increasingly demanding to support all these models.

Thanks!

[asomoza]

The images have the workflows in them, you can just drag them into comfyui.

It's fine if you don't have the time, maybe later we can ask the community to do them and just guide them though it.

The mask I used was simple but it worked fine with SDXL.

[vikm2o]

I am facing an issue while adapting Differential Diffusion for SD3. exx8/differential-diffusion#28

prepare_latents in SD3 Pipeline produces latents of this shape torch.Size([1, 16, 80, 120]) for 100 steps in this line https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3_img2img.py#L646

In SD2 its torch.Size([101, 4, 80, 120]) for 100 steps.

[asomoza]

@vikm2o did you find a solution to your problem?

I had a little time today and did a quick custom pipeline for SD3, this is my example of the apple to pear I use for testing:

source	mask	diffusers	comfyui

I used another mask for comfyui because it doesn't work with normal grayscale masks. When I have more time I'll test with a real inpainting but it seems to work.

I had to make some core modifications so I'll sort those out first before sharing, diff-diff doesn't work without doing some changes in the scheduler (that's why you are getting the first dimension with 1 instead of the steps)

[vikm2o]

@asomoza amazing! Please share when you are ready. I would love to try.

[asomoza]

@vikm2o I opened the PRs to be able to use it. I didn't know if you wanted to open a PR for the community pipeline, if you want to do it, just tell me and I'll close mine.

[asomoza]

Also I did the same example with the dog, I think it works really well most of the time:

original	result 1	result 2

[vikm2o]

@vikm2o I opened the PRs to be able to use it. I didn't know if you wanted to open a PR for the community pipeline, if you want to do it, just tell me and I'll close mine.

It's fine @asomoza . Mine wasn't ready for PR anyway !