Inpainting example is using data from the masked parts with strength 1

[leszekhanusz]

Describe the bug

It is my understanding that if we provide strength=1 to the inpainting pipeline, that the model would try to fill the missing parts without taking into account what was in the image before (where the mask is).

It does not appear to be the case and is really apparent when trying to zoomout of a picture.
In that case, the prediction will very often fill this area with white.

Using the following mask (open the image in a new tab to see the white border):

I create prediction first by keeping the white border in the init_image, then I create prediction by filling the white border with similar colors (with gimp).

keeping white border:

Filling white border with gimp:

Here is my result with the white border:

Here is my result with the filled border:

It is clear that the model is keeping data about the parts which should be cropped by the mask.

Reproduction

The code I used to generate the grids is here

Add 3 images above named zoomout_mask.png zoomout_white.png zoomout_filled.png in the /tmp/inpainting folder and run the above script.

Logs

No response

System Info

diffusers version: commit 5cbed8e0d157f65d3ddc2420dfd09f2df630e978

[nagolinc]

@leszekhanusz Interesting.

I added a hack here (main...nagolinc:diffusers:inpainting-fix) that fixes this. But I wish I understood better why it was happening in the first place.

(this is using white borders and strength=1.0 and my changes)

[leszekhanusz]

Thanks for the quick answer!
But it seems like we are still missing something, you can clearly see where the mask is on the pictures and that should not be the case.
I tried to print the latents to see what they look like along the iterations and I got something like this:

What I don't understand is why we are using noise for the unmasked parts. Shouldn't we force the unmasked parts of the image in the latents at each step? For the image to image pipeline, that makes sense, because the final image will change in that zone, but for inpainting shouldn't we force the latents to have the unmasked parts untouched.
So the first latents should be something like this:

[nagolinc]

@leszekhanusz Adding the un-noised latents was what I originally tried, but @jackloomen pointed out here (#241 (comment)) that was causing the final inpainted image to be blurry.

Worth noting that @EteriaAI has apparently figured out a much better method (https://twitter.com/EteriaAI/status/1563018093055553539), so hopefully we'll find out what that is soon.

What I don't understand is why we are using noise for the unmasked parts

The NN that denoises expects to have noised latents at some step tn. If you use clean latents behavior will be undefined - and therefore blurry or with artifacts.

The version I use doesn't even have strength tbh, it has no value for inpainting.

also be careful to make sure the mask tensor scalars are 0.0 or 1.0 and nothing in between.

inpainting is working fine for me - with some semantic difficulties in the results, but sadly there is some difficulty with resampling - that is, backtracking with noise to have the model redo steps with more information from later down the pipeline.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.