[Bug][StableDiffusion] Allow loading models with Float16 in `load_pipeline_from_original_stable_diffusion_ckpt`

[Stax124]

Describe the bug

Models loaded from .ckpt / .safetensors file are using Float32 automatically, with no obvious way to change this behavior.

Reproduction

• Import load_pipeline_from_original_stable_diffusion_ckpt from diffusers.pipelines.stable_diffusion.convert_from_ckpt
• Load model from .ckpt file
• Observe extreme VRAM usage compared to Float16

Logs

No response

System Info

Diffusers: 0.14.0
OS: Linux (Kubuntu 22.04)
Python 3.10.6

[Stax124]

On a second look, it seems that an option to disable the safety checker is missing as well.

EDIT: I was able to manually edit that function to cast all required components by calling .half() on them, I will try to create a PR fixing these two issues. Can any maintainer tell me if there is a utility function that is used for this purpose, or should I just write one from scratch?

[sayakpaul]

Thanks for your report! It'd be helpful to provide a minimally reproducible snippet as well.

While instantiating a pipeline, you can specify safety_checker=None to disable it.

Also, note that while instantiating a pipeline, you can specify the torch_dtype argument to be torch.float16.

[Stax124]

Thanks for your report! It'd be helpful to provide a minimally reproducible snippet as well.

While instantiating a pipeline, you can specify safety_checker=None to disable it.

Also, note that while instantiating a pipeline, you can specify the torch_dtype argument to be torch.float16.

No, you cannot if you try to load it from .ckpt, as it will always load the safety_checker by default. The same goes with the dtype as it cannot be specified. Please take a look at my PR which should fix this problem.

PS: Function was renamed to download_from_original_stable_diffusion_ckpt in the version 0.15.0dev

[svjack]

Thanks for your report! It'd be helpful to provide a minimally reproducible snippet as well.
While instantiating a pipeline, you can specify safety_checker=None to disable it.
Also, note that while instantiating a pipeline, you can specify the torch_dtype argument to be torch.float16.

No, you cannot if you try to load it from .ckpt, as it will always load the safety_checker by default. The same goes with the dtype as it cannot be specified. Please take a look at my PR which should fix this problem.

PS: Function was renamed to download_from_original_stable_diffusion_ckpt in the version 0.15.0dev

Why this function renamed as 'download_from_original_stable_diffusion_ckpt'
it has the prefix "download", but seems no download working codes from internet.
simply read and open files from local

[patrickvonplaten]

yeah maybe we should indeed rename this function