Data type mismatch when using stable diffusion in fp16

[ParadoxZW]

Describe the bug

When run following code to try stable diffusion v1.5,

from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
       "local_project_path/stable-diffusion-v1-5", 
        torch_dtype=torch.float16, revision="fp16"
)
pipe = pipe.to("cuda")

prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt).images[0]  

I got the error

File "{conda_env_path}/lib/python3.9/site-packages/transformers/models/clip/modeling_clip.py", line 260, in forward
    attn_output = torch.bmm(attn_probs, value_states)
RuntimeError: expected scalar type Half but found Float

Reproduction

No response

Logs

No response

System Info

• diffusers version: 0.6.0
• Platform: Linux-4.4.0-31-generic-x86_64-with-glibc2.27
• Python version: 3.9.11
• PyTorch version (GPU?): 1.12.0 (True)
• Huggingface_hub version: 0.10.1
• Transformers version: 4.20.1
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

[patrickvonplaten]

Hey @ParadoxZW,

Could you try to install diffusers from master:

pip install git+https://github.com/huggingface/diffusers.git

=> I cannot reprodure this error from master

[vanewu]

I had the same problem with version 0.6.x package and master branch.

[dblunk88]

I don't know if that is still the case, but fp16 used to have to be run in autocast

[vanewu]

Yes, I added autocast and computed in this scope . The problem was solved.

[dblunk88]

Yes, I added autocast and computed in this scope . The problem was solved.

That being said, diffusers has been going away from autocast. Devs, any fix for that without using autocast?

[patrickvonplaten]

Looking into it!

[patrickvonplaten]

I cannot reproduce the error - the code snippet runs fine for me. My version is:

- `diffusers` version: 0.7.0.dev0
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.13
- Python version: 3.7.12
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Huggingface_hub version: 0.10.1
- Transformers version: 4.24.0.dev0
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

@ParadoxZW could you maybe try to upgrade transformers to the newest version - the problem seems to come from transformers here actually and not diffusers:

pip install --upgrade transformers

Also @dblunk88 is 100% right, we do not recommend using autocast anymore - instead one should use "pure" FP16 as is done in the code example above.

[bonlime]

upgrading transformers worked for me

[pcuenca]

Same thing happens from time to time: https://discuss.huggingface.co/t/error-expected-scalar-type-half-but-found-float/25685/3?u=pcuenq. @patrickvonplaten perhaps we'd need to mention it in the docs, I'll open a PR later unless somebody else wants to do it :)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[PrinceRay7]

@patrickvonplaten I'm quite confused. In Diffusers docs, it is strongly disencouraged to use torch.autocast, however, in accelerate scripts, when training with the flag fp16, it is still implemented by using autocast. So i really wonder what should we do when training diffusion models. It seems like accelerate is very necessary and now it conflicts so I'm confused now. Looking forward to your reply, thanks!