Don't override torch_dtype and don't use when quantization_config is set#11039

DN6 merged 5 commits intohuggingface:mainhuggingface/diffusers:mainfrom

hlky:torch-dtype-quantization-confighlky/diffusers:torch-dtype-quantization-configCopy head branch name to clipboard