[Core] better support offloading when side loading is enabled.#4855

sayakpaul merged 13 commits intomainhuggingface/diffusers:mainfrom

improve-side-offloadinghuggingface/diffusers:improve-side-offloadingCopy head branch name to clipboard