[tests] enable cpu offload test in torchao without compilation.#12704

sayakpaul merged 3 commits intomainhuggingface/diffusers:mainfrom

torchao-offloadhuggingface/diffusers:torchao-offloadCopy head branch name to clipboard