I wanted to post here that running the SDXL training script with the latest branch and training crashed consistently with SIGFPE (floating point exception).
I'm using an Nvidia T4, CUDA 12 and Torch 2.10. Among other settings, I used FP16 mixed precision training and the Prodigy optimizer. Using Prodigy specifically caused the floating point crash.
After trying many different things, I finally was able to resolve the issue by installing Torch 2.9.1.
I wanted to post here that running the SDXL training script with the latest branch and training crashed consistently with SIGFPE (floating point exception).
I'm using an Nvidia T4, CUDA 12 and Torch 2.10. Among other settings, I used FP16 mixed precision training and the Prodigy optimizer. Using Prodigy specifically caused the floating point crash.
After trying many different things, I finally was able to resolve the issue by installing Torch 2.9.1.