SDXL Lora training: testing results are different from validation results

[yanjk3]

I am trying to train an SDXL Lora to generate ’Kuromi‘ by using the train_text_to_image_lora_sdxl.py. The training is done and the validation results are as I expected.
However, the testing results are weird and different from the validation results.

Here is my training script:
accelerate launch train_text_to_image_lora_sdxl.py --pretrained_model_name_or_path='/my_local_path/stable-diffusion-xl-base-1.0' --pretrained_vae_model_name_or_path='/my_local_path/models--madebyollin--sdxl-vae-fp16-fix' --train_data_dir='./kuromi_data' --validation_prompt='A Kuromi wearing a purple top hat, holding a magic wand, unfolds her wings, grins mischievously, surrounded by skeletons, with a cute anime style, ultra-high quality, close-up.' --output_dir='./training_results/kuromi/' --num_train_epochs=150 --train_batch_size=4 --gradient_accumulation_steps=4 --mixed_precision='fp16' --checkpointing_steps=1

[yanjk3]

Note, I have not modified any code in the train_text_to_image_lora_sdxl.py, and even if the sampling step and random seed are all the same for validation and test, the mentioned problem still exists.
Now I suspect that there may be an issue with the part that saves the lora parameters.

[yanjk3]

I also suspect that there may be an issue loading the lora parameters. Has anyone else encountered a similar problem?

[sayakpaul]

#6225 should have fixed it. Could you please check again?

[yanjk3]

@sayakpaul Cool, that's the reason. Many thanks and close it~