Use HTTP-based `upload_folder` in training scripts

[Wauplin]

While investigating #2851 (getting error message when using push_to_hub in a training), I realized that most of the training scripts rely on Repository to push training data to the Hub at the end of the training (train_text_to_image_flax, train_instruct_pix2pix, train_dreambooth,...).

I guess this is due to some copy-pasting from another script (and that's fine since it works). However, I think it would be better to use upload_folder instead of Repository. The differences I see are:

• upload_folder is expected to be ~1.3x to 1.6x faster than git push
• if the repo already exists, there is no need to clone it locally before starting the training. With upload_folder, the "clone" step is avoided
• upload_folder do not have a sense of "local repo". This is a problem if we want the users to have a local copy of the repo at the end of their training with a commit history. With upload_folder we would still have the artifacts locally but not as a git repo, just as a normal folder.
• upload_folder is more "user-friendly" with a progress bar compared to running "git push" in the background without any logs
• Repository has the advantage of being able to run in the background. However, since the scripts only pushes once at the end of the training, it's not a problem to have a "blocking" step.

@patrickvonplaten @pcuenca @sayakpaul what's your opinion on that? If you also think it makes sense, I'd be glad to open a PR. Just let me know if there are some specificities I need to know.

Note: about #2851 itself and if we keep Repository, I think it would be best to use repo.push_to_hub(..., blocking=True) in all the scripts in order to properly block the main thread at the end of the training.

[sayakpaul]

Thanks so much @Wauplin for this investigation!

Personally, I find upload_folder() to be much friendlier UX-wise. So, I would be supportive of the idea.

[pcuenca]

Hi @Wauplin, thanks a lot for the excellent investigation and the proposal! I agree that the short-term solution for #2851 would be to wait for the upload to finish instead of pushing in the background.

Regarding upload_folder, I think it's great. My question is what happens if the user performs various training runs using the same script but changing the hyperparameters between runs. Would upload_folder work in this scenario? Would it replace the previous contents of the same folder in the repo? Would the new upload be registered in the repo as a new "commit" so the user can go back to artifacts from previous training runs? I think this would be a very frequent use case of the training script, so I'm trying to understand user expectations here :)

[Wauplin]

@pcuenca I'll try to answer your questions 1 by 1 but please let me know if I missed something.

what happens if the user performs various training runs using the same script but changing the hyperparameters between runs. Would upload_folder work in this scenario?

By default, all files from output_dir will be committed in a single commit. LFS files that are already existing remote will not be uploaded twice (but still be in the commit). This is already what's happening with the current repo.push_to_hub implementation since under the hood it does git add .+git commit+git push.

Would it replace the previous contents of the same folder in the repo?

Only for the files that have changed. The files that didn't change will still be present in the repo after upload_folder. Note that this is exactly what would happen using Repository.push_to_hub since old files are not explicitly deleted.

If you are afraid of having some logs files from a previous training for example, you need to delete them from the local folder (in pipeline.save_pretrained ?). That's also the case with Repository -and currently not done afaik-. If you do delete some files locally between re-runs, you can then use the delete_patterns parameter in upload_folder to delete remote files while pushing new files in a single commit.

Example:

api.upload_folder(
    folder_path=args.output_dir,
    repo_id=...,
    allow_patterns="*.txt", # Upload all local text files
    delete_patterns="*.txt", # Delete all remote text files in the same commit
)

(note: delete_patterns requires huggingface_hub>=0.13)

Would the new upload be registered in the repo as a new "commit" so the user can go back to artifacts from previous training runs?

Yes everything is uploaded as a single commit. You can even create a tag programmatically to have an explicit name to pass as "revision" (which is also the case with Repository).

[pcuenca]

That sounds great! Deleting old files sounds a nice-to-have, but the default behaviour is very much what I would expect. Thanks for the clarifications!

[Wauplin]

@pcuenca @sayakpaul is it good to go then? If yes, I'll give it a try this week. My plan is to update the training scripts to use upload_folder and keep the same behavior as now.

Since they all share a lot of code in common, I'm not planning to test them all. Do you have a procedure when updating the training scripts or do you trust it's working? -although I guess it's less critical than introducing a bug in diffusers itself-

[sayakpaul]

Looks good to me but I would like to have @patrickvonplaten's inputs as well.

[patrickvonplaten]

Def happy to switch to upload_folder if it has only advantages at pretty much no downsides.

One thing that's quite important for me is that the upload can happen asynchronously, i.e. I don't want people to start training and then only after 20k steps the script fails because the upload didn't work. E.g. if uploading the model works for whatever reason (bad internet), the training still needs to be able to continue.

[Wauplin]

One thing that's quite important for me is that the upload can happen asynchronously, i.e. I don't want people to start training and then only after 20k steps the script fails because the upload didn't work. E.g. if uploading the model works for whatever reason (bad internet), the training still needs to be able to continue.

Yes, seems reasonable to me as well. But in all the training scripts I've seen, the push_to_hub phase only happens at the end of the training. Do you have an example of training script where weights are pushed while training? (I know there are in transformers so definitely agree with the use case, I just need to find one)

[patrickvonplaten]

Here is one :-) https://github.com/huggingface/transformers/blob/d5de578c2227250d615f73a8fb88a5ce7f1743be/examples/flax/language-modeling/run_t5_mlm_flax.py#L956

BTW @yiyixuxu we should have something similar for the controlnet sprint

[Wauplin]

Cool, thanks for the link.

For now I'll focus on diffusers training scripts that do not require background tasks and in parallel start to see what we can do in huggingface_hub to make non-blocking calls as convenient as Repository (getting back huggingface/huggingface_hub#939 :) )

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.