Expand documentation for LORA with locally stored models

[nicemicro]

Description of LORA fine tuning does not cover all the bases
I am fairly new to Stable Diffusion, but have some experience using other ML projects, so in general, configuring a dataset and fine tuning is not an issue, however, I run into a block here trying to finetune LORA on a local model, as the --pretrained_model_name_or_path switch expects a JSON file, but this behavior is undocumented for the script.

A solution I'd like
A chapter for the documentation in diffusers/examples/text_to_image/README.md that describes or at least links to other documentation on what file path should be specifies for using a locally downloaded model for the basis of LORA.

Alternatives I've considered
I've read through the code, and I can see that the args.pretrained_model_name_or_path is used in many function calls, so I assume people more familiar with Stable Diffusion would be able to find the documentation, but it is hardly ideal for someone a bit less knowledgeable.

Additional context
Nowadays many different SD models exist that one can get from the internet. Using LORA to fine tune one that is almost there for one's personal use case would be ideal from a time and wasted computational power perspective. Therefore having documentation on how to do LORA fine tuning on locally downloaded models (safetensors, ckpt) would be nice.

[patrickvonplaten]

Hey @nicemicro,

Yes I agree we should try to improve our docs here. Maybe @sayakpaul you could take this in one go next week once you have some time to adapt the LoRA training script :-)

[sayakpaul]

I think it's unrelated to LoRA but related to mentioning the expectation of what pretrained_model_name_or_path should be in general, no?

pretrained_model_name_or_path remains unchanged irrespective of LoRA. Or am I missing something?

For clarification on the docs side, I'd like to defer to @yiyixuxu.

[patrickvonplaten]

Guess it could make sense to generally update the LoRA docs here a bit once we support LoRA text encoder training and LoRA A1111 format loading. Maybe just a PR similar to: #3068. Also cc @stevhliu @yiyixuxu

[nicemicro]

@sayakpaul yes, you are right, it is not directly related to LoRA. If there is a documentation on what pretrained_model_name_or_path expects, I somehow did not find, and by extension, others might not find it either, so at least a link would be useful in my opinion.

[sayakpaul]

I see what you are saying.

All the pipelines (like StableDiffusionPipeline) inherit from DiffusionPipeline. There we have:

Any diffusion pipeline that is loaded with from_pretrained() will automatically detect the pipeline type, e.g. StableDiffusionPipeline and consequently load each component of the pipeline and pass them into the init function of the pipeline, e.g. init().

And there you have: https://huggingface.co/docs/diffusers/v0.15.0/en/api/diffusion_pipeline#diffusers.DiffusionPipeline.from_pretrained.

Does this help?

Cc @stevhliu

[stevhliu]

Thanks for the feedback!

For all the training examples, I can clarify that setting the MODEL_NAME environment variable is really just specifying the argument to from_pretrained.pretrained_model_name_or_path and maybe also include a link to it.

[nicemicro]

@sayakpaul @stevhliu
Thank you for the quick reactions!
I think if it specified what command/variable it uses from what other package, that would be helpful, and a direct link to the relevant documentation would be amazing!