Add IP Adapter training script and update the docs with instructions #7196
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| #### Usage Example: | ||
| ``` | ||
| accelerate launch --num_processes 8 --multi_gpu --mixed_precision "fp16" \ | ||
| tutorial_train_ip-adapter.py \ | ||
| --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5/" \ | ||
| --image_encoder_path="{image_encoder_path}" \ | ||
| --data_json_file="{data.json}" \ | ||
| --data_root_path="{image_path}" \ | ||
| --mixed_precision="fp16" \ | ||
| --resolution=512 \ | ||
| --train_batch_size=8 \ | ||
| --dataloader_num_workers=4 \ | ||
| --learning_rate=1e-04 \ | ||
| --weight_decay=0.01 \ | ||
| --output_dir="{output_dir}" \ | ||
| --save_steps=10000 | ||
| ``` |
There was a problem hiding this comment.
Lets first provide a single-GPU example command and then proceed to multi-GPU.
| import torch | ||
|
|
||
| # Load the trained model checkpoint | ||
| ckpt = "checkpoint-50000/pytorch_model.bin" |
There was a problem hiding this comment.
Let's default to serializing in .safetensors please as it's a more secure file-format.
| - `map_location="cpu"`: Specifies that the model should be loaded onto the CPU. | ||
| - `image_proj_sd`: Dictionary to store the components related to image projection. | ||
| - `ip_sd`: Dictionary to store the components related to the IP adapter. | ||
| - `"unet"`, `"image_proj_model"`, `"adapter_modules"`: Prefixes indicating components of the model. No newline at end of file |
There was a problem hiding this comment.
But we are not showing actual inference here no? I think we are just showing a part of the process. How to run inference with "ip_adapter.bin" is missing.
| block_id = int(name[len("down_blocks.")]) | ||
| hidden_size = unet.config.block_out_channels[block_id] | ||
| if cross_attention_dim is None: | ||
| attn_procs[name] = AttnProcessor() |
There was a problem hiding this comment.
We should use AttnProcessor2_0() when using PyTorch 2.0 as it's more memory and compute efficient.
There was a problem hiding this comment.
It's imported AttnProcessor2_0 as AttnProcessor
| "to_k_ip.weight": unet_sd[layer_name + ".to_k.weight"], | ||
| "to_v_ip.weight": unet_sd[layer_name + ".to_v.weight"], | ||
| } | ||
| attn_procs[name] = IPAttnProcessor(hidden_size=hidden_size, cross_attention_dim=cross_attention_dim, num_tokens=num_tokens) |
There was a problem hiding this comment.
Also imported IPAttnProcessor2_0 as IPAttnProcessor
There was a problem hiding this comment.
Better would be to first check if it's using PyTorch 2. and then dynamically selecting it.
An example:
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Thanks! I think the structure of the training scripts deviate quite a bit from how our official training scripts are written. For now, let's maybe put these under research_projects?
|
Sure! |
|
And I'll work on the suggested changes |
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
|
Can you give information about the json file? Can you share a sample json file? How should I prepare a dataset? |
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
|
To start, maybe we could add this to |
|
@AMohamedAakhil we would very much like to add this to |
|
Sure @sayakpaul |
|
Thanks! Please let us know once done! |
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
5 participants
What does this PR do?
This PR adds the original IP Adapter training scripts, and also updates the documentation with instructions on how to use it.
Fixes # (issue)
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.