[`PEFT` / `LoRA`] PEFT integration - text encoder by younesbelkada · Pull Request #5058 · huggingface/diffusers

younesbelkada · 2023-09-15T12:59:21Z

What does this PR do?

Tackles the PEFT integration step by step #4780 , the first step being the ability to leverage PEFT for the text encoder

cc @patrickvonplaten @sayakpaul @pacman100

Still draft for now

to check correctness, I run

from diffusers import StableDiffusionPipeline
import os
import torch

path = "runwayml/stable-diffusion-v1-5"
lora_id = "takuma104/lora-test-text-encoder-lora-target"

pipe = StableDiffusionPipeline.from_pretrained(path, torch_dtype=torch.float16)
pipe.load_lora_weights(lora_id)
pipe = pipe.to("cuda")

prompt = "a red sks dog"

images = pipe(
    prompt=prompt, 
    num_inference_steps=15, 
    cross_attention_kwargs={"scale": 0.5},
    generator=torch.manual_seed(0)
).images

for i, image in enumerate(images):
    file_name = f"aa_{i}"
    os.makedirs("images-integration", exist_ok=True)
    path = os.path.join("images-integration", f"{file_name}-test-text-encoder.png")
    image.save(path)

and this should return on main:

sayakpaul · 2023-09-15T13:07:10Z

@younesbelkada let me know early feedback / PR jamming is needed.

younesbelkada · 2023-09-15T13:12:53Z

src/diffusers/loaders.py


+        is_peft_lora_compatible = version.parse(importlib.metadata.version("transformers")) > version.parse("4.33.1")
+
+        if not is_peft_lora_compatible:


@sayakpaul @patrickvonplaten what is the approach we want to follow here? For now I am forcing users to use latest transformers.

I think Patrick has already proposed a nice plan i.e., we follow a deprecation cycle and then completely remove the older methods.

younesbelkada · 2023-09-15T14:50:30Z

src/diffusers/loaders.py


        # Remove any existing hooks.
-        if is_accelerate_available() and is_accelerate_version(">=", "0.17.0.dev0"):
+        if is_accelerate_available() and version.parse(importlib.metadata.version("accelerate")) >= version.parse("0.17.0"):


For some reason this test was always failing on my end with the previous logic, I replaced it with that one which is a one-liner (maybe there was a silent bug in the is_accelerate_version method

younesbelkada · 2023-09-15T15:22:21Z

tests/models/test_lora_layers.py

        # Outputs shouldn't match.
        self.assertFalse(torch.allclose(torch.from_numpy(orig_image_slice), torch.from_numpy(lora_image_slice)))

+    @unittest.skip("this is an old test")


I need to re-write the tests here as they depend on private methods I have removed

younesbelkada · 2023-09-18T11:29:51Z

src/diffusers/loaders.py

                        rank.update({rank_key_fc2: text_encoder_lora_state_dict[rank_key_fc2].shape[1]})

+                # for diffusers format you always get the same rank everywhere
+                # is it possible to load with PEFT


cc @pacman100 for multi-rank

Is this supposed to be a question? :D

src/diffusers/loaders.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

younesbelkada · 2023-09-18T14:08:22Z

tests/models/test_lora_layers.py

+    # TODO: @younesbelkada add save / load tests with text encoder
+    # TODO: @younesbelkada add public method to attach adapters in text encoder


Don't forget this

src/diffusers/loaders.py

sayakpaul · 2023-09-19T09:56:16Z

src/diffusers/loaders.py

+                    attention_modules = text_encoder_attn_modules(text_encoder)
+                    text_encoder_lora_state_dict = convert_old_state_dict_to_peft(
+                        attention_modules, text_encoder_lora_state_dict
+                    )


💯 the way to do it!

sayakpaul · 2023-09-19T09:57:21Z

src/diffusers/loaders.py

+                # New diffusers format to PEFT
+                elif any("lora_linear_layer" in k for k in text_encoder_lora_state_dict.keys()):
+                    attention_modules = text_encoder_attn_modules(text_encoder)
+                    text_encoder_lora_state_dict = convert_diffusers_state_dict_to_peft(


Yeah this part could be bit confusing for the maintainers to understand as to why we're using two different peft methods. Maybe if we name the previous one a bit more explicitly, it will help clear the confusion.

Actually @younesbelkada @sayakpaul , I'd change the logic here slightly for now:

We convert all the different formats first to diffusers format (this way this method can be leveraged by the deprecated method). And then we convert all the differnt formats to peft.
So we should IMO have to methods:

# 1. convert to diffusers format text_encoder_lora_state_dict = convert_state_dict_to_diffusers(attention_modules, text_encoder_lora_state_dict) # 2. convert to peft format text_encoder_lora_state_dict = convert_diffusers_to_peft(attention_modules, text_encoder_lora_state_dict)

Then when saving we can add the reverse method:

# 1. convert PEFT to diffusers: text_encoder_lora_state_dict = convert_peft_to_diffusers(attention_modules, text_encoder_lora_state_dict)

This has some advantages:

1.) Easier to read

2.) Easier to maintain / test

3.) Easier to add new conversions

Makes totally sense !

Proposed something in f8e87f6 , we could easily extend that to other formats if needed. Let me know what do you think

sayakpaul · 2023-09-19T09:58:19Z

src/diffusers/loaders.py

+                lora_rank = list(rank.values())[0]
+                alpha = lora_scale * lora_rank


Maybe a comment to add a justification on why this makes sense?

src/diffusers/loaders.py

… into peftpart-1

BenjaminBossan

Just a couple of comments from my side. I don't know enough about diffusers to comment on the bigger picture, so it's mostly local stuff.

Also, I would like to ensure that not too many deprecation messages are shown if a user just uses their normal code with the latest diffusers release. Personally, I would be quite annoyed to get 5 deprecation messages that are basically for the same thing. Maybe that's not happening, in that case please disregard this comment :)

BenjaminBossan · 2023-09-21T11:27:05Z

setup.py

    "torchvision",
    "transformers>=4.25.1",
    "urllib3<=2.0.0",
+    "peft>=0.5.0"


Not sure what the exact plan for releasing the updates is, but does it make sense to require peft>=0.5.0 when further below, USE_PEFT_BACKEND requires >0.5?

I added it here so that the CI will automatically fall back to PEFT tests once we make a new release on PEFT. I can also remove that and make a PR on diffusers once we make a release.
In both cases after the PEFT release we'll have to update that line to use PEFT >= 0.6.0

src/diffusers/loaders.py

BenjaminBossan · 2023-09-21T11:40:21Z

src/diffusers/loaders.py

+                    if patch_mlp:
+                        target_modules += ["fc1", "fc2"]
+
+                    lora_config = LoraConfig(r=lora_rank, target_modules=target_modules, lora_alpha=alpha)


I guess this will be replaced once huggingface/peft#873 is merged? If so, I'd leave a # TODO comment about it.

src/diffusers/utils/import_utils.py

BenjaminBossan · 2023-09-21T12:05:53Z

src/diffusers/utils/peft_utils.py

+    import torch
+
+
+def recurse_remove_peft_layers(model):


I wonder if we can somehow use unload from PEFT? The logic seems pretty similar. Maybe it would work with some small tweaks?

unload is only available on LoRAModel: https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py#L673 which currently makes it not usable at layer-level I think

Yeah, it would require some tweaks on PEFT side. Maybe we can add those later to avoid code duplication, it should be fine as is for this PR.

yes, we could think of exposing a public method that will be leveraged by unload method in LoraModel and use that later in diffusers

src/diffusers/utils/state_dict_utils.py

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

setup.py

src/diffusers/dependency_versions_table.py

patrickvonplaten

Played around with different inference scenarios and everything works as expected. PR is good to merge for me. As a final thing to make sure we test the new PEFT implementation it would be nice if we could add:

python -m pip install git+https://github.com/huggingface/peft.git
python -m pip install git+https://github.com/huggingface/transformers.git

here:

diffusers/.github/workflows/pr_tests.yml

Line 75 in d558811

python -m pip install git+https://github.com/huggingface/accelerate.git

and here:
https://github.com/huggingface/diffusers/blob/d558811b2654ad89c93b3eea10120f3878fec2be/.github/workflows/push_tests.yml#L66C9-L66C80

pacman100 · 2023-09-22T07:20:57Z

Hello @younesbelkada , could you update this PR from main?
I am trying to raise a PR to merge my changes into this branch but my branch has been recently updated from the main leading to huge diff

… into peftpart-1

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* more fixes * up * up * style * add in setup * oops * more changes * v1 rzfactor CI * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * few todos * protect torch import * style * fix fuse text encoder * Update src/diffusers/loaders.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * replace with `recurse_replace_peft_layers` * keep old modules for BC * adjustments on `adjust_lora_scale_text_encoder` * nit * move tests * add conversion utils * remove unneeded methods * use class method instead * oops * use `base_version` * fix examples * fix CI * fix weird error with python 3.8 * fix * better fix * style * Apply suggestions from code review Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * add comment * Apply suggestions from code review Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * conv2d support for recurse remove * added docstrings * more docstring * add deprecate * revert * try to fix merge conflicts * v1 tests * add new decorator * add saving utilities test * adapt tests a bit * add save / from_pretrained tests * add saving tests * add scale tests * fix deps tests * fix lora CI * fix tests * add comment * fix * style * add slow tests * slow tests pass * style * Update src/diffusers/utils/import_utils.py Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com> * circumvents pattern finding issue * left a todo * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * update hub path * add lora workflow * fix --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

more fixes

ba24f2a

younesbelkada marked this pull request as draft September 15, 2023 12:59

up

c17634c

up

2a6e535

younesbelkada commented Sep 15, 2023

View reviewed changes

younesbelkada added 4 commits September 15, 2023 13:13

style

01f6d1d

add in setup

5a150b2

oops

961e776

more changes

cdbe739

younesbelkada commented Sep 15, 2023

View reviewed changes

younesbelkada commented Sep 18, 2023

View reviewed changes

v1 rzfactor CI

691368b

patrickvonplaten reviewed Sep 18, 2023

View reviewed changes

src/diffusers/loaders.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Sep 18, 2023

View reviewed changes

src/diffusers/loaders.py Outdated Show resolved Hide resolved

younesbelkada and others added 4 commits September 18, 2023 16:00

Apply suggestions from code review

7918851

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

few todos

14db139

Merge branch 'main' into peftpart-1

c06c40b

protect torch import

d56a14d

younesbelkada commented Sep 18, 2023

View reviewed changes

younesbelkada added 3 commits September 18, 2023 14:29

style

ec87c19

fix fuse text encoder

40a6028

Merge remote-tracking branch 'upstream/main' into peftpart-1

0c62ef3

sayakpaul reviewed Sep 19, 2023

View reviewed changes

src/diffusers/loaders.py Outdated Show resolved Hide resolved

sayakpaul reviewed Sep 19, 2023

View reviewed changes

src/diffusers/loaders.py Outdated Show resolved Hide resolved

sayakpaul reviewed Sep 19, 2023

View reviewed changes

src/diffusers/loaders.py Show resolved Hide resolved

younesbelkada added 9 commits September 21, 2023 08:05

Merge branch 'peftpart-1' of https://github.com/younesbelkada/diffusers…

d1d1f22

… into peftpart-1

fix lora CI

24b38a8

fix tests

651de85

add comment

26af3ea

fix

6e3d33b

style

6903825

add slow tests

7aadd30

slow tests pass

aedffbf

style

2a64c88

BenjaminBossan reviewed Sep 21, 2023

View reviewed changes

younesbelkada and others added 4 commits September 21, 2023 14:30

Update src/diffusers/utils/import_utils.py

557404f

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

Apply suggestions from code review

5de8506

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

circumvents pattern finding issue

65fe519

left a todo

9303d04

patrickvonplaten mentioned this pull request Sep 21, 2023

support fuse/unfuse multiple lora #5129

Closed

6 tasks

Merge branch 'main' into peftpart-1

29bfc56

patrickvonplaten reviewed Sep 21, 2023

View reviewed changes

setup.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Sep 21, 2023

View reviewed changes

src/diffusers/dependency_versions_table.py Outdated Show resolved Hide resolved

patrickvonplaten marked this pull request as ready for review September 21, 2023 22:27

patrickvonplaten approved these changes Sep 21, 2023

View reviewed changes

younesbelkada and others added 6 commits September 22, 2023 07:21

Merge remote-tracking branch 'upstream/main' into peftpart-1

90974ea

Merge branch 'peftpart-1' of https://github.com/younesbelkada/diffusers…

053c827

… into peftpart-1

Apply suggestions from code review

03f7431

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

update hub path

d4637f7

add lora workflow

17ac967

fix

5295aa2

younesbelkada merged commit 493f952 into huggingface:main Sep 22, 2023


		is_peft_lora_compatible = version.parse(importlib.metadata.version("transformers")) > version.parse("4.33.1")

		if not is_peft_lora_compatible:

		# TODO: @younesbelkada add save / load tests with text encoder
		# TODO: @younesbelkada add public method to attach adapters in text encoder

		lora_rank = list(rank.values())[0]
		alpha = lora_scale * lora_rank

Conversation

younesbelkada commented Sep 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

sayakpaul commented Sep 15, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patrickvonplaten left a comment

Choose a reason for hiding this comment

Uh oh!

pacman100 commented Sep 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Comments

younesbelkada commented Sep 15, 2023 •

edited

Loading