Add variant to transformers #21332

patrickvonplaten · 2023-01-27T09:17:59Z

What does this PR do?

This PR adds a "variant" keyword argument to PyTorch's from_pretrained and save_pretrained so that multiple weight variants can be saved in the model repo.

You can try it out by running:

from transformers import CLIPTextModel

path = "huggingface/the-no-branch-repo"  # or ./text_encoder if local

print("This should work!:")
model = CLIPTextModel.from_pretrained(path, subfolder="text_encoder", variant="no_ema")
print("This should work!:")
model = CLIPTextModel.from_pretrained(path, subfolder="text_encoder", variant="fp16")
print("This should work!:")
model = CLIPTextModel.from_pretrained(path, subfolder="text_encoder")
print("This should NOT work!:")
model = CLIPTextModel.from_pretrained(path, subfolder="text_encoder", variant="other")

From this repo: https://huggingface.co/huggingface/the-no-branch-repo/tree/main/text_encoder . The repo is a dummy stable diffusion model and folder structure looks as follows:

├── feature_extractor
│   └── preprocessor_config.json
├── load.py
├── model_index.json
├── safety_checker
│   ├── config.json
│   └── pytorch_model.bin
├── save.py
├── scheduler
│   └── scheduler_config.json
├── text_encoder
│   ├── config.json
│   ├── pytorch_model.bin
│   ├── pytorch_model.fp16.bin
│   └── pytorch_model.no_ema.bin
├── tokenizer
│   ├── merges.txt
│   ├── special_tokens_map.json
│   ├── tokenizer_config.json
│   └── vocab.json
├── unet
│   ├── config.json
│   └── diffusion_pytorch_model.bin
└── vae
    ├── config.json
    └── diffusion_pytorch_model.bin

cc @pcuenca @patil-suraj @sgugger @LysandreJik @julien-c @osanseviero

[Update] This PR should be ready for merge

Bumps [onnx](https://github.com/onnx/onnx) from 1.11.0 to 1.13.0. - [Release notes](https://github.com/onnx/onnx/releases) - [Changelog](https://github.com/onnx/onnx/blob/main/docs/Changelog.md) - [Commits](onnx/onnx@v1.11.0...v1.13.0) --- updated-dependencies: - dependency-name: onnx dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>

HuggingFaceDocBuilderDev · 2023-01-27T09:43:27Z

The documentation is not available anymore as the PR was closed or merged.

pcuenca

I did a quick first pass. I'm not sure about all the design implications for transformers, so I just pointed out small comments and suggestions. Will review and test in depth after design is frozen :)

src/transformers/modeling_utils.py

pcuenca · 2023-01-27T11:39:34Z

src/transformers/modeling_utils.py

+
+                tf_weights_1_path = get_local_path(TF_WEIGHTS_NAME + ".index")
+                tf2_weights_2_path = get_local_path(TF2_WEIGHTS_NAME)
+                flax_weights_path = get_local_path(FLAX_WEIGHTS_NAME)


We should do the same thing for Flax weights (add the suffix) in modeling_flax_utils, I suppose (at least in diffusers).

src/transformers/modeling_utils.py

pcuenca · 2023-01-27T11:42:55Z

src/transformers/modeling_utils.py

+                            filename = WEIGHTS_NAME + variant_suffix
                            resolved_archive_file = cached_file(
-                                pretrained_model_name_or_path, WEIGHTS_NAME, **cached_file_kwargs
+                                pretrained_model_name_or_path, WEIGHTS_NAME + variant_suffix, **cached_file_kwargs


Why not just filename here?

src/transformers/modeling_utils.py

pcuenca · 2023-01-27T11:47:47Z

src/transformers/modeling_utils.py

                        f" same name. Otherwise, make sure '{pretrained_model_name_or_path}' is the correct path to a"
-                        f" directory containing a file named {WEIGHTS_NAME}, {TF2_WEIGHTS_NAME}, {TF_WEIGHTS_NAME} or"
-                        f" {FLAX_WEIGHTS_NAME}."
+                        f" directory containing a file named {WEIGHTS_NAME + variant_suffix}, {TF2_WEIGHTS_NAME},"


Add {WEIGHTS_NAME} too if we accept the suggestion mentioned before.

julien-c

pytorch_model.{variant}.bin sounds better to me, to keep the file-extension (not so important for .bin, but more important for .h5, .safetensors or any other format)

Note that this is different from pytorch_model.bin.index.json (sharding scheme) as that file is only a slice (shard) of a bigger valid file, i.e. is not really valid by itself

Wauplin · 2023-01-27T13:42:07Z

pytorch_model.{variant}.bin sounds better to me, to keep the file-extension (not so important for .bin, but more important for .h5, .safetensors or any other format)

Even for .bin files, I'd say it's good to keep the file extension as it does not break the LFS property for existing .gitattributes files (see huggingface/the-no-branch-repo where bin files are uploaded as regular).

sgugger

Thanks for the PoC! I left a couple of comments.

src/transformers/modeling_utils.py

examples/research_projects/decision_transformer/requirements.txt

src/transformers/modeling_utils.py

Co-authored-by: Lucain <lucainp@gmail.com> Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

patrickvonplaten · 2023-01-31T13:35:13Z

Failing test is unrelated. Think this PR is good for merge.

@Wauplin @julien-c good for you?

The resulting folder structure now looks as described in the PR statement: #21332 (comment)

julien-c

haven't checked the code, but the file structure lgtm!

Wauplin

Looks good to me as well ! :)

LysandreJik

LGTM, thanks for the PR @patrickvonplaten!

patrickvonplaten · 2023-02-01T08:21:49Z

Thanks for the reviews! Merging

NielsRogge · 2023-02-07T13:13:11Z

cc @sgugger would it be possible to add this feature to push_to_hub as well?

I'd like to use it for BLIP-2. For the moment it seems the only way to do this is calling save_pretrained("...", variant="fp16") and then manually upload the PyTorch checkpoint to the model repo

sgugger · 2023-02-07T14:10:36Z

Happy to review a PR.

patrickvonplaten force-pushed the variant_saving_format branch from 62865ff to 818fa15 Compare January 27, 2023 09:20

patrickvonplaten changed the title ~~Add variant to transformers~~ [RFC] Add variant to transformers Jan 27, 2023

patrickvonplaten requested review from LysandreJik, julien-c, osanseviero, patil-suraj, pcuenca and sgugger January 27, 2023 09:33

pcuenca reviewed Jan 27, 2023

View reviewed changes

julien-c reviewed Jan 27, 2023

View reviewed changes

sgugger reviewed Jan 27, 2023

View reviewed changes

patrickvonplaten added 2 commits January 30, 2023 17:27

adapt

d0ab748

finish

d9b31ba

patrickvonplaten force-pushed the variant_saving_format branch from 141cfa2 to d9b31ba Compare January 30, 2023 15:37

patrickvonplaten commented Jan 30, 2023

View reviewed changes

examples/research_projects/decision_transformer/requirements.txt Outdated Show resolved Hide resolved

patrickvonplaten added 2 commits January 30, 2023 16:37

Update examples/research_projects/decision_transformer/requirements.txt

6837e8f

up

e48ca3a

Wauplin reviewed Jan 30, 2023

View reviewed changes

src/transformers/modeling_utils.py Outdated Show resolved Hide resolved

sgugger approved these changes Jan 30, 2023

View reviewed changes

src/transformers/modeling_utils.py Outdated Show resolved Hide resolved

patrickvonplaten and others added 2 commits January 30, 2023 22:00

add tests

a06ecfa

Apply suggestions from code review

284666b

Co-authored-by: Lucain <lucainp@gmail.com> Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

patrickvonplaten changed the title ~~[RFC] Add variant to transformers~~ Add variant to transformers Jan 30, 2023

fix test

d1f4c7d

julien-c reviewed Jan 31, 2023

View reviewed changes

Wauplin reviewed Jan 31, 2023

View reviewed changes

LysandreJik approved these changes Jan 31, 2023

View reviewed changes

patrickvonplaten merged commit 90cddfa into main Feb 1, 2023

patrickvonplaten deleted the variant_saving_format branch February 1, 2023 08:21

patrickvonplaten mentioned this pull request Feb 10, 2023

[Variant] Add "variant" as input kwarg so to have better UX when downloading no_ema or fp16 weights huggingface/diffusers#2305

Merged

3 tasks

pcuenca mentioned this pull request Aug 29, 2023

Add some info on dtypes huggingface/blog#1425

Merged

Add variant to transformers #21332

Add variant to transformers #21332

Uh oh!

Conversation

patrickvonplaten commented Jan 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

HuggingFaceDocBuilderDev commented Jan 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pcuenca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pcuenca Jan 27, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pcuenca Jan 27, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pcuenca Jan 27, 2023

Choose a reason for hiding this comment

Uh oh!

julien-c left a comment

Choose a reason for hiding this comment

Uh oh!

Wauplin commented Jan 27, 2023

Uh oh!

sgugger left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patrickvonplaten commented Jan 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

julien-c left a comment

Choose a reason for hiding this comment

Uh oh!

Wauplin left a comment

Choose a reason for hiding this comment

Uh oh!

LysandreJik left a comment

Choose a reason for hiding this comment

Uh oh!

patrickvonplaten commented Feb 1, 2023

Uh oh!

NielsRogge commented Feb 7, 2023

Uh oh!

sgugger commented Feb 7, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

patrickvonplaten commented Jan 27, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Jan 27, 2023 •

edited

Loading

patrickvonplaten commented Jan 31, 2023 •

edited

Loading