Fixed quantization error in modules that use register_buffer by DonggeunYu · Pull Request #29203 · huggingface/transformers

DonggeunYu · 2024-02-22T06:17:53Z

What does this PR do?

Fixed quantization error in modules that use register_buffer.
When using self.register_buffer, it is included in state_dict, but not in _parameters, which is why first validate that tensor_name exists in _parameters before searching for it by tensor_name in _parameters.

Fixes #29201

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

…tr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

younesbelkada

Great catch ! can you confirm the 4bit tests pass for you ?

RUN_SLOW=1 pytest tests/quantization/bnb/test_4bit.py

If they pass I think we can safely merge the PR, depending on your hardware you might get some failure with respect to generated texts, for that you can simply check if you get the same failures between main branch and yours

DonggeunYu · 2024-02-22T07:09:19Z

@younesbelkada

RUN_SLOW=1 pytest tests/quantization/bnb/test_4bit.py on my branch output:

=================================== short test summary info ===================================
FAILED tests/quantization/bnb/test_4bit.py::BaseSerializationTest::test_serialization - ValueError: Supplied state dict for model.decoder.layers.0.fc1.weight does not contain `bi...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_fp4_double_safe - ValueError: Supplied state dict for model.decoder.layers.0.fc1.weight does not contain `bi...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_fp4_double_unsafe - ValueError: Supplied state dict for model.decoder.layers.0.self_attn.k_proj.weight does no...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_fp4_single_safe - ValueError: Supplied state dict for model.decoder.layers.0.fc1.weight does not contain `bi...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_fp4_single_unsafe - ValueError: Supplied state dict for model.decoder.layers.0.self_attn.k_proj.weight does no...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_nf4_double_unsafe - ValueError: Supplied state dict for model.decoder.layers.0.self_attn.k_proj.weight does no...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_nf4_single_safe - ValueError: Supplied state dict for model.decoder.layers.0.fc1.weight does not contain `bi...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_nf4_single_unsafe - ValueError: Supplied state dict for model.decoder.layers.0.self_attn.k_proj.weight does no...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_serialization - ValueError: Supplied state dict for model.decoder.layers.0.fc1.weight does not contain `bi...
FAILED tests/quantization/bnb/test_4bit.py::BloomSerializationTest::test_serialization - ValueError: Supplied state dict for transformer.h.0.mlp.dense_4h_to_h.weight does not cont...
FAILED tests/quantization/bnb/test_4bit.py::GPTSerializationTest::test_serialization - ValueError: Supplied state dict for transformer.h.0.attn.c_attn.weight does not contain `b...
============= 11 failed, 27 passed, 1 skipped, 29 warnings in 1628.99s (0:27:08) ==============

Main branch output:

=================================== short test summary info ===================================
FAILED tests/quantization/bnb/test_4bit.py::BaseSerializationTest::test_serialization - ValueError: Supplied state dict for model.decoder.layers.0.fc1.weight does not contain `bi...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_fp4_double_safe - ValueError: Supplied state dict for model.decoder.layers.0.fc1.weight does not contain `bi...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_fp4_double_unsafe - ValueError: Supplied state dict for model.decoder.layers.0.self_attn.k_proj.weight does no...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_fp4_single_safe - ValueError: Supplied state dict for model.decoder.layers.0.fc1.weight does not contain `bi...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_fp4_single_unsafe - ValueError: Supplied state dict for model.decoder.layers.0.self_attn.k_proj.weight does no...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_nf4_double_unsafe - ValueError: Supplied state dict for model.decoder.layers.0.self_attn.k_proj.weight does no...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_nf4_single_safe - ValueError: Supplied state dict for model.decoder.layers.0.fc1.weight does not contain `bi...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_nf4_single_unsafe - ValueError: Supplied state dict for model.decoder.layers.0.self_attn.k_proj.weight does no...
FAILED tests/quantization/bnb/test_4bit.py::ExtendedSerializationTest::test_serialization - ValueError: Supplied state dict for model.decoder.layers.0.fc1.weight does not contain `bi...
FAILED tests/quantization/bnb/test_4bit.py::BloomSerializationTest::test_serialization - ValueError: Supplied state dict for transformer.h.0.mlp.dense_4h_to_h.weight does not cont...
FAILED tests/quantization/bnb/test_4bit.py::GPTSerializationTest::test_serialization - ValueError: Supplied state dict for transformer.h.0.attn.c_attn.weight does not contain `b...
============= 11 failed, 27 passed, 1 skipped, 29 warnings in 1102.77s (0:18:22) ==============

SunMarc

LGTM !

younesbelkada

Thanks again @DonggeunYu !
Just to confirm one last point, could your re-run the bnb tests with bitsandbytes==0.42.0 ?

HuggingFaceDocBuilderDev · 2024-02-23T04:16:44Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

DonggeunYu · 2024-02-23T04:43:01Z

Thanks again @DonggeunYu ! Just to confirm one last point, could your re-run the bnb tests with bitsandbytes==0.42.0 ?

@younesbelkada
But, the same 11 failed...

DonggeunYu · 2024-03-04T14:02:31Z

🥲
#29329

amyeroberts · 2024-03-07T17:38:36Z

@DonggeunYu Am I correct in understanding this PR was superceded by #29329? If so, apologies for the delay in reviewing this, and we should close this PR. Please don't hesitate to ping me on a PR if there's a delay in response - thee's a lot of notifications and sometimes they slip through the cracks.

DonggeunYu · 2024-03-08T07:04:17Z

@DonggeunYu Am I correct in understanding this PR was superceded by #29329? If so, apologies for the delay in reviewing this, and we should close this PR. Please don't hesitate to ping me on a PR if there's a delay in response - thee's a lot of notifications and sometimes they slip through the cracks.

Yes, that's right. I understand. Thank you 😊

DonggeunYu and others added 14 commits February 14, 2024 15:41

Update ms_deform_attn_cuda.cu

95ff15b

Update ms_deform_attn_cuda.cuh

10d0337

Update modeling_deformable_detr.py

84eaeef

Update src/transformers/models/deformable_detr/modeling_deformable_de…

7773d5c

…tr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

Update modeling_deformable_detr.py

9b3789c

python utils/check_copies.py --fix_and_overwrite

ad0e298

Fix dtype missmatch error

743b518

Update test_modeling_deformable_detr.py

6a7beb9

Update test_modeling_deformable_detr.py

350cf84

Update modeling_deformable_detr.py

4379ec7

Update modeling_deformable_detr.py

2657833

Merge branch 'huggingface:main' into main

ff794c2

Update quantizer_bnb_4bit.py

619cf83

Update quantizer_bnb_8bit.py

fb81208

younesbelkada reviewed Feb 22, 2024

View reviewed changes

SunMarc self-requested a review February 22, 2024 16:53

SunMarc approved these changes Feb 22, 2024

View reviewed changes

younesbelkada approved these changes Feb 23, 2024

View reviewed changes

younesbelkada requested a review from amyeroberts February 23, 2024 03:59

DonggeunYu closed this Mar 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed quantization error in modules that use register_buffer#29203

Fixed quantization error in modules that use register_buffer#29203
DonggeunYu wants to merge 14 commits intohuggingface:mainfrom
DonggeunYu:main

DonggeunYu commented Feb 22, 2024

Uh oh!

younesbelkada left a comment

Uh oh!

DonggeunYu commented Feb 22, 2024 •

edited

Loading

Uh oh!

SunMarc left a comment

Uh oh!

younesbelkada left a comment

Uh oh!

HuggingFaceDocBuilderDev commented Feb 23, 2024

Uh oh!

DonggeunYu commented Feb 23, 2024 •

edited

Loading

Uh oh!

DonggeunYu commented Mar 4, 2024

Uh oh!

amyeroberts commented Mar 7, 2024

Uh oh!

DonggeunYu commented Mar 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

DonggeunYu commented Feb 22, 2024

What does this PR do?

Before submitting

Who can review?

Uh oh!

younesbelkada left a comment

Choose a reason for hiding this comment

Uh oh!

DonggeunYu commented Feb 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SunMarc left a comment

Choose a reason for hiding this comment

Uh oh!

younesbelkada left a comment

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Feb 23, 2024

Uh oh!

DonggeunYu commented Feb 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DonggeunYu commented Mar 4, 2024

Uh oh!

amyeroberts commented Mar 7, 2024

Uh oh!

DonggeunYu commented Mar 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

DonggeunYu commented Feb 22, 2024 •

edited

Loading

DonggeunYu commented Feb 23, 2024 •

edited

Loading