Skip to content

Add Wav2Vec2 Adapter Weights to Flax#15521

Closed
sanchit-gandhi wants to merge 34 commits intohuggingface:masterfrom
sanchit-gandhi:flax-wav2vec2
Closed

Add Wav2Vec2 Adapter Weights to Flax#15521
sanchit-gandhi wants to merge 34 commits intohuggingface:masterfrom
sanchit-gandhi:flax-wav2vec2

Conversation

@sanchit-gandhi
Copy link
Copy Markdown
Contributor

@sanchit-gandhi sanchit-gandhi commented Feb 4, 2022

What does this PR do?

Fixes #15476

  • Adds an adapter to the Flax Wav2Vec2 model to reduce the time dimension of the extracted feature vectors beyond that of the standard Wav2Vec2 model. The encoder's output hidden states thus have a time context window that is more similar to that of a subword token instead of just a character.
  • Shape and values of Flax output logits match those of the PyTorch model.
  • Flax model uses all PyTorch model weights, including those of the adapter. Running the script in Add Adapter Weighs to Flax #15476 resolved to yield identical results (within 4e-2 threshold).

@HuggingFaceDocBuilder
Copy link
Copy Markdown

HuggingFaceDocBuilder commented Feb 4, 2022 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

@sanchit-gandhi sanchit-gandhi changed the title Flax wav2vec2 Add Wav2Vec2 Adapter Weights to Flax Feb 4, 2022
stas00 and others added 3 commits February 4, 2022 11:15
@stas00
[deepspeed docs] Megatron-Deepspeed info (#15488)
31be2f4
@sgugger @NielsRogge @LysandreJik
Standardize semantic segmentation models outputs (#15469)
ac6aa10 
* Standardize instance segmentation models outputs

* Rename output

* Update src/transformers/modeling_outputs.py

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Add legacy argument to the config and model forward

* Update src/transformers/models/beit/modeling_beit.py

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

* Copy fix in Segformer

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
@stas00 @sgugger
[deepspeed docs] DeepSpeed ZeRO Inference (#15486)
8ce1330 
* [deepspeed docs] DeepSpeed ZeRO Inference

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* tweak

* deal with black

* extra cleanup, better comments

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
patrickvonplaten
patrickvonplaten reviewed Feb 7, 2022
View reviewed changes
Comment thread src/transformers/modeling_flax_pytorch_utils.py Outdated
patrickvonplaten
patrickvonplaten reviewed Feb 7, 2022
View reviewed changes
Comment thread src/transformers/models/wav2vec2/modeling_flax_wav2vec2.py Outdated
patrickvonplaten
patrickvonplaten reviewed Feb 7, 2022
View reviewed changes
Comment thread src/transformers/models/wav2vec2/modeling_flax_wav2vec2.py Outdated
patrickvonplaten
patrickvonplaten reviewed Feb 7, 2022
View reviewed changes
Comment thread src/transformers/models/wav2vec2/modeling_flax_wav2vec2.py Outdated
patrickvonplaten
patrickvonplaten reviewed Feb 7, 2022
View reviewed changes
Comment thread src/transformers/models/wav2vec2/modeling_flax_wav2vec2.py
patrickvonplaten
patrickvonplaten reviewed Feb 7, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@patrickvonplaten patrickvonplaten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool PR! Looks more or less good to me. Left some final comments and then I think we can merge :-)

patrickvonplaten and others added 9 commits February 7, 2022 15:35
@patrickvonplaten
[ASR pipeline] correct asr pipeline for seq2seq models (#15541)
5f1918a
@patrickvonplaten
[torch_int_div] Correct true division in generation (#15498)
c47d259 
* [torch_int_div] Correct true division in generation

* up

* up
@NielsRogge @sgugger
Add ConvNeXT (#15277)
84eec9e 
* First draft

* Add conversion script

* Improve conversion script

* Improve docs and implement tests

* Define model output class

* Fix tests

* Fix more tests

* Add model to README

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Apply more suggestions from code review

* Apply suggestions from code review

* Rename dims to hidden_sizes

* Fix equivalence test

* Rename gamma to gamma_parameter

* Clean up conversion script

* Add ConvNextFeatureExtractor

* Add corresponding tests

* Implement feature extractor correctly

* Make implementation cleaner

* Add ConvNextStem class

* Improve design

* Update design to also include encoder

* Fix gamma parameter

* Use sample docstrings

* Finish conversion, add center cropping

* Replace nielsr by facebook, make feature extractor tests smaller

* Fix integration test

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
@anton-l @sgugger
[Trainer] Deeper length checks for IterableDatasetShard (#15539)
75b13f8 
* Unused import

* Make `has_length()` torch-independent to use in callbacks

* Update src/transformers/trainer_utils.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
@anton-l @patrickvonplaten
Add ASR CTC streaming example (#15309)
a459f7f 
* Single-epoch run

* Apply suggestions from code review

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Infinite dataset

* Trainer fix + distributed benchmark

* Benchmark fix

* unused import

* interleaved splits

* interleaved splits

* has_length util

* Move to research projects

* Leftover Sized checks

* Bump min version

* Unused import

* Revert trainer changes

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
@FremyCompany @patrickvonplaten
Wav2Vec2 models must either throw or deal with add_apater (#15409)
7a1412e 
* Wav2Vec2 models must either throw or deal with add_apater

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Add pre-add_adapter backwards compatibility

* Add pre-add_adapter backwards compatibility

* Fix issue in tests/test_modeling_wav2vec2.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
@sanchit-gandhi
Revert "Handle PyTorch to Flax conversion of 1D convolutions"
8255163 
This reverts commit 48d8aef.
@lewtun
Remove Longformers from ONNX-supported models (#15273)
6775b21
@ydshieh
Fix TF T5/LED missing cross attn in retrun values (#15511)
131e258 
* add cross attn to outputs

* add cross attn to outputs for TFLED

* add undo padding

* remove unused import

* fix style

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
@sanchit-gandhi sanchit-gandhi deleted the flax-wav2vec2 branch February 7, 2022 16:49
@sanchit-gandhi sanchit-gandhi restored the flax-wav2vec2 branch February 7, 2022 16:50
@sanchit-gandhi sanchit-gandhi reopened this Feb 7, 2022
sanchit-gandhi and others added 17 commits February 7, 2022 18:07
@sanchit-gandhi @patrickvonplaten
Add adapter arg
6900a67 
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
@ydshieh
Make TF Wav2Vec2 outputs the same as PT's version (#15530)
ad1d3c4 
* fix outputs

* fix for CTC

* fix doc

* make style

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
@sanchit-gandhi @patrickvonplaten
Remove layer drop
c460cfe 
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
@sanchit-gandhi @patrickvonplaten
Remove layer drop
1e34903 
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
@sanchit-gandhi @patrickvonplaten
Remove layer drop
521bd9d 
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
@stevhliu
Create a custom model guide (#15489)
552f8d3 
* 📝 add config section

* 📝 finish first draft

* 📝 add feature extractor and processor

* 🖍 apply feedback from review

* 📝 minor edits

* last review
@michaelbenayoun
FX tracing improvement (#14321)
0fe17f3 
* Change the way tracing happens, enabling dynamic axes out of the box

* Update the tests and modeling xlnet

* Add the non recoding of leaf modules to avoid recording more values for the methods to record than what will be seen at tracing time (which would otherwise desynchronize the recorded values and the values that need to be given to the proxies during tracing, causing errors).

* Comments and making tracing work for gpt-j and xlnet

* Refactore things related to num_choices (and batch_size, sequence_length)

* Update fx to work on PyTorch 1.10

* Postpone autowrap_function feature usage for later

* Add copyrights

* Remove unnecessary file

* Fix issue with add_new_model_like

* Apply suggestions
@aaron-dunamu @lewtun
electra is added to onnx supported model (#15084)
87d08af 
* electra is added to onnx supported model

* add google/electra-base-generator for test onnx module

Co-authored-by: Lewis Tunstall <lewis.c.tunstall@gmail.com>
@patil-suraj
[GPTJ] fix docs (#15558)
0acd84f
@ydshieh
Force use_cache to be False in PyTorch (#15385)
6a5472a 
* use_cache = False for PT models if labels is passed

* Fix for BigBirdPegasusForConditionalGeneration

* add warning if users specify use_cache=True

* Use logger.warning instead of warnings.warn

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
@sanchit-gandhi
Add Wav2Vec2 Adapter Weights to Flax
34da433
@sanchit-gandhi
Comment update for 1 and 2D convolutions
1b5cb64
@sanchit-gandhi
Remove layer drop
b131023
@sanchit-gandhi
Correct Flax int div implementation
6c3be19
@sanchit-gandhi
Flax int div
05f64ec
@sanchit-gandhi
Correct Flax int div
269f02a
@sanchit-gandhi
flake reformat
c5fd053
@patrickvonplaten
Copy link
Copy Markdown
Contributor

patrickvonplaten commented Feb 8, 2022

I think the commit history is messed up here - the best is usually to just reopen a new PR and to just extract your changes from this PR.

@sanchit-gandhi sanchit-gandhi deleted the flax-wav2vec2 branch February 8, 2022 17:23
@patrickvonplaten patrickvonplaten mentioned this pull request Feb 9, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@patrickvonplaten patrickvonplaten patrickvonplaten left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add Adapter Weighs to Flax