Skip to content

[TF Led] Fix flaky TF Led test#9513

Merged
patrickvonplaten merged 1 commit intohuggingface:masterfrom
patrickvonplaten:fix_test_led
Jan 11, 2021
Merged

[TF Led] Fix flaky TF Led test#9513
patrickvonplaten merged 1 commit intohuggingface:masterfrom
patrickvonplaten:fix_test_led

Conversation

@patrickvonplaten
Copy link
Copy Markdown
Contributor

@patrickvonplaten patrickvonplaten commented Jan 11, 2021

What does this PR do?

The reason why the TF LED test is flaky was not fully fixed in: #9459
and is actually the following:

Currently the decoder_attention_mask can have a 0 at its first input:

decoder_attention_mask[:, 0] == 0

Since the decoder uses a causal mask, this however leads to problems as a softmax over only very large negative numbers in computed. Now since TF and PT use slightly different large numbers, we can see significant differences between the models. The solution is to make sure that the decoder_attention_mask used for the tf_pt_equivalence test cannot be zero at the first position (I've done the same changes for all TFBart models in: #9497 and also made sure in #9497 that the TF templates are correctly updated )

@patrickvonplaten
Copy link
Copy Markdown
Contributor Author

cc @LysandreJik @sgugger @jplu

@patrickvonplaten patrickvonplaten merged commit 6c8ec2a into huggingface:master Jan 11, 2021
@sgugger
Copy link
Copy Markdown
Collaborator

sgugger commented Jan 11, 2021

Thanks for fixing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants