Skip to content

[roberta] add sdpa to roberta and xlm-roberta#31754

Closed
kiszk wants to merge 4 commits intohuggingface:mainfrom
kiszk:sdpa_roberta
Closed

[roberta] add sdpa to roberta and xlm-roberta#31754
kiszk wants to merge 4 commits intohuggingface:mainfrom
kiszk:sdpa_roberta

Conversation

@kiszk
Copy link
Copy Markdown
Contributor

@kiszk kiszk commented Jul 2, 2024

What does this PR do?

This PR enables sdpa in RoBERTa and XLM-RoBERTa. Since sdpa is already added to BERT thru #28802, this PR follows that PR.

Tests for RoBERTa are done by the followings at ModelTesterMixin.

  • test_eager_matches_sdpa_generate - test_eager_matches_sdpa_inference_0_float16
  • test_eager_matches_sdpa_inference_0_bfloat16
  • test_eager_matches_sdpa_inference_0_float32

Fixes #31752

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

@ArthurZucker @hackyon

@kiszk kiszk changed the title add sdpa to roberta and xlm-roberta [roberta] add sdpa to roberta and xlm-roberta Jul 2, 2024
@kiszk
Copy link
Copy Markdown
Contributor Author

kiszk commented Jul 10, 2024

I realized this PR has been already submitted to support SDPA for RoBERTa-based models.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Aug 3, 2024

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@github-actions github-actions Bot closed this Aug 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Suport sdpa for RoBERTa and XLM-RoBERTa models

1 participant