Skip to content

Conversation

@vgrau98
Copy link
Contributor

@vgrau98 vgrau98 commented Apr 27, 2024

Fixes #7227

Description

Integrate the private SABlock from monai/networks/nets/transformer.py to monai/networks/blocks/selfattention.py

Add:

  • causal self attention
  • flash attention

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

Signed-off-by: vgrau98 <victor.grau93@gmail.com>
Signed-off-by: vgrau98 <victor.grau93@gmail.com>
Signed-off-by: vgrau98 <victor.grau93@gmail.com>
@vgrau98 vgrau98 closed this Apr 27, 2024
@vgrau98 vgrau98 deleted the genai-SABlock-integration branch April 27, 2024 11:22
@vgrau98
Copy link
Contributor Author

vgrau98 commented Apr 27, 2024

Duplicate with #7715

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant