add multi-modal (vision + language) transformers #2936

ahatamiz · 2021-09-13T10:11:44Z

Signed-off-by: ahatamizadeh ahatamizadeh@nvidia.com

Description

This pull request adds the full pipeline and support for multimodal (vision + language ) transformers. The transformers implementation follow Huggingface repository.

Status

Ready

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

ahatamiz · 2021-09-13T10:19:04Z

/black

wyli

thanks, please some initial comments inline

monai/networks/nets/vltransformer.py

wyli · 2021-09-13T11:26:42Z

monai/networks/nets/vltransformer.py

+            num_mixed_layers: number of mixed transformer layers.
+        """
+        super().__init__()
+        bert_config = type(


please make it a module-level variable `, like

MONAI/monai/networks/nets/efficientnet.py

Line 28 in c50235c

efficientnet_params = {

One issue here is that module-level variable does not work properly when trying to initialize model weights from bert checkpoint in huggingface library.

how to replicate this issue, could you please elaborate?

or you can make the dictionary (in line 232-255) a global variable as the default,

def __init__( self, num_language_layers: int = 2, num_vision_layers: int = 2, num_mixed_layers: int = 2, bert_config: Dict = BERT_CONFIG, )

it'll leave some flexibility for tuning the parameters for the end users.

One way is to simply change the object-based definition to a dictionary-based config for BERT model, but class BertPreTrainedModel is not compatible with this. I am using this class directly from Huggingface with very minor modifications.

For your suggestion to give more flexibility, I think we need to keep the original BETR configurations to be able to use the pre-trained BERT weights (at least for this release). Pre-training a BERT type model on text data from scratch could be costly. Any suggestions ?

are you sure that we need to keep exactly this version of config in order to load the pretrained?

Basically, I first tried with different configurations. But the model will not be initialized from the BERT pre-trained checkpoint,so takes longer for training.

my concern is mainly about the hard-coded config. for example, as a user if I want to set 'attention_probs_dropout_prob': 0.2 and "transformers_version": "4.10.1" for fine-tuning of the pretrained model, how can I set this MultiModal class without rewriting everything from scratch?

setup.cfg

ahatamiz · 2021-09-13T14:34:19Z

@wyli it seems like some check fails due to the following which is not related to multi-modality ?

/home/runner/work/MONAI/MONAI/monai/data/grid_dataset.py:144:13: B904 Within an `except` clause, raise exceptions with `raise ... from err` or `raise ... from None` to distinguish them from errors in exception handling.  See https://docs.python.org/3/tutorial/errors.html#exception-chaining for details.
/home/runner/work/MONAI/MONAI/monai/data/utils.py:286:9: B904 Within an `except` clause, raise exceptions with `raise ... from err` or `raise ... from None` to distinguish them from errors in exception handling.  See https://docs.python.org/3/tutorial/errors.html#exception-chaining for details.
/home/runner/work/MONAI/MONAI/monai/data/utils.py:297:9: B904 Within an `except` clause, raise exceptions with `raise ... from err` or `raise ... from None` to distinguish them from errors in exception handling.  See https://docs.python.org/3/tutorial/errors.html#exception-chaining for details.
/home/runner/work/MONAI/MONAI/monai/utils/aliases.py:74:13: B904 Within an `except` clause, raise exceptions with `raise ... from err` or `raise ... from None` to distinguish them from errors in exception handling.  See https://docs.python.org/3/tutorial/errors.html#exception-chaining for details.
/home/runner/work/MONAI/MONAI/monai/transforms/inverse_batch_transform.py:102:13: B904 Within an `except` clause, raise exceptions with `raise ... from err` or `raise ... from None` to distinguish them from errors in exception handling.  See https://docs.python.org/3/tutorial/errors.html#exception-chaining for details.
/home/runner/work/MONAI/MONAI/monai/_extensions/loader.py:37:9: B904 Within an `except` clause, raise exceptions with `raise ... from err` or `raise ... from None` to distinguish them from errors in exception handling.  See https://docs.python.org/3/tutorial/errors.html#exception-chaining for details.
/home/runner/work/MONAI/MONAI/tests/utils.py:77:9: B904 Within an `except` clause, raise exceptions with `raise ... from err` or `raise ... from None` to distinguish them from errors in exception handling.  See https://docs.python.org/3/tutorial/errors.html#exception-chaining for details.
7     B904 Within an `except` clause, raise exceptions with `raise ... from err` or `raise ... from None` to distinguish them from errors in exception handling.  See https://docs.python.org/3/tutorial/errors.html#exception-chaining for details.

wyli · 2021-09-13T14:36:36Z

yes, I'm merging a fix for that flake8 issue very soon #2940

ahatamiz · 2021-09-13T14:38:20Z

Hi @wyli

Another issue that comes up is with the optional import ( or just import). For some reason transformers is not being installed and cannot be imported:

monai.utils.module.OptionalImportError: from transformers.models.bert.modeling_bert import BertEmbeddings (No module named 'transformers').

Thanks

ahatamiz · 2021-09-13T14:39:34Z

yes, I'm merging a fix for that flake8 issue very soon #2940

Thanks much.

ahatamiz · 2021-09-13T14:40:47Z

/black

wyli · 2021-09-13T15:07:40Z

BertEmbeddings

how to replicate this issue?

I tried and the following works

Python 3.7.10 (default, Feb 26 2021, 10:16:00) 
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from monai.utils import optional_import
>>> bert_embeddings = optional_import("transformers.models.bert.modeling_bert", name="BertEmbeddings")[0]
>>> bert_embeddings
<class 'transformers.models.bert.modeling_bert.BertEmbeddings'>
>>>

wyli · 2021-09-13T15:23:15Z

/build

ahatamiz · 2021-09-13T15:24:28Z

BertEmbeddings

how to replicate this issue?

I tried and the following works

Python 3.7.10 (default, Feb 26 2021, 10:16:00) 
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from monai.utils import optional_import
>>> bert_embeddings = optional_import("transformers.models.bert.modeling_bert", name="BertEmbeddings")[0]
>>> bert_embeddings
<class 'transformers.models.bert.modeling_bert.BertEmbeddings'>
>>>

Thanks @wyli. I pushed a new PR based on this for optional imports.

ahatamiz · 2021-09-13T15:34:53Z

BertEmbeddings

how to replicate this issue?
I tried and the following works

Python 3.7.10 (default, Feb 26 2021, 10:16:00) 
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from monai.utils import optional_import
>>> bert_embeddings = optional_import("transformers.models.bert.modeling_bert", name="BertEmbeddings")[0]
>>> bert_embeddings
<class 'transformers.models.bert.modeling_bert.BertEmbeddings'>
>>>

Thanks @wyli. I pushed a new PR based on this for optional imports.

Hi @wyli , seems like this did not resolve the issue as some of the checks failed.

wyli · 2021-09-13T15:46:31Z

BertEmbeddings

how to replicate this issue?
I tried and the following works
Python 3.7.10 (default, Feb 26 2021, 10:16:00) 
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from monai.utils import optional_import
>>> bert_embeddings = optional_import("transformers.models.bert.modeling_bert", name="BertEmbeddings")[0]
>>> bert_embeddings
<class 'transformers.models.bert.modeling_bert.BertEmbeddings'>
>>> 
Thanks @wyli. I pushed a new PR based on this for optional imports.
Hi @wyli , seems like this did not resolve the issue as some of the checks failed.

Please add the test to this list to exclude it for minimum environment tests

MONAI/tests/min_tests.py

Line 138 in 76dc5c2

"test_vit",

ahatamiz · 2021-09-13T15:59:57Z

BertEmbeddings

how to replicate this issue?
I tried and the following works
Python 3.7.10 (default, Feb 26 2021, 10:16:00) 
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from monai.utils import optional_import
>>> bert_embeddings = optional_import("transformers.models.bert.modeling_bert", name="BertEmbeddings")[0]
>>> bert_embeddings
<class 'transformers.models.bert.modeling_bert.BertEmbeddings'>
>>> 
Thanks @wyli. I pushed a new PR based on this for optional imports.
Hi @wyli , seems like this did not resolve the issue as some of the checks failed.
Please add the test to this list to exclude it for minimum environment tests

MONAI/tests/min_tests.py

Line 138 in 76dc5c2

"test_vit",

Sure.

ahatamiz · 2021-09-13T16:07:23Z

/black

ahatamiz · 2021-09-13T16:27:41Z

/black

ahatamiz · 2021-09-13T16:28:13Z

/build

ahatamiz · 2021-09-13T16:51:02Z

/black

ahatamiz · 2021-09-13T16:51:12Z

/build

ahatamiz · 2021-09-13T17:15:11Z

/black

ahatamiz · 2021-09-13T17:15:18Z

/build

ahatamiz · 2021-09-13T17:36:06Z

/black

Signed-off-by: Nic Ma <nma@nvidia.com> Co-authored-by: Wenqi Li <wenqil@nvidia.com> Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

* [DLMED] enhance the pad mode Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update all the tensor pad related Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix error tests Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] fix GPU tests Signed-off-by: Nic Ma <nma@nvidia.com> * [DLMED] update according to comments Signed-off-by: Nic Ma <nma@nvidia.com> Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

ahatamiz · 2021-09-13T18:58:52Z

/black

ahatamiz · 2021-09-13T18:59:00Z

/build

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

ahatamiz · 2021-09-15T22:10:02Z

I close this pull request due to many conflicts between the two branches. Will push a new one.

wyli · 2021-09-15T22:20:30Z

Hi @ahatamiz, why the close? the PR looks nice, sorry for the confusion that my previous comments might have created

wyli · 2021-09-15T22:22:37Z

the latest update with the flexible bert_config looks good to me.

ahatamiz · 2021-09-16T05:42:39Z

Hi @wyli. Sorry about this. dealing with so many conflicts in this PR was a bit time-consuming. I had to submit another one.

ahatamiz requested review from Nic-Ma and wyli September 13, 2021 10:12

ahatamiz self-assigned this Sep 13, 2021

ahatamiz added the Feature request label Sep 13, 2021

ahatamiz added this to the Multi Modality Support milestone Sep 13, 2021

This was linked to issues Sep 13, 2021

Add transformer-based NLP pipelines #2776

Closed

Add transformer-based multi-modality pipelines #2777

Closed

wyli reviewed Sep 13, 2021

View reviewed changes

ahatamiz linked an issue Sep 13, 2021 that may be closed by this pull request

Add BERT-like Transformers #2774

Closed

ahatamiz removed a link to an issue Sep 13, 2021

Add BERT-like Transformers #2774

Closed

Nic-Ma and others added 16 commits September 13, 2021 11:57

[DLMED] add dict version shuffle (#2918)

75c6402

Signed-off-by: Nic Ma <nma@nvidia.com> Co-authored-by: Wenqi Li <wenqil@nvidia.com> Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

Add device to ToTensor (#2926)

176a0fe

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

5627f23

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

c9c632a

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

2cd138e

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

4b5bdd0

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

9662e81

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

f2f4f91

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

97c518c

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

6fb363a

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

ea89c1f

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

51abbf1

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

f3b7c5b

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

19f0e98

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

8829c95

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

ahatamiz added 3 commits September 15, 2021 13:55

add multimodal transformers

7be790d

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

add multimodal transformers

7187c69

Signed-off-by: ahatamizadeh <ahatamizadeh@nvidia.com>

Resolved merge conflict by incorporating both suggestions.

939a2b7

ahatamiz closed this Sep 15, 2021

ahatamiz deleted the 2775_multimodality_v1 branch September 15, 2021 22:10

This was unlinked from issues Sep 16, 2021

Add transformer-based NLP pipelines #2776

Closed

Add transformer-based multi-modality pipelines #2777

Closed

ahatamiz removed this from the Multi Modality Support milestone Sep 16, 2021

add multi-modal (vision + language) transformers #2936

add multi-modal (vision + language) transformers #2936

Uh oh!

Conversation

ahatamiz commented Sep 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Status

Types of changes

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

wyli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wyli Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

ahatamiz Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

wyli Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

wyli Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

ahatamiz Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

wyli Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

ahatamiz Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

wyli Sep 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

wyli commented Sep 13, 2021

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

wyli commented Sep 13, 2021

Uh oh!

wyli commented Sep 13, 2021

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

wyli commented Sep 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

ahatamiz commented Sep 13, 2021

Uh oh!

ahatamiz commented Sep 13, 2021

ahatamiz commented Sep 13, 2021 •

edited

Loading

wyli Sep 13, 2021 •

edited

Loading

wyli commented Sep 13, 2021 •

edited

Loading

ahatamiz commented Sep 16, 2021 •

edited

Loading