add multi-modal (vision + language) transformers

[ahatamiz]

Fixes #2775
Fixes #2776
Fixes #2777 .

Status

This pull request adds the full pipeline and support for multimodal (vision + language ) transformers. The transformers implementation follow Huggingface repository.

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[ahatamiz]

/build

[ahatamiz]

/black

[ahatamiz]

/build

[ahatamiz]

/black

[ahatamiz]

/build

[ahatamiz]

/black

[ahatamiz]

/build

[ahatamiz]

/black

[ahatamiz]

Hi @wyli , this PR allows for highly flexible specification of the model for BERT config. Users can simply specify them without any restrictions.

It has passed all the tests. Let me know if anything else is needed, otherwise should be good for merging.

Thanks

[wyli]

Hi @wyli , this PR allows for highly flexible specification of the model for BERT config. Users can simply specify them without any restrictions.

It has passed all the tests. Let me know if anything else is needed, otherwise should be good for merging.

Thanks

thanks, it looks good to me. please remove the changes in monai/_version.py, they are probably automatically generated. (or I can do it later today)

[ahatamiz]

Hi @wyli , this PR allows for highly flexible specification of the model for BERT config. Users can simply specify them without any restrictions.
It has passed all the tests. Let me know if anything else is needed, otherwise should be good for merging.
Thanks

thanks, it looks good to me. please remove the changes in monai/_version.py, they are probably automatically generated. (or I can do it later today)

Thanks @wyli. Yes I think there were automatically generated. They maybe triggered to be generated again in future PRs.

[ahatamiz]

/build

[ahatamiz]

/black

[ahatamiz]

Hi @wyli , this PR allows for highly flexible specification of the model for BERT config. Users can simply specify them without any restrictions.
It has passed all the tests. Let me know if anything else is needed, otherwise should be good for merging.
Thanks

thanks, it looks good to me. please remove the changes in monai/_version.py, they are probably automatically generated. (or I can do it later today)

Thanks @wyli. Yes I think there were automatically generated. They maybe triggered to be generated again in future PRs.

Just submitted another PR to address this.

[wyli]

I've added some basic docs, still this module needs better paper references and general info. this could be done in some follow-up PRs.

[wyli]

/build

[wyli]

looks good to me, it requires follow-up documentation/tutorial to show the usages