NVTX Range Transform

[bhashemian]

Fixes #2754

Description

This PR adds Range transform which accept a transform as an input and pushes an NVTX range before that transform, and pops it afterwards.

Status

Ready

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[Nic-Ma]

Looks good to me.
@wyli what do you think?

Thanks.

[bhashemian]

@wyli, if you don't have any comment on this, I'll merge the PR. Please let me know. Thanks.

[wyli]

thanks @drbeh, I have a quick question, what will happen if we have the same type(transform).__name__ msg for different nvtx transforms?

[bhashemian]

thanks @drbeh, I have a quick question, what will happen if we have the same type(transform).__name__ msg for different nvtx transforms?

If we are profiling the same transform that is used multiple times in the compose, like ToTensor, currently, the only way to distinguish them is to assign different name to it. For instance: Range(ToTensor(), "ToTensor_1") instead of Range(ToTensor()).