transform documentation enhancements

[Spenhouet]

Is your feature request related to a problem? Please describe.

Until now we are using MONAI for our training pipeline and torchio for data augmentations (offline preprocessed).
The data augmentations torchio provides are very medically relevant.
Their documentation, naming of the augmentations and the visual examples are very helpful.

Original	Random blur
Random flip	Random noise
Random affine transformation	Random elastic transformation
Random bias field artifact	Random motion artifact
Random spike artifact	Random ghosting artifact
Random Anisotropy	Random Swap

We struggled finding equivalent methods in MONAI and even when we did, we were not certain that these are actually the same methods.
We therefore never bothered using MONAI for data augmentations.
We would like to replace torchio with MONAI to further reduce our dependencies.

Describe the solution you'd like

1. We wish that MONAI would provide the same transformations as torchio does.
2. We also wish that MONAI provides a equally good documentation and uses medical terms for naming for these transforms.
3. MONAI should add visual examples for these transforms (in the form of GIFs if possible)

While this issue is meant as a feature request, I see that it has a wide scope. It is also a question in that I might just not see the equivalents.
Let's see were MONAI currently is. I will to try find the equivalent transforms of both packages.

torchio	MONAI
RandomAnisotropy	Downsampling + Upsampling with Spacing
RandomBlur	RandGaussianSmooth
RandomFlip	RandFlip
RandomNoise	RandGaussianNoise
RandomAffine	RandAffine
RandomElasticDeformation	Rand3DElastic
RandomBiasField	RandBiasField
RandomMotion	?
RandomSpike	RandKSpaceSpikeNoise
RandomGhosting	?
RandomGamma	RandAdjustContrast
RandomSwap	?

Having visual examples in the documentation of the transforms would really help.

Other open issues on data augmentations I could find:

• Monai equivalent to tio.OneOf transform? (ETA 8/13) #1847
• GridDistortion data augmentation #2186
• Augmentation Transform | InCut & OutCut Dropping (ETA 08/27/21) #2707
• Augmentation Transform | Patch Shuffling (ETA 8/27) #2701
• histogram equalisation transform? #2691
• RandFlip operates on all axes or none #2245
• random plane sampling transform #1993

I would like to collect all task which are needed for MONAI to match torchio on its documentation and transforms.

The following tasks will be necessary to match torchio:

• Implement RandAnisotropy similar to RandomAnisotropy. This should be implemented with a sequential down and up sampling via Spacing.
• Implement a RandMotion similar to RandomMotion. See also MRI k-Space Motion Artefact Augmentation: Model Robustness and Task-Specific Uncertainty.
• Implement a RandPatchSwap similar to RandomSwap. Coordinate with Augmentation Transform | Patch Shuffling (ETA 8/27) #2701 where a RandPatchShuffle is proposed.
• Implement a RandGhosting similar to . See also Ghost Artifacts - Why do motion artifacts often form into discrete ghosts?
• Add example images to documentation of transforms (especially the augmentation transforms). If possible create GIFs like it is done in torchio (see above).

[Nic-Ma]

Hi @Spenhouet ,

Thanks for your great feedback!
I think that are very good points and welcome contributions.
Maybe we can work together to achieve them.

Thanks.

[Spenhouet]

Hi @Nic-Ma,

this is definitely not something I will have time for any time soon but I did hope I could get this going.
The first thing needed would be a precise list of steps / small TODOs so that the community can open up pull request for individual tasks.

From the table I included, is there something where I missed the MONAI equivalent? Is the monai.RandKSpaceSpikeNoise the same as the torchio.RandomSpike?

What would be preferred here:

1. Have this issue as a big issue which will track all tasks and pull request?
2. or have this issue to identify all tasks and then create individual issues for every task?

[Nic-Ma]

Hi @Spenhouet ,

1. I think RandKSpaceSpikeNoise is the correct mapping.
2. RandomBlur may map to MONAI RandGaussianSmooth transform.
3. RandomAnisotropy may be achieved by MONAI Spacing or Resize transforms.
4. RandomMotion may map to MONAI RandBiasField transform? I didn't check the details carefully.
5. RandomGamma maps to MONAI RandAdjustContrast transform.
6. RandomSwap is an interesting feature, I think @finalelement is working on it in ticket: Augmentation Transform | Patch Shuffling (ETA 8/27) #2701.

So only RandomGhosting of your list is not existing.

I think you can make a clear list in this ticket and then submit sub-tickets to track every separate concrete task?
Every PR can close a sub-ticket in theory. @wyli @ericspod @rijobro do you guys have other opinions?

Thanks in advance.

[Spenhouet]

Hi @Nic-Ma,

• We should consider renaming RandGaussianSmooth to RandGaussianBlur since that seems much more common. https://en.wikipedia.org/wiki/Gaussian_blur
• Yes, RandomAnisotropy is a downsampling + upsampling with Spacing. Nonetheless, we should implement it as a method since it fulfills a very important purpose - simulating lower resolution input during training. This helps with 2D vs 3D MR acquisition.
• I don't see how RandBiasField relates to RandomMotion. These are very different operations. So I assume a RandomMotion is also missing. See MRI k-Space Motion Artefact Augmentation: Model Robustness and Task-Specific Uncertainty
• RandomSwap is different to what is proposed in Augmentation Transform | Patch Shuffling (ETA 8/27) #2701. One is about taking blocks and putting them at different locations and one is about mixing pixels within random blocks.

I'm not sure how sub tickets in GitHub work but I will create and maintain a task list in my original post. In the comments we can discuss necessary and potential steps. I will add everything to the task list which seems decided to me. Everything else should be discussed first.

[Spenhouet]

• We should also think about renaming Rand2DElastic and Rand3DElastic to Rand2DElasticDeformation and Rand3DElasticDeformation since this is a much more precise and intuitive name
• One thing I like about torchio is that they explain everything in medical relevant terms. I believe the MONAI documentation could take a page from their book on that and improve all the existing doc strings. This might seem like unnecessary work but I strongly believe that it is important work.
• Another thing is that they always make sure to cite relevant work in the individual transform topics. Doing the same would make MONAI more reputable.

[Nic-Ma]

Hi @Spenhouet ,

Thanks for your great suggestions! Very valuable feedback for MONAI project.
And I think maybe we can start from non-breaking feature requests to add the new transforms you listed?
@wyli @ericspod @rijobro What do you guys think about the GIF demo of medical transforms?

Thanks.

[Nic-Ma]

About RandomSwap, I just confirmed it's different from MONAI ticket #2701 , the latter is to shuffle the content of patches.
So this RandomSwap transform is also not in our radar so far.

Thanks.

[diazandr3s]

This is a great point.

CC @fepegar

[fepegar]

Hi all! Thanks for pinging, @diazandr3s. I have a couple of comments:

1. Feel free to ping me in any discussions or questions related to TorchIO, I'm more than happy to help and collaborate. See e.g. Monai equivalent to tio.OneOf transform? (ETA 8/13) #1847, Random Patch Dataset #606, How to integrate transforms from BatchGenerator, TorchIO and Rising #493
2. I'm not sure it's necessary to spend resources in implementing more and more transforms. TorchIO and MONAI can be mixed more or less easily, as shown in
   a. Integrate 3rd party transforms into a MONAI program
   b. TorchIO + MONAI + PyTorch Lightning

[aylward]

Two thoughts on these visualizations:

1. would be great if the generated examples/images/documentation could also be part of the MONAI testing framework.

2. perhaps the visualizations (and documentation) could be generated in jupyter notebooks. A nice way of embedding interactive 2D and 3D visualizations of numpy, tensor, and other 2D and 3D structures in jupyter notebooks is via itkWidgets:
   https://github.com/InsightSoftwareConsortium/itkwidgets
   and
   https://blog.kitware.com/monai-and-itkwidgets-getting-started/

• Also, there is a nice way of capturing jupyter notebook visualizations generated by itkWidgets and saving them as pngs/gifs, etc.

[diazandr3s]

@aylward do you know whether itkwidgets work in Google Colab as well?
Another option for visualization is to have a Slicer module to try/test image transformations in Slicer. Similar to what is being created for TorchIO https://github.com/fepegar/SlicerTorchIO

[aylward]

@diazandr3s - great point...we are in the process of adding Colab support in collaboration with the NIH/NCI. It does work with Binder and other cloud-based jupyter notebooks, but google changed the jupyter notebook just enough such that itkWidgets and some (many?) other jupyter notebook / jupyter lab widgets don't quite work (they don't crash, but they simply don't appear in a cell).

The nice part about these widgets is that they are embedded in the notebooks, and you can use automated cell capturing to generate images and movies. So, it might be possible to automate the image generation for docs and for testing using the same notebook.

To capture a view do...

from itkwidgets import view
import ipywebrtc as webrtc
... read your image / numpy array / tensor / ...
... generate or read your label image ...
view1 = view(image=<your_image>, label_image=<your_label_image>, cmap=itkwidgets.cm.viridis, gradient_opacity=0.9, background=[0.5,0.5,0.5], slicing_planes=True)
view1

image_recorder1 = webrtc.ImageRecorder(stream=view1, filename='<your_filename.png>', autosave=True)
image_recorder1.recording=True

For all of the view options, use

view?

Enjoy.

[Spenhouet]

Hi @fepegar, first of I would like to thank you for your great library. As I already wrote above I really value the documentation and that it is highly medically relevant.
Sorry that I did not tag you on this issue. No bad intention.

I'm not sure it's necessary to spend resources in implementing more and more transforms. TorchIO and MONAI can be mixed more or less easily

Having MONAI transform wrappers around torch.io transforms would be an option but this would add torch.io as dependency to MONAI and that is a decision I have no say in but doubt that it is likely.
For us, as a medtech software company, reducing dependencies is crucial. For each SOUP (software of unknown provenance) we have to perform validations, risk assessments, .... and then again on updates. This is a lot of work for a Startup.

I agree that this seemingly binds "unnecessary" resources but having a single framework instead of multiple is way better for us (and probably many more). Currently there are way to many individual projects on the same topics and combining resources would be the best. But it is how it is. We did see the most potential for our use case in MONAI and therefore went with it. But MONAI doesn't yet provide the same data augmentations as torch.io does why we are still dependent on torch.io. Therefore my wish to close this gap.