diff --git a/docs/images/fast_training.png b/docs/images/fast_training.png index d0584b9dac..34e47bcb21 100644 Binary files a/docs/images/fast_training.png and b/docs/images/fast_training.png differ diff --git a/docs/images/threaddataloader.png b/docs/images/threaddataloader.png new file mode 100644 index 0000000000..565df8d0d4 Binary files /dev/null and b/docs/images/threaddataloader.png differ diff --git a/docs/source/highlights.md b/docs/source/highlights.md index 141c0846d1..b84e93ff2d 100644 --- a/docs/source/highlights.md +++ b/docs/source/highlights.md @@ -16,7 +16,7 @@ The overall architecture and modules are shown in the following figure: The rest of this page provides more details for each module. * [Data I/O, processing and augmentation](#medical-image-data-i-o-processing-and-augmentation) -* [Datasets](#datasets) +* [Datasets and DataLoader](#datasets-and-dataloader) * [Loss functions](#losses) * [Optimizers](#optimizers) * [Network architectures](#network-architectures) @@ -25,7 +25,7 @@ The rest of this page provides more details for each module. * [Result writing](#result-writing) * [Workflows](#workflows) * [Research](#research) -* [GPU acceleration](#gpu-acceleration) +* [Performance optimization and GPU acceleration](#performance-optimization-and-gpu-acceleration) * [Applications](#applications) ## Medical image data I/O, processing and augmentation @@ -56,8 +56,15 @@ transformations. These currently include, for example: [2D transforms tutorial](https://github.com/Project-MONAI/tutorials/blob/master/modules/transforms_demo_2d.ipynb) shows the detailed usage of several MONAI medical image specific transforms. ![2d transform examples](../images/medical_transforms.png) -### 3. Fused spatial transforms and GPU acceleration -As medical image volumes are usually large (in multi-dimensional arrays), pre-processing performance affects the overall pipeline speed. MONAI provides affine transforms to execute fused spatial operations, supports GPU acceleration via native PyTorch for high performance. + +### 3. Transforms support both NumPy array and PyTorch Tensor (CPU or GPU accelerated) +From MONAI v0.7 we introduced PyTorch `Tensor` based computation in transforms, many transforms already support both `numpy array` and `Tensor` data. + +To accelerate the transforms, a common approach is to leverage GPU parallel-computation. Users can first convert input data into GPU Tensor by `ToTensor` or `EnsureType` transform, then the following transforms can execute on GPU based on PyTorch `Tensor` APIs. +GPU transform tutorial is available at [Spleen fast training tutorial](https://github.com/Project-MONAI/tutorials/blob/master/acceleration/fast_training_tutorial.ipynb). + +### 4. Fused spatial transforms +As medical image volumes are usually large (in multi-dimensional arrays), pre-processing performance affects the overall pipeline speed. MONAI provides affine transforms to execute fused spatial operations. For example: ```py @@ -67,20 +74,21 @@ affine = Affine( scale_params=(1.2, 1.2), translate_params=(200, 40), padding_mode='zeros', - device=torch.device('cuda:0') ) # convert the image using bilinear interpolation new_img = affine(image, spatial_size=(300, 400), mode='bilinear') ``` Experiments and test results are available at [Fused transforms test](https://github.com/Project-MONAI/tutorials/blob/master/acceleration/transform_speed.ipynb). -Currently, all the geometric image transforms (Spacing, Zoom, Rotate, Resize, etc.) are designed based on the PyTorch native interfaces. [Geometric transforms tutorial](https://github.com/Project-MONAI/tutorials/blob/master/modules/3d_image_transforms.ipynb) indicates the usage of affine transforms with 3D medical images. +Currently, all the geometric image transforms (Spacing, Zoom, Rotate, Resize, etc.) are designed based on the PyTorch native interfaces. So all of them support GPU acceleration via `GPU Tensor` operations for high performance. + +[Geometric transforms tutorial](https://github.com/Project-MONAI/tutorials/blob/master/modules/3d_image_transforms.ipynb) indicates the usage of affine transforms with 3D medical images. ![3d transform examples](../images/affine.png) -### 4. Randomly crop out batch images based on positive/negative ratio +### 5. Randomly crop out batch images based on positive/negative ratio Medical image data volume may be too large to fit into GPU memory. A widely-used approach is to randomly draw small size data samples during training and run a “sliding window” routine for inference. MONAI currently provides general random sampling strategies including class-balanced fixed ratio sampling which may help stabilize the patch-based training process. A typical example is in [Spleen 3D segmentation tutorial](https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/spleen_segmentation_3d.ipynb), which achieves the class-balanced sampling with `RandCropByPosNegLabel` transform. -### 5. Deterministic training for reproducibility +### 6. Deterministic training for reproducibility Deterministic training support is necessary and important for deep learning research, especially in the medical field. Users can easily set the random seed to all the random transforms in MONAI locally and will not affect other non-deterministic modules in the user's program. For example: @@ -99,16 +107,16 @@ Users can also enable/disable deterministic at the beginning of training program monai.utils.set_determinism(seed=0, additional_settings=None) ``` -### 6. Multiple transform chains +### 7. Multiple transform chains To apply different transforms on the same data and concatenate the results, MONAI provides `CopyItems` transform to make copies of specified items in the data dictionary and `ConcatItems` transform to combine specified items on the expected dimension, and also provides `DeleteItems` transform to delete unnecessary items to save memory. Typical usage is to scale the intensity of the same image into different ranges and concatenate the results together. ![multiple transform chains](../images/multi_transform_chains.png) -### 7. Debug transforms with DataStats +### 8. Debug transforms with DataStats When transforms are combined with the "compose" function, it's not easy to track the output of a specific transform. To help debug errors in the composed transforms, MONAI provides utility transforms such as `DataStats` to print out intermediate data properties such as `data shape`, `value range`, `data value`, `Additional information`, etc. It's a self-contained transform and can be integrated into any transform chain. -### 8. Post-processing transforms for model output +### 9. Post-processing transforms for model output MONAI also provides post-processing transforms for handling the model outputs. Currently, the transforms include: - Adding an activation layer (Sigmoid, Softmax, etc.). - Converting to discrete values (Argmax, One-Hot, Threshold value, etc), as below figure (b). @@ -119,12 +127,19 @@ MONAI also provides post-processing transforms for handling the model outputs. C After decollating the batch data of model output and applying the post-processing transforms, it's easier to compute metrics, save model output into files or visualize data in the TensorBoard. [Postprocessing transforms tutorial](https://github.com/Project-MONAI/tutorials/blob/master/modules/postprocessing_transforms.ipynb) shows an example with several main transforms for post-processing. ![post-processing transforms](../images/postprocessing_transforms.png) -### 9. Integrate third-party transforms +### 10. Integrate third-party transforms The design of MONAI transforms emphasis code readability and usability. It works for array data or dictionary-based data. MONAI also provides `Adaptor` tools to accommodate different data format for 3rd party transforms. To convert the data shapes or types, utility transforms such as `ToTensor`, `ToNumpy`, `SqueezeDim` are also provided. So it's easy to enhance the transform chain by seamlessly integrating transforms from external packages, including: `ITK`, `BatchGenerator`, `TorchIO` and `Rising`. For more details, please check out the tutorial: [integrate 3rd party transforms into MONAI program](https://github.com/Project-MONAI/tutorials/blob/master/modules/integrate_3rd_party_transforms.ipynb). -### 10. IO factory for medical image formats +In digital pathology training, due to the immense burden of loading images, the CPU is preoccupied by loading images and cannot catch up with preparing the data. This causes the pipeline to become IO bound and results in under-utilization of GPU. To overcome this bottleneck, [cuCIM](https://github.com/rapidsai/cucim) has implemented an optimized version of several common transforms that we are using in digital pathology pipeline. These transforms are natively being run on GPU and act on CuPy arrays. MONAI provides `CuCIM` and `RandCuCIM` adapters to integrate the `cuCIM` library. For instance: +```py +RandCuCIM(name="color_jitter", brightness=64.0 / 255.0, contrast=0.75, saturation=0.25, hue=0.04) +CuCIM(name="scale_intensity_range", a_min=0.0, a_max=255.0, b_min=-1.0, b_max=1.0) +``` +It has shown a significant speed up in pathology training metastasis detection model. + +### 11. IO factory for medical image formats Many popular image formats exist in the medical domain, and they are quite different with rich metadata information. To easily handle different medical image formats in the same pipeline, [MONAI provides `LoadImage` transform](https://github.com/Project-MONAI/tutorials/blob/master/modules/load_medical_images.ipynb), which can automatically choose image readers based on the supported suffixes and in the following priority order: - User-specified reader at runtime when calling this loader. - Registered readers from the latest to the first in the list. @@ -134,13 +149,13 @@ The `ImageReader` API is quite straightforward, users can easily extend it for t With these pre-defined image readers, MONAI can load images in formats: `NIfTI`, `DICOM`, `PNG`, `JPG`, `BMP`, `NPY/NPZ`, etc. -### 11. Save transform data into NIfTI or PNG files +### 12. Save transform data into NIfTI or PNG files To convert images into files or debug the transform chain, MONAI provides `SaveImage` transform. Users can inject this transform into the transform chain to save the results. -### 12. Automatically ensure `channel-first` data shape +### 13. Automatically ensure `channel-first` data shape Medical images have different shape formats. They can be `channel-last`, `channel-first` or even `no-channel`. We may, for example, want to load several `no-channel` images and stack them as `channel-first` data. To improve the user experience, MONAI provided an `EnsureChannelFirst` transform to automatically detect data shape according to the meta information and convert it to the `channel-first` format consistently. -### 13. Invert spatial transforms and test-time augmentations +### 14. Invert spatial transforms and test-time augmentations It is often desirable to invert the previously applied spatial transforms (resize, flip, rotate, zoom, crop, pad, etc.) within the deep learning workflows, for example, to resume to the original imaging space after processing the image data in a normalized data space. Many spatial transforms are enhanced with an `inverse` operation since in v0.5. The [model inference tutorial](https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/torch/unet_inference_dict.py) shows a basic example. If the pipeline includes random transformations, users may want to observe the effect that these transformations have on the output. The typical approach is that we pass the same input through the transforms multiple times with different random realizations. Then use the inverse transforms to move all the results to a common space, and calculate the metrics. MONAI provided `TestTimeAugmentation` for this feature, which by default will calculate the `mode`, `mean`, `standard deviation` and `volume variation coefficient`. @@ -153,7 +168,7 @@ If the pipeline includes random transformations, users may want to observe the e (2) The TTA results of `mode`, `mean` and `standard deviation`: ![test time augmentation](../images/tta.png) -## Datasets +## Datasets and DataLoader ### 1. Cache IO and transforms data to accelerate training Users often need to train the model with many (potentially thousands of) epochs over the data to achieve the desired model quality. A native PyTorch implementation may repeatedly load data and run the same preprocessing steps for every epoch during training, which can be time-consuming and unnecessary, especially when the medical image volumes are large. @@ -221,6 +236,11 @@ The `partition_dataset` utility in MONAI can perform different types of partitio CSV tables are often used in additional to image data to incorporate adjunct information, such as patient demographics, lab results, image acquisition parameters and other non-image data, MONAI provides `CSVDataset` to load CSV files and `CSVIterableDataset` to load large CSV files with scalable data access. In addition to the regular preprocessing transform while loading, it also supports multiple CSV files loading, joining tables, rows and columns selection and grouping. [CSVDatasets tutorial](https://github.com/Project-MONAI/tutorials/blob/master/modules/csv_datasets.ipynb) shows detailed usage examples. +### 9. `ThreadDataLoader` vs. `DataLoader` +If the transforms are light-weighted, especially when we cache all the data in RAM, the multiprocessing of PyTorch `DataLoader` may cause unnecessary IPC time and cause the drop of GPU utilization after every epoch. MONAI provides `ThreadDataLoader` which executes the transforms in a separate thread: +![threaddataloader](../images/threaddataloader.png) +a `ThreadDataLoader` example is available at [Spleen fast training tutorial](https://github.com/Project-MONAI/tutorials/blob/master/acceleration/fast_training_tutorial.ipynb). + ## Losses There are domain-specific loss functions in the medical imaging research which are not typically used in generic computer vision tasks. As an important module of MONAI, these loss functions are implemented in PyTorch, such as `DiceLoss`, `GeneralizedDiceLoss`, `MaskedDiceLoss`, `TverskyLoss`, `FocalLoss`, `DiceCELoss`, and `DiceFocalLoss`, etc. @@ -249,7 +269,7 @@ add_module('conv1', conv_type(in_channels, out_channels, kernel_size=1, bias=Fal ``` ### 2. Implementation of generic 2D/3D networks -And there are several 1D/2D/3D-compatible implementations of intermediate blocks and generic networks, such as UNet, DynUNet, DenseNet, GAN, AHNet, VNet, SENet(and SEResNet, SEResNeXt), SegResNet, EfficientNet, Attention-based networks. All the networks can support PyTorch serialization pipeline based on `torch.jit.script`. +And there are several 1D/2D/3D-compatible implementations of intermediate blocks and generic networks, such as UNet, DynUNet, DenseNet, GAN, AHNet, VNet, SENet(and SEResNet, SEResNeXt), SegResNet, EfficientNet, Attention-based transformer networks. All the networks can support PyTorch serialization pipeline based on `torch.jit.script`. ### 3. Network adapter to finetune final layers Instead of training from scratch, we often leverage the existing models, and finetune the final layers of a network for new learning tasks. MONAI provides a `NetAdapter` to easily replace the last layer of a model by a convolutional layer or a fully-connected layer. A typical usage example is to adapt [Torchvision models trained with ImageNet](https://pytorch.org/vision/stable/models.html) for other learning tasks. @@ -366,10 +386,15 @@ G. Wang, X. Liu, C. Li, Z. Xu, J. Ruan, H. Zhu, T. Meng, K. Li, N. Huang, S. Zha Wentao Zhu, Can Zhao, Wenqi Li, Holger Roth, Ziyue Xu, and Daguang Xu (2020) "LAMP: Large Deep Nets with Automated Model Parallelism for Image Segmentation." MICCAI 2020 (Early Accept, paper link: https://arxiv.org/abs/2006.12575) ![LAMP UNet](../images/unet-pipe.png) -## GPU acceleration +## Performance optimization and GPU acceleration +Typically, model training is a time-consuming step during deep learning development, especially in medical imaging applications. Volumetric medical images are usually large (as multi-dimensional arrays) and the model training process can be complex. Even with powerful hardware (e.g. CPU/GPU with large RAM), it is not easy to fully leverage them to achieve high performance. MONAI provides a [fast training guide](https://github.com/Project-MONAI/tutorials/blob/master/acceleration/fast_model_training_guide.md) to achieve the best performance. + NVIDIA GPUs have been widely applied in many areas of deep learning training and evaluation, and the CUDA parallel computation shows obvious acceleration when comparing to traditional computation methods. To fully leverage GPU features, many popular mechanisms raised, like automatic mixed precision (AMP), distributed data parallel, etc. MONAI can support these features and provides rich examples. -### 1. Auto mixed precision(AMP) +### 1. Profiling the pipelines +First of all, MONAI provides several methods based on `DLProf`, `Nsight`, `NVTX` and `NVML` for users to analyze their programs to identify the performance bottleneck. The analyses include operation-based GPU activity and overall GPU activity during model training. They will greatly help users manage computing bottlenecks and provide insights for the area to be improved for better computing efficiency. The detailed example is shown in the [performance profiling tutorial]( https://github.com/Project-MONAI/tutorials/blob/master/performance_profiling/profiling_train_base_nvtx.ipynb). + +### 2. Auto mixed precision(AMP) In 2017, NVIDIA researchers developed a methodology for mixed-precision training, which combined single-precision (FP32) with half-precision (e.g. FP16) format when training a network, and it achieved the same accuracy as FP32 training using the same hyperparameters. For the PyTorch 1.6 release, developers at NVIDIA and Facebook moved mixed precision functionality into PyTorch core as the AMP package, `torch.cuda.amp`. @@ -379,16 +404,16 @@ MONAI workflows can easily set `amp=True/False` in `SupervisedTrainer` or `Super We also executed the same test program on NVIDIA A100 GPU with the same software environment, obtained faster results: ![amp a100 results](../images/amp_training_a100.png) More details is available at [AMP training tutorial](https://github.com/Project-MONAI/tutorials/blob/master/acceleration/automatic_mixed_precision.ipynb). -We also tried to combine AMP with `CacheDataset` and `Novograd` optimizer to achieve the fast training in MONAI, able to obtain approximately 12x speedup compared with a Pytorch native implementation when the training converges at a validation mean dice of 0.93. Benchmark for reference: +We also tried to combine `AMP` with `CacheDataset`, `GPU cache`, `GPU transforms`, `ThreadDataLoader`, `DiceCE` loss function and `Novograd` optimizer to achieve the fast training in MONAI, able to obtain approximately `200x` speedup compared with a Pytorch native implementation when the training converges at a validation mean dice of `0.95`. Benchmark for reference: ![fast training results](../images/fast_training.png) More details is available at [Fast training tutorial](https://github.com/Project-MONAI/tutorials/blob/master/acceleration/fast_training_tutorial.ipynb). -### 2. Distributed data parallel +### 3. Distributed data parallel Distributed data parallel is an important feature of PyTorch to connect multiple GPU devices on single or multiple nodes to train or evaluate models. The distributed data parallel APIs of MONAI are compatible with native PyTorch distributed module, pytorch-ignite distributed module, Horovod, XLA, and the SLURM platform. MONAI provides demos for reference: train/evaluate with PyTorch DDP, train/evaluate with Horovod, train/evaluate with Ignite DDP, partition dataset and train with SmartCacheDataset, as well as a real world training example based on Decathlon challenge Task01 - Brain Tumor segmentation. The demo contains distributed caching, training, and validation. We obtained performance benchmarks for reference (based on PyTorch 1.6, CUDA 11, NVIDIA V100 GPUs): ![distributed training results](../images/distributed_training.png) -### 3. C++/CUDA optimized modules +### 4. C++/CUDA optimized modules To further accelerate the domain-specific routines in the workflows, MONAI C++/CUDA implementation are introduced as extensions of the PyTorch native implementations. MONAI provides the modules using [the two ways of building C++ extensions from PyTorch](https://pytorch.org/tutorials/advanced/cpp_extension.html#custom-c-and-cuda-extensions): - via `setuptools`, for modules including `Resampler`, `Conditional random field (CRF)`, `Fast bilateral filtering using the permutohedral lattice`. @@ -396,6 +421,26 @@ MONAI provides the modules using [the two ways of building C++ extensions from P The following figure shows results of MONAI's Gaussian mixture models applied to tissue and surgical tools segmentation: ![Gaussian mixture models as a postprocessing step](../images/gmm_feature_set_comparison_s.png) +### 5. Cache IO and transforms data to GPU memory +Even with `CacheDataset`, we usually need to copy the same data to GPU memory for GPU random transforms or network computation in every epoch. An efficient approach is to cache the data to GPU memory directly, then every epoch can start from GPU computation immediately. + +For example: +```py +train_transforms = [ + LoadImaged(...), + AddChanneld(...), + Spacingd(...), + Orientationd(...), + ScaleIntensityRanged(...), + EnsureTyped(..., data_type="tensor"), + ToDeviced(..., device="cuda:0"), + RandCropByPosNegLabeld(...), +] +dataset = CacheDataset(..., transform=train_trans) +``` +Here we convert to PyTorch `Tensor` with `EnsureTyped` transform and move data to GPU with `ToDeviced` transform. `CacheDataset` caches the transform results until `ToDeviced`, so it is in GPU memory. Then in every epoch, the program fetches cached data from GPU memory and only executes the random transform `RandCropByPosNegLabeld` on GPU directly. +GPU caching example is available at [Spleen fast training tutorial](https://github.com/Project-MONAI/tutorials/blob/master/acceleration/fast_training_tutorial.ipynb). + ## Applications The research area of medical image deep learning is expanding fast. To apply the latest achievements into applications, MONAI contains many application components to build end-to-end solutions or prototypes for other similar use cases. @@ -417,3 +462,8 @@ Starting from v0.5.0, MONAI provides experimental features for building learning The following figure shows the registration of CT images acquired at different time points for a single patient using MONAI: ![3d registration](../images/3d_paired.png) + +### 4. Reproducing the state-of-the-art Kaggle competition solutions +[A reimplementation](https://github.com/Project-MONAI/tutorials/tree/master/kaggle/RANZCR/4th_place_solution) of the 4th place solution of RANZCR CLiP - Catheter and Line Position Challenge in Kaggle: https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification + +The original solution is produced by Team Watercooled, and the authors are Dieter (https://www.kaggle.com/christofhenkel) and Psi (https://www.kaggle.com/philippsinger).