CutMix transform breaking change in v1.4.0

[maxime915]

The behavior of the CutMix transform on labels was changed significantly from v1.3.2 to v1.4.0rc1. This change originates from #7813, in which I have left a comment, but I'm creating an issue to increase visibility.

Here is a script using the GlaS dataset.

import pathlib
from PIL import Image
import matplotlib.pyplot as plt
import numpy as np
import torch
from monai.transforms.regularization.dictionary import CutMixd

np.random.seed(0)
torch.manual_seed(0)

# take the 5 first training samples from glas
indices = list(range(1, 6))
glas_dir = pathlib.Path.home() / "datasets" / "Warwick QU Dataset (Released 2016_07_08)"
images = [np.array(Image.open(glas_dir / f"train_{idx}.bmp")).copy() for idx in indices]
labels = [np.array(Image.open(glas_dir / f"train_{idx}_anno.bmp")).copy() for idx in indices]
for lbl_np in labels:
    lbl_np[lbl_np > 0] = 1  # binary segmentation

# create a batch for cutmixd
img = torch.stack([torch.from_numpy(img).permute(2, 0, 1) for img in images])
lbl = torch.stack([torch.from_numpy(lbl).view(lbl.shape + (1,)).permute(2, 0, 1) for lbl in labels])
data = {"image": img, "segmentation": lbl}

transform = CutMixd(keys=("image",), batch_size=len(img), label_keys=("segmentation",))
transform.mixer.set_random_state(100)
data = transform(data)

fig, axes = plt.subplots(1, 2)
axes[0].imshow(data["image"][0].permute(1, 2, 0).numpy())
axes[0].set_axis_off()
axes[1].imshow(data["segmentation"][0].permute(1, 2, 0).numpy())
axes[1].set_axis_off()
plt.show()

Before, the label produced was a linear combination of the two labels.

In v1.4.0.rc1 (bottom) it is created similarly to the images.

The paper (https://arxiv.org/abs/1905.04899) mentioned in #2872 only describe how to transform a classification. While I have no authority on this subject, I believe the new implementation makes a lot more sense than the previous implementation.

However, warning users in a release note for this fixed bug / breaking change may be a good idea, if this change is to be kept.

Thank you for your consideration.

Environment

Ensuring you use the relevant python executable, please paste the output of:

python -c "import monai; monai.config.print_debug_info()"

For v1.4.0rc1

================================
Printing MONAI config...
================================
MONAI version: 1.4.0rc1
Numpy version: 1.26.4
Pytorch version: 2.4.0
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 34ce94db424445b38eb56a6c842e55a2122d4a9d
MONAI __file__: /home/<username>/anaconda3/envs/tsw24/lib/python3.9/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
ITK version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: 5.2.1
scikit-image version: 0.24.0
scipy version: 1.13.1
Pillow version: 10.4.0
Tensorboard version: NOT INSTALLED or UNKNOWN VERSION.
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.19.0
tqdm version: 4.66.5
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 6.0.0
pandas version: 2.2.2
einops version: 0.8.0
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.
clearml version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 22.04.5 LTS
Platform: Linux-6.8.0-45-generic-x86_64-with-glibc2.35
Processor: x86_64
Machine: x86_64
Python version: 3.9.19
Process name: pt_main_thread
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 14
Num logical CPUs: 20
Num usable CPUs: 20
CPU usage (%): [10.4, 7.9, 9.4, 7.9, 98.4, 8.7, 9.5, 7.9, 9.4, 8.7, 8.0, 8.7, 9.4, 7.9, 9.4, 8.7, 8.7, 8.6, 7.9, 8.7]
CPU freq. (MHz): 890
Load avg. in last 1, 5, 15 mins (%): [3.6, 4.3, 3.6]
Disk usage (%): 78.7
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 31.0
Available memory (GB): 22.1
Used memory (GB): 7.4

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 11.8
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: None
cuDNN version: 90100
Current device: 0
Library compiled for CUDA architectures: ['sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_37', 'sm_90', 'compute_37']
GPU 0 Name: NVIDIA GeForce RTX 3050 Laptop GPU
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 16
GPU 0 Total memory (GB): 3.8
GPU 0 CUDA capability (maj.min): 8.6

for v1.3.2

================================
Printing MONAI config...
================================
MONAI version: 1.3.2
Numpy version: 1.26.4
Pytorch version: 2.4.0
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 59a7211070538586369afd4a01eca0a7fe2e742e
MONAI __file__: /home/<username>/anaconda3/envs/tsw24/lib/python3.9/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
ITK version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: 5.2.1
scikit-image version: 0.24.0
scipy version: 1.13.1
Pillow version: 10.4.0
Tensorboard version: NOT INSTALLED or UNKNOWN VERSION.
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.19.0
tqdm version: 4.66.5
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 6.0.0
pandas version: 2.2.2
einops version: 0.8.0
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.
clearml version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 22.04.5 LTS
Platform: Linux-6.8.0-45-generic-x86_64-with-glibc2.35
Processor: x86_64
Machine: x86_64
Python version: 3.9.19
Process name: pt_main_thread
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 14
Num logical CPUs: 20
Num usable CPUs: 20
CPU usage (%): [11.3, 6.6, 13.6, 6.5, 83.1, 6.5, 20.7, 7.8, 11.0, 8.2, 13.0, 12.9, 13.0, 11.7, 12.4, 9.8, 9.8, 11.0, 9.7, 11.0]
CPU freq. (MHz): 1131
Load avg. in last 1, 5, 15 mins (%): [3.2, 4.1, 3.6]
Disk usage (%): 78.7
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 31.0
Available memory (GB): 22.0
Used memory (GB): 7.4

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 11.8
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: None
cuDNN version: 90100
Current device: 0
Library compiled for CUDA architectures: ['sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_37', 'sm_90', 'compute_37']
GPU 0 Name: NVIDIA GeForce RTX 3050 Laptop GPU
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 16
GPU 0 Total memory (GB): 3.8
GPU 0 CUDA capability (maj.min): 8.6

[KumoLiu]

Hi @maxime915,

Thank you for reporting this issue!

You're right—there is indeed a behavior change here. As I mentioned in the PR: #7813 (comment)

In MONAI, we expect the same seed to be used across all keys in the dictionary-based version of the transforms. We didn’t include a specific note regarding this because the change aligns with the behavior of other transforms in MONAI, and we considered it an expected improvement. Apologies if this caused any confusion.

Your report will help bring more attention to this change, and I’ll make sure to enhance the docstring in the class to better explain this behavior.

Thanks again!

[maxime915]

Thank you for the prompt reply.

There is one more thing I do not understand about this PR, can you help me ? I see that there were modification regarding the params field of CutMix, but I also see that for the processing of the label, the old code in CutMix.__call__used CutMix.apply_on_label whereas it now uses CutMix.apply (the same as the image). Is this change related to the determinism as well ? Looking at the implementation of apply and apply_on_label, both methods seem to be deterministic after the update.

For clarity, here is a link to the implementation in v1.3.2 :

MONAI/monai/transforms/regularization/array.py

Line 142 in 59a7211

return (augmented, self.apply_on_labels(labels)) if labels is not None else augmented

And here is a link to the implementation in v1.4.0 :

MONAI/monai/transforms/regularization/array.py

Lines 161 to 163 in 76ef9f4

	if labels is not None:
	augmented_label = convert_to_dst_type(self.apply(labels_t), dst=labels)[0]
	return (augmented, augmented_label) if labels is not None else augmented

[KumoLiu]

Yes, that's what I mentioned here: #7813 (comment)
We want ensure every key in the dictionary transform works as expected.