Multiple patches transform for MIL

[myron]

I need a spatial transform for MIL that takes an input (ndarray) NxCxWxH,
loops over the first dimension (patches) and randomly flips (see below)

What are my options with the current monai functionality?

1. I understand that in the previous transform I can output a list of dicts (each having one patch) to be able to use RandomFlip (form monai). But in this case I don't know how to handle "label", all these patches have the same label (class) and I suppose I could repeat the same label (as a workaround) in each dict, but then at the end I need to recombine (concat) these patches and have only 1 single label

furthermore, the output of all transforms should be in the 5D shape BxNxCxWxH (where B is the batch dim, N is number of patches). and with this workaround, I get 4D: BNxCxWxH, and labels will be mixed up too

2. sequential application of several flips, creates copies , which I'm trying to avoid

class BatchedRandFlip():

    def __init__( self, keys, p=0.5) :
        super().__init__()
        self.keys = keys
        self.p=p

    def __call__(self, data):

        d = dict(data)

        for key in self.keys:
            images = d[key]
            images2=[]

            for i in range(images.shape[0]):

                img = images[i]
                if np.random.random() > self.p: img = np.flip(img, axis=1)
                if np.random.random() > self.p: img = np.flip(img, axis=2)
                if np.random.random() > self.p: img = np.transpose(img, axes=(0,2,1))
                images2.append(img)

            d[key] = np.stack(images2, axis=0)

        return d

[Nic-Ma]

Hi @myron ,

Thanks very much for your detailed explanation of the issue you are facing.
@wyli and I totally went through your code for this problem and we think it's better to do something like:

1. Define your patch creation transform in the similar way of RandCropByPosNegLabeld, return a list of dictionaries and label will be copied to every dictionary.
2. All the following transforms will automatically apply to every dictionary of above list.
3. Please define your own collate_fn in PyTorch Dataloader to stack the patches and only use the label of the first dictionary in the list, something like:

def list_data_collate(batch: Sequence):
    for i, item in enumerate(batch):
        data = item[0]
        data["image"] = torch.stack([i["image"] for i in item], dim=0)
        batch[i] = data
    return default_collate(batch)

dataloader = DataLoader(..., collate_fn=list_data_collate)

Then the DataLoader output will be a 5 dims data with expected labels.

What do you think?

Thanks.

[myron]

So this looks very 'hack-ish'.
To make this work , I'll have to modify the Tiling transform to create list of dicts, by copying some elements (that are not in keys argument), and splitting all ndarrays that are in the keys argument, a bit convoluted logic. And most of these keys are just copies (and are later ignored, just to make it work) Then to use this additional custom collate function. And after all, it will still make memory copies, and will be slower.

I think I prefer just to keep the custom Transform (as above) in the tutorial.

[bhashemian]

So this looks very 'hack-ish'. To make this work , I'll have to modify the Tiling transform to create list of dicts, by copying some elements (that are not in keys argument), and splitting all ndarrays that are in the keys argument, a bit convoluted logic. And most of these keys are just copies (and are later ignored, just to make it work) Then to use this additional custom collate function. And after all, it will still make memory copies, and will be slower.

I think I prefer just to keep the custom Transform (as above) in the tutorial.

@myron, are you saying that MONAI's structure is not capable of handling pipelines like multi-instance learning? If so, it is very concerning since it would have made MONAI limited in the scope!

@Nic-Ma @wyli can you weigh in here? Is MONAI approach way more inefficient than what Andriy is already doing?

[myron]

Monai is new to MIL, so there is not much support yet. we can be increasing the support over time. I think for now a simple way is sufficient, given the time constraint

[Nic-Ma]

Hi @myron and @drbeh ,

Thanks for the deep discussion!
The reason I suggested to make a list of dictionaries:

1. In current way, if users want to try other MONAI transforms for the extracted patches, they need to develop their own transform like your BatchedRandFlip. But if you use a list of dictionaries, all the following transforms can be any regular MONAI transform. I think we build this MIL module is to provide a flexible reference for users to develop more interesting things based on it, instead of just recording our best practice.
2. In your case, when creating a list of dictionaries, the image is extracted, we only need to deepcopy other things(classification label, meta, etc.), they are very small and should not affect performance much.
3. collate_fn is the standard method in PyTorch DataLoader to support complicated data batching, not a hack way.

What do you think?

Thanks.

[myron]

I have implemented it via collate and RandFlipd transforms (from monai).
It is 15% slower, probably because of

• memory copies in RandFlipd.
• the final intensity normalization transform (ScaleIntensityRangeD) before would run once on a single large np.array , and now that we split it into batches that transforms runs separately for each patch, which is slower. Similar any other transforms we may add after creating a list of dicts, will be slower

before:

       train_transform = transforms.Compose(
            [
                load_image_transform,
                LabelEncodeIntegerGraded(keys=["label"], num_classes=args.num_classes), #encode label
                TileOnGridd(keys=["image"], tile_count=args.max_tiles, tile_size=args.tile_size, tile_random_offset=True, tile_all=False, tile_background_val=255, return_list_of_dicts=False),
                SimpleTileFlip(keys=["image"]),
                transforms.ScaleIntensityRangeD(keys=["image"], a_min=np.float32(255), a_max=np.float32(0), b_min=np.float32(0), b_max=np.float32(1), clip=False),
                transforms.ToTensord(keys=["image", "label"]),
            ]
        )

after

       train_transform = transforms.Compose(
            [
                load_image_transform,
                LabelEncodeIntegerGraded(keys=["label"], num_classes=args.num_classes), #encode label
                TileOnGridd(keys=["image"], tile_count=args.max_tiles, tile_size=args.tile_size, tile_random_offset=True, tile_all=False, tile_background_val=255, return_list_of_dicts=True),
                transforms.RandFlipd(keys=["image"], spatial_axis=0, prob=0.5),
                transforms.RandFlipd(keys=["image"], spatial_axis=1, prob=0.5),
                transforms.RandRotate90d(keys=["image"], prob=0.5),
                transforms.ScaleIntensityRangeD(keys=["image"], a_min=np.float32(255), a_max=np.float32(0), b_min=np.float32(0), b_max=np.float32(1), clip=False),
                transforms.ToTensord(keys=["image", "label"]),
            ]
        )

and I had to add extra function into Tutorial (which is also not ideal, since we want to minimize the number of custom functions)

from torch.utils.data.dataloader import default_collate
def list_data_collate(batch: collections.abc.Sequence):
    for i, item in enumerate(batch):
        data = item[0]
        data["image"] = torch.stack([ix["image"] for ix in item], dim=0)
        batch[i] = data
    return default_collate(batch)

do we think it's an acceptable solution with 15% speed drop?

[myron]

merged #3237