Including tiles coordinates in TileOnGrid transform

[vale-salvatelli]

Hi MONAI team! Here a feature request concerning Pathology that would really help us.

Problem description:

• My team would like to develop a deepmil pipeline where WSI slides are tiled on the fly reusing as much components as possible from MONAI.
• TileOnGrid is the transform currently offered by the library and suggested in the tutorial for this task.
• The problem we encountered is that this transformation does not return the coordinates of the tiles. This makes impossible to reconstruct model outputs that depends on the coordinates, such as attention maps.
• Attention maps are an essential tool for model interpretation.

Suggested feature:
A transformation having all the options in TileOnGrid (e.g. random offset, randomized tiles selection, overlapping tiles) that returns also the tiles coordinates with respect to a fixed reference system (e.g. top corner of the slide).

Alternative solution
One alternative solution is not to tile on the fly. Pre-tiling comes however with other drawbacks: it requires extensive pre-processing, reloading files in memory and overall it offers less modelling flexibility because the tiling parameters cannot be model parameters.

[Nic-Ma]

Hi @vale-salvatelli ,

Thanks very much for your interest and feedback here.
@drbeh @myron What do you think about it? Could you please help share some comments?

Thanks in advance.

[bhashemian]

Hi @vale-salvatelli, that's a great point! Thanks!
Let me think about it and come back to you. Until prioritizing this feature, I will add it to the backlog of MOANI Pathology.

[vale-salvatelli]

Thanks all for the speedy reply. @drbeh looking forward to hear what you think

[Nic-Ma]

Hi @drbeh ,

In my simple mind, I feel maybe you can refer to the GridPatch as an example of patch & coordinate outputs:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/data/utils.py#L220

Thanks.

[bhashemian]

Thank you @Nic-Ma for your suggestion.

One point that we need to consider here is that, unlike radiology images, patch (tile or regions a native concepts in whole slide imaging, and we can make this process much simpler. Here is my thinking and approach to address this issue:

1. Given a coordinate, a single patch handling (like loading and padding) is being handled by WSIReader via its backend libraries like OpenSlide and cuCIM.
2. Its duty of WSIReader to return the information about the patches when relying on it for patching. WSIReader to return patch information #4098
3. Then for this scenario (extract patches directly from WSI), it makes sense to delegate the task of generating random patches to a dataset. So we can implement a RandomPatchWSIDataset (in addition to PatchWSIDataset), which generate and extract random patches from whole slide images. Whole slide dataset to extract random patches #4099
4. On the other hand, if patching is being done for an image in the memory rather than on the file system, the process becomes similar to other applications in MONAI. However, for specifically for TileOnGrid, we can track the location of all patches and returning the list of locations corresponding to each patch as a metadata in the transform call.
   @vale-salvatelli, can this solution solve your issue?

Anyways, we are refactoring GridOnTile for the future release and if there is anything in this transform that can make your use case easier to implement, we would love to hear.

[Nic-Ma]

Hi @drbeh ,

Thanks for your proposal, it overall looks good to me to put the random logic in a dataset.
And I think maybe you can refer to the existing GridPatchDataset:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/data/grid_dataset.py#L134
@wyli @ericspod What do you think?

Thanks.

[ericspod]

What we've done in other areas of MONAI is have random patch selection done by transforms like RandSpatialCrop. Unless there's a specific need to have a randomised dataset I would avoid this since it breaks from that existing architectural choice.

[bhashemian]

What we've done in other areas of MONAI is have random patch selection done by transforms like RandSpatialCrop. Unless there's a specific need to have a randomised dataset I would avoid this since it breaks from that existing architectural choice.

Hi @ericspod, thanks for your feedback. Unfortunately, these logics do not work in digital pathology where we are dealing with whole slide imaging (WSI). Here we do not load the entire image to memory so that we can randomly crop or extract patches in the transforms, instead we load a specific patch from WSI in data loading part. Thus, the randomness needs to be appear in the datasets. Does it make sense?

[Nic-Ma]

Hi @ericspod ,

I feel maybe this feature is similar to our random CSVIteratbleDataset?
https://github.com/Project-MONAI/MONAI/blob/dev/monai/data/iterable_dataset.py#L122
So I think maybe @drbeh you can take a look how IterableDataset supports an extremely big dataset?
And also leverage the ShuffleBuffer?
https://github.com/Project-MONAI/MONAI/blob/dev/monai/data/iterable_dataset.py#L64

Please feel free to correct me if I understood anything.

Thanks.