Compositional Patch-based Inference

[bhashemian]

Currently, SlidingWindowInferer and SlidingWindowHoVerNetInferer rely on sliding_window_inferer, which lacks flexibility and structure to generalize to other use cases. For instance, different way of splitting image into patches (e.g., whole slide images or patch filtering), or different way of merging the result patches together.

To make the inference on patches more modular and extensible, I have prepared the following design and would appreciate your feedbacks on it.

In this way, similar behavior to SlidingWindowInferer:

SlidingWindowInferer(roi_shape=(256, 256), overlap=0.5, mode="gaussian", sigma_scale=1.0, sw_batch_size=8)

can be achieved as following:

PatchInference(splitter=SlidingPatchSplitter(patch_size=(256, 256), overlap=0.5), merger=GaussianMerger(sigma=1.0), batch_size=8)

while it is easy to extend this via composition of splitter and merger.

For instance, use the maximum value for overlapping regions:

PatchInference(splitter=SlidingPatchSplitter(patch_size=(256, 256), overlap=0.5), merger=MaxMerger())

or use a whole slide image as an input:

PatchInference(splitter=WSISlidingPatchSplitter(patch_size=(256, 256), overlap=0.5), merger=MaxMerger())

Here, I mentioned few concrete classes for Splitter and Merger as example but many more concrete classes can implemented to extend the usage of PatchInference.

PS: I would suggest to change the name of Inferer base class to Inference since "inferer" does not exist in English vocabulary.

CC: @Nic-Ma @ericspod @wyli @rijobro

[Nic-Ma]

Hi @drbeh ,

To be backward compatible, I would suggest to:

1. Keep the Inferer class, Yan, Eric, Wenqi and I defined this signature at the very beginning of MONAI.
2. Keep the SlidingWindowInferer as a typical (or simplified) case of PatchInference.

@wyli @ericspod What do you guys think?

Thanks.

[ericspod]

I agree with @Nic-Ma that we should keep the existing Inferer classes in some form. One thing to decide is if "window" or "patch" is the word we want to use to describe a subcomponent of an input image used as network input and produced as network output, we used "window" in our class name already so I'd stick to that if they're synonyms. We should refactor the existing class though as you're describing to put the window selection and merging into separate modular units, I'm pretty sure this can be backwards compatible as well..

[bhashemian]

Thank you @Nic-Ma and @ericspod for your feedback, so

1. I will keep the Inferer base class as is.
2. We can reimplement all the existing inferer classes (e.g., SlidingWindowInfer) with full backward compatibility based on PatchInference.
3. There are several options to name this smaller size image extracted from the larger image (ROI, window, region, patch, etc.). In pathology use cases, we tried to use a consistent naming since this notion is extensively being used. In pathology WG, we found "patch" to be a more general notion than window, more specific than ROI, and used in deep learning community too (e.g., "Patches Are All You Need?"). I would suggest to use patch here too (while we can still have specific-purposed classes like SlidingWindowInferer), which would also make sense when we want to have random patches for the inference (e.g., some MIL use cases). What do you think?

[Nic-Ma]

Hi @wyli @ericspod ,

Could you please help share some comments about the proposal?

Thanks in advance.

[wyli]

I think a patch merger or patch aggregator makes sense, potentially it could be generic
and be used together with the patch datasets and patch sampling transforms (https://github.com/Project-MONAI/MONAI/blob/dev/monai/data/grid_dataset.py#L133).

For the splitter and PatchInference wrapper, perhaps it's too early to unify the APIs... we haven't explored different patch sampling strategies for example coarse to fine search, handling multiple input resolutions, incorporating efficient IO approaches such as cucim's implementation.

[bhashemian]

Hi @wyli,

Thanks for your feedback. I agree that there are many ways to split an image to patches but those are implementation details and can be handled by concrete classes of Splitter. The Splitter just define the API and not how the patches are extracted so it can be general enough to cover any use case that I know, including multiple resolutions, GDS, random patches, sliding patches, WSI, 2D, 3D, etc. Please let me know if there is any specific use case that I am missing to address.

Thakns

[bhashemian]

Thanks everyone for the feedbacks and the great discussion that we had today. Based on what we agreed, I will create followup issues and proceed with the implementation of this design.