Perceptual loss not working when one of the dimensions in a 3D volume is too small

[virginiafdez]

When one of the dimensions in a 3D volume is too small (which can occur, in images with small number of slices - i.e. cardiac images), the generation of pseudo-3d slices along all directions gives error because it results in 2D slices with one very small dimension.

We could automatise the choice of the directions that are taken into account for the calculation as a function of the size.

IE: volume of dimensions [64, 64, 12]. Because slices of size [64, 12] will not work due to the small width, only axial cuts ([64, 64]) will be taken into account.

[marksgraham]

Hi @virginiafdez

In PR #158 Walter has added full 3D perceptual loss - I've just checked, and it works with e.g. shapes of (64,64,12). Hopefully it will be merged soon and you can try it out by setting is_fake_3d=False

[virginiafdez]

As I discussed with @marksgraham, if is_fake_3d = False, a 3D network will be used to calculate the loss instead of the original 2D one, removing the slicing issue (I still need to test it).
Still, just in case, since the user might not know a priori that having a too little dimension might cause this problem, we could automate the process by:

1. When first calling the perceptual loss, calculating whether all the input dimensions are divisible by 2^n_downsamplings of the 2D network.
2. If not, maybe raise a user Warning / ValueError, telling the user that there will be issues and that it is suggested to set is_fake_3d to False, or doing it directly.

[Warvito]

One suggestion would be to include the axis selection (similar to what is available at the vqvae repo), allowing the user to specify to remove any axis with a low number of slices if wanted. This would allow using the fake 3d perceptual loss, even in your example.

Regarding how the loss should behave if the user tries to use an axis that the pre-trained network cannot process, I think it is important to educate the user about any problem (with a warning or an error). I think both, letting the execution crash or fixing it with a warning, would be okay. The only other example that I have in mind that implements the perceptual loss is the LPIPS package and them just let it crash returning the error of the own network. @marksgraham do you have any preference in this point?

[marksgraham]

Hi @virginiafdez @Warvito

My preference would be us raising an informative warning, instead of leaving the network itself to crash. If as Walter suggests we also allow for axis selection for the 2D loss then warning could inform users they could either drop the problematic axis or switch to the full 3D loss

[Warvito]

Note: Similar problem is find if the number of channels is not compatible with the pretrained network