Layer norm can't be used

[danieltudosiu]

Describe the bug
If I set norm="LAYER" in Conv or ResidualUnit I get the following error:

  File "/home/danieltudosiu/projects/vqvae/venv/lib/python3.6/site-packages/monai/networks/blocks/convolutions.py", line 135, in __init__
    dropout_dim=dropout_dim,
  File "/home/danieltudosiu/projects/vqvae/venv/lib/python3.6/site-packages/monai/networks/blocks/acti_norm.py", line 94, in __init__
    op_dict["N"] = norm_type(**kw_args)
TypeError: __init__() missing 1 required positional argument: 'normalized_shape'

To Reproduce

from monai.networks.blocks import ResidualUnit

a = ResidualUnit(
    dimensions=3,
    in_channels=1,
    out_channels=1,
    norm="LAYER"
)

Expected behavior
To allow me to use "LAYER" normalization.

Environment

Printing MONAI config...

MONAI version: 0.4.0
Numpy version: 1.19.5
Pytorch version: 1.8.0
MONAI flags: HAS_EXT = False, USE_COMPILED = False
MONAI rev id: 0563a44

Optional dependencies:
Pytorch Ignite version: 0.4.2
Nibabel version: 3.2.1
scikit-image version: NOT INSTALLED or UNKNOWN VERSION.
Pillow version: 8.1.2
Tensorboard version: 2.4.1
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.9.0
ITK version: NOT INSTALLED or UNKNOWN VERSION.
tqdm version: 4.59.0
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 5.8.0

For details about installing the optional dependencies, please visit:
https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

Printing system config...

System: Linux
Linux version: Ubuntu 18.04.5 LTS
Platform: Linux-5.4.0-65-generic-x86_64-with-Ubuntu-18.04-bionic
Processor: x86_64
Machine: x86_64
Python version: 3.6.9
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 16
Num logical CPUs: 32
Num usable CPUs: 32
CPU usage (%): [33.3, 33.3, 30.4, 33.3, 34.8, 30.4, 30.4, 30.4, 30.4, 21.7, 33.3, 33.3, 30.4, 33.3, 33.3, 31.8, 30.4, 28.6, 0.0, 4.2, 4.5, 22.7, 34.8, 30.4, 30.4, 31.8, 8.3, 13.6, 34.8, 34.8, 37.5, 100.0]
CPU freq. (MHz): 2530
Load avg. in last 1, 5, 15 mins (%): [4.3, 4.5, 4.3]
Disk usage (%): 48.4
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 62.7
Available memory (GB): 34.9
Used memory (GB): 26.4

Printing GPU config...

Num GPUs: 2
Has CUDA: True
CUDA version: 10.2
cuDNN enabled: True
cuDNN version: 7605
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_70']
Info for GPU: 1
Name: GeForce GT 1030
Is integrated: False
Is multi GPU board: False
Multi processor count: 3
Total memory (GB): 1.9
Cached memory (GB): 0.0
Allocated memory (GB): 0.0
CUDA capability (maj.min): 6.1

Additional context
I am aware it just needs a kwargs so we can manually pass the shape.

[wyli]

you could do

>>> a = ResidualUnit(3, 1, 1, norm=("layer", {"normalized_shape": (10, 10, 10)}))
>>> a
ResidualUnit(
  (conv): Sequential(
    (unit0): Convolution(
      (conv): Conv3d(1, 1, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
      (adn): ADN(
        (N): LayerNorm((10, 10, 10), eps=1e-05, elementwise_affine=True)
        (A): PReLU(num_parameters=1)
      )
    )
    (unit1): Convolution(
      (conv): Conv3d(1, 1, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
      (adn): ADN(
        (N): LayerNorm((10, 10, 10), eps=1e-05, elementwise_affine=True)
        (A): PReLU(num_parameters=1)
      )
    )
  )
  (residual): Identity()
)

[danieltudosiu]

Oh, I wasn't aware of that usage. Sorry!

[wyli]

ok, sure I'll update the docstrings...

[function2-llx]

Sorry to pop up this old issue, but how about adapting the get_norm_layer for LayerNorm? E.g., appending:

    if has_option(norm_type, "normalized_shape") and "normalized_shape" not in kw_args:
        kw_args["normalized_shape"] = channels