integration TestDynUNetWithInstanceNorm3dNVFuser

[wyli]

Describe the bug
logs using pytorch image 22.04 https://github.com/Project-MONAI/MONAI/runs/6306011370?check_suite_focus=true

(| NVIDIA-SMI 495.29.05 Driver Version: 495.29.05 CUDA Version: 11.5 )

ERROR: test_consistency_0 (tests.test_dynunet.TestDynUNetWithInstanceNorm3dNVFuser)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/parameterized/parameterized.py", line 533, in standalone_func
test_compute_0 (tests.test_prepare_batch_default_dist.DistributedPrepareBatchDefault) (10.5s)
test_compute_1 (tests.test_prepare_batch_default_dist.DistributedPrepareBatchDefault) (10.6s)
test_verify_0__tmp_tmp_HuvzKkaSgJ_tests_testing_data_metadata_json (tests.test_bundle_verify_net.TestVerifyNetwork) (11.2s)
    return func(*(a + p.args), **p.kwargs)
  File "/tmp/tmp.HuvzKkaSgJ/tests/test_dynunet.py", line 146, in test_consistency
    result_fuser = net_fuser(input_tensor)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/monai/networks/nets/dynunet.py", line 268, in forward
    out = self.skip_layers(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/monai/networks/nets/dynunet.py", line 46, in forward
    downout = self.downsample(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/monai/networks/blocks/dynunet_block.py", line 80, in forward
    out = self.norm1(out)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/apex/normalization/instance_norm.py", line 138, in forward
    out = InstanceNormNVFuserFunction.apply(
  File "/opt/conda/lib/python3.8/site-packages/apex/normalization/instance_norm.py", line 16, in forward
    instance_norm_nvfuser_cuda = importlib.import_module("instance_norm_nvfuser_cuda")
  File "/opt/conda/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1166, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /opt/conda/lib/python3.8/site-packages/instance_norm_nvfuser_cuda.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZTVN5torch3jit5fuser4cuda3kir6KernelE

----------------------------------------------------------------------

[Nic-Ma]

Hi @yiheng-wang-nv ,

Could you please help investigate and fix this InstanceNorm3dNVFuser issue in PyTorch docker 22.04?

Thanks in advance.

[yiheng-wang-nv]

Hi @Nic-Ma and @wyli , my initial thought is that the issue is due to the driver version, with driver version 510.47.03, docker 22.04 does not raise the same issue. For the ci/cd machine, is it possible to test with different driver version?

[wyli]

thanks, I think blossom tests are with the different drivers. could you please discuss this with @pxLi? (there's no need to update the github CI because we aim to use the most compatible driver to cover as many cases as possible)

at the same time it seems this condition is not good enough to detect a suitable environment, could you

MONAI/monai/networks/layers/factories.py

Line 266 in 209db9e

if not has_nvfuser:

could you help improve?

[pxLi]

FYI internal blossom hosts are mostly w/ driver 465.07. This depends on your guys what driver versions we should cover in tests. Please let me know if you have specific requirement, we could try find available machines in public pool to test before actually asking blossom guys to help modify drivers version on our entitled machines, thanks

[yiheng-wang-nv]

Hi @Nic-Ma and @wyli , my initial thought is that the issue is due to the driver version, with driver version 510.47.03, docker 22.04 does not raise the same issue. For the ci/cd machine, is it possible to test with different driver version?

Updates: I checked more about this issue, and with Peixin's help, we found that it may not the cuda/driver/torch version/docker issue, same environment is used for the cron-docker test (https://github.com/Project-MONAI/MONAI/runs/6306012243?check_suite_focus=true) and the docker image is based on 22.04.
So far, the issue is avoided by adding more checks, but I'm not sure if there is any other issue within the env building part of cron-pip.

[wyli]

Updates: I checked more about this issue, and with Peixin's help, we found that it may not the cuda/driver/torch version/docker issue, same environment is used for the cron-docker test (https://github.com/Project-MONAI/MONAI/runs/6306012243?check_suite_focus=true) and the docker image is based on 22.04. So far, the issue is avoided by adding more checks, but I'm not sure if there is any other issue within the env building part of cron-pip.

thanks for looking into this, now it's clear that
reinstalling pytorch/torchvision in the docker breaks the apex customized module

MONAI/.github/workflows/cron.yml

Line 134 in 557c129

python -m pip uninstall -y torch torchvision

MONAI/.github/workflows/cron.yml

Line 149 in 557c129

python -m pip install $name[all]

and Yiheng's PR makes the call robust in this case #4241