CuFileDriver cannot be open

[wyli]

Describe the bug
running nvidia-smi gives a warning CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
https://github.com/Project-MONAI/MONAI/runs/2130671421?check_suite_focus=true#step:6:294

To Reproduce
appeared in several recent testing results, but all tests finished normally

[wyli]

could you help look into this? @Nic-Ma I saw this today but don't know the root cause

[Nic-Ma]

Hi @behxyz ,

I think you know something about this issue, could you please help take a look?

Thanks in advance.

[bhashemian]

Hi @wyli and @Nic-Ma,
It is not related to nvidia-smi and running nvidia-smi does not give this warning. This is an expected message from cuCIM. Since our docker does not have libcufile for GPU Direct Storage, it falls back to use POSIX IO, so it gives a warning. It doesn't have any effect on the functionality of our codes, so nothing to worry about. I talked to the developer and it can be removed in the next release but the implication of removing this message needs to be examined first.

[wyli]

ok, do you know which line is triggering this warning? there are many repeated warning messages of the same during the tests. if it's from cuCIM, is it possible to mute it unless we actually use the relevant module?

[bhashemian]

This message comes from C++ level of cuCIM, and we cannot silence it. It is produced right after import cucim in image_reader. Since there are many tests for image_reader, we will see it multiple times.
It's a warning that we should skip for now and maybe request to remove this warning from cucim library.

[wyli]

we should move the cucim import to line 652 in the reader as a workaround? https://github.com/Project-MONAI/MONAI/blob/master/monai/data/image_reader.py#L652

[wyli]

the notebooks are giving too many outputs

Executing:  22%|██▏       | 5/23 [00:06<00:24,  1.37s/cell]
Warning: :  39%|███▉      | 9/23 [00:17<00:30,  2.20s/cell][warning] CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.

Warning: :  52%|█████▏    | 12/23 [00:25<00:25,  2.30s/cell][warning] CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.

Warning: :  57%|█████▋    | 13/23 [00:32<00:30,  3.04s/cell][warning] CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.

Warning: :  65%|██████▌   | 15/23 [00:49<00:38,  4.80s/cell][warning] CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.
Warning:  CuFileDriver cannot be open. Falling back to use POSIX file IO APIs.

[bhashemian]

@wyli, I see. Which notebook is this?

[wyli]

./acceleration/transform_speed.ipynb from the tutorials, logs here: https://github.com/Project-MONAI/MONAI/runs/2130772126?check_suite_focus=true

[bhashemian]

Right, I checked the notebook and it uses LoadImage, and since it uses multi-processing it appears multiple times. In single processing, it should appear only once.

Moving the import to where you suggested break the functionality of the __init__. Let me see what we can do.

[bhashemian]

Looking into the workaround, I realized in order for displacing cucim import works is to remove WSIReader from LoadImage. We are not using cucim or openslide in the same way that we are using other image readers (we rarely load the whole slide image), so I think #1695 was unnecessary and is causing problem for what we are not using. I'll submit a PR to fix this as a temporary workaround.

[bhashemian]

@wyli, @Nic-Ma
I talked to Gigon (the cucim developer) again, and he kindly agreed to create and push a minor release to PyPI for us to remove this message. I'll update the dependencies once it is published.

[wyli]

thanks @behxyz , I also agree with you that we won't have many use cases for #1695 , it's somewhat unnecessary

[bhashemian]

thanks @behxyz , I also agree with you that we won't have many use cases for #1695 , it's somewhat unnecessary

I submitted a PR to address this just in case, #1796.