[Torch, CI] Upgrade to PyTorch 1.10

[masahi]

PyTorch 1.10 has just been released. We are now at PT 1.7 which we upgraded to one year ago. I think it is a good time for another update.
https://pytorch.org/blog/pytorch-1.10-released/

Recently I've been sensing growing interest in more tight integration with PyTorch, for example using TVM as a backend in a PyTorch-based application (#8777) or using TVM for training acceleration. I'm looking forward to seeing more development in this space, in addition to continuing to support the traditional usage of TVM as a e2e inference solution for PT models. From these point of view, I believe having our PT support up-to-date and actively maintained is increasingly important.

• Figure out what's broken, any API change
• Investigate the symbol crash issue [Bug] PyTorch and TVM loading problem due to conflicting LLVM symbols #9362
• Send out necessary fix
• Upgrade the CI GPU image

[masahi]

Tracing MaskRCNN is broken with this release pytorch/vision#4158 (comment)

cc @hgt312

UPDATE: If I import PyTorch before TVM, the error I got during tracing is gone. MaskRCNN still traces correctly.

[masahi]

With some simple fix, MaskRCNN can be correctly converted to Relay and pass the test. Other tests are also looking good, only less than five of them are broken but seems easy to fix:

FAILED test_forward.py::test_forward_deform_conv - tvm._ffi.base.TVMError: Traceback (most recent call last):
FAILED test_forward.py::test_forward_linspace - tvm._ffi.base.TVMError: Traceback (most recent call last):
FAILED test_forward.py::test_forward_nll_loss - NotImplementedError: The following operators are not implemented: ['aten::nll_loss_nd']

So it looks like it's going to be an easy upgrade.

[lhutton1]

Tracing MaskRCNN is broken with this release pytorch/vision#4158 (comment)

cc @hgt312

UPDATE: If I import PyTorch before TVM, the error I got during tracing is gone. MaskRCNN still traces correctly.

Thanks for looking into the upgrade @masahi, this will be very useful! I've also experienced the import order issue while attempting to trace a PyTorch model. For example,

import tvm.driver.tvmc as tvmc
import torch

traced_model = tvmc.load("mobilenetv2_quantized.pth", shape_dict={"input0": [1, 3, 224, 224]})

Results in the following:

munmap_chunk(): invalid pointer
Aborted (core dumped)

Yet after changing the order of imports, it works perfectly fine. The reason I bring this up is because sometimes its not possible to simply swap the order of imports to fix the issue. For example, someone using tvmc directly from the command line. Are you aware of any other ways this could be fixed?

[masahi]

@lhutton1 Unfortunately, I don't have a solution for this problem. The error I got with PT 1.10 + TVM seemed related to LLVM (it says Unable to find target for this triple (no targets are registered) and dies during tracing). I also got Aborted (core dumped) error upon process exit.

Also I've never hit this problem with older PyTorch versions + TVM. But ONNX + PT did cause me some pain in the past, which was also fixed by swapping the import order, onnx/onnx#2394 (comment). I wonder if this is the same dynamic loader related issue. I'd love to dig deep if this is something we can fix from our side.

Related issues from PT repo
pytorch/pytorch#2507
pytorch/pytorch#19739

[masahi]

It might be due to our use of RTLD_GLOBAL, e.g.

tvm/python/tvm/_ffi/base.py

Line 57 in dfe4ceb

lib = ctypes.CDLL(lib_path[0], ctypes.RTLD_GLOBAL)

@lhutton1 Try replacing above with ctypes.CDLL(lib_path[0]), it fixes the issue on my end (mask rcnn test works without swapping the import order).

See related issues in other repos
dmlc/dgl#2255
pytorch/pytorch#28536
pytorch/pytorch#3059

[lhutton1]

Thanks @masahi for looking into this, it's interesting to see many other repos having similar issues. Apologies for checking late, but I can confirm the change you suggested above fixes the issue :) - Just to add for completeness, the PyTorch version I'm using currently is 1.8.1 .

[masahi]

Upgrade to PT v1.10 is now complete.