Windows link failures in Debug mode

[larroy]

Description

Some template functions need to be defined in the compilation unit, right now can't build in debug mode and thus fix issues with GPU builds in windows.

[473/482] Linking CXX shared library mxnet_52.dll
FAILED: mxnet_52.dll mxnet_52.lib
cmd.exe /C "cd . && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_dll --intdir=CMakeFiles\mxnet_52.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100162~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100162~1.0\x64\mt.exe --manifests  -- C:\PROGRA~2\MICROS~1\2017\COMMUN~1\VC\Tools\MSVC\1
413~1.261\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\mxnet_52.rsp  /out:mxnet_52.dll /implib:mxnet_52.lib /pdb:mxnet_52.pdb /dll /version:0.0 /machine:x64  /debug /INCREMENTAL   && cd ."
LINK Pass 1: command "C:\PROGRA~2\MICROS~1\2017\COMMUN~1\VC\Tools\MSVC\1413~1.261\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\mxnet_52.rsp /out:mxnet_52.dll /implib:mxnet_52.lib /pdb:mxnet_52.pdb /dll /version:0.0 /machine:x64 /debug /INCREMENTAL /MANIFEST /MANIFESTFILE:CMakeFiles\
mxnet_52.dir/intermediate.manifest CMakeFiles\mxnet_52.dir/manifest.res" failed (exit code 1120) with the following output:
   Creating library mxnet_52.lib and object mxnet_52.exp
image_random.cc.obj : error LNK2019: unresolved external symbol "void __cdecl mxnet::op::image::ToTensorImplCUDA<class mshadow::bfloat::bf16_t,struct mshadow::Tensor<struct mshadow::gpu,3,class mshadow::bfloat::bf16_t>,struct mshadow::Tensor<struct mshadow::gpu,3,float> >(struct msh
adow::Stream<struct mshadow::gpu> *,struct mshadow::Tensor<struct mshadow::gpu,3,class mshadow::bfloat::bf16_t>,struct mshadow::Tensor<struct mshadow::gpu,3,float>,int,float)" (??$ToTensorImplCUDA@Vbf16_t@bfloat@mshadow@@U?$Tensor@Ugpu@mshadow@@$02Vbf16_t@bfloat@2@@3@U?$Tensor@Ugpu@
mshadow@@$02M@3@@image@op@mxnet@@YAXPEAU?$Stream@Ugpu@mshadow@@@mshadow@@U?$Tensor@Ugpu@mshadow@@$02Vbf16_t@bfloat@2@@4@U?$Tensor@Ugpu@mshadow@@$02M@4@HM@Z) referenced in function "void __cdecl mxnet::op::image::ToTensorOpForward<struct mshadow::cpu>(struct nnvm::NodeAttrs const &,s
truct mxnet::OpContext const &,class std::vector<class mxnet::TBlob,class std::allocator<class mxnet::TBlob> > const &,class std::vector<enum mxnet::OpReqType,class std::allocator<enum mxnet::OpReqType> > const &,class std::vector<class mxnet::TBlob,class std::allocator<class mxnet:
:TBlob> > const &)" (??$ToTensorOpForward@Ucpu@mshadow@@@image@op@mxnet@@YAXAEBUNodeAttrs@nnvm@@AEBUOpContext@2@AEBV?$vector@VTBlob@mxnet@@V?$allocator@VTBlob@mxnet@@@std@@@std@@AEBV?$vector@W4OpReqType@mxnet@@V?$allocator@W4OpReqType@mxnet@@@std@@@7@2@Z)
image_random.cc.obj : error LNK2019: unresolved external symbol "void __cdecl mxnet::op::image::ToTensorImplCUDA<class mshadow::bfloat::bf16_t,struct mshadow::Tensor<struct mshadow::gpu,4,class mshadow::bfloat::bf16_t>,struct mshadow::Tensor<struct mshadow::gpu,4,float> >(struct msh
adow::Stream<struct mshadow::gpu> *,struct mshadow::Tensor<struct mshadow::gpu,4,class mshadow::bfloat::bf16_t>,struct mshadow::Tensor<struct mshadow::gpu,4,float>,int,float)" (??$ToTensorImplCUDA@Vbf16_t@bfloat@mshadow@@U?$Tensor@Ugpu@mshadow@@$03Vbf16_t@bfloat@2@@3@U?$Tensor@Ugpu@
mshadow@@$03M@3@@image@op@mxnet@@YAXPEAU?$Stream@Ugpu@mshadow@@@mshadow@@U?$Tensor@Ugpu@mshadow@@$03Vbf16_t@bfloat@2@@4@U?$Tensor@Ugpu@mshadow@@$03M@4@HM@Z) referenced in function "void __cdecl mxnet::op::image::ToTensorOpForward<struct mshadow::cpu>(struct nnvm::NodeAttrs const &,s
truct mxnet::OpContext const &,class std::vector<class mxnet::TBlob,class std::allocator<class mxnet::TBlob> > const &,class std::vector<enum mxnet::OpReqType,class std::allocator<enum mxnet::OpReqType> > const &,class std::vector<class mxnet::TBlob,class std::allocator<class mxnet:
:TBlob> > const &)" (??$ToTensorOpForward@Ucpu@mshadow@@@image@op@mxnet@@YAXAEBUNodeAttrs@nnvm@@AEBUOpContext@2@AEBV?$vector@VTBlob@mxnet@@V?$allocator@VTBlob@mxnet@@@std@@@std@@AEBV?$vector@W4OpReqType@mxnet@@V?$allocator@W4OpReqType@mxnet@@@std@@@7@2@Z)
image_random.cc.obj : error LNK2019: unresolved external symbol "void __cdecl mxnet::op::image::NormalizeImplCUDA<class mshadow::bfloat::bf16_t>(struct mshadow::Stream<struct mshadow::gpu> *,class mshadow::bfloat::bf16_t const *,class mshadow::bfloat::bf16_t *,int,int,int,int,int,fl
oat,float,float,float,float,float)" (??$NormalizeImplCUDA@Vbf16_t@bfloat@mshadow@@@image@op@mxnet@@YAXPEAU?$Stream@Ugpu@mshadow@@@mshadow@@PEBVbf16_t@bfloat@4@PEAV564@HHHHHMMMMMM@Z) referenced in function "void __cdecl mxnet::op::image::NormalizeOpForward<struct mshadow::cpu>(struct
 nnvm::NodeAttrs const &,struct mxnet::OpContext const &,class std::vector<class mxnet::TBlob,class std::allocator<class mxnet::TBlob> > const &,class std::vector<enum mxnet::OpReqType,class std::allocator<enum mxnet::OpReqType> > const &,class std::vector<class mxnet::TBlob,class s
td::allocator<class mxnet::TBlob> > const &)" (??$NormalizeOpForward@Ucpu@mshadow@@@image@op@mxnet@@YAXAEBUNodeAttrs@nnvm@@AEBUOpContext@2@AEBV?$vector@VTBlob@mxnet@@V?$allocator@VTBlob@mxnet@@@std@@@std@@AEBV?$vector@W4OpReqType@mxnet@@V?$allocator@W4OpReqType@mxnet@@@std@@@7@2@Z)
image_random.cc.obj : error LNK2019: unresolved external symbol "void __cdecl mxnet::op::image::NormalizeBackwardImplCUDA<class mshadow::bfloat::bf16_t>(struct mshadow::Stream<struct mshadow::gpu> *,class mshadow::bfloat::bf16_t const *,class mshadow::bfloat::bf16_t *,int,int,int,in
t,int,float,float,float)" (??$NormalizeBackwardImplCUDA@Vbf16_t@bfloat@mshadow@@@image@op@mxnet@@YAXPEAU?$Stream@Ugpu@mshadow@@@mshadow@@PEBVbf16_t@bfloat@4@PEAV564@HHHHHMMM@Z) referenced in function "void __cdecl mxnet::op::image::NormalizeOpBackward<struct mshadow::cpu>(struct nnv
m::NodeAttrs const &,struct mxnet::OpContext const &,class std::vector<class mxnet::TBlob,class std::allocator<class mxnet::TBlob> > const &,class std::vector<enum mxnet::OpReqType,class std::allocator<enum mxnet::OpReqType> > const &,class std::vector<class mxnet::TBlob,class std::
allocator<class mxnet::TBlob> > const &)" (??$NormalizeOpBackward@Ucpu@mshadow@@@image@op@mxnet@@YAXAEBUNodeAttrs@nnvm@@AEBUOpContext@2@AEBV?$vector@VTBlob@mxnet@@V?$allocator@VTBlob@mxnet@@@std@@@std@@AEBV?$vector@W4OpReqType@mxnet@@V?$allocator@W4OpReqType@mxnet@@@std@@@7@2@Z)
resize.cc.obj : error LNK2019: unresolved external symbol "void __cdecl mxnet::op::image::ResizeImplCUDA<class mshadow::bfloat::bf16_t,struct mshadow::Tensor<struct mshadow::gpu,3,class mshadow::bfloat::bf16_t>,float>(struct mshadow::Stream<struct mshadow::gpu> *,struct mshadow::Ten
sor<struct mshadow::gpu,3,class mshadow::bfloat::bf16_t>,struct mshadow::Tensor<struct mshadow::gpu,3,class mshadow::bfloat::bf16_t>)" (??$ResizeImplCUDA@Vbf16_t@bfloat@mshadow@@U?$Tensor@Ugpu@mshadow@@$02Vbf16_t@bfloat@2@@3@M@image@op@mxnet@@YAXPEAU?$Stream@Ugpu@mshadow@@@mshadow@@
U?$Tensor@Ugpu@mshadow@@$02Vbf16_t@bfloat@2@@4@1@Z) referenced in function "void __cdecl mxnet::op::image::Resize<struct mshadow::cpu>(struct nnvm::NodeAttrs const &,struct mxnet::OpContext const &,class std::vector<class mxnet::TBlob,class std::allocator<class mxnet::TBlob> > const
 &,class std::vector<enum mxnet::OpReqType,class std::allocator<enum mxnet::OpReqType> > const &,class std::vector<class mxnet::TBlob,class std::allocator<class mxnet::TBlob> > const &)" (??$Resize@Ucpu@mshadow@@@image@op@mxnet@@YAXAEBUNodeAttrs@nnvm@@AEBUOpContext@2@AEBV?$vector@VT
Blob@mxnet@@V?$allocator@VTBlob@mxnet@@@std@@@std@@AEBV?$vector@W4OpReqType@mxnet@@V?$allocator@W4OpReqType@mxnet@@@std@@@7@2@Z)
resize.cc.obj : error LNK2019: unresolved external symbol "void __cdecl mxnet::op::image::ResizeImplCUDA<class mshadow::bfloat::bf16_t,struct mshadow::Tensor<struct mshadow::gpu,4,class mshadow::bfloat::bf16_t>,float>(struct mshadow::Stream<struct mshadow::gpu> *,struct mshadow::Ten
sor<struct mshadow::gpu,4,class mshadow::bfloat::bf16_t>,struct mshadow::Tensor<struct mshadow::gpu,4,class mshadow::bfloat::bf16_t>)" (??$ResizeImplCUDA@Vbf16_t@bfloat@mshadow@@U?$Tensor@Ugpu@mshadow@@$03Vbf16_t@bfloat@2@@3@M@image@op@mxnet@@YAXPEAU?$Stream@Ugpu@mshadow@@@mshadow@@
U?$Tensor@Ugpu@mshadow@@$03Vbf16_t@bfloat@2@@4@1@Z) referenced in function "void __cdecl mxnet::op::image::Resize<struct mshadow::cpu>(struct nnvm::NodeAttrs const &,struct mxnet::OpContext const &,class std::vector<class mxnet::TBlob,class std::allocator<class mxnet::TBlob> > const
 &,class std::vector<enum mxnet::OpReqType,class std::allocator<enum mxnet::OpReqType> > const &,class std::vector<class mxnet::TBlob,class std::allocator<class mxnet::TBlob> > const &)" (??$Resize@Ucpu@mshadow@@@image@op@mxnet@@YAXAEBUNodeAttrs@nnvm@@AEBUOpContext@2@AEBV?$vector@VT
Blob@mxnet@@V?$allocator@VTBlob@mxnet@@@std@@@std@@AEBV?$vector@W4OpReqType@mxnet@@V?$allocator@W4OpReqType@mxnet@@@std@@@7@2@Z)
np_multinomial_op.cc.obj : error LNK2019: unresolved external symbol "void __cdecl mxnet::op::CheckPvalGPU<class mshadow::bfloat::bf16_t>(struct mxnet::OpContext const &,class mshadow::bfloat::bf16_t *,int)" (??$CheckPvalGPU@Vbf16_t@bfloat@mshadow@@@op@mxnet@@YAXAEBUOpContext@1@PEAV
bf16_t@bfloat@mshadow@@H@Z) referenced in function "void __cdecl mxnet::op::NumpyMultinomialForward<struct mshadow::cpu>(struct nnvm::NodeAttrs const &,struct mxnet::OpContext const &,class std::vector<class mxnet::TBlob,class std::allocator<class mxnet::TBlob> > const &,class std::
vector<enum mxnet::OpReqType,class std::allocator<enum mxnet::OpReqType> > const &,class std::vector<class mxnet::TBlob,class std::allocator<class mxnet::TBlob> > const &)" (??$NumpyMultinomialForward@Ucpu@mshadow@@@op@mxnet@@YAXAEBUNodeAttrs@nnvm@@AEBUOpContext@1@AEBV?$vector@VTBlo
b@mxnet@@V?$allocator@VTBlob@mxnet@@@std@@@std@@AEBV?$vector@W4OpReqType@mxnet@@V?$allocator@W4OpReqType@mxnet@@@std@@@6@2@Z)
mxnet_52.dll : fatal error LNK1120: 7 unresolved externals
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "C:\Users\Administrator\mxnet\ci\build_windows.py", line 288, in <module>
    sys.exit(main())
  File "C:\Users\Administrator\mxnet\ci\build_windows.py", line 276, in main
    windows_build(args)
  File "C:\Users\Administrator\mxnet\ci\build_windows.py", line 191, in windows_build
    check_call(cmd, shell=True)
  File "C:\Python37\lib\subprocess.py", line 328, in check_call

Error Message

(Paste the complete error message. Please also include stack trace by setting environment variable DMLC_LOG_STACK_TRACE_DEPTH=10 before running your script.)

To Reproduce

(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)

Steps to reproduce

(Paste the commands you ran that produced the error.)

Add CMAKE_BUILD_TYPE=Debug in build_windows.py and build -f WIN_GPU

What have you tried to solve it?

Removed the offending operators and was able to compile successfully.

Environment

We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:

curl --retry 10 -s https://raw.githubusercontent.com/dmlc/gluon-nlp/master/tools/diagnose.py | python

# paste outputs here

[larroy]

@mxnet-label-bot add [breaking]

[leezu]

@yajiedesign is this error due to the Workaround for DLL size limitation on Windows (#16980) ?

[yajiedesign]

It shouldn't be. The size limit won't go wrong at this step.The problem is really strange. Check the subprojects are updated and recompile after clear.

[yajiedesign]

In addition, you can try to generate vs project to build.

[ChaiBapchya]

@larroy so in your opinion what can be done to fix this?
While trying to resolve windows-gpu issue here #17916 getting similar errors on windows-cpu side (migrating from vs2015 to 2019 & toolchain from 32bit to 64bit) hence curious

[flygragon]

Is there the same bug with release version on windows？ I encountered the same problem when building debug library on windows.

[larroy]

I think this can be fixed easily by moving the definition / inserting declaration between header and cc files. I fixed the first error like that, but didn't have time to fix all. This will make the symbol present in the compilation unit so the linker can find it and the error will go away. It's simple to fix.

[sugurbaris]

@larroy What do you mean by "moving the definition / inserting declaration between header and cc files."? I encountered a similar error and couldn't find a solution for this. (for x64 build on VS 2015)

[larroy]

I think the function needs to be forward declared or defined in a way that the windows compiler sees the symbol, it's not the case now.

[sa-mustafa]

I think the function needs to be forward declared or defined in a way that the windows compiler sees the symbol, it's not the case now.

The functions are already forward-declared. For example, ToTensorImplCUDA/NormalizeImplCUDA/NormalizeBackwardImplCUDA are properly declared in image_random-inl.h.
I don't know how the release build succeeds without nagging. Anyway, to work around it you should manually instantiate the template functions in the corresponding cuda ('.cu') file. For example, add the following lines to image_random.cu :

template
void ToTensorImplCUDA
    <mshadow::bfloat::bf16_t, Tensor<gpu, 3, mshadow::bfloat::bf16_t>, Tensor<gpu, 3, float>>
    (mshadow::Stream<gpu>*,
      const Tensor<gpu, 3, mshadow::bfloat::bf16_t>,
      const Tensor<gpu, 3, float>,
      const int, 
      const float);

template
void ToTensorImplCUDA
    <mshadow::bfloat::bf16_t, Tensor<gpu, 4, mshadow::bfloat::bf16_t>, Tensor<gpu, 4, float>>
    (mshadow::Stream<gpu>*,
      const Tensor<gpu, 4, mshadow::bfloat::bf16_t>,
      const Tensor<gpu, 4, float>,
      const int, 
      const float);

Do this for other functions too (CheckPvalGPU in np_multinomial_op.cu and ResizeImplCUDA in resize.cu, NormalizeImplCUDA/NormalizeBackwardImplCUDA in image_random.cu).
That's all.

[tomicfish]

I had the same problem.
configuration：
apache-mxnet-src-1.7.0-incubating, win10(64bit) +vs2017+cuda10.1+cudnn v8.1, openblas v0.2.14, opencv-4.5.1, Anaconda3(python 3.7).
I generated vs2017 project to build. In Release mode, it worked fine. In Debug mode, I encountered the same problem.

1>------ Start build Project: mxnet_75, Configuration: Debug x64 ------
1> Creating library D:/mxnet/apache-mxnet-src-1.7.0-incubating/buildcpp/Debug/mxnet_75.lib and object D:/mxnet/apache-mxnet-src-1.7.0-incubating/buildcpp/Debug/mxnet_75.exp
1>image_random.cc.obj : error LNK2019: unresolved external symbol "void __cdecl mxnet::op::image::ToTensorImplCUDA<class mshadow::bfloat::bf16_t,struct mshadow::Tensor<struct mshadow::gpu,3,class mshadow::bfloat::bf16_t>,struct mshadow::Tensor<struct mshadow::gpu,3,float> >(struct mshadow::Stream *,struct mshadow::Tensor<struct mshadow::gpu,3,class mshadow::bfloat::bf16_t>,struct mshadow::Tensor<struct mshadow::gpu,3,float>,int,float)" (??$ToTensorImplCUDA@Vbf16_t@bfloat@mshadow@@U?$Tensor@Ugpu@mshadow@@$02Vbf16_t@bfloat@2@@3@U?$Tensor@Ugpu@mshadow@@$02M@3@@image@op@mxnet@@YAXPEAU?$Stream@Ugpu@mshadow@@@mshadow@@U?$Tensor@Ugpu@mshadow@@$02Vbf16_t@bfloat@2@@4@U?$Tensor@Ugpu@mshadow@@$02M@4@HM@Z)，referenced in function "void __cdecl mxnet::op::image::ToTensorOpForward(struct nnvm::NodeAttrs const &,struct mxnet::OpContext const &,class std::vector<class mxnet::TBlob,class std::allocator > const &,class std::vector<enum mxnet::OpReqType,class std::all
ocator > const &,class std::vector<class mxnet::TBlob,class std::allocator > const &)" (??$ToTensorOpForward@Ucpu@mshadow@@@image@op@mxnet@@YAXAEBUNodeAttrs@nnvm@@AEBUOpContext@2@AEBV?$vector@VTBlob@mxnet@@v?$allocator@VTBlob@mxnet@@@std@@@std@@aebv?$vector@W4OpReqType@mxnet@@v?$allocator@W4OpReqType@mxnet@@@std@@@7@2@Z)