Windows Python Layer Fix #3915

[mlamarre]

This is a fix for the issue load python layer error on windows #3915

The changes are limited but they are not trivial. I don't expect this PR to get merged quickly. There might be a simpler more clever way to do this. Also, I'm not 100% Appveyor and Travis will pass because I tested this only on Windows.

The issue is that on Windows networks with Python layers can't be trained with the caffe command line tool. This error is returned:
'TypeError: No to_python (by-value) converter found for C++ type: class caffe::LayerParameter'

It seems that on Windows, the embedded interpreter can't convert C++ class that are wrapped in the _caffe module (DLL). I didn't find the root cause for this which is most likely in Boost.Python.

The fix is to embed the _caffe module inside the caffe command line application and to use the embedded module when it's present in pycaffe.py and init.py.

Embedding the _caffe module requires simple changes to _caffe.cpp, caffe.cpp and the project file (caffe.vcxproj)

The changes to pycaffe.py and init.py are simple but may feel a bit intrusive for a Windows-only fix.

[IsaacYangSLA]

@mlamarre, thanks for the information about this issue, especially the links in #3915. One thing I tried before was to register LayerParameter in _caffe.cpp, and check if LayerParameter is still registered before calling bp::object layer = module.attr(param.python_param().layer().c_str())(param); Just like what you mentioned, it showed LayerParameter was not registered. Then I registered that again before calling bp::object layer = module.attr(param.python_param().layer().c_str())(param); I got signature mismatch. Therefore, I got a feeling the link in your another post https://wiki.python.org/moin/boost.python/CrossExtensionModuleDependencies caused the issue we encountered in Windows environment.

[EhsanSHV]

Hi Mathieu,
Than you for your post. I have applied all changes as presented in " First part of a Python layer fix\n ..." but it only changes the error into "no python class is registered for c++ class, class caffe:: layerparameter". do you have any suggesstion ?

[mlamarre]

Force the use of embedded_caffe by editing python/caffe/init.py and python/caffe/pycaffe.py, i.e. remove the code that fallbacks to the original bindings. If embedded_caffe doesn't import, it's because something is not right with your caffe.exe build.

[EhsanSHV]

Dear Mathieu,
Thank you for your help. its working fine now :)

[willyd]

@mlamarre New windows branch maintainer here. I won't merge this as according to my tests training nets with caffe.exe with python layers works provided that we use a dynamically linked boost.python because the type registry is shared among python modules see this. I tested with my CMake build and it works. See the soon to be merged PR #3990.

[boxoq-rex]

@mlamarre Thank you for your post. I've forced the use of embedded_caffe as the aforementioned suggestion. And yet I got "No module named embedded_caffe". Maybe there's something wrong with my caffe.exe build. Do you have any suggestion on this? I really appreciate it.

[mlamarre]

@boxoq-rex I haven't touched Caffe in the last 6 months. @willyd maintains the windows branch it has pre-built binaries for many variants. If you need to build your own, starting from his CMake files which uses the dynamically linked boost.python is a better solution than my workaround.

[woshimo]

@EhsanSHV Hi, I also have the error "no python class is registered for c++ class, class caffe:: layerparameter", do you have any suggestion to fix it?

[xunhen]

Force the use of embedded_caffe by editing python/caffe/init.py and python/caffe/pycaffe.py, i.e. remove the code that fallbacks to the original bindings. If embedded_caffe doesn't import, it's because something is not right with your caffe.exe build.

When I change the code following your ,but it onlys changes the error into "no python class is registered for c++ class, class caffe:: layerparameter". And next you suggest Force the use of embedded_caffe,but what i should change?

[xunhen]

Dear Mathieu,
Thank you for your help. its working fine now :)

Dear,
Can you tell me how can i fix the problem about the registered