load python layer error on windows

[xingyizhou]

I have successfully set up the caffe windows on a newly installed windows10, following the instruction. When I use python interface and import caffe, some warnings occur:
>>> import caffe caffe\pycaffe.py:13: RuntimeWarning: to-Python converter for class boost::shared_ptr<class caffe::Net<float> > already registered; second conversion method ignored. from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \ caffe\pycaffe.py:13: RuntimeWarning: to-Python converter for class boost::shared_ptr<class caffe::Blob<float> > already registered; second conversion method ignored. from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \ caffe\pycaffe.py:13: RuntimeWarning: to-Python converter for class boost::shared_ptr<class caffe::Solver<float> > already registered; second conversion method ignored. from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
But it can load net and solver anyway.

However, when I load a net with a python layer, an error occurs
'TypeError: No to_python (by-value) converter found for C++ type: class caffe::LayerParameter'

I am using Anaconda2.4 and I have also tried Miniconda2, it reports the same problem. All other settings are set as default. It seems the problem comes from boost, but how to fix it? Any analyses or suggestions are highly appreciated.

[tinyloop]

I met the same issue.

[tinyloop]

I am using Miniconda2 and boost 1_59_0. The same error occured.

[TomC868]

I noticed that the April 28th, 2016 commit fixed the pycaffe runtime warnings you were getting.

However, I'm still getting the same error you were when I load a net with a Python layer:
'TypeError: No to_python (by-value) converter found for C++ type: class caffe::LayerParameter'

[IsaacYangSLA]

I have the same 'TypeError' issue. However, it only happened when I ran caffe.exe with Python Layer. The same solver.prototxt can be loaded when I ran python, import caffe and created an instance of solver with that file.

[TomC868]

Isaac's comment is right. Only being able to use python input layers with python is probably expected, I just didn't think about that.

[ferrouswheel]

I also ran into this when working on a custom data layer in Python.

My environment is:

• Windows 10.
• System python is Python 2.7.3 from python.org
• However I was using Anaconda (python 2.7.11), and configured it for use using the Microsoft branch of caffe (by editing CommonSettings.props). Microsoft's branch uses boost python 1.59 from nuget.

I thought the error was due to Anaconda being a different version that didn't match the version that boost.python was compiled against. So I tracked down the caffe-required python libs from http://www.lfd.uci.edu/~gohlke/pythonlibs/ and installed them in my system python. Then I recompiled caffe from scratch.

This didn't help, but I did try running the tests and they all worked fine! Curious - at least I had something to work with. Then I found this issue... phew!

@TomC868 I don't think this should be considered expected behaviour. From a user or new developer perspective it'd be quite unintuitive. If I'm using Python layers then the caffe executable should be able to train/test nets with them, or give an obvious error telling the user it's not possible to use them outside of python.

[mlamarre]

I investigated this further. This may be a windows only issue.
I can train a network with a Python layer in Ubuntu using
caffe train -solver solver.prototxt
The same file gives the error reported by @TomC868
TypeError: No to_python (by-value) converter found for C++ type: class caffe::LayerParameter
I tried adding boost::ref in layer_factory.cpp like this:
bp::object layer = module.attr(param.python_param().layer().c_str())(boost::ref(param));
This only changes the error message to:
TypeError: No Python class registered for C++ class class caffe::LayerParameter
Following the Boost.Python answer by Flexo (7 votes) found here:
https://stackoverflow.com/questions/9040669/how-can-i-implement-a-c-class-in-python-to-be-called-by-c
I added this declaration in caffe.cpp just before the main():

#ifdef WITH_PYTHON_LAYER
BOOST_PYTHON_MODULE(_caffe) {
    bp::class_<caffe::LayerParameter>("LayerParameter", bp::no_init);
}
#endif

This had no effect on the error. This was a long shot since this is not the same scenario, the module _caffe is external while in the stackoverflow example it's embedded in the executable.

Since this works on Ubuntu, I wonder if this is related to this text:
https://wiki.python.org/moin/boost.python/CrossExtensionModuleDependencies

I will continue trying to solve this for a couple of hours before throwing the towel.

Is there a difference between the main and windows branches that I'm not aware of?

Did someone try boost python 1.61 with the windows branch?

[willyd]

Please see my comment about this in PR #4424.

[willyd]

Please confirm that the issue persists with the CMake build.

[mlamarre]

For my part, I'm not using Caffe anymore and the new build is so different, it's not something I can test quickly. If the issue author is not responding, you can close this.

[s0302102]

@willyd Although this bug seems to be fixed by #4424 (or maybe on 3 May, 2016,can't remember which PR, just use BP_REGISTER_PTR(....) something like this), but actually this bug still exists in Windows Caffe. If I use batch script to run the training (with layers in python), it will work. However, if I use VS Debug mode, then the program will crash and the same problem occurs.

[GuitarYourself]

@s0302102 can you share your detail in batch script way to run training with python layers? I use .bat and the error still exist!
My bat file is as follows:
SET GLOG_logtostderr=1
caffe.exe train --solver=solver.prototxt
pause