SolverPool for MultiGPU in Python

[cdoersch]

A simple interface which allows you to use the new MultiGPU functionality from inside python. Begin by calling pool=get_solver_pool(solver_file,gpu_list), where solver_file is a string for the file name of the solver definition prototxt, and gpu_list is a python list of ints specifying the gpus to use. pool will then have the methods pool.step(int), which lets you step all solvers in the pool in parallel for the specified number of iterations, and pool.solvers which is a list of python solver objects.

There is currently no documentation or tests. This has been well tested on an older version of the MultiGPU version of caffe. There have been several boilerplate changes since that version (most notably, P2Psync::run is no longer static, which changed how SolverPool interacts with it). I have updated this PR such that it compiles and the tests pass, but it is still possible that I made some mistake while updating the boilerplate. Feedback is appreciated.

@cypof parallel.cpp line 396 now calls SetDevice, since I couldn't find another good place to do this. This means that SetDevice gets called twice when you run tools/caffe, which is maybe a bit confusing. Let me know if I can get rid of the SetDevice call in tools/caffe.

[cypof]

Thanks @cdoersch. I would not remove SetDevice in caffe.cpp, as the root solver gets created after that call. I don't think it is confusing, and the call first checks the current device so it's OK to call it multiple times.

[DelightRun]

+1, That's really what we need.
So when will this PR be merged?

[cdoersch]

Someone just needs to update this PR and resolve the merge conflicts, and maybe add some testing code. I haven't had a need for multi-gpu recently, so I haven't kept up with it (I currently don't have time to do so unless it's a part of my research). It shouldn't be too hard to make it work, though.

[shelhamer]

Closing as replaced by #4563, but thanks for the pycaffe integration of the multi-GPU setup at the time.