Multi-GPU

[shelhamer]

This is my packaging of #2114 for merge. I figured @cypof @thatguymike and company had made plenty of revisions and that I could help.

This PR is ready to use for data parallel training of networks but

• parallel IO is only coordinated for lmdb / leveldb through a DataReader
• it cannot be satisfactorily tested with the current design

which are resolved for merge by #2903.

• fix multi-GPU solver test failures see Multi-GPU #2870 (comment)
  • ensure test data determinism Test solvers on fixed hdf5 data #2887
  • make ParallelDataLayer interface in place of DataReader @ronghanghu
  • test single solver / multi-solver batch equivalence
• propagate the singleton state to threads without member variables
• adopt persistent check left as TODO
• reconcile with Fix weight sharing #2866
• replace CHECK(false) with LOG(FATAL)
• drop timing code
• squash history to clear fixes and such

@cypof @thatguymike @longjon @jeffdonahue please take a look.
@cdoersch could you fire up your parallel training test again?

Reviews and testing by the community are welcome!

[thatguymike]

All of my quick sanity tests are passing. Despite knowing that by default this is weak scaling, e.g. the specified batch size in the train_val.prototxt is multiplied by the number of GPUs you choose to run on, I forgot that when validating accuracy graphs. I still fear that is going to bite users.

[cypof]

OK, training works for me. The thread launch code is much better without the fields, that's great.

[shelhamer]

Thanks for testing @thatguymike and @cypof. My short test worked so once we hear from @cdoersch about the ec2 test I think this is ready to merge.

[buaaliyi]

must add 'timer.Start();' here to restart timer, or line 266 that timing for grads maybe incorrect.

[shelhamer]

Added to line 224 before forward + backward, thanks.

[cdoersch]

Training seems to be working fine on ec2.

[shelhamer]

After discussion with @longjon we decided the timing code is too intrusive to bundle in this change. I have stripped it but archived the branch with timing at shelhamer/caffe:time-multi_gpu. It could be re-introduced in a future PR.

[jeffdonahue]

probably a bit late to comment on this, and to me not necessary for merge, but these conditional LOG(INFO) calls could be made a bit more compact using LOG_IFs, e.g. LOG_IF(INFO, Caffe::root_solver()) << "Iteration..."

[jeffdonahue]

This call to params() and the two other calls below should be replaced with learnable_params() after #2866, I think? (I was debating whether the public params() method should just be removed, or if params() should just return learnable_params_, or...)

[shelhamer]

Agreed. Making the switch seems to have no effect though and I have the same test failures before and after.

[shelhamer]

@cypof @thatguymike it turns out #2114 was not rigorously checking solver updates; see #2114 (comment). Fixing the test net targets reveals that all the LeastSquaresUpdate multi-GPU tests fail. I still need to peer at this more closely to see if the issue is with multi-GPU itself or a subtlety of RNG in the tests, but this is now blocked on figuring it out. That is, the multiple solvers could be drawing targets that are not equivalent to the sequential order.

Apart from the tests, my experiments to check parallel training on real nets make progress so there's hope.

#2866 is not the problem as the same failures show up in the multi-GPU branch before the latest rebase when the test is fixed. This can be seen in shelhamer/caffe:old-multi_gpu.

[shelhamer]

I'm fairly positive this is a test artifact due to the random Gaussian targets. The multiple solvers can't reproduce random draws equivalent to the single solver sequence:

• The worker solvers don't inherit the root seed in the current test since the seed is set in the singleton, and not the solver param (test_gradient_based_solver.cpp:183). They make their own nondeterministic draws.
• If the test is rewritten to make the workers inherit the seed, they'll all draw the same targets and still be wrong w.r.t. to single solver equivalent.

The solution seems to be making the solver tests take fixed external data, such as the hdf5 data used in the sample_data.h5 used in the HDF5DataLayerTest. See #2887

[shelhamer]

This is now based on #2887 but the multi-GPU solver tests still fail. I believe this is because DataReader only knows lmdb + leveldb so the hdf5 inputs are identical among the WorkerSolvers. This should be checked, and then DataReader could be extended to hdf5. However this does raise issues with the DataReader design since layers that do / do not support it will have different behavior for parallelism.

[cypof]

I was off yesterday, but looking at it now.

[shelhamer]

Everyone see #2903 for the rigorously tested and passing multi-GPU branch. @ronghanghu has developed a parallel data layer solution.

[ronghanghu]

Merged in #2903