How does NVCaffe MultiGPU Solver Works?

[linnanwang]

Hello,

I saw each GPU is assigned a solver; however, I 'm not sure how do you average the gradients from multiGPUs. Is it at each iterations, every GPU fetch the images of batch/GPUs and compute the gradient. Then the root solver retrieve the rest of gradients and average together. Then it goes to the next iteration. Is it correct? Thank you

[lukeyeager]

I'll point you to the original Multi-GPU pull request, which has a large discussion about implementation:
BVLC#2114

Note that some changes were made before it was finally merged into BVLC (and later, then this fork):
BVLC#2903