Deduplicate solver regularization, logging, and local rates and decays

[shelhamer]

This simplifies the solver code by de-duplicating shared logic.

• ca81667 is taken from Refactor solvers regularization and logging code #2397 by @cypof but renames Solver::Iteration() to Solver::MakeUpdate()
  to verb.
• a85f7f1 is a minor follow-up to simplify Decouple the computational batch size and minibatch size by accumulating gradients #1977

I plan to merge this shortly to make way for an updated #1977.

[jeffdonahue]

I just looked over this -- looks great! This refactoring was much needed. Thanks @cypof and @shelhamer. Only issue I see is I'm not sure about the verb in MakeUpdate -- I think DoUpdate or PerformUpdate would be more suggestive of what it does. Personally when I first saw the name MakeUpdate in the PR text, I guessed its purpose was that of ComputeUpdateValue.

[jeffdonahue]

Actually, I guess it is replacing ComputeUpdateValue, but also doing net_->Update() at the end. Would it not be better to keep calling it ComputeUpdateValue and just keep net_->Update at the end of the body of Step?

[cypof]

No the net_->Update() needs to be in, so that it is not executed by solvers that are not of type SGDSolver. This way, only the root solver in a parallel setup will apply the update. That was actually the race I introduced when I split the big PR into small ones. Another name could be ApplyGradients?

[jeffdonahue]

Okay, I see -- thanks for the explanation @cypof. I like Apply but also like Update; how about ApplyUpdate?

[shelhamer]

Thanks for the comments @cypof and @jeffdonahue. I went with ApplyUpdate. Please merge if this looks right to you, Jeff.

[shelhamer]

p.s. ignore the travis push check -- the travis Pr check is the one to heed. The push check was triggered by my accidental push to BVLC/caffe and then my deleting the branch made it fail.

[jeffdonahue]

Cool, LGTM