RMSprop clean up and rebase

[ronghanghu]

Rebased and adapted RMSprop implementation #1890 to the new solver interface #2518 and #1977. The original author is @erogol. Pulled against master instead of dev.

The RMSprop solver is based on G. Hinton's lecture (http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf). Param gradients are divided by average root mean square of gradients in recent batches. It can be seen as a mini-batch version of using only the sign of gradients.

Update rule:

MeanSquare(t) = rms_decay * MeanSquare(t-1) + (1 - rms_decay) * gradient(t)^2
param_update(t) = gradient(t) / (sqrt(MeanSquare(t)) + delta)

Momentum is not supported for RMSprop solver, as in #1890.

[erogol]

thanks for handling this :)

[jeffdonahue]

These should be declared as constants (e.g. const Dtype kRMSDecay = 0) like the other args to make the meaning clear.

[jeffdonahue]

Thanks @erogol for the original work and thanks @ronghanghu for the rebase. This looks good except as noted above.

[ronghanghu]

@jeffdonahue OK, I'll handle them. Thanks for the comments!

[ronghanghu]

Fixed those issues. I expect this PR to be merged after #2856 and #2782.

[jeffdonahue]

Cool, LGTM. @ronghanghu feel free to merge whenever it's easiest for you, before or after the other two PRs.

[ronghanghu]

Took a further rebase on #2866. Authorship preserved for @erogol in commit

Ready to merge.

[shelhamer]

Could you set the RMS decay here, instead of introducing the decay argument to least squares and snapshotting tests? Since it is unique to this solver I think it is best handled here.

[shelhamer]

@ronghanghu Sorry I didn't catch this earlier, but I have a suggestion for the RMS decay parameter in the tests. Instead of introducing another argument and setting it for every test, this param could be set by the RMSProp test class for encapsulation. Could you send a follow-up PR to make this change?

[ronghanghu]

@shelhamer Yes, I can send another PR to do that. Adam solver is also going to introduce a momentum2 parameter, which can be handle in the same way (put into InitSolver()).

Addressed in #2888.