Handle divergence/NaNs gracefully

[longjon]

Bad data scaling and/or bad learning rates often cause training to diverge, producing NaN values and losses that quickly propagate throughout the net.

While occasional divergence is a fact of life, Caffe could do a better job of handling this situation, both in terms of user experience (there is no point in continuing to perform gradient descent on a NaN loss), and in terms of code correctness (pooling layer and probably others do not handle NaN comparisons correctly, which actually causes memory to be written out-of-bounds due to a default argmax index of -1).

See the issue with pooling layer at #1333 (and thanks @tleyden for pointing out the crash).

[sguada]

I'm encountering a problem I never had before. So usually the loss either diverge soon and generate NaNs or start converging and never diverges again. However, recently in some experiments I'm encountering the problem that the loss start converging, and after a while 45K iterations diverges.
We need to do something about this, in addition to stop, maybe we could log what caused the NaN to appear.
Also maybe adding gradient clipping could help, @jeffdonahue could you do a PR with the gradient clipping?

[bhack]

I also found this problem with some network configuration and learning rate. Not at 45k but on some earlier iteration.