Lrelu layer

[qipeng]

Implemented the Leaky ReLU unit described in this paper

Maas, Andrew L., Awni Y. Hannun, and Andrew Y. Ng. "Rectifier nonlinearities improve neural network acoustic models." ICML Workshop on Deep Learning for Audio, Speech, and Language Processing. 2013.

which shares similar sparse activation properties with the ReLU, but was shown easier to optimize.

[jeffdonahue]

Thanks for this contribution @qipeng!

Unless there is a noticeable performance impact on the existing ReLU layer, I think this would be better implemented as a simple generalization of ReLU than a new layer. In particular you could add to caffe.proto a message ReLUParameter which has a field optional float negative_coeff = ... [default = 0]. This also needs unit tests (see src/caffe/test/test_neuron_layer.cpp). Will merge if these changes are made and tests pass. Thanks!

[qipeng]

Hi Jeff (@jeffdonahue ), I've merged the code back into ReLULayer and added unit test. Please let me know if anything further needs to be done.

[bhack]

I think you need to rebase from dev.

[qipeng]

Closing and moving to #740