Merged
Conversation
Member
|
Testing in a minute is sweet! Now all I have to do is bring up the build bot and our tests will be perpetual. Thanks Jeff! |
Merged
mitmul
pushed a commit
to mitmul/caffe
that referenced
this pull request
Sep 30, 2014
Gradient check speedups
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes to the gradient checker that improve performance pretty significantly without any cost to unit test coverage.
(1) 8901fe7 moves the analytic gradient computation outside the loop over bottom features and stores it throughout all finite differencing computation (extra storage should be pretty minimal since the unit tests use relatively tiny blobs).
(2) 061b0db adds a method
GradientCheckEltwisethat replacesGradientCheckExhaustivein many layers. In any layer where we have element-wise computation where each output is a function only of the input at the corresponding index (e.g., neuron layers), we only need to do finite-differencing to compute the diagonal of the jacobian and we know all other entries are zero.In my tests of the entire unit test suite (
./build/test/test_all.testbinon a k40) this gave a 73% speedup from 238,610 ms @ dev down to 66,322 ms.