NesterovSolverTest fails in one case

[to3i]

I just pulled the current dev branch and runtest fails in the case of NesterovSolver (see details below). I tested on Ubuntu 14.04 with CUDA 6.5, GTX 770 and 340.29 installed. Since #741 is a new addition, maybe this issue concerns different setups as well.

src/caffe/test/test_gradient_based_solver.cpp:244: Failure
The difference between expected_updated_weight and solver_updated_weight is 0.0072021484375, which exceeds error_margin, where
expected_updated_weight evaluates to -6.3511962890625,
solver_updated_weight evaluates to -6.343994140625, and
error_margin evaluates to 0.0063439942896366119.
[  FAILED  ] NesterovSolverTest/2.TestLeastSquaresUpdateWithMomentumMultiIter, where TypeParam = caffe::FloatGPU (37 ms)
...
...
[----------] Global test environment tear-down
[==========] 810 tests from 167 test cases ran. (80183 ms total)
[  PASSED  ] 809 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] NesterovSolverTest/2.TestLeastSquaresUpdateWithMomentumMultiIter, where TypeParam = caffe::FloatGPU

[mohomran]

Haven't been able to reproduce the error on Debian 7.5 (wheezy) with CUDA 5.0, a GTX Titan and driver version 331.38.

[ducha-aiki]

to3i, have the same issue. My configuration is Ubuntu 14.04 with CUDA 6.5, GTX 780, drivers 340.29.

[ RUN ] NesterovSolverTest/2.TestLeastSquaresUpdateWithMomentumMultiIter
src/caffe/test/test_gradient_based_solver.cpp:244: Failure
The difference between expected_updated_weight and solver_updated_weight is
0.0072021484375, which exceeds error_margin, where
expected_updated_weight evaluates to -6.3511962890625,
solver_updated_weight evaluates to -6.343994140625, and
error_margin evaluates to 0.0063439942896366119.
[ FAILED ] NesterovSolverTest/2.TestLeastSquaresUpdateWithMomentumMultiIter, where > TypeParam = caffe::FloatGPU (50 ms)
...
[----------] Global test environment tear-down
[==========] 822 tests from 173 test cases ran. (102580 ms total)
[ PASSED ] 821 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] NesterovSolverTest/2.TestLeastSquaresUpdateWithMomentumMultiIter, where > TypeParam = caffe::FloatGPU

[qipeng]

@jeffdonahue This seems like a precision issue... Can we just relax the precision requirement a bit for this one case? I can make the changes if necessary.

[jeffdonahue]

Yup, it's almost definitely just a precision issue. Since there's only one failure it's probably happening on the last iteration (as the errors will accumulate across iterations), so you might be able to fix it just by dropping the number of iterations from 5 to 4 for the Nesterov test (src/caffe/test/test_gradient_based_solver.cpp:482). I've never seen this though, so it would be good if someone who has the failure can confirm the fix.

[qipeng]

Should be addressed in #1040 .

[shelhamer]

Closing for fix in #1040.

[to3i]

Thanks for the fast response! However, #1040 does not solve the issue.

I just tested on my setup as you suggested earlier. The error already occurs on the third iteration of TestLeastSquaresUpdateWithMomentumMultiIter (NesterovSolverTest). Test passes for kNumIters = 2 or kPrecision = 2e-3;.

[jeffdonahue]

@to3i or anyone else with this problem feel free to reasonably increase whichever error margin (the one on line 244, based on the error message) fails for you to a point that makes it pass and send a PR. thanks. closing the issue (edit: oh, it was already closed and I reopened it...)