test_gradient_based_solver fails #22
Description
I have a problem with make runtest failing on SGDSolver and NesterovSolver. I looked at the git repository of BVLC/caffe (BVLC/caffe#3109) and there somebody was referring to a problem coming from the same file test_gradient_based_solver.cpp. In the comments people were writing that it was caused by multiple GPUs present in the system or the fact that Intel MKL's float point operations (such as matrix multiplication) are non-deterministic by default.
Regarding my system, I am running Caffe cloned from github on 22nd of December 2015 on Ubuntu 15.10 with Radeon R9 290 (4GB) and i7-4770K CPU @ 3.50GHz, AMDAPPSDK-3.0. Four tests failed.
If anybody knows how to make them pass or what causes the problem it would be great.
$ fglrxinfo
display: :0 screen: 0
OpenGL vendor string: Advanced Micro Devices, Inc.
OpenGL renderer string: AMD Radeon R9 200 Series
OpenGL version string: 4.5.13399 Compatibility Profile Context 15.201.1151
/OpenCL-caffe/build$ make runtest
[ 1%] Built target proto
[ 55%] Built target caffe
[ 55%] Built target gtest
[100%] Built target test.testbin
Current device id: 0
Note: Google Test filter = -*GPU*
Note: Randomizing tests' orders with a seed of 3162 .
...................
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00015804916620254517, which exceeds error_margin, where
expected_param evaluates to 0.013755172491073608,
accum_param evaluates to 0.013913221657276154, and
error_margin evaluates to 0.00013755173131357878.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00015809759497642517, which exceeds error_margin, where
expected_param evaluates to 0.00093255564570426941,
accum_param evaluates to 0.0010906532406806946, and
error_margin evaluates to 9.3255566753214225e-06.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00015811249613761902, which exceeds error_margin, where
expected_param evaluates to -0.010214578360319138,
accum_param evaluates to -0.010056465864181519, and
error_margin evaluates to 0.00010056465544039384.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00015810877084732056, which exceeds error_margin, where
expected_param evaluates to -0.0109538733959198,
accum_param evaluates to -0.010795764625072479, and
error_margin evaluates to 0.0001079576468328014.
src/caffe/test/test_gradient_based_solver.cpp:315: Failure
The difference between expected_bias and accum_bias is 0.020383119583129883, which exceeds error_margin, where
expected_bias evaluates to -1.2308368682861328,
accum_bias evaluates to -1.2512199878692627, and
error_margin evaluates to 0.01230836845934391.
[ FAILED ] SGDSolverTest/2.TestLeastSquaresUpdateWithEverythingAccum, where TypeParam = caffe::GPUDevice<float> (72 ms)
.....................
src/caffe/test/test_gradient_based_solver.cpp:315: Failure
The difference between expected_bias and accum_bias is 16.290860759216979, which exceeds error_margin, where
expected_bias evaluates to 30.680854565170151,
accum_bias evaluates to 14.389993805953171, and
error_margin evaluates to 0.14389993805953172.
[ FAILED ] NesterovSolverTest/3.TestLeastSquaresUpdateWithEverythingAccum, where TypeParam = caffe::GPUDevice<double> (12 ms)
.....................
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.0009438430034522105, which exceeds error_margin, where
expected_param evaluates to -0.0075809667410185466,
accum_param evaluates to -0.0085248097444707571, and
error_margin evaluates to 7.5809667410185473e-05.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345218275, which exceeds error_margin, where
expected_param evaluates to 0.073903926620193378,
accum_param evaluates to 0.072960083616741195, and
error_margin evaluates to 0.00072960083616741196.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345218275, which exceeds error_margin, where
expected_param evaluates to 0.057955563277547251,
accum_param evaluates to 0.057011720274095068, and
error_margin evaluates to 0.00057011720274095074.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345218275, which exceeds error_margin, where
expected_param evaluates to -0.040239520053002617,
accum_param evaluates to -0.0411833630564548, and
error_margin evaluates to 0.00040239520053002618.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345230765, which exceeds error_margin, where
expected_param evaluates to -0.054015869066066119,
accum_param evaluates to -0.054959712069518427, and
error_margin evaluates to 0.00054015869066066115.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345244643, which exceeds error_margin, where
expected_param evaluates to -0.035164120485524253,
accum_param evaluates to -0.036107963488976699, and
error_margin evaluates to 0.00035164120485524255.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345243255, which exceeds error_margin, where
expected_param evaluates to 0.023867411055095877,
accum_param evaluates to 0.022923568051643445, and
error_margin evaluates to 0.00022923568051643446.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345229377, which exceeds error_margin, where
expected_param evaluates to -0.023262755953808445,
accum_param evaluates to -0.024206598957260739, and
error_margin evaluates to 0.00023262755953808447.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.0009438430034522105, which exceeds error_margin, where
expected_param evaluates to 0.022473440985742582,
accum_param evaluates to 0.021529597982290372, and
error_margin evaluates to 0.00021529597982290373.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345204397, which exceeds error_margin, where
expected_param evaluates to -0.094111263940639428,
accum_param evaluates to -0.095055106944091472, and
error_margin evaluates to 0.00094111263940639428.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345215499, which exceeds error_margin, where
expected_param evaluates to -0.052540014489286579,
accum_param evaluates to -0.053483857492738734, and
error_margin evaluates to 0.0005254001448928658.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.0009438430034522105, which exceeds error_margin, where
expected_param evaluates to -0.074735299450877965,
accum_param evaluates to -0.075679142454330176, and
error_margin evaluates to 0.00074735299450877965.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.0009438430034522105, which exceeds error_margin, where
expected_param evaluates to -0.032339898640959791,
accum_param evaluates to -0.033283741644412002, and
error_margin evaluates to 0.00032339898640959791.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345218275, which exceeds error_margin, where
expected_param evaluates to 0.089805274972429433,
accum_param evaluates to 0.08886143196897725, and
error_margin evaluates to 0.00088861431968977251.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345243255, which exceeds error_margin, where
expected_param evaluates to 0.079982388023672302,
accum_param evaluates to 0.079038545020219869, and
error_margin evaluates to 0.00079038545020219866.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345229377, which exceeds error_margin, where
expected_param evaluates to -0.046441101129402113,
accum_param evaluates to -0.047384944132854406, and
error_margin evaluates to 0.00046441101129402113.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345215499, which exceeds error_margin, where
expected_param evaluates to 0.073243776031034963,
accum_param evaluates to 0.072299933027582808, and
error_margin evaluates to 0.00072299933027582806.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345218275, which exceeds error_margin, where
expected_param evaluates to 0.036783959999803301,
accum_param evaluates to 0.035840116996351118, and
error_margin evaluates to 0.0003584011699635112.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345218275, which exceeds error_margin, where
expected_param evaluates to 0.074800868345276994,
accum_param evaluates to 0.073857025341824811, and
error_margin evaluates to 0.00073857025341824811.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345229377, which exceeds error_margin, where
expected_param evaluates to -0.087903557731515064,
accum_param evaluates to -0.088847400734967358, and
error_margin evaluates to 0.00087903557731515067.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.0009438430034522105, which exceeds error_margin, where
expected_param evaluates to -0.09313550033631654,
accum_param evaluates to -0.094079343339768751, and
error_margin evaluates to 0.00093135500336316546.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345232153, which exceeds error_margin, where
expected_param evaluates to 0.0023059492350265554,
accum_param evaluates to 0.0013621062315742338, and
error_margin evaluates to 1.3621062315742339e-05.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.0009438430034522105, which exceeds error_margin, where
expected_param evaluates to 0.018183528710792568,
accum_param evaluates to 0.017239685707340358, and
error_margin evaluates to 0.00017239685707340358.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345237704, which exceeds error_margin, where
expected_param evaluates to 0.040265922015119665,
accum_param evaluates to 0.039322079011667288, and
error_margin evaluates to 0.00039322079011667289.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.00094384300345218275, which exceeds error_margin, where
expected_param evaluates to 0.091959157638004163,
accum_param evaluates to 0.09101531463455198, and
error_margin evaluates to 0.00091015314634551982.
src/caffe/test/test_gradient_based_solver.cpp:315: Failure
The difference between expected_bias and accum_bias is 0.11647201251983841, which exceeds error_margin, where
expected_bias evaluates to -1.9010368863451557,
accum_bias evaluates to -1.7845648738253173, and
error_margin evaluates to 0.017845648738253173.
[ FAILED ] SGDSolverTest/3.TestLeastSquaresUpdateWithEverythingAccum, where TypeParam = caffe::GPUDevice<double> (16 ms)
.....................
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.057497024536132812, which exceeds error_margin, where
expected_param evaluates to -5.7565555572509766,
accum_param evaluates to -5.6990585327148438, and
error_margin evaluates to 0.056990586221218109.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.057497501373291016, which exceeds error_margin, where
expected_param evaluates to -5.6071248054504395,
accum_param evaluates to -5.5496273040771484, and
error_margin evaluates to 0.055496271699666977.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.057498931884765625, which exceeds error_margin, where
expected_param evaluates to -5.7071495056152344,
accum_param evaluates to -5.6496505737304688, and
error_margin evaluates to 0.056496504694223404.
src/caffe/test/test_gradient_based_solver.cpp:308: Failure
The difference between expected_param and accum_param is 0.057498931884765625, which exceeds error_margin, where
expected_param evaluates to -5.3606395721435547,
accum_param evaluates to -5.3031406402587891, and
error_margin evaluates to 0.053031407296657562.
src/caffe/test/test_gradient_based_solver.cpp:315: Failure
The difference between expected_bias and accum_bias is 4.0414938926696777, which exceeds error_margin, where
expected_bias evaluates to -9.2879724502563477,
accum_bias evaluates to -5.2464785575866699, and
error_margin evaluates to 0.052464786916971207.
[ FAILED ] NesterovSolverTest/2.TestLeastSquaresUpdateWithEverythingAccum, where TypeParam = caffe::GPUDevice<float> (19 ms)
....................
[ FAILED ] 4 tests, listed below:
[ FAILED ] SGDSolverTest/2.TestLeastSquaresUpdateWithEverythingAccum, where TypeParam = caffe::GPUDevice<float>
[ FAILED ] SGDSolverTest/3.TestLeastSquaresUpdateWithEverythingAccum, where TypeParam = caffe::GPUDevice<double>
[ FAILED ] NesterovSolverTest/2.TestLeastSquaresUpdateWithEverythingAccum, where TypeParam = caffe::GPUDevice<float>
[ FAILED ] NesterovSolverTest/3.TestLeastSquaresUpdateWithEverythingAccum, where TypeParam = caffe::GPUDevice<double>