Clear Scratch Diffs to Prevent Contaminating Backward through Splits

[shelhamer]

Certain layers (SoftmaxWithLossLayer, SigmoidCrossEntropyLossLayer, and AccuracyLayer) save memory during forward by making use of bottom diffs that are otherwise unused or overwritten during backward. The trouble is that these scratch diffs can be mistakenly propagated by backward through split layers. All of the top diffs of a split are accumulated even when backward is not called for the losses (when their loss weight is zero) or accuracy (which has no backward step). This was missed at first because it requires the interaction of split layers and the backward pruning that prevents computation of unnecessary gradients.

This fix zeros out the scratch diffs to prevent this kind of contamination. This requires a little computation but not much at all. This is preferable to requiring more memory for internal buffers because the further memory usage might cause an unexpected crash for a previously good configuration.

See #2895 (comment) for the first explanation of this issue.

Related:

• fix Incorrect gradient from a SoftmaxWithLossLayer with loss_weight 0 #2895, fix New Accuracy Layer on GPU interferes with training #5981
• close Fix the bug of incorrect gradient when loss_weight set to zero #3868, close Hotfix for accuracy interfering with training #5987, close Put the acc_data in a new syncedmemory block #6141

Credits:

• Thanks @vlomonaco for reporting the Accuracy layer issue in New Accuracy Layer on GPU interferes with training #5981.
• Thanks @Noiredd for proposing an internal buffer fix in Hotfix for accuracy interfering with training #5987 and @sonack for another version of the same in Put the acc_data in a new syncedmemory block #6141
• Thanks @orzlibo for proposing the clearing fix for losses in clear bottom diffs after used as temp memory in softmax/sigmoid loss … #6186. Note the
  cherry-pick of that fix in this PR.

[Noiredd]

This is a neat, systemic solution. Performance impact of caffe_gpu_set is minimal: on a theoretical network making heavy use of Accuracy it doubled the forward pass time... from 0.78ms to 1.60ms - IMO negligible. The same network ate 703 MB RAM with my hotfix, comparable to 526 MB with this PR. On more typical classification setups the differences are unnoticeable.
My vote goes for merging this.

I have added a reference to close #6141 which attempted to solve the Accuracy layer problem by allocating a raw SyncedMemory object to act as an internal buffer. Thanks @sonack!

[shelhamer]

Thanks all for your work on this and thanks again @Noiredd for the final review!