Saving memory by reusing col_buffer_.

[philkr]

For fully convolutional nets the col_buffer_ in the BaseColvolutionLayer can get quite large (1GB or more in total). Currently this temporary buffer is allocated for each convolution separately. This PR introduces a TemporaryMemory class which allows all temporary memory to be shared between convolutional layers. This could be used for other layers that use temporary memory, but I couldn't find any other memory hungry ones...
This saves up to 1GB of memory for fully convolutional VGG models.

[philkr]

Travis seems to dislike cudaMalloc when built WITH_CUDA=true ("CUDA driver version is insufficient for CUDA runtime version"). @shelhamer any ideas on how to work around this?

[shelhamer]

@philkr Check out #1291 for the quick hack edition of this idea -- although it of course does not play nicely with parallelism. Not sure what's up with Travis and cudaMalloc off the top of my head.

[philkr]

Currently this PR doesn't play too nicely with parallelism. It will just throw an exception if two threads are trying to access the temporary memory concurrently.
But I couldn't see where in caffe this would happen...
Edit: It now should work with concurrent memory accesses.

[nian-liu]

Hi philkr,
I have tested your code, there seems to be a bug while using cuDNN engine. When a cuDNN forward operation is runing, it will report access violation error, while the codes running well with the CAFFE engine.

[philkr]

@nian-liu this seems a bit odd, as cudnn doesn't seem to be using col_buffer_. What version of cuDNN are you using?
Let me see if I can install cudnn to test this...

[nian-liu]

@philkr I am using cuDNN v1.

[shelhamer]

Closing since master has drifted and this was always potentially a bit dangerous. Thanks for the bespoke conv memory pool all the same Philipp!