Ensure consistency between memory alloc and free

[ronghanghu]

Add a bool flag to record whether a host memory is allocated using malloc or cudaMallocHost, and free correspondingly using this flag, instead of depending on Caffe::mode(), which is mutable during runtime.

This should fix #3053 .

[ronghanghu]

I hope to merge this PR, if no one opposes. This ensures that memories are allocated and freed consistently.

[shelhamer]

Let's hold off a bit. Although this addresses the immediate issue the root cause is that Nets do not know their own state / mode and adding more state to SyncedMem isn't necessarily how we want to work around that. However I might be convinced this is the way to go for now since it works.

[ronghanghu]

Although this addresses the immediate issue the root cause is that Nets do not know their own state / mode and adding more state to SyncedMem isn't necessarily how we want to work around that.

I agree. In the spirit of #1500 I think it is better to have mode and device in net (or in layers if we plan to have individual mode/device for each layer).

[ronghanghu]

However, I think even the net knows their state/mode/device, it is still not enough, because in order to make SyncMemory and Blob self-contained data structures (not depending on the net holding it), SyncMemory class needs to know things like how memory is allocated, on which device etc.

So personally I still think a bool flag in this PR could be sort of necessary for SyncMemory. But I am open to opinions and suggestions on this issue.

[longjon]

Can we remove this whitespace change from the commit?

[ronghanghu]

Resolved.

[shelhamer]

Alright, this is good to merge for solving the immediate issue and we will further plan #1500 for diffusing mode + device through Net, Layer, and Blob.

[ronghanghu]

This should be good now :)