Prefetching HDF5DataLayer (currently broken)#1362
Closed
jeffdonahue wants to merge 2 commits intoBVLC:devfrom
Closed
Prefetching HDF5DataLayer (currently broken)#1362jeffdonahue wants to merge 2 commits intoBVLC:devfrom
jeffdonahue wants to merge 2 commits intoBVLC:devfrom
Conversation
Member
|
Closing in favor of follow-up #2271. |
ronghanghu
pushed a commit
to ronghanghu/caffe
that referenced
this pull request
Aug 9, 2015
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Posting this hoping someone will have debugging tips or suggestions on how to fix it. I modified the HDF5 reading interface to have two methods:
HDF5PrepareBlobwhich pre-shapes a Blob for a particular number of rows (or -1 to indicate all rows in the HDF5 file), andHDF5ReadRowsToBlobto read a particular number of rows from an HDF5 file to a Blob, starting at a particular offset of each.The intention of this interface is to enable reading HDF5 data without ever allocating additional memory -- the main thread calls
HDF5PrepareBlobinLayerSetUp(followed bymutable_cpu_data) to pre-allocate the blob data, then the prefetch thread callsHDF5ReadRowsToBlobto fill the presized Blob with however many floats it needs.On my machine, the new interface works perfectly in the first commit where it's all executed in the main thread (passes the unit tests, at least..), but in the second commit, when I use the prefetch thread, there seem to be all sorts of non-deterministic problems, so I must be reading/writing some memory I'm not supposed to be reading/writing in the prefetch thread.
valgrind sometimes points to my use of
c_str()to access filename inhdf5_data_layer.cpp:32-- it often reports an invalid read fromH5Fopenor the print statement of the filename on line 33.