Improve attribute read time in ADIOS2 backend#613
Conversation
118fea8 to
eabb290
Compare
Use the fact that std::map sorts string lexicographically This allows faster lookup by prefix. Dito for variables
eabb290 to
a5074ef
Compare
ax3l
left a comment
There was a problem hiding this comment.
Thanks, just some minor comments.
Did you see/measure how this affects read/open performance? Any benchmark to share?
|
ping @franzpoeschel just a few slight updates needed for merge :) |
|
Time measurements, performed with a Series containing 30 iterations as written by the LaserWakefield example of PIConGPU, with 688 variables and 4264 attributes: Before this PR: After this PR: For comparison, when using file-based iteration layout, before this PR: And afterwards: |
|
Really nice, thank you very much! |
ax3l
left a comment
There was a problem hiding this comment.
Looks great!
If you like, feel free to update CHANGELOG.md on the way, otherwise I'll do it on release.
|
@ax3l @franzpoeschel cool, great job! Thanks for sharing the good news. |
Previously, the methods
listDatasets,listPathsandlistAttributesissued calls toadios2::IO::GetAvailableAttributesand scanned them for the pattern of interest. Two issues:adios2::IO::GetAvailableAttributesis an operation linear in the size of the attributes map since the map is returned by value.The same holds for those methods' usage of
adios2::IO::GetAvailableVariables.This PR solves these issues respectively by:
std::map<std::string>can be efficiently queried for prefixes, see here.