Skip to content

Improve attribute read time in ADIOS2 backend#613

Merged
ax3l merged 2 commits intoopenPMD:devfrom
franzpoeschel:topic-adios2-better-attributes
Dec 6, 2019
Merged

Improve attribute read time in ADIOS2 backend#613
ax3l merged 2 commits intoopenPMD:devfrom
franzpoeschel:topic-adios2-better-attributes

Conversation

@franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented Nov 15, 2019

Previously, the methods listDatasets, listPaths and listAttributes issued calls to adios2::IO::GetAvailableAttributes and scanned them for the pattern of interest. Two issues:

  1. Calling adios2::IO::GetAvailableAttributes is an operation linear in the size of the attributes map since the map is returned by value.
  2. Scanning for the pattern of interest was done via a linear search.

The same holds for those methods' usage of adios2::IO::GetAvailableVariables.
This PR solves these issues respectively by:

  1. Buffering the aforementioned maps and reloading them only as soon as attributes/variables are modified.
  2. Using the fact that std::map<std::string> can be efficiently queried for prefixes, see here.

@franzpoeschel franzpoeschel force-pushed the topic-adios2-better-attributes branch from 118fea8 to eabb290 Compare November 15, 2019 10:17
Use the fact that std::map sorts string lexicographically

This allows faster lookup by prefix.

Dito for variables
@franzpoeschel franzpoeschel force-pushed the topic-adios2-better-attributes branch from eabb290 to a5074ef Compare November 15, 2019 10:17
@ax3l ax3l self-requested a review November 15, 2019 22:23
@ax3l ax3l self-assigned this Nov 15, 2019
Copy link
Member

@ax3l ax3l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just some minor comments.

Did you see/measure how this affects read/open performance? Any benchmark to share?

@ax3l
Copy link
Member

ax3l commented Nov 26, 2019

ping @franzpoeschel just a few slight updates needed for merge :)

@franzpoeschel
Copy link
Contributor Author

Time measurements, performed with a Series containing 30 iterations as written by the LaserWakefield example of PIConGPU, with 688 variables and 4264 attributes:

Before this PR:

> time ./openSeries 
Opened series stream

real    0m21,834s
user    0m21,666s
sys     0m0,053s

After this PR:

> time ./openSeries 
Opened series stream

real    0m0,516s
user    0m0,462s
sys     0m0,012s

For comparison, when using file-based iteration layout, before this PR:

> time ./openSeries 
Opened series stream%T

real    0m0,713s
user    0m0,694s
sys     0m0,017s

And afterwards:

> time ./openSeries 
Opened series stream%T

real    0m0,167s
user    0m0,157s
sys     0m0,009s

@ax3l
Copy link
Member

ax3l commented Dec 5, 2019

Really nice, thank you very much!
@williamfgc have you seen this already? :)

Copy link
Member

@ax3l ax3l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!
If you like, feel free to update CHANGELOG.md on the way, otherwise I'll do it on release.

@ax3l ax3l added the workaround label Dec 5, 2019
@williamfgc
Copy link

@ax3l @franzpoeschel cool, great job! Thanks for sharing the good news.

@ax3l ax3l merged commit 50c9dc3 into openPMD:dev Dec 6, 2019
@franzpoeschel franzpoeschel deleted the topic-adios2-better-attributes branch January 28, 2021 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants