Unify Random-Access API and Streaming API into Series::snapshots()#1592
Merged
franzpoeschel merged 97 commits intoopenPMD:devfrom Dec 10, 2024
Merged
Unify Random-Access API and Streaming API into Series::snapshots()#1592franzpoeschel merged 97 commits intoopenPMD:devfrom
franzpoeschel merged 97 commits intoopenPMD:devfrom
Conversation
0ee77a2 to
73336c9
Compare
1e862b7 to
ab0991d
Compare
Comment on lines
+100
to
+235
| switch (series.iterationEncoding()) | ||
| { | ||
| using IE = IterationEncoding; | ||
| case IE::fileBased: | ||
| series.readFileBased(); | ||
| break; | ||
| case IE::groupBased: | ||
| case IE::variableBased: { | ||
| Parameter<Operation::OPEN_FILE> fOpen; | ||
| fOpen.name = series.get().m_name; | ||
| series.IOHandler()->enqueue(IOTask(&series, fOpen)); | ||
| series.IOHandler()->flush(internal::defaultFlushParams); | ||
| using PP = Parameter<Operation::OPEN_FILE>::ParsePreference; | ||
| switch (*fOpen.out_parsePreference) | ||
| { | ||
| case PP::PerStep: | ||
| series.advance(AdvanceMode::BEGINSTEP); | ||
| series.readGorVBased( | ||
| /* do_always_throw_errors = */ false, /* init = */ true); | ||
| break; | ||
| case PP::UpFront: | ||
| series.readGorVBased( | ||
| /* do_always_throw_errors = */ false, /* init = */ true); | ||
| /* | ||
| * In linear read mode (where no parsing at all is done upon | ||
| * constructing the Series), it might turn out after parsing | ||
| * that what we expected to be a group-based Series was in fact | ||
| * a single file of a file-based Series. | ||
| * (E.g. when opening "data00000100.h5" directly instead of | ||
| * "data%T.h5") | ||
| * So we need to check the current value of | ||
| * `iterationEncoding()` once more. | ||
| */ | ||
| if (series.iterationEncoding() != IterationEncoding::fileBased) | ||
| { | ||
| series.advance(AdvanceMode::BEGINSTEP); | ||
| } | ||
| break; | ||
| } | ||
| data.parsePreference = *fOpen.out_parsePreference; | ||
| break; | ||
| } | ||
| } |
Check notice
Code scanning / CodeQL
Long switch case
3845ed7 to
b5e4217
Compare
b5e4217 to
9987204
Compare
667ee24 to
f5d8433
Compare
f5d8433 to
26e5a49
Compare
Comment on lines
+88
to
+117
| // increment/decrement | ||
| // ChildClass &operator++(); | ||
| // ChildClass &operator--(); | ||
| // ChildClass &operator++(int); | ||
| // ChildClass &operator--(int); |
Check notice
Code scanning / CodeQL
Commented-out code
Comment on lines
+82
to
+109
| // dereference | ||
| // value_type const &operator*() const = 0; | ||
| // value_type &operator*() = 0; |
Check notice
Code scanning / CodeQL
Commented-out code
| } | ||
| // ~Series intentionally not yet called | ||
|
|
||
| // std::cout << "READ " << filename << std::endl; |
Check notice
Code scanning / CodeQL
Commented-out code
| auxiliary::getEnvNum("OPENPMD_TEST_NFILES_MAX", 1030); | ||
| std::string filename = | ||
| "../samples/many_iterations/many_iterations_%T." + ext; | ||
| // std::cout << "WRITE " << filename << std::endl; |
Check notice
Code scanning / CodeQL
Commented-out code
This was referenced Mar 19, 2024
19395b0 to
5b85307
Compare
for more information, see https://pre-commit.ci
Proper return() is supported beginning with CMake 3.25 only
for more information, see https://pre-commit.ci
a73c4c6 to
d278794
Compare
d278794 to
16c0f73
Compare
16c0f73 to
a6df22f
Compare
a6df22f to
cad227a
Compare
Contributor
Author
|
The changes introduced with this PR are mostly (1) internal and (2) a new API. I will go ahead and merge this now, as many of the internal changes are the groundwork for new features laid out in the PR description. The new API should be treated as experimental until release. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
So far, there were two distinct APIs for accessing Iterations:
Series::iterationsSeries::writeIterations()andSeries::readIterations()This PR introduces a new API
Series::snapshots()that returns a unified container API. The unified container API can be implemented internally via random-access and via synchronous access, i.e. by using steps.This is part of an effort to bring both workflows and their functionalities closer together. While this redesigned container API does not yet go all the way along this road, it lays the fundament for such extensibility by (1) cleanly separating the API from its implementation and by (2) redesigning and cleaning up the implementation of the step-based workflow.
Preliminary/experimental support for reopening closed Iterations is now available, especially for file-based workflows.
Note that there are still caveats about reopening Iterations. Especially in ADIOS2, a step cannot be modified once closed, so writing new data is only possible into a new step. However, the Iteration handle currently does not delete old data definitions upon closing, meaning that there will be invalid data definitions in the handle.
In future, a closed Iteration will be removed from internal data structures (but users may keep it around for longer than that in their own data structures).
Future features to be built on top of this include:
SetStepSelection()in ADIOS2writeIterations()andreadIterations()are now implemented as aliases/wrappers toSeries::snapshots(). Due to the past somewhat unfortunate decision to introduceIndexedIterationas iteration type forreadIterations(), the method name cannot be reused without breaking compatibility.writeIterations()however is just a wrapper forsnapshots()with default arguments.TODO:
usingdefinitions, empty headerssnapshots(Snapshots::RandomAccess)vssnapshots(Snapshots::CollectiveAccess)or sosnapshots().begin()-> reset to start, add sth likesnapshots().current(), alternatively keep the current semantics and addsnapshots().reset()?defer_iteration_parsingin READ_LINEAR modeDiff: franzpoeschel/openPMD-api@shared-attributable-data...topic-iterator