openPMD · ax3l · Feb 13, 2023 · Jun 8, 2022 · Jul 6, 2022 · Jul 6, 2022
diff --git a/docs/source/dev/design.rst b/docs/source/dev/design.rst
@@ -23,7 +23,7 @@ Therefore, enabling users to handle hierarchical, self-describing file formats w
 
 .. literalinclude:: IOTask.hpp
    :language: cpp
-   :lines: 44-62
+   :lines: 48-78
 
 Every task is designed to be a fully self-contained description of one such atomic operation. By describing a required minimal step of work (without any side-effect), these operations are the foundation of the unified handling mechanism across suitable file formats.
 The actual low-level exchange of data is implemented in ``IOHandlers``, one per file format (possibly two if handlingi MPI-parallel work is possible and requires different behaviour).

diff --git a/docs/source/usage/streaming.rst b/docs/source/usage/streaming.rst
@@ -22,6 +22,7 @@ C++
 ^^^
 
 The reading end of the streaming API is activated through use of ``Series::readIterations()`` instead of accessing the field ``Series::iterations`` directly.
+Use of ``Access::READ_LINEAR`` mode is recommended.
 The returned object of type ``ReadIterations`` can be used in a C++11 range-based for loop to iterate over objects of type ``IndexedIteration``.
 This class extends the ``Iteration`` class with a field ``IndexedIteration::iterationIndex``, denoting this iteration's index.
 
@@ -40,6 +41,7 @@ Python
 ^^^^^^
 
 The reading end of the streaming API is activated through use of ``Series.read_iterations()`` instead of accessing the field ``Series.iterations`` directly.
+Use of ``Access.read_linear`` mode is recommended.
 The returned object of type ``ReadIterations`` can be used in a Python range-based for loop to iterate over objects of type ``IndexedIteration``.
 This class extends the ``Iteration`` class with a field ``IndexedIteration.iteration_index``, denoting this iteration's index.
 

diff --git a/docs/source/usage/workflow.rst b/docs/source/usage/workflow.rst
@@ -7,14 +7,41 @@ The openPMD-api distinguishes between a number of different access modes:
 
 * **Create mode**: Used for creating a new Series from scratch.
   Any file possibly existing in the specified location will be overwritten.
-* **Read-only mode**: Used for reading from an existing Series.
+* Two distinct read modes: **Read-random-access mode** and **Read-linear mode**.
+  (Specification of **Read-only mode** is equivalent to read-random-access mode.)
+  Both modes are used for reading from an existing Series.
   No modifications will be made.
+
+  Differences between both modes:
+
+  * When intending to use ``Series::readIterations()`` (i.e. step-by-step reading of iterations, e.g. in streaming), then **linear read mode** is preferred and always supported.
+    Data is parsed inside ``Series::readIterations()``, no data is available right after opening the Series.
+    Global attributes are available directly after calling ``Series::readIterations()``, Iterations and all their corresponding data become available by use of the returned Iterator, e.g. in a foreach loop.
+  * Otherwise (i.e. for random-access workflows), **random-access read mode** is required, but works only in backends that support random access.
+    Data is parsed and available right after opening the Series.
+
+  In both modes, parsing of iterations can be deferred with the JSON/TOML option ``defer_iteration_parsing``.
+
+  Detailed rules:
+
+  1. In backends that have no notion of IO steps (all except ADIOS2), *random-access read mode* can always be used.
+  2. In backends that can be accessed either in random-access or step-by-step, the chosen access mode decides which approach is used.
+     Examples are the BP4 and BP5 engines of ADIOS2.
+  3. In streaming backends, random-access is not possible.
+     When using such a backend, the access mode will be coerced automatically to *linear read mode*.
+     Use of Series::readIterations() is mandatory for access.
+  4. Reading a variable-based Series is only fully supported with *linear access mode*.
+     If using *random-access read mode*, the dataset will be considered to only have one single step.
+     If the dataset only has one single step, this is guaranteed to work as expected.
+     Otherwise, it is undefined which step's data is returned.
+
 * **Read/Write mode**: Creates a new Series if not existing, otherwise opens an existing Series for reading and writing.
   New datasets and iterations will be inserted as needed.
   Not fully supported by all backends:
 
   * ADIOS1: Automatically coerced to *Create* mode if the file does not exist yet and to *Read-only* mode if it exists.
   * ADIOS2: Automatically coerced to *Create* mode if the file does not exist yet and to *Read-only* mode if it exists.
+
   Since this happens on a per-file level, this mode allows to read from existing iterations and write to new iterations at the same time in file-based iteration encoding.
 * **Append mode**: Restricted mode for appending new iterations to an existing Series that is supported by all backends at least in file-based iteration encoding, and by all but ADIOS1 in other encodings.
   The API is equivalent to that of the *Create* mode, meaning that no reading is supported whatsoever.

diff --git a/examples/10_streaming_read.cpp b/examples/10_streaming_read.cpp
@@ -19,7 +19,7 @@ int main()
         return 0;
     }
 
-    Series series = Series("electrons.sst", Access::READ_ONLY);
+    Series series = Series("electrons.sst", Access::READ_LINEAR);
 
     for (IndexedIteration iteration : series.readIterations())
     {

diff --git a/examples/10_streaming_read.py b/examples/10_streaming_read.py
@@ -17,7 +17,7 @@
         print("SST engine not available in ADIOS2.")
         sys.exit(0)
 
-    series = io.Series("simData.sst", io.Access_Type.read_only,
+    series = io.Series("simData.sst", io.Access_Type.read_linear,
                        json.dumps(config))
 
     # Read all available iterations and print electron position data.

diff --git a/include/openPMD/IO/ADIOS/ADIOS2IOHandler.hpp b/include/openPMD/IO/ADIOS/ADIOS2IOHandler.hpp
@@ -164,7 +164,7 @@ class ADIOS2IOHandlerImpl
     void extendDataset(
         Writable *, Parameter<Operation::EXTEND_DATASET> const &) override;
 
-    void openFile(Writable *, Parameter<Operation::OPEN_FILE> const &) override;
+    void openFile(Writable *, Parameter<Operation::OPEN_FILE> &) override;
 
     void
     closeFile(Writable *, Parameter<Operation::CLOSE_FILE> const &) override;
@@ -213,6 +213,10 @@ class ADIOS2IOHandlerImpl
 
     void availableChunks(
         Writable *, Parameter<Operation::AVAILABLE_CHUNKS> &) override;
+
+    void
+    deregister(Writable *, Parameter<Operation::DEREGISTER> const &) override;
+
     /**
      * @brief The ADIOS2 access type to chose for Engines opened
      * within this instance.
@@ -248,7 +252,10 @@ class ADIOS2IOHandlerImpl
      */
     std::string m_userSpecifiedExtension;
 
-    ADIOS2Schema::schema_t m_schema = ADIOS2Schema::schema_0000_00_00;
+    /*
+     * Empty option: No schema has been explicitly selected, use default.
+     */
+    std::optional<ADIOS2Schema::schema_t> m_schema;
 
     enum class UseSpan : char
     {
@@ -267,7 +274,11 @@ class ADIOS2IOHandlerImpl
 
     inline SupportedSchema schema() const
     {
-        switch (m_schema)
+        if (!m_schema.has_value())
+        {
+            return SupportedSchema::s_0000_00_00;
+        }
+        switch (m_schema.value())
         {
         case ADIOS2Schema::schema_0000_00_00:
             return SupportedSchema::s_0000_00_00;
@@ -276,7 +287,7 @@ class ADIOS2IOHandlerImpl
         default:
             throw std::runtime_error(
                 "[ADIOS2] Encountered unsupported schema version: " +
-                std::to_string(m_schema));
+                std::to_string(m_schema.value()));
         }
     }
 
@@ -331,11 +342,11 @@ class ADIOS2IOHandlerImpl
      * @return first parameter: the operators, second parameters: whether
      * operators have been configured
      */
-    std::optional<std::vector<ParameterizedOperator> >
+    std::optional<std::vector<ParameterizedOperator>>
     getOperators(json::TracingJSON config);
 
     // use m_config
-    std::optional<std::vector<ParameterizedOperator> > getOperators();
+    std::optional<std::vector<ParameterizedOperator>> getOperators();
 
     std::string fileSuffix(bool verbose = true) const;
 
@@ -361,7 +372,7 @@ class ADIOS2IOHandlerImpl
      */
     std::unordered_map<
         InvalidatableFile,
-        std::unique_ptr<detail::BufferedActions> >
+        std::unique_ptr<detail::BufferedActions>>
         m_fileData;
 
     std::map<std::string, adios2::Operator> m_operators;
@@ -455,8 +466,8 @@ namespace detail
 
     template <typename T>
     inline constexpr bool IsUnsupportedComplex_v =
-        std::is_same_v<T, std::complex<long double> > ||
-        std::is_same_v<T, std::vector<std::complex<long double> > >;
+        std::is_same_v<T, std::complex<long double>> ||
+        std::is_same_v<T, std::vector<std::complex<long double>>>;
 
     struct DatasetReader
     {
@@ -581,7 +592,8 @@ namespace detail
             Parameter<Operation::AVAILABLE_CHUNKS> &params,
             adios2::IO &IO,
             adios2::Engine &engine,
-            std::string const &varName);
+            std::string const &varName,
+            bool allSteps);
 
         template <int n, typename... Params>
         static void call(Params &&...);
@@ -630,7 +642,7 @@ namespace detail
     };
 
     template <>
-    struct AttributeTypes<std::complex<long double> >
+    struct AttributeTypes<std::complex<long double>>
     {
         static void createAttribute(
             adios2::IO &,
@@ -663,13 +675,13 @@ namespace detail
     };
 
     template <>
-    struct AttributeTypes<std::vector<std::complex<long double> > >
+    struct AttributeTypes<std::vector<std::complex<long double>>>
     {
         static void createAttribute(
             adios2::IO &,
             adios2::Engine &,
             detail::BufferedAttributeWrite &,
-            const std::vector<std::complex<long double> > &)
+            const std::vector<std::complex<long double>> &)
         {
             throw std::runtime_error(
                 "[ADIOS2] Internal error: no support for long double complex "
@@ -687,7 +699,7 @@ namespace detail
         }
 
         static bool attributeUnchanged(
-            adios2::IO &, std::string, std::vector<std::complex<long double> >)
+            adios2::IO &, std::string, std::vector<std::complex<long double>>)
         {
             throw std::runtime_error(
                 "[ADIOS2] Internal error: no support for long double complex "
@@ -696,7 +708,7 @@ namespace detail
     };
 
     template <typename T>
-    struct AttributeTypes<std::vector<T> >
+    struct AttributeTypes<std::vector<T>>
     {
         static void createAttribute(
             adios2::IO &IO,
@@ -734,7 +746,7 @@ namespace detail
     };
 
     template <>
-    struct AttributeTypes<std::vector<std::string> >
+    struct AttributeTypes<std::vector<std::string>>
     {
         static void createAttribute(
             adios2::IO &IO,
@@ -772,7 +784,7 @@ namespace detail
     };
 
     template <typename T, size_t n>
-    struct AttributeTypes<std::array<T, n> >
+    struct AttributeTypes<std::array<T, n>>
     {
         static void createAttribute(
             adios2::IO &IO,
@@ -986,14 +998,14 @@ namespace detail
          * Hence, next to the actual file name, also store the name for the
          * IO.
          */
-        std::string const m_IOName;
+        std::string m_IOName;
         adios2::ADIOS &m_ADIOS;
         adios2::IO m_IO;
         /**
          * The default queue for deferred actions.
          * Drained upon BufferedActions::flush().
          */
-        std::vector<std::unique_ptr<BufferedAction> > m_buffer;
+        std::vector<std::unique_ptr<BufferedAction>> m_buffer;
         /**
          * Buffer for attributes to be written in the new (variable-based)
          * attribute layout.
@@ -1017,7 +1029,7 @@ namespace detail
          * We must store them somewhere until the next PerformPuts/Gets, EndStep
          * or Close in ADIOS2 to avoid use after free conditions.
          */
-        std::vector<std::unique_ptr<BufferedAction> > m_alreadyEnqueued;
+        std::vector<std::unique_ptr<BufferedAction>> m_alreadyEnqueued;
         adios2::Mode m_mode;
         /**
          * The base pointer of an ADIOS2 span might change after reallocations.
@@ -1027,7 +1039,7 @@ namespace detail
          * retrieval of the updated base pointer.
          * This map is cleared upon flush points.
          */
-        std::map<unsigned, std::unique_ptr<I_UpdateSpan> > m_updateSpans;
+        std::map<unsigned, std::unique_ptr<I_UpdateSpan>> m_updateSpans;
         PreloadAdiosAttributes preloadAttributes;
 
         /*
@@ -1049,6 +1061,10 @@ namespace detail
          */
         bool optimizeAttributesStreaming = false;
 
+        using ParsePreference =
+            Parameter<Operation::OPEN_FILE>::ParsePreference;
+        ParsePreference parsePreference = ParsePreference::UpFront;
+
         using AttributeMap_t = std::map<std::string, adios2::Params>;
 
         BufferedActions(ADIOS2IOHandlerImpl &impl, InvalidatableFile file);
@@ -1257,13 +1273,6 @@ namespace detail
          */
         std::string m_engineType;
 
-        /**
-         * See documentation for StreamStatus::Parsing.
-         * Will be set true under the circumstance described there in order to
-         * indicate that the first step should only be opened after parsing.
-         */
-        bool delayOpeningTheFirstStep = false;
-
         /*
          * ADIOS2 does not give direct access to its internal attribute and
          * variable maps, but will instead give access to copies of them.
@@ -1279,6 +1288,11 @@ namespace detail
         std::optional<AttributeMap_t> m_availableAttributes;
         std::optional<AttributeMap_t> m_availableVariables;
 
+        /*
+         * Cannot write attributes right after opening the engine
+         * https://github.com/ornladios/ADIOS2/issues/3433
+         */
+        bool initializedDefaults = false;
         /*
          * finalize() will set this true to avoid running twice.
          */
@@ -1289,7 +1303,11 @@ namespace detail
             return m_impl->schema();
         }
 
+        void create_IO();
+
         void configure_IO(ADIOS2IOHandlerImpl &impl);
+        void configure_IO_Read(std::optional<bool> userSpecifiedUsesteps);
+        void configure_IO_Write(std::optional<bool> userSpecifiedUsesteps);
 
         using AttributeLayout = ADIOS2IOHandlerImpl::AttributeLayout;
         inline AttributeLayout attributeLayout() const

diff --git a/include/openPMD/IO/ADIOS/CommonADIOS1IOHandler.hpp b/include/openPMD/IO/ADIOS/CommonADIOS1IOHandler.hpp
@@ -59,7 +59,7 @@ class CommonADIOS1IOHandlerImpl : public AbstractIOHandlerImpl
         Writable *, Parameter<Operation::CREATE_DATASET> const &) override;
     void extendDataset(
         Writable *, Parameter<Operation::EXTEND_DATASET> const &) override;
-    void openFile(Writable *, Parameter<Operation::OPEN_FILE> const &) override;
+    void openFile(Writable *, Parameter<Operation::OPEN_FILE> &) override;
     void
     closeFile(Writable *, Parameter<Operation::CLOSE_FILE> const &) override;
     void availableChunks(
@@ -84,6 +84,8 @@ class CommonADIOS1IOHandlerImpl : public AbstractIOHandlerImpl
     void
     listDatasets(Writable *, Parameter<Operation::LIST_DATASETS> &) override;
     void listAttributes(Writable *, Parameter<Operation::LIST_ATTS> &) override;
+    void
+    deregister(Writable *, Parameter<Operation::DEREGISTER> const &) override;
 
     void close(int64_t);
     void close(ADIOS_FILE *);

diff --git a/include/openPMD/IO/AbstractIOHandlerImpl.hpp b/include/openPMD/IO/AbstractIOHandlerImpl.hpp
@@ -202,6 +202,12 @@ class AbstractIOHandlerImpl
                         deref_dynamic_cast<Parameter<O::KEEP_SYNCHRONOUS> >(
                             i.parameter.get()));
                     break;
+                case O::DEREGISTER:
+                    deregister(
+                        i.writable,
+                        deref_dynamic_cast<Parameter<O::DEREGISTER> >(
+                            i.parameter.get()));
+                    break;
                 }
             }
             catch (...)
@@ -354,8 +360,7 @@ class AbstractIOHandlerImpl
      * root group "/" of the hierarchy in the opened file. The Writable should
      * be marked written when the operation completes successfully.
      */
-    virtual void
-    openFile(Writable *, Parameter<Operation::OPEN_FILE> const &) = 0;
+    virtual void openFile(Writable *, Parameter<Operation::OPEN_FILE> &) = 0;
     /** Open all contained groups in a path, possibly recursively.
      *
      * The operation should overwrite existing file positions, even when the
@@ -567,6 +572,16 @@ class AbstractIOHandlerImpl
     void
     keepSynchronous(Writable *, Parameter<Operation::KEEP_SYNCHRONOUS> param);
 
+    /** Notify the backend that the Writable has been / will be deallocated.
+     *
+     * The backend should remove all references to this Writable from internal
+     * data structures. Subtle bugs might be possible if not doing this, since
+     * new objects might be allocated to the now-freed address.
+     * The Writable pointer must not be dereferenced.
+     */
+    virtual void
+    deregister(Writable *, Parameter<Operation::DEREGISTER> const &param) = 0;
+
     AbstractIOHandler *m_handler;
 }; // AbstractIOHandlerImpl
 } // namespace openPMD