Add ADIOS2 IO Backend

CMakeLists.txt

@@ -59,7 +59,7 @@ openpmd_option(MPI            "Parallel, Multi-Node I/O for clusters"     AUTO)
 openpmd_option(JSON           "JSON backend (.json files)"                AUTO)
 openpmd_option(HDF5           "HDF5 backend (.h5 files)"                  AUTO)
 openpmd_option(ADIOS1         "ADIOS1 backend (.bp files)"                AUTO)
-openpmd_option(ADIOS2         "ADIOS2 backend (.bp files)"                 OFF)
+openpmd_option(ADIOS2         "ADIOS2 backend (.bp files)"                AUTO)
 openpmd_option(PYTHON         "Enable Python bindings"                    AUTO)
 
 option(openPMD_HAVE_PKGCONFIG        "Generate a .pc file for pkg-config"   ON)
@@ -341,7 +341,10 @@ set(IO_SOURCE
         src/IO/HDF5/ParallelHDF5IOHandler.cpp
         src/IO/JSON/JSONIOHandler.cpp
         src/IO/JSON/JSONIOHandlerImpl.cpp
-        src/IO/JSON/JSONFilePosition.cpp)
+        src/IO/JSON/JSONFilePosition.cpp
+        src/IO/ADIOS/ADIOS2IOHandler.cpp
+        src/IO/ADIOS/ADIOS2Auxiliary.cpp
+        src/IO/InvalidatableFile.cpp)
 set(IO_ADIOS1_SEQUENTIAL_SOURCE
         src/auxiliary/Filesystem.cpp
         src/IO/ADIOS/ADIOS1IOHandler.cpp)

README.md

@@ -105,7 +105,7 @@ Optional I/O backends:
 * [JSON](https://en.wikipedia.org/wiki/JSON)
 * [HDF5](https://support.hdfgroup.org/HDF5) 1.8.13+
 * [ADIOS1](https://www.olcf.ornl.gov/center-projects/adios) 1.13.1+
-* [ADIOS2](https://github.com/ornladios/ADIOS2) 2.4.0+ (*not yet implemented*)
+* [ADIOS2](https://github.com/ornladios/ADIOS2) 2.4.0+
 
 while those can be built either with or without:
 * MPI 2.1+, e.g. OpenMPI 1.6.5+ or MPICH2
@@ -212,15 +212,14 @@ CMake controls options with prefixed `-D`, e.g. `-DopenPMD_USE_MPI=OFF`:
 | `openPMD_USE_JSON`           | **AUTO**/ON/OFF  | JSON backend (`.json` files)                                                 |
 | `openPMD_USE_HDF5`           | **AUTO**/ON/OFF  | HDF5 backend (`.h5` files)                                                   |
 | `openPMD_USE_ADIOS1`         | **AUTO**/ON/OFF  | ADIOS1 backend (`.bp` files)                                                 |
-| `openPMD_USE_ADIOS2`         | AUTO/ON/**OFF**  | ADIOS2 backend (`.bp` files) <sup>1</sup>                                    |
+| `openPMD_USE_ADIOS2`         | **AUTO**/ON/OFF  | ADIOS2 backend (`.bp` files)                                                 |
 | `openPMD_USE_PYTHON`         | **AUTO**/ON/OFF  | Enable Python bindings                                                       |
-| `openPMD_USE_INVASIVE_TESTS` | ON/**OFF**       | Enable unit tests that modify source code <sup>2</sup>                       |
-| `openPMD_USE_VERIFY`         | **ON**/OFF       | Enable internal VERIFY (assert) macro independent of build type <sup>3</sup> |
+| `openPMD_USE_INVASIVE_TESTS` | ON/**OFF**       | Enable unit tests that modify source code <sup>1</sup>                       |
+| `openPMD_USE_VERIFY`         | **ON**/OFF       | Enable internal VERIFY (assert) macro independent of build type <sup>2</sup> |
 | `PYTHON_EXECUTABLE`          | (first found)    | Path to Python executable                                                    |
 
-<sup>1</sup> *not yet implemented*
-<sup>2</sup> *e.g. changes C++ visibility keywords, breaks MSVC*
-<sup>3</sup> *this includes most pre-/post-condition checks, disabling without specific cause is highly discouraged*
+<sup>1</sup> *e.g. changes C++ visibility keywords, breaks MSVC*
+<sup>2</sup> *this includes most pre-/post-condition checks, disabling without specific cause is highly discouraged*
 
 Additionally, the following libraries are shipped internally.
 The following options allow to switch to external installs:

docs/source/backends/adios2.rst

@@ -0,0 +1,77 @@
+.. _backends-adios2:
+
+ADIOS2 Backend
+==============
+
+openPMD supports writing to and reading from ADIOS2 ``.bp`` files.
+For this, the installed copy of openPMD must have been built with support for the ADIOS2 backend.
+To build openPMD with support for ADIOS2, use the CMake option ``-DopenPMD_USE_ADIOS2=ON``.
+For further information, check out the :ref:`installation guide <install>`,
+:ref:`build dependencies <development-dependencies>` and the :ref:`build options <development-buildoptions>`.
+
+
+I/O Method
+----------
+
+ADIOS2 has several engines for alternative file formats and other kinds of backends, yet natively writes to ``.bp`` files. At the moment, the openPMD API exclusively uses the BPFile engine.
+We currently leverage the default ADIOS2 transport parameters, i.e. ``POSIX`` on Unix systems and ``FStream`` on Windows.
+
+
+Backend-Specific Controls
+-------------------------
+
+The following environment variables control ADIOS2 I/O behavior at runtime.
+Fine-tuning these is especially useful when running at large scale.
+
+===================================== ======= ===================================================================
+environment variable                  default description
+===================================== ======= ===================================================================
+``OPENPMD_ADIOS2_HAVE_PROFILING``      ``1``   Turns on/off profiling information right after a run.
+``OPENPMD_ADIOS2_HAVE_METADATA_FILE``  ``1``   Online creation of the adios journal file (``1``: yes, ``0``: no).
+``OPENPMD_ADIOS2_NUM_SUBSTREAMS``      ``0``   Number of files to be created, 0 indicates maximum number possible.
+===================================== ======= ===================================================================
+
+Please refer to the `ADIOS2 manual, section 5.1 <https://media.readthedocs.org/pdf/adios2/latest/adios2.pdf>`_ for details.
+
+
+Best Practice at Large Scale
+----------------------------
+
+A good practice at scale is to disable the online creation of the metadata file.
+After writing the data, run ``bpmeta`` on the (to-be-created) filename to generate the metadata file offline (repeat per iteration for file-based encoding). 
+This metadata file is needed for reading, while the actual heavy data resides in ``<metadata filename>.dir/`` directories.
+Note that such a tool is not yet available for ADIOS2, but the ``bpmeta`` utility provided by ADIOS1 is capable of processing files written by ADIOS2.
+
+Further options depend heavily on filesystem type, specific file striping, network infrastructure and available RAM on the aggregator nodes.
+A good number for substreams is usually the number of contributing nodes divided by four.
+
+For fine-tuning at extreme scale or for exotic systems, please refer to the ADIOS2 manual and talk to your filesystem admins and the ADIOS2 authors.
+Be aware that extreme-scale I/O is a research topic after all.
+
+Selected References
+-------------------
+
+* Hasan Abbasi, Matthew Wolf, Greg Eisenhauer, Scott Klasky, Karsten Schwan, and Fang Zheng.
+  *Datastager: scalable data staging services for petascale applications,*
+  Cluster Computing, 13(3):277–290, 2010.
+  `DOI:10.1007/s10586-010-0135-6 <https://doi.org/10.1007/s10586-010-0135-6>`_
+
+* Ciprian Docan, Manish Parashar, and Scott Klasky.
+  *DataSpaces: An interaction and coordination framework or coupled simulation workflows,*
+  In Proc. of 19th International Symposium on High Performance and Distributed Computing (HPDC’10), June 2010.
+  `DOI:10.1007/s10586-011-0162-y <https://doi.org/10.1007/s10586-011-0162-y>`_
+
+* Qing Liu, Jeremy Logan, Yuan Tian, Hasan Abbasi, Norbert Podhorszki, Jong Youl Choi, Scott Klasky, Roselyne Tchoua, Jay Lofstead, Ron Oldfield, Manish Parashar, Nagiza Samatova, Karsten Schwan, Arie Shoshani, Matthew Wolf, Kesheng Wu, and Weikuan Yu.
+  *Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks,*
+  Concurrency and Computation: Practice and Experience, 26(7):1453–1473, 2014.
+  `DOI:10.1002/cpe.3125 <https://doi.org/10.1002/cpe.3125>`_
+
+* Robert McLay, Doug James, Si Liu, John Cazes, and William Barth.
+  *A user-friendly approach for tuning parallel file operations,*
+  In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC'14, pages 229–236, IEEE Press, 2014.
+  `DOI:10.1109/SC.2014.24 <https://doi.org/10.1109/SC.2014.24>`_
+
+* Axel Huebl, Rene Widera, Felix Schmitt, Alexander Matthes, Norbert Podhorszki, Jong Youl Choi, Scott Klasky, and Michael Bussmann.
+  *On the Scalability of Data Reduction Techniques in Current and Upcoming HPC Systems from an Application Perspective,*
+  ISC High Performance 2017: High Performance Computing, pp. 15-29, 2017.
+  `arXiv:1706.00522 <https://arxiv.org/abs/1706.00522>`_, `DOI:10.1007/978-3-319-67630-2_2 <https://doi.org/10.1007/978-3-319-67630-2_2>`_

docs/source/dev/buildoptions.rst

@@ -18,18 +18,16 @@ CMake Option                   Values          Description
 ``openPMD_USE_JSON``           **AUTO**/ON/OFF JSON backend (``.json`` files)
 ``openPMD_USE_HDF5``           **AUTO**/ON/OFF HDF5 backend (``.h5`` files)
 ``openPMD_USE_ADIOS1``         **AUTO**/ON/OFF ADIOS1 backend (``.bp`` files)
-``openPMD_USE_ADIOS2``         AUTO/ON/**OFF** ADIOS2 backend (``.bp`` files) :sup:`1`
+``openPMD_USE_ADIOS2``         **AUTO**/ON/OFF ADIOS2 backend (``.bp`` files)
 ``openPMD_USE_PYTHON``         **AUTO**/ON/OFF Enable Python bindings
-``openPMD_USE_INVASIVE_TESTS`` ON/**OFF**      Enable unit tests that modify source code :sup:`2`
-``openPMD_USE_VERIFY``         **ON**/OFF      Enable internal VERIFY (assert) macro independent of build type :sup:`3`
+``openPMD_USE_INVASIVE_TESTS`` ON/**OFF**      Enable unit tests that modify source code :sup:`1`
+``openPMD_USE_VERIFY``         **ON**/OFF      Enable internal VERIFY (assert) macro independent of build type :sup:`2`
 ``PYTHON_EXECUTABLE``          (first found)   Path to Python executable
 ============================== =============== ========================================================================
 
-:sup:`1` *not yet implemented*
+:sup:`1` e.g. changes C++ visibility keywords, breaks MSVC
 
-:sup:`2` e.g. changes C++ visibility keywords, breaks MSVC
-
-:sup:`3` this includes most pre-/post-condition checks, disabling without specific cause is highly discouraged
+:sup:`2` this includes most pre-/post-condition checks, disabling without specific cause is highly discouraged
 
 
 Shared or Static

docs/source/dev/dependencies.rst

@@ -30,7 +30,7 @@ Optional: I/O backends
 * `JSON <https://en.wikipedia.org/wiki/JSON>`_
 * `HDF5 <https://support.hdfgroup.org/HDF5>`_ 1.8.13+
 * `ADIOS1 <https://www.olcf.ornl.gov/center-projects/adios>`_ 1.13.1+
-* `ADIOS2 <https://github.com/ornladios/ADIOS2>`_ 2.4.0+ (*not yet implemented*)
+* `ADIOS2 <https://github.com/ornladios/ADIOS2>`_ 2.4.0+
 
 while those can be build either with or without:
 

docs/source/index.rst

@@ -95,6 +95,7 @@ Backends
 
    backends/json
    backends/adios1
+   backends/adios2
 
 Utilities
 ---------
@@ -104,4 +105,3 @@ Utilities
    :hidden:
 
    utilities/benchmark.rst
-