TOML Backend by franzpoeschel · Pull Request #1436 · openPMD/openPMD-api

franzpoeschel · 2023-05-08T15:33:05Z

Extracted from #1277, since that PR contains thematically related, but clearly distinct items.

TODO:

Performance testing: TOML Parsing is quite slow in toml11. As this is more for lightweight data with user interaction, this is not a huge problem.
Reduce full test suite
Docs
Merge Better handling for file extensions #1473 first

test/SerialIOTest.cpp

franzpoeschel · 2023-05-08T16:17:15Z

We will not be able to activate TOML for our broad test suite as TOML serialization and deserialization are both quite slow.
For the intended usage, this is not a large problem as TOML is intended for small, handwritten datasets.

The following example is the write_and_read_many_iterations test that writes a file-based Series of 1030 iterations per backend. This first profile (created with google-perftools) is without the TOML backend activated:

With TOML backend activated, most time is spent parsing ~1000 TOML files:

src/IO/JSON/JSONIOHandlerImpl.cpp

TOML is not shown as available on NVIDIA compilers

ToruNiina/toml11#205

ax3l · 2023-08-17T03:22:08Z

src/IO/JSON/JSONIOHandlerImpl.cpp

+            }
+#if defined(__INTEL_COMPILER)
+/*
+ * ICPC has trouble with if constexpr, thinking that return statements are


I think this is likely with EDG frontents in general and might also affect nvcc. I remember we saw this there as well and reported it at some point (should be fixed in newer >CUDA 12 versions).

So, we should change the macro to something more precise?
If yes, do you know what would be the right check?

I think I was mumbling here and kept this for future reference, in case we see a warning from it. Nothing to do now.

ax3l · 2023-08-17T03:23:31Z

src/IO/JSON/JSONIOHandlerImpl.cpp

-            dset["data"] = initializeNDArray(parameter.extent);
            break;
        }
+        // TOML does not support nulls, so initialize with zero


Interesting discussion: toml-lang/toml#30

Probably the central bit of that discussion is "TOML is intended for configuration", which is exactly the use case for which we are adding the TOML backen.
If users insist on writing entire datasets with TOML, that's fine too, but it will be initialized with 0.
Otherwise, TOML is mostly intended for usage in conjunction with the follow-up #1493 which only writes the dataset metadata.

Agreed, that makes total sense. Should we be a bit more explicit in our guidance in docs/source/backends/json.rst to avoid that users mistake it as a full-fledged, high-performance data backend?

Yes, I'll add something. Also, in #1493 we could think about enabling the abbreviated modes by default in TOML.

Great idea about the default with #1493, yes! :)

test/python/unittest/API/APITest.py

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

ax3l

Small suggestions on motivation.
Generalizing a bit and clarifying.

docs/source/backends/json.rst

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

ax3l

This is great, thank you! 🎉

* dev: Fix CMake: HDF5 Libs are PUBLIC (openPMD#1520) Fix `chmod` in `download_samples.sh` (openPMD#1518) CI: Old CTest (openPMD#1519) Python: Fix ODR Violation (openPMD#1521) replace extent in weighting and displacement (openPMD#1510) CMake: Warn and Continue on Empty HDF5_VERSION (openPMD#1512) Replace openPMD_Datatypes global with function (openPMD#1509) Streaming examples: Set WAN as default transport (openPMD#1511) TOML Backend (openPMD#1436) make it possible to manually set chunks when loading dask arrays (openPMD#1477) [pre-commit.ci] pre-commit autoupdate (openPMD#1504) Optional debugging output for AbstractIOHandlerImpl::flush() (openPMD#1495) Python: 3.8+ (openPMD#1502) # Conflicts: # .github/workflows/linux.yml # src/binding/python/Series.cpp

franzpoeschel added the backend label May 8, 2023

github-advanced-security bot found potential problems May 8, 2023

View reviewed changes

test/SerialIOTest.cpp Fixed Show fixed Hide fixed

franzpoeschel force-pushed the topic-toml-backend branch from e3b9179 to ad4183c Compare May 9, 2023 10:00

github-advanced-security bot found potential problems May 9, 2023

View reviewed changes

src/IO/JSON/JSONIOHandlerImpl.cpp Fixed Show fixed Hide fixed

src/IO/JSON/JSONIOHandlerImpl.cpp Fixed Show fixed Hide fixed

franzpoeschel force-pushed the topic-toml-backend branch 2 times, most recently from a001ac8 to bc141aa Compare May 10, 2023 15:59

franzpoeschel force-pushed the topic-toml-backend branch 2 times, most recently from 8ef584e to 620cf53 Compare May 24, 2023 14:44

franzpoeschel force-pushed the topic-toml-backend branch from 620cf53 to 25a57de Compare June 27, 2023 08:45

franzpoeschel force-pushed the topic-toml-backend branch 3 times, most recently from 922f674 to 3d0c786 Compare July 10, 2023 09:28

franzpoeschel mentioned this pull request Jul 10, 2023

Parallel JSON #1475

Merged

4 tasks

franzpoeschel force-pushed the topic-toml-backend branch from 3d0c786 to 09012fd Compare July 24, 2023 09:27

franzpoeschel mentioned this pull request Aug 4, 2023

JSON/TOML backend: introduce abbreviated IO modes #1493

Merged

5 tasks

franzpoeschel force-pushed the topic-toml-backend branch from 09012fd to fe4e835 Compare August 4, 2023 13:55

franzpoeschel added 7 commits August 10, 2023 11:38

TOML backend

30c98f5

Add documentation for TOML

3cd65e6

Fixes for long double and long integer types

9140921

Only run TOML tests if TOML is available

c3ab89d

TOML is not shown as available on NVIDIA compilers

Deactivate long double entirely for JSON/TOML

1bfe6a1

CI fix: unused variable

761da4c

Hide/deactivate/warn Toml backend on nvcc compilers

48af319

ToruNiina/toml11#205

franzpoeschel force-pushed the topic-toml-backend branch from fe4e835 to 48af319 Compare August 10, 2023 09:38

ax3l self-requested a review August 17, 2023 03:19

ax3l self-assigned this Aug 17, 2023

ax3l added the backend: TOML label Aug 17, 2023

ax3l reviewed Aug 17, 2023

View reviewed changes

test/python/unittest/API/APITest.py Outdated Show resolved Hide resolved

ax3l reviewed Aug 17, 2023

View reviewed changes

test/python/unittest/API/APITest.py Show resolved Hide resolved

ax3l mentioned this pull request Aug 17, 2023

Add JSON schema #1426

Merged

4 tasks

franzpoeschel and others added 2 commits August 17, 2023 15:42

Usage notes for JSON / TOML

f087215

Update comment in test/python/unittest/API/APITest.py

9b3860c

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

ax3l reviewed Aug 17, 2023

View reviewed changes

docs/source/backends/json.rst Outdated Show resolved Hide resolved

docs/source/backends/json.rst Outdated Show resolved Hide resolved

Update documentation text

7d85f22

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

ax3l approved these changes Aug 17, 2023

View reviewed changes

ax3l enabled auto-merge (squash) August 17, 2023 15:42

ax3l merged commit 9ec90b6 into openPMD:dev Aug 17, 2023

Conversation

franzpoeschel commented May 8, 2023 • edited by ax3l Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

franzpoeschel commented May 8, 2023

Uh oh!

Uh oh!

Uh oh!

ax3l Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

franzpoeschel Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

ax3l Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

ax3l Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

franzpoeschel Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

ax3l Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

franzpoeschel Aug 17, 2023

Choose a reason for hiding this comment

Uh oh!

ax3l Aug 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ax3l left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ax3l left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

franzpoeschel commented May 8, 2023 •

edited by ax3l

Loading

ax3l Aug 17, 2023 •

edited

Loading