[WIP] Use JSON/TOML template for defining openPMD metadata in a config file by franzpoeschel · Pull Request #1277 · openPMD/openPMD-api

franzpoeschel · 2022-05-17T15:56:54Z

Not relevant for next release

Until now: Simulations specify their metadata in-code via API calls
With this PR: In some workflows (e.g. experiments) there is no omniscient simulation, but metadata is instead input by the experimentors via configuration files, using the API is not a good workflow for that

Idea: We already have a JSON backend, use an openPMD-conforming JSON dataset to define only metadata. With this, the configuration file will be just another openPMD dataset.
Then, add some functionality to initialize an empty Series from such a metadata file.

TODO:

https://github.com/franzpoeschel/openPMD-api/compare/topic-json-short-modes..topic-json-template

franzpoeschel · 2022-05-18T12:28:26Z

An openPMD dataset in TOML:

[platform_byte_widths]
USHORT = 2
ULONG = 8
BOOL = 1
CLONG_DOUBLE = 32
LONGLONG = 8
CFLOAT = 8
CHAR = 1
DOUBLE = 8
CDOUBLE = 16
SHORT = 2
UCHAR = 1
FLOAT = 4
INT = 4
ULONGLONG = 8
UINT = 4
LONG = 8
LONG_DOUBLE = 16

[data]

[data.0]

[data.0.meshes]

[data.0.meshes.E]

[data.0.meshes.E.x]
datatype = "FLOAT"
data = [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]

[data.0.meshes.E.x.attributes]

[data.0.meshes.E.x.attributes.unitSI]
value = 1.0
datatype = "DOUBLE"

[data.0.meshes.E.x.attributes.position]
value = [0.0]
datatype = "VEC_DOUBLE"

[data.0.meshes.E.attributes]

[data.0.meshes.E.attributes.timeOffset]
value = 0.0
datatype = "FLOAT"

[data.0.meshes.E.attributes.gridUnitSI]
value = 1.0
datatype = "DOUBLE"

[data.0.meshes.E.attributes.gridSpacing]
value = [1.0]
datatype = "VEC_DOUBLE"

[data.0.meshes.E.attributes.gridGlobalOffset]
value = [0.0]
datatype = "VEC_DOUBLE"

[data.0.meshes.E.attributes.unitDimension]
value = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
datatype = "ARR_DBL_7"

[data.0.meshes.E.attributes.geometry]
value = "cartesian"
datatype = "STRING"

[data.0.meshes.E.attributes.dataOrder]
value = "C"
datatype = "STRING"

[data.0.meshes.E.attributes.axisLabels]
value = ["x"]
datatype = "VEC_STRING"

[data.0.attributes]

[data.0.attributes.timeUnitSI]
value = 1.0
datatype = "DOUBLE"

[data.0.attributes.time]
value = 0.0
datatype = "DOUBLE"

[data.0.attributes.dt]
value = 1.0
datatype = "DOUBLE"

[attributes]

[attributes.softwareVersion]
value = "0.15.0-dev"
datatype = "STRING"

[attributes.software]
value = "openPMD-api"
datatype = "STRING"

[attributes.openPMDextension]
value = 0
datatype = "UINT"

[attributes.meshesPath]
value = "meshes/"
datatype = "STRING"

[attributes.iterationFormat]
value = "many_iterations_%T"
datatype = "STRING"

[attributes.iterationEncoding]
value = "fileBased"
datatype = "STRING"

[attributes.openPMD]
value = "1.1.0"
datatype = "STRING"

[attributes.date]
value = "2022-05-18 12:20:23 +0000"
datatype = "STRING"

[attributes.basePath]
value = "/data/%T/"
datatype = "STRING"

franzpoeschel · 2022-05-19T12:01:00Z

This is now a simplified TOML openPMD template, created by {"json":{"mode": "template"}}:

[data]

[data.meshes]

[data.meshes.temperature]
extent = [5, 5]
datatype = "FLOAT"

[data.meshes.temperature.attributes]
timeOffset = 0.0
# Explicit datatype can still be used if needed
unitSI = {"value" = 1.0, "datatype" = "FLOAT"}
position = [0.0]
gridUnitSI = 1.0
gridSpacing = [1.0]
gridGlobalOffset = [0.0]
unitDimension = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
geometry = "cartesian"
dataOrder = "C"
axisLabels = ["x"]

[data.attributes]
timeUnitSI = 1.0
snapshot = 0
time = 0.0
dt = 1.0

[attributes]
softwareVersion = "0.15.0-dev"
software = "openPMD-api"
openPMDextension = 0
meshesPath = "meshes/"
iterationFormat = "/data"
iterationEncoding = "variableBased"
openPMD = "1.1.0"
date = "2022-05-19 11:55:07 +0000"
basePath = "/data"

Differences to regular JSON/TOML openPMD datasets:

Platform byte width table is missing
Attributes don't explicitly store their datatypes, datatypes are dynamically (and a bit heuristically) restored from what is there.
No actual datasets can be written, instead just the extent is stored.

Template mode is also available in json:

{
  "attributes": {
    "basePath": "/data",
    "date": "2022-05-19 12:00:09 +0000",
    "iterationEncoding": "variableBased",
    "iterationFormat": "/data",
    "meshesPath": "meshes/",
    "openPMD": "1.1.0",
    "openPMDextension": 0,
    "software": "openPMD-api",
    "softwareVersion": "0.15.0-dev"
  },
  "data": {
    "attributes": {
      "dt": 1,
      "snapshot": 0,
      "time": 0,
      "timeUnitSI": 1
    },
    "meshes": {
      "temperature": {
        "attributes": {
          "axisLabels": [
            "x"
          ],
          "dataOrder": "C",
          "geometry": "cartesian",
          "gridGlobalOffset": [
            0
          ],
          "gridSpacing": [
            1
          ],
          "gridUnitSI": 1,
          "position": [
            0
          ],
          "timeOffset": 0,
          "unitDimension": [
            0,
            0,
            0,
            0,
            0,
            0,
            0
          ],
          "unitSI": 1
        },
        "datatype": "FLOAT",
        "extent": [
          5,
          5
        ]
      }
    }
  }
}

franzpoeschel · 2022-05-19T15:27:50Z

Longer example:

[data]

[data.particles]

[data.particles.e]

[data.particles.e.positionOffset]

[data.particles.e.positionOffset.z]

[data.particles.e.positionOffset.z.attributes]
value = 3.14
unitSI = 1.0
shape = [5, 5]

[data.particles.e.positionOffset.y]

[data.particles.e.positionOffset.y.attributes]
value = 3.14
unitSI = 1.0
shape = [5, 5]

[data.particles.e.positionOffset.x]

[data.particles.e.positionOffset.x.attributes]
value = 3.14
unitSI = 1.0
shape = [5, 5]

[data.particles.e.positionOffset.attributes]
unitDimension = [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
timeOffset = 0.0

[data.particles.e.position]

[data.particles.e.position.z]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.position.z.attributes]
unitSI = 1.0

[data.particles.e.position.y]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.position.y.attributes]
unitSI = 1.0

[data.particles.e.position.x]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.position.x.attributes]
unitSI = 1.0

[data.particles.e.position.attributes]
unitDimension = [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
timeOffset = 0.0

[data.particles.e.particlePatches]

[data.particles.e.particlePatches.numParticlesOffset]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.numParticlesOffset.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.numParticles]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.numParticles.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.offset]

[data.particles.e.particlePatches.offset.z]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.offset.z.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.offset.y]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.offset.y.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.offset.x]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.offset.x.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.offset.attributes]
unitDimension = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

[data.particles.e.particlePatches.extent]

[data.particles.e.particlePatches.extent.z]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.extent.z.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.extent.y]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.extent.y.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.extent.x]
extent = [5, 5]
datatype = "FLOAT"

[data.particles.e.particlePatches.extent.x.attributes]
unitSI = 1.0

[data.particles.e.particlePatches.extent.attributes]
unitDimension = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

[data.meshes]

[data.meshes.temperature]
extent = [5, 5]
datatype = "FLOAT"

[data.meshes.temperature.attributes]
timeOffset = 0.0
unitSI = 1.0
position = [0.0]
gridUnitSI = 1.0
gridSpacing = [1.0]
gridGlobalOffset = [0.0]
unitDimension = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
geometry = "cartesian"
dataOrder = "C"
axisLabels = ["x"]

[data.meshes.E]

[data.meshes.E.z]
extent = [5, 5]
datatype = "FLOAT"

[data.meshes.E.z.attributes]
unitSI = 1.0
position = [0.0]

[data.meshes.E.y]
extent = [5, 5]
datatype = "FLOAT"

[data.meshes.E.y.attributes]
unitSI = 1.0
position = [0.0]

[data.meshes.E.x]
extent = [5, 5]
datatype = "FLOAT"

[data.meshes.E.x.attributes]
unitSI = 1.0
position = [0.0]

[data.meshes.E.attributes]
timeOffset = 0.0
gridUnitSI = 1.0
gridSpacing = [1.0]
gridGlobalOffset = [0.0]
unitDimension = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
geometry = "cartesian"
dataOrder = "C"
axisLabels = ["x"]

[data.attributes]
timeUnitSI = 1.0
snapshot = 0
time = 0.0
dt = 1.0

[attributes]
softwareVersion = "0.15.0-dev"
particlesPath = "particles/"
software = "openPMD-api"
openPMDextension = 0
meshesPath = "meshes/"
iterationFormat = "/data"
iterationEncoding = "variableBased"
openPMD = "1.1.0"
date = "2022-05-19 15:26:37 +0000"
basePath = "/data"

franzpoeschel · 2022-07-21T16:38:54Z

Notes for myself on the recent reodering of commits:

5 3ee509de (HEAD -> topic-json-template, origin/topic-json-template) Properly deal with undefined datasets
2 06da2d58 Make JSON and TOML look like two different backends
5 960ab21a Initialize Dataset definitions from template
5 b88bae67 Initialize Series attributes from template
3 6302a33c Fix NVHPC Toml11 open mode
2 d825008b Fix precision-losing type conversion
4 da960a23 Enable .toml tests in generic tests
4 0398b86f Extend example
3 7332996e Windows compatibility
x 85527799 Add and use Attribute::getOptional<T>()
1 64cde966 Template mode: Fill with zero upon read
1 fa483843 Write/read shorthand attributes without explicit datatype
3 bd8da013 CI fixes
1 d802d2ac Don't write platform datatype size table in template mode
2 cba71f7f Use .toml as filename extension
2 b019a7d1 TOML as alternative backend for JSON backend
1 4b25de8c Select template mode via JSON param
1 8ef4753f Add template mode to JSON backend

franzpoeschel · 2022-09-15T09:23:37Z

src/RecordComponent.cpp

+        //     throw error::WrongAPIUsage(
+        //         "[RecordComponent] Must set specific datatype (Use "
+        //         "resetDataset call).");
+        // }


Note that this check was inactive up to now, since RecordComponentData::RecordComponentData initialized that field with Datatype::CHAR. Using an optional would make these things more obvious and avoid such pitfalls.
To be done in a different PR though

#1316 now uses std::optional

examples/14_toml_template.cpp

src/IO/JSON/JSONIOHandlerImpl.cpp

test/SerialIOTest.cpp

Has the same implementation as JSON anyway, and it makes tests run into timeouts otherwise.

franzpoeschel · 2026-01-08T11:57:16Z

The relevant parts of this have been merged separately. The remainder can be closed as not planned.

franzpoeschel added backend: JSON api: new additions to the API labels May 17, 2022

franzpoeschel force-pushed the topic-json-template branch from f10fc90 to c63c06a Compare May 18, 2022 12:26

franzpoeschel force-pushed the topic-json-template branch 3 times, most recently from 0d475a5 to 1a23a03 Compare May 19, 2022 11:54

franzpoeschel force-pushed the topic-json-template branch from 1a23a03 to 656e740 Compare May 19, 2022 12:33

franzpoeschel mentioned this pull request May 19, 2022

Add Attribute::getOptional<T>() and use to add some more dynamic datatype conversions at read time #1278

Merged

franzpoeschel force-pushed the topic-json-template branch 5 times, most recently from 475be7b to ee8bdf1 Compare May 23, 2022 11:14

franzpoeschel changed the title ~~Use JSON/TOML template for defining openPMD metadata in a config file~~ [WIP] Use JSON/TOML template for defining openPMD metadata in a config file May 23, 2022

franzpoeschel force-pushed the topic-json-template branch 3 times, most recently from d32fff3 to 376bc2a Compare July 5, 2022 09:23

franzpoeschel force-pushed the topic-json-template branch 2 times, most recently from 3ee509d to a662865 Compare July 21, 2022 16:35

franzpoeschel force-pushed the topic-json-template branch 2 times, most recently from 1db63f6 to 55b72f8 Compare July 29, 2022 09:00

franzpoeschel force-pushed the topic-json-template branch 2 times, most recently from b07a2a8 to a41c2c6 Compare August 17, 2022 09:21

franzpoeschel commented Sep 15, 2022

View reviewed changes

franzpoeschel mentioned this pull request Oct 4, 2022

RecordComponent: Properly handle uninitialized datasets #1316

Merged

6 tasks

franzpoeschel force-pushed the topic-json-template branch from a41c2c6 to c6a369c Compare October 12, 2022 09:45

franzpoeschel force-pushed the topic-json-template branch from 0b26e4f to cf5c134 Compare August 4, 2023 14:10

github-advanced-security bot found potential problems Aug 4, 2023

View reviewed changes

examples/14_toml_template.cpp Fixed Show fixed Hide fixed

franzpoeschel force-pushed the topic-json-template branch from cf5c134 to 7518036 Compare August 10, 2023 09:39

franzpoeschel force-pushed the topic-json-template branch from 7518036 to d4fe5c3 Compare September 5, 2023 11:51

franzpoeschel force-pushed the topic-json-template branch from d4fe5c3 to 94f2797 Compare October 12, 2023 13:19

github-advanced-security bot found potential problems Oct 12, 2023

View reviewed changes

src/IO/JSON/JSONIOHandlerImpl.cpp Fixed Show fixed Hide fixed

franzpoeschel force-pushed the topic-json-template branch 2 times, most recently from 2db2d3a to 3133351 Compare November 20, 2023 16:37

github-advanced-security bot found potential problems Nov 20, 2023

View reviewed changes

src/IO/JSON/JSONIOHandlerImpl.cpp Fixed Show fixed Hide fixed

franzpoeschel force-pushed the topic-json-template branch 3 times, most recently from 2b2bbb6 to 367798c Compare December 22, 2023 18:35

franzpoeschel force-pushed the topic-json-template branch from 367798c to c497fef Compare January 4, 2024 11:48

franzpoeschel force-pushed the topic-json-template branch from fb75b46 to f0e2e21 Compare January 23, 2024 16:20

franzpoeschel force-pushed the topic-json-template branch from 8ed49ca to 57a5fb2 Compare February 5, 2024 10:32

franzpoeschel force-pushed the topic-json-template branch from 57a5fb2 to 771a644 Compare February 14, 2024 10:16

franzpoeschel force-pushed the topic-json-template branch 3 times, most recently from 1307f10 to b5eecbb Compare February 29, 2024 13:34

franzpoeschel force-pushed the topic-json-template branch from b5eecbb to 2f6b7d1 Compare March 26, 2024 11:17

github-advanced-security bot found potential problems Jun 7, 2024

View reviewed changes

test/SerialIOTest.cpp Fixed Show fixed Hide fixed

franzpoeschel added 8 commits November 21, 2025 11:19

Initialize Series attributes and datasets from template

a82d409

Further testing (@todo: cleanup)

f309d2a

Extend example

d7b8c8c

Only opt-into tests for TOML

2a417ae

Has the same implementation as JSON anyway, and it makes tests run into timeouts otherwise.

Adapt this to changed SCALAR API

ee2998c

Avoid CI warnings

74bd097

Fix after rebasing

b85db02

Ensure that E.z is there

e343049

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Use JSON/TOML template for defining openPMD metadata in a config file#1277

[WIP] Use JSON/TOML template for defining openPMD metadata in a config file#1277
franzpoeschel wants to merge 8 commits intoopenPMD:devfrom
franzpoeschel:topic-json-template

franzpoeschel commented May 17, 2022 •

edited

Loading

Uh oh!

franzpoeschel commented May 18, 2022 •

edited

Loading

Uh oh!

franzpoeschel commented May 19, 2022 •

edited

Loading

Uh oh!

franzpoeschel commented May 19, 2022

Uh oh!

franzpoeschel commented Jul 21, 2022

Uh oh!

franzpoeschel Sep 15, 2022

Uh oh!

franzpoeschel May 9, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

franzpoeschel commented Jan 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

franzpoeschel commented May 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

franzpoeschel commented May 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

franzpoeschel commented May 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

franzpoeschel commented May 19, 2022

Uh oh!

franzpoeschel commented Jul 21, 2022

Uh oh!

franzpoeschel Sep 15, 2022

Choose a reason for hiding this comment

Uh oh!

franzpoeschel May 9, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

franzpoeschel commented Jan 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

franzpoeschel commented May 17, 2022 •

edited

Loading

franzpoeschel commented May 18, 2022 •

edited

Loading

franzpoeschel commented May 19, 2022 •

edited

Loading