Use OpenPMD standard for output data

[jlvay]

There is a new standard for storing particle and field data that is being adopted by a growing number of Particle-In-Cell codes in the U.S. and Europe. The name of the standard is OpenPMD and its github repo is at OpenPMD standard (https://github.com/openPMD). It can be used with HDF5 or other self-describing data formats and general purpose writer/reader are in development in Python, C++ and FORTRAN. It may be a good standard to adopt for PlasmaPy for I/O. Also, the standard can be extended to accommodates needs that may not be covered yet. It also enables extensions for more specialized subtopics.

[namurphy]

This sounds like a great idea to me! I am strongly in favor of adopting open standards as much as we can. Many thanks to @jlvay for proposing this.

A few of the things I am wondering about are:

• How compatible would this standard be with our adoption of astropy.units?
• Are there any other standards for representing PIC simulation results? As far as I know, there aren't any other detailed specifications like the OpenPMD standard that are in wide use, though I don't do PIC simulations. We definitely want functionality for the open PICKSC codes, and to coordinate with them too.
• How can we best avoid duplication of functionality? (e.g., common methods for atomic data, common classes for representing different species, etc.)
• Is there a similar standard for representing simulations of plasmas using fluid models? We would probably need a way to compare fluid and PIC models directly, since that is often done by several of us who study magnetic reconnection.
• Can we also work with yt after their demeshening? I did notice that there is an openPMD-yt repository.

@PlasmaPy/plasmapy-community

Possibly related to: #87, #12, PlasmaPy/PlasmaPy-PLEPs#6, and possibly #52

[RemiLehe]

Hi @namurphy,

Thanks for your interest in openPMD. Here are a few answers to your questions:

• In the openPMD standard, each dataset can be stored in whichever units the user (or the corresponding PIC codes) sees fit, but then the metadata must contain a coefficient (unitSI) which provides the conversion factor for SI units. The unit dimension (unitDimension) is also stored in the metadata (as a combination of the 7 fundamental dimensions: length, mass, time, electric current, thermodynamic temperature, etc.). So I think the information is there, but one would need to write the corresponding code to convert this information to an astropy.units object.

• Regarding other standards for the output of PIC codes: there are certainly other standards (e.g. Osiris and VSim, for instance, each have their own standard). But to my knowledge, openPMD is the only one which is openly accessible and which has been adopted by several different PIC codes as their default standard for production runs. It would definitely be great to collaborate with the PICKSC codes on it, and their is currently an effort to have QuickPIC compatible with openPMD.

• Regarding fluid models: the openPMD standard can represent both particles and mesh data. I guess a fluid model would use only the mesh part, and I think that such a code could use the standard as is. On the other hand, a Vlasov code would not be able to use the current version of the standard (because the standard does not currently support nd-phase space data ; but this is planned for version 2 of the openPMD standard, hopefully coming soon.)

• yt can already read openPMD files (starting with version 3.4), although the demeshening would certainly make it even better!

[RemiLehe]

Pinging @ax3l (main developer of the openPMD standard)

[ax3l]

just shimming in to let you know I read the discussion and support effort ✨
thx to @RemiLehe for already answering everything so far :)

yt demeshing

uh, that was a pain to implement for our particles and would definitely be better demeshed :) thx for letting me know of the effort, missed that on the yt list.

quote talk: "must know where particles are on disk" -> openPMD particlePatches :)

[namurphy]

Thank you, @RemiLehe and @ax3l! Overall, I think my main question was about whether or not there were any disadvantages to adopting OpenPMD. Right now I can only think of advantages! We'll probably need to plan compatibility with astropy.units a bit more carefully but I think that will not a significant barrier.

We're working on setting up a repository for PlasmaPy Enhancement Proposals, though we still need to establish the process by which they get submitted, reviewed, and decided upon. Adopting the OpenPMD standard would make a good topic for one of these, especially since we would then have a design document that we can refer to during the development process. If we decide to write one, then I would be happy to help out with drafting it.

[StanczakDominik]

I think we can close this with a resounding "heck yeah" :)