Add HDF5 data from w2dynamics

[CaptainSifff]

An HDF5 file generated by w2dynamics: https://github.com/w2dynamics/w2dynamics

[pdurbin]

@CaptainSifff thanks!! Please keep them coming! 🎉

[pdurbin]

Unlike the HDF5 file in PR #34 I'm able to open this one using the NetCDF Java library we've added to Dataverse IQSS/dataverse#9152

Here's the metadata I'm able to extract in plain text and XML formats (see also IQSS/dataverse#9153 ):

• ncdump.txt
• ncml.xml.txt

Here's a little preview of each:

plain text

$ head ncdump.txt 
netcdf /tmp/EDcheck_CIRO-2022-12-08-Thu-20-03-14.hdf5 {
  group: .axes {
    variables:
      double iw(400);

      double iwb-g4(1);

      double iwb-p2(201);

      double iwb-p3(201);
$ tail -6 ncdump.txt
  // global attributes:
  :outfile-version = 2L, 2L; // long
  :code-version = "1", "1", "3";
  :git-revision = "d02d8c9973fec34540061ea3ec5705bbd25bdc5b\n";
  :run-date = "Thu Dec  8 20:03:14 2022";
}

XML

$ head ncml.xml.txt 
<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="file:FIXME">
  <group name=".axes">
    <variable name="iw" shape="400" type="double" />
    <variable name="iwb-g4" shape="1" type="double" />
    <variable name="iwb-p2" shape="201" type="double" />
    <variable name="iwb-p3" shape="201" type="double" />
    <variable name="iwf-g2" shape="2" type="double" />
    <variable name="iwf-g4" shape="2" type="double" />
    <variable name="iwf-p3" shape="200" type="double" />
$ tail ncml.xml.txt
        <variable name="value" shape="" type="double" />
      </group>
    </group>
    <attribute name="desc" value="Stat iteration no. 1" />
  </group>
  <attribute name="outfile-version" type="long" value="2 2" />
  <attribute name="code-version" value="1|1|3" separator="|" />
  <attribute name="git-revision" value="d02d8c9973fec34540061ea3ec5705bbd25bdc5b&#xA;" />
  <attribute name="run-date" value="Thu Dec  8 20:03:14 2022" />
</netcdf>

[pdurbin]

It turns out @JR-1991 knows something about HDF5 and even added support for it in easyDataverse: https://github.com/gdcc/easyDataverse

I asked him to recommend a tool that would let me see folder hierarchy and he suggested HDFView from https://confluence.hdfgroup.org/display/HDFVIEW/HDFView

It works great on this the file (from this PR) from @CaptainSifff:

I also tried brew install hdf5 to get h5dump. Here's the output using the "contents" flag:

% h5dump --contents EDcheck_CIRO-2022-12-08-Thu-20-03-14.hdf5
HDF5 "EDcheck_CIRO-2022-12-08-Thu-20-03-14.hdf5" {
FILE_CONTENTS {
 group      /
 group      /.axes
 dataset    /.axes/iw
 dataset    /.axes/iwb-g4
 dataset    /.axes/iwb-p2
 dataset    /.axes/iwb-p3
 dataset    /.axes/iwf-g2
 dataset    /.axes/iwf-g4
 dataset    /.axes/iwf-p3
 dataset    /.axes/pos-iw
 dataset    /.axes/tau
 dataset    /.axes/tau-g4
 dataset    /.axes/taubin
 dataset    /.axes/tauf
 dataset    /.axes/tausus
 group      /.config
 group      /.environment
 group      /.quantities
 group      /.quantities/accept-flavourchange
 group      /.quantities/accept-glob
 group      /.quantities/accept-ins
 group      /.quantities/accept-ins4
 group      /.quantities/accept-pair-tau
 group      /.quantities/accept-rem
 group      /.quantities/accept-rem4
 group      /.quantities/accept-shift
 group      /.quantities/contrib-sst
 group      /.quantities/contrib-state
 group      /.quantities/energies-eigenstates
 group      /.quantities/fiw
 group      /.quantities/ftau
 group      /.quantities/ftau-full
 group      /.quantities/g0iw
 group      /.quantities/g0iw-full
 group      /.quantities/giw
 group      /.quantities/giw-full
 group      /.quantities/gleg
 group      /.quantities/gleg-full
 group      /.quantities/gtau
 group      /.quantities/gtau-full
 group      /.quantities/hist
 group      /.quantities/hist-seg
 group      /.quantities/hist-sst
 group      /.quantities/hist-state
 group      /.quantities/lhist
 group      /.quantities/muimp
 group      /.quantities/ntau-n0
 group      /.quantities/occ
 group      /.quantities/occbasis-mapping
 group      /.quantities/rhist
 group      /.quantities/sign
 group      /.quantities/sign-sst
 group      /.quantities/sign-state
 group      /.quantities/siw
 group      /.quantities/siw-full
 group      /.quantities/smom
 group      /.quantities/smom-full
 group      /.quantities/ssts-states
 group      /.quantities/time-qmc
 group      /.quantities/time-sampling
 group      /.quantities/time-simulation
 group      /.quantities/time-warmup
 group      /start
 group      /stat-001
 group      /stat-001/ineq-001
 group      /stat-001/ineq-001/accept-flavourchange
 dataset    /stat-001/ineq-001/accept-flavourchange/error
 dataset    /stat-001/ineq-001/accept-flavourchange/value
 group      /stat-001/ineq-001/accept-glob
 dataset    /stat-001/ineq-001/accept-glob/error
 dataset    /stat-001/ineq-001/accept-glob/value
 group      /stat-001/ineq-001/accept-ins
 dataset    /stat-001/ineq-001/accept-ins/error
 dataset    /stat-001/ineq-001/accept-ins/value
 group      /stat-001/ineq-001/accept-ins4
 dataset    /stat-001/ineq-001/accept-ins4/error
 dataset    /stat-001/ineq-001/accept-ins4/value
 group      /stat-001/ineq-001/accept-pair-tau
 dataset    /stat-001/ineq-001/accept-pair-tau/error
 dataset    /stat-001/ineq-001/accept-pair-tau/value
 group      /stat-001/ineq-001/accept-rem
 dataset    /stat-001/ineq-001/accept-rem/error
 dataset    /stat-001/ineq-001/accept-rem/value
 group      /stat-001/ineq-001/accept-rem4
 dataset    /stat-001/ineq-001/accept-rem4/error
 dataset    /stat-001/ineq-001/accept-rem4/value
 group      /stat-001/ineq-001/accept-shift
 dataset    /stat-001/ineq-001/accept-shift/error
 dataset    /stat-001/ineq-001/accept-shift/value
 group      /stat-001/ineq-001/contrib-sst
 dataset    /stat-001/ineq-001/contrib-sst/error
 dataset    /stat-001/ineq-001/contrib-sst/value
 group      /stat-001/ineq-001/contrib-state
 dataset    /stat-001/ineq-001/contrib-state/error
 dataset    /stat-001/ineq-001/contrib-state/value
 group      /stat-001/ineq-001/energies-eigenstates
 dataset    /stat-001/ineq-001/energies-eigenstates/error
 dataset    /stat-001/ineq-001/energies-eigenstates/value
 group      /stat-001/ineq-001/fiw
 dataset    /stat-001/ineq-001/fiw/value
 group      /stat-001/ineq-001/ftau
 dataset    /stat-001/ineq-001/ftau/value
 group      /stat-001/ineq-001/ftau-full
 dataset    /stat-001/ineq-001/ftau-full/value
 group      /stat-001/ineq-001/g0iw
 dataset    /stat-001/ineq-001/g0iw/value
 group      /stat-001/ineq-001/g0iw-full
 dataset    /stat-001/ineq-001/g0iw-full/value
 group      /stat-001/ineq-001/giw
 dataset    /stat-001/ineq-001/giw/value
 group      /stat-001/ineq-001/giw-full
 dataset    /stat-001/ineq-001/giw-full/error
 dataset    /stat-001/ineq-001/giw-full/value
 group      /stat-001/ineq-001/gleg
 dataset    /stat-001/ineq-001/gleg/error
 dataset    /stat-001/ineq-001/gleg/value
 group      /stat-001/ineq-001/gleg-full
 dataset    /stat-001/ineq-001/gleg-full/error
 dataset    /stat-001/ineq-001/gleg-full/value
 group      /stat-001/ineq-001/gtau
 dataset    /stat-001/ineq-001/gtau/error
 dataset    /stat-001/ineq-001/gtau/value
 group      /stat-001/ineq-001/gtau-full
 dataset    /stat-001/ineq-001/gtau-full/error
 dataset    /stat-001/ineq-001/gtau-full/value
 group      /stat-001/ineq-001/hist
 dataset    /stat-001/ineq-001/hist/error
 dataset    /stat-001/ineq-001/hist/value
 group      /stat-001/ineq-001/hist-seg
 dataset    /stat-001/ineq-001/hist-seg/error
 dataset    /stat-001/ineq-001/hist-seg/value
 group      /stat-001/ineq-001/hist-sst
 dataset    /stat-001/ineq-001/hist-sst/error
 dataset    /stat-001/ineq-001/hist-sst/value
 group      /stat-001/ineq-001/hist-state
 dataset    /stat-001/ineq-001/hist-state/error
 dataset    /stat-001/ineq-001/hist-state/value
 group      /stat-001/ineq-001/lhist
 dataset    /stat-001/ineq-001/lhist/error
 dataset    /stat-001/ineq-001/lhist/value
 group      /stat-001/ineq-001/muimp
 dataset    /stat-001/ineq-001/muimp/value
 group      /stat-001/ineq-001/ntau-n0
 dataset    /stat-001/ineq-001/ntau-n0/error
 dataset    /stat-001/ineq-001/ntau-n0/value
 group      /stat-001/ineq-001/occ
 dataset    /stat-001/ineq-001/occ/value
 group      /stat-001/ineq-001/occbasis-mapping
 dataset    /stat-001/ineq-001/occbasis-mapping/error
 dataset    /stat-001/ineq-001/occbasis-mapping/value
 group      /stat-001/ineq-001/rhist
 dataset    /stat-001/ineq-001/rhist/error
 dataset    /stat-001/ineq-001/rhist/value
 group      /stat-001/ineq-001/sign
 dataset    /stat-001/ineq-001/sign/error
 dataset    /stat-001/ineq-001/sign/value
 group      /stat-001/ineq-001/sign-sst
 dataset    /stat-001/ineq-001/sign-sst/error
 dataset    /stat-001/ineq-001/sign-sst/value
 group      /stat-001/ineq-001/sign-state
 dataset    /stat-001/ineq-001/sign-state/error
 dataset    /stat-001/ineq-001/sign-state/value
 group      /stat-001/ineq-001/siw
 dataset    /stat-001/ineq-001/siw/value
 group      /stat-001/ineq-001/siw-full
 dataset    /stat-001/ineq-001/siw-full/error
 dataset    /stat-001/ineq-001/siw-full/value
 group      /stat-001/ineq-001/smom
 dataset    /stat-001/ineq-001/smom/value
 group      /stat-001/ineq-001/smom-full
 dataset    /stat-001/ineq-001/smom-full/error
 dataset    /stat-001/ineq-001/smom-full/value
 group      /stat-001/ineq-001/ssts-states
 dataset    /stat-001/ineq-001/ssts-states/error
 dataset    /stat-001/ineq-001/ssts-states/value
 group      /stat-001/ineq-001/time-qmc
 dataset    /stat-001/ineq-001/time-qmc/error
 dataset    /stat-001/ineq-001/time-qmc/value
 group      /stat-001/ineq-001/time-sampling
 dataset    /stat-001/ineq-001/time-sampling/error
 dataset    /stat-001/ineq-001/time-sampling/value
 group      /stat-001/ineq-001/time-simulation
 dataset    /stat-001/ineq-001/time-simulation/error
 dataset    /stat-001/ineq-001/time-simulation/value
 group      /stat-001/ineq-001/time-warmup
 dataset    /stat-001/ineq-001/time-warmup/error
 dataset    /stat-001/ineq-001/time-warmup/value
 link       /stat-last -> /stat-001
 }
}

[JR-1991]

@pdurbin so far the library expects that the metadata schema fits to the metadatablocks present in the Dataverse installation. There exists a NetCDF Python package that parses the file accordingly. I can write a mapping that automates the upload to Dataverse using EasyDataverse. Do you already know which fields should be mapped to metadata blocks?

[pdurbin]

@JR-1991 so far @atrisovic have mostly been talking about populating geospatial fields (e.g. "bounding box"). Please see our design doc about this: https://groups.google.com/g/dataverse-community/c/rAdXa7hVeH8/m/TnBtdOxJAwAJ

You've attended one of our calls already and we'd love to have you back to discuss this! 😄

[JR-1991]

Perfect! The demo Dataverse instance already has the geospatial block, will go through the design doc and test it there. Going to open a PR at EasyDataverse for this.

[CaptainSifff]

The output of hdfview looks great. This is similar to what I had in mind dataverse to browse the different dataset parts. It also works on the data from MARQOV and ALF, no?

[pdurbin]

@CaptainSifff yes, both the MARQOV (#34) and ALF (#36) data look good in HDFView (as far as I can tell):

Thanks again for that sample data!

[pdurbin]

@CaptainSifff oh, also, the conversation is moving over here: IQSS/dataverse#9442 😄