Extract metadata from NetCDF and HDF5 files as XML in NcML format

[pdurbin]

Assuming PR #9152 is merged we'll have a library in place to start extracting XML from NetCDF and HDF5 files.

The supported XML format is called NcML and is described here: https://docs.unidata.ucar.edu/netcdf-java/current/userguide/ncml_overview.html

Yesterday there was general agreement among devs that it would be fine to save the XML as a derivative or aux file.

This will open the door for previewing the file as raw XML to start.

Additionally, we could work on created a dedicated previewer that shows the data in a nicer way than raw XML.

The code we write will look something like this:

String ncml = netcdfFile.toNcml(file.getName());

Here's the output for an HDF5 file at src/test/resources/hdf/hdf5/vlen_string_dset (from the PR above):

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="file:vlen_string_dset">
  <variable name="DS1" shape="4" type="String" />
</netcdf>

Here's part of the output for a NetCDF file at src/test/resources/netcdf/madis-raob.nc (also from the PR above):

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="file:madis-raob.nc">
  <dimension name="recNum" length="1" isUnlimited="true" />
  <dimension name="manLevel" length="22" />
  <dimension name="sigTLevel" length="150" />
  <dimension name="sigWLevel" length="76" />
  <dimension name="mWndNum" length="4" />
  <dimension name="mTropNum" length="4" />
  <dimension name="staNameLen" length="50" />
  <dimension name="QCcheckNum" length="10" />
  <dimension name="QCcheckNameLen" length="60" />
  <dimension name="maxStaticIds" length="1000" />
  <dimension name="totalIdLen" length="50" />
  <dimension name="nInventoryBins" length="32" />
  <variable name="nStaticIds" shape="" type="int">
    <attribute name="_FillValue" type="int" value="0" />
  </variable>
  <variable name="staticIds" shape="maxStaticIds totalIdLen" type="char">
    <attribute name="_FillValue" value="" />
  </variable>
  <variable name="lastRecord" shape="maxStaticIds" type="int">
    <attribute name="_FillValue" type="int" value="-1" />
  </variable>
  <variable name="invTime" shape="recNum" type="int">
    <attribute name="_FillValue" type="int" value="0" />
  </variable>
...

Here's the full XML/NcML output: madis-ncml.xml.txt

---

2.5 years ago @qqmyers made some suggestions for previewing XML files at IQSS/dataverse.harvard.edu#70 (comment) . Here's his comment:

"FWIW: Something like https://www.jqueryscript.net/other/tree-xml-viewer-formatter.html adapted with the wiki instructions at https://github.com/GlobalDataverseCommunityConsortium/dataverse-previewers/wiki/How-to-create-a-previewer might be a quick win. (I didn't search too hard for an XML viewer - there could be better libraries out there to start from.)"

[pdurbin]

Just a quick update that I have some uncommitted code locally that extracts an NcML file (XML) from a NetCDF file and saves it as an auxiliary file:

The code is very hacky and probably doesn't work with S3. I'm been chatting with @landreev about the best place to put it. Probably a dedicated method the gets called right after ingestService.startIngestJobsForDataset and put in the same bean.

[pdurbin]

I cleaned up the code a bit, added a lot of TODOs, and pushed it to a branch so that @landreev and others can take a look: 711dc63

[mreekie]

Discussion/Notes
Leonid - When we expand this system to extract other types of metadata from files, we may consider moving this to an asynchronous queue. But for now it's out of scope.

Part of what's left is deciding if it's a goal to support direct upload.

When you don't do direct upload, there is a temporary file.
Direct - browser to S3 directly, no temporary file to have in the interim.

Two PRs will come out of this, one for viewer, one for dataverse.

This is hard coded right now. It doesn't use a queue such as JMS (which we use for file ingest).
Does not support direct upload.

Some installations will care about these formats some will. Is this the beginning of a new class of plugins?

Size for this sprint was a 33.
Size what's left at an 80

TODO: Phil put a sentence or two into laying out better than what mike did the scope of this work.
TODO: sounds like we need a follow-on investigation on the bigger picture which is outside the scope of this issue.