extract metadata (NcML XML) from NetCDF/HDF5 files, new "requirements" option for external tools

[pdurbin]

What this PR does / why we need it:

From talking to researchers who use NetCDF and HDF5 files, we believe there is value in extracting metadata from those files to give some insight about their contents.

I plan to also make a pull request in the https://github.com/gdcc/dataverse-previewers repo to add a preview tool for the extracted file, which is in NcML format.

Which issue(s) this PR closes:

• Closes Extract metadata from NetCDF and HDF5 files as XML in NcML format #9153
• Relates to Add NcML previewer to gdcc repo #9242

Special notes for your reviewer:

Additional scope we could consider:

• an API endpoint to extract NcML from existing NetCDF and HDF5 files
• have the "requirements" option apply in more places than file level preview tools
• making the extraction feature modular

Suggestions on how to test this:

• try various NetCDF and HDF5 files and see if NcML aux files are created
• try the new "requirements" option for external tools
• use the proposed NcML previewer in this PR: add NcML previewer #17 gdcc/dataverse-previewers#18

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

No but here's an example of two HDF5 files where one doesn't show the eyeball because it doesn't meet the requirement of having a aux file:

Is there a release notes update needed for this change?:

Yes, included.

Additional documentation:

Included.

[pdurbin]

The failing test is this:

java.lang.AssertionError: expected:<201> but was:<500>
at edu.harvard.iq.dataverse.api.HarvestingClientsIT.testHarvestingClientRun(HarvestingClientsIT.java:184)

I just clicked "build now" to see if that fixes it.

[mreekie]

Update. This was reviewed by Stephen, but is on hold pending some code tweaks from Phil.
Will go back to review by Stephen after that.

[mreekie]

update: Permissions are being discussed.
Stephen/Phil To keep this simple for now, superuser permissions will be required.

[pdurbin]

an API endpoint to extract NcML from existing NetCDF and HDF5 files

I just added this in e2066c8.

I'm ready for more review so I'm unassigning myself.

[pdurbin]

@sekmiller and talked a bit about the new "requirements" feature for external tools. I also removed an extraneous println he caught. Finally, I merged the latest from develop in.

[sekmiller]

Spoke with Phil to get a fuller understanding of "meetsRequirements" on the external tool. passing to qa

[mreekie]

size=10 for the next sprint

[coveralls]

Coverage: 20.0% (-0.01%) from 20.013% when pulling 2302989 on 9153-extract-metadata into 1bef93a on develop.

[pdurbin]

I'm not sure why I'm seeing this at https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/job/PR-9239/8/consoleFull

TASK [dataverse : fire off installer] ******************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "/usr/bin/python3 /tmp/dvinstall/install.py -f --config_file=default.config --noninteractive > /tmp/dvinstall/install.out 2>&1", "delta": "0:02:23.416798", "end": "2023-01-11 23:03:08.721725", "msg": "non-zero return code", "rc": 1, "start": "2023-01-11 23:00:45.304927", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

[kcondon]

Generally works good. Found 2 issues:

1. Extracting metadata with api endpoint works on draft dataset but returns 404 on published one.
   [Kevin] Retested and this also works. What was confusing is the list aux file endpoint shows code":404,"message":"API endpoint does not exist on this server." when really, it's just that the aux file isn't there. When I correctly extracted it and ran the list again, it returned ok. The other part of the confusion is the external tool isn't appearing for a number of the test files so it gives the impression extract didn't work and you then need to do a list aux file.

2. One test hdf file is processed and a NcML aux file is produced but not displaying eye icon for auxfiles previewer tool.
   [Kevin] I'm seeing many more that don't show the external tool aux file viewer icon, though the aux file is extracted and confirmed by list aux file. Might be worth looking at what the tool thinks when it fails?

[kcondon]

Since the first issue was not reproducible and the second affects an external tool, we're merging and will work on the external tool side of things separately.