Add support for yoda-H5

[mhabedan]

Replacement of #884 to make the CI run properly.
The changes of this PR are

• Include .yoda.h5 in supported file formats for download requests/urls.
• Update documentation accordingly.
• Add test cases for .yoda.h5.
• As discussed in
  Add yoda-hdf5 support hepdata-converter#64 (comment), this does not add .yoda.h5 to the dropdown menus.

[mhabedan]

Two observations:

1. Apparently, if I push the branch from my fork onto the HEPData remote, the CI for the HEPData branch is mirrored to the fork. So the CI in Add support for yoda-H5 #884 now gives the same result as the CI here.
2. The current CI failure does not seem to be connected to this PR.

[GraemeWatt]

1. Apparently, if I push the branch from my fork onto the HEPData remote, the CI for the HEPData branch is mirrored to the fork. So the CI in Add support for yoda-H5 #884 now gives the same result as the CI here.

Hmm, strange, I don't see a reason for this and it might be a bug in the GitHub web interface. But maybe you can exploit the mirroring to avoid opening new PRs for #878 and #886 as long as the two branches are kept identical, then I could close this PR and merge the original PR #884 instead? Edit: actually, not sure this is a good idea and the pull_request CI job wouldn't run unless you opened a new PR with the same name.

2. The current CI failure does not seem to be connected to this PR.

I re-ran the CI with the same result. All tests pass on the main branch, so I think the failure is connected to your changes. Accessing "View raw logs" gives two test failures in:

FAILED tests/records_test.py::test_get_record_contents - AssertionError: assert '2014-08-11T17:25:55' == '2014-08-11 17:25:55'
FAILED tests/records_test.py::test_get_json_ld - AssertionError: ...

Not sure about the first failure, but for the second you need to modify download_types to add yoda.h5 in the JSON-LD format. Please try to run the tests locally and check they pass before pushing to GitHub.

[mhabedan]

1. Okay, then I'll open new PRs for Analysis JSON schema #878 and Add GAMBIT analysis JSON #886 as soon as I get around to them.
2. I applied the fix you suggested for the second one. But the first one is giving my headaches. It seems to be directly caused by this string replacement and goes away when I remove the " " -> "T" replacement. I don't really see why adding another download format should cause the date format to change. Do you have any idea what might be causing this? Could the test in this form be outdated?

[coveralls]

coverage: 84.058% (+0.3%) from 83.732%
when pulling 716624c on mh-yodaH5
into 9541b29 on main.

[GraemeWatt]

The issue is that get_record_contents falls back to the database if the record is not in the OpenSearch index. The logs (or running the tests locally) gave an error message ERROR:hepdata.ext.opensearch.api:Bulk insert failed when trying to add a record to the index:

'error': {'type': 'illegal_argument_exception', 'reason': "can't merge a non object mapping [access_urls.links.yoda] with an object mapping"}

This was caused by the . in yoda.h5 in FORMATS, so I replaced the . by - where it caused problems in commit 06ac6ae. Tests are passing now.

[Copilot]

Pull Request Overview

This PR adds support for the YODA-H5 file format (yoda.h5) to the HEPData platform. YODA-H5 is a parallel computing-optimized variant of the YODA format for high-energy physics data.

• Add .yoda.h5 to supported file formats across download endpoints and configuration
• Update documentation to describe the new format and its parallel computing benefits
• Add comprehensive test coverage for the new format in JSON-LD metadata generation

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
hepdata/config.py	Add yoda.h5 to supported formats list
hepdata/modules/converter/views.py	Update routes and documentation to support yoda.h5 format
hepdata/modules/records/utils/data_processing_utils.py	Refactor URL generation to include yoda.h5 format
hepdata/modules/records/utils/json_ld.py	Add yoda.h5 to JSON-LD metadata format mappings
hepdata/ext/opensearch/document_enhancers.py	Include yoda.h5 in search document URL generation
hepdata/modules/theme/templates/hepdata_theme/pages/formats.html	Update documentation with yoda.h5 format description
tests/records_test.py	Add test case for yoda.h5 in JSON-LD metadata
tests/e2e/test_general.py	Add end-to-end test coverage for yoda.h5 format
hepdata/version.py	Bump development version

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[GraemeWatt]

@codecov-ai-reviewer review

[GraemeWatt]

@codecov-ai-reviewer test

[ghost]

On it! Codecov is generating unit tests for this PR.

[GraemeWatt]

@codecov-ai-reviewer test

[ghost]

On it! Codecov is generating unit tests for this PR.

[mhabedan]

The issue is that get_record_contents falls back to the database if the record is not in the OpenSearch index. The logs (or running the tests locally) gave an error message ERROR:hepdata.ext.opensearch.api:Bulk insert failed when trying to add a record to the index:

'error': {'type': 'illegal_argument_exception', 'reason': "can't merge a non object mapping [access_urls.links.yoda] with an object mapping"}

This was caused by the . in yoda.h5 in FORMATS, so I replaced the . by - where it caused problems in commit 06ac6ae. Tests are passing now.

Ah, I somehow completely missed that error. Thanks a lot for fixing the problem!