Expose export formats in native API by julian-schneider · Pull Request #10739 · IQSS/dataverse

julian-schneider · 2024-08-02T12:57:12Z

What this PR does / why we need it: The PR adds a method that exposes the full list of available exporters (display- and format names). This is especially useful for instances that have added custom formats.

Which issue(s) this PR closes: I know no pre-existing issues for this.

Suggestions on how to test this: New method found here: /api/info/exportFormats

Does this PR introduce a user interface change? If mockups are available, please link/include them here: No, just API and documentation changes.

API answer looks like this:

{
  "status": "OK",
  "data": [
    {
      "displayName": "OAI_ORE",
      "formatName": "OAI_ORE"
    },
    {
      "displayName": "DataCite",
      "formatName": "Datacite"
    },
    {
      "displayName": "Dublin Core",
      "formatName": "oai_dc"
    },
    {
      "displayName": "OpenAIRE",
      "formatName": "oai_datacite"
    },
    {
      "displayName": "Schema.org JSON-LD",
      "formatName": "schema.org"
    },
    {
      "displayName": "DDI",
      "formatName": "ddi"
    },
    {
      "displayName": "Dublin Core",
      "formatName": "dcterms"
    },
    {
      "displayName": "DDI HTML Codebook",
      "formatName": "html"
    },
    {
      "displayName": "JSON",
      "formatName": "dataverse_json"
    },
    {
      "displayName": "DDI",
      "formatName": "oai_ddi"
    }
  ]
}

Rendered docs entry looks like this:

Live version: https://dataverse-guide--10739.org.readthedocs.build/en/10739/api/native-api.html#get-export-formats

Is there a release notes update needed for this change?:
New API method for listing the available exporters. Found at /api/info/exportFormats, produces display names and format names.

pdurbin

@julian-schneider good idea! Thanks for the pull request. I left you some comments.

pdurbin · 2024-08-02T13:39:33Z

src/main/java/edu/harvard/iq/dataverse/api/Info.java

+    @GET
+    @Path("exportFormats")
+    public Response getExportFormats() {
+        JsonArrayBuilder responseModel = Json.createArrayBuilder();


Did you consider returning an object instead? Just this week I pushed a commit to return via API if feature flags are enabled or not and I played with returning an array but ended up liking object better. Here's the code: 771d85a (for #10732)

I'm suggesting something like this:

{ "status": "OK", "data": { "OAI_ORE": { "displayName": "OAI_ORE" }, "oai_dc": { "displayName": "Dublin Core" } } }

Alternatively, perhaps you could add a second endpoint for getting info about a single export format (e.g. /api/info/exportFormats/oai_dc). That way you can easily get info on that format instead of having to search through an array to find the element that matches oai_dc.

Just some thoughts! 😄 Maybe the array is actually better.

Yeah an object is probably better, especially with the increased amount of properties, so I switched this in cc99729

Related: https://softwareengineering.stackexchange.com/questions/311773/is-returning-a-json-object-instead-an-array-via-rest-good-or-bad-style

src/main/java/edu/harvard/iq/dataverse/api/Info.java

doc/sphinx-guides/source/api/native-api.rst

pdurbin · 2024-08-02T13:47:10Z

doc/sphinx-guides/source/api/native-api.rst

+Get Export Formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Get the available export formats. The response contains a list of objects each containing the display name and format name for an available exporter.


Does this include external exporters? I assume so. Can we please mention they are included as well?

As far as I know, all exporters, standard and custom, are managed by the ExportService, and are consequently all listed. I improved the docs in 3b2c8e5.

src/main/java/edu/harvard/iq/dataverse/api/Info.java

julian-schneider · 2024-08-02T14:20:13Z

Thanks for your feedback @pdurbin - I will be absent for two weeks, so some time will pass before I revise this. See you then!

…ing response as JSON object

julian-schneider · 2024-08-28T09:15:53Z

The response is now an object, and contains more properties per format, as suggested.

New response looks like this:

(expand response)

{
  "status": "OK",
  "data": {
    "OAI_ORE": {
      "displayName": "OAI_ORE",
      "mediaType": "application/json",
      "isHarvestable": false,
      "isVisibleInUserInterface": true
    },
    "Datacite": {
      "displayName": "DataCite",
      "mediaType": "application/xml",
      "isHarvestable": true,
      "isVisibleInUserInterface": true,
      "XMLNameSpace": "http://datacite.org/schema/kernel-3",
      "XMLSchemaLocation": "http://datacite.org/schema/kernel-3 http://schema.datacite.org/meta/kernel-3/metadata.xsd",
      "XMLSchemaVersion": "3.0"
    },
    "oai_dc": {
      "displayName": "Dublin Core",
      "mediaType": "application/xml",
      "isHarvestable": true,
      "isVisibleInUserInterface": false,
      "XMLNameSpace": "http://www.openarchives.org/OAI/2.0/oai_dc/",
      "XMLSchemaLocation": "http://www.openarchives.org/OAI/2.0/oai_dc.xsd",
      "XMLSchemaVersion": "2.0"
    },
    "oai_datacite": {
      "displayName": "OpenAIRE",
      "mediaType": "application/xml",
      "isHarvestable": true,
      "isVisibleInUserInterface": true,
      "XMLNameSpace": "http://datacite.org/schema/kernel-4",
      "XMLSchemaLocation": "http://schema.datacite.org/meta/kernel-4.1/metadata.xsd",
      "XMLSchemaVersion": "4.1"
    },
    "schema.org": {
      "displayName": "Schema.org JSON-LD",
      "mediaType": "application/json",
      "isHarvestable": false,
      "isVisibleInUserInterface": true
    },
    "ddi": {
      "displayName": "DDI",
      "mediaType": "application/xml",
      "isHarvestable": false,
      "isVisibleInUserInterface": true,
      "XMLNameSpace": "ddi:codebook:2_5",
      "XMLSchemaLocation": "https://ddialliance.org/Specification/DDI-Codebook/2.5/XMLSchema/codebook.xsd",
      "XMLSchemaVersion": "2.5"
    },
    "dcterms": {
      "displayName": "Dublin Core",
      "mediaType": "application/xml",
      "isHarvestable": false,
      "isVisibleInUserInterface": true,
      "XMLNameSpace": "http://purl.org/dc/terms/",
      "XMLSchemaLocation": "http://dublincore.org/schemas/xmls/qdc/dcterms.xsd",
      "XMLSchemaVersion": "2.0"
    },
    "html": {
      "displayName": "DDI HTML Codebook",
      "mediaType": "text/html",
      "isHarvestable": false,
      "isVisibleInUserInterface": true
    },
    "dataverse_json": {
      "displayName": "JSON",
      "mediaType": "application/json",
      "isHarvestable": true,
      "isVisibleInUserInterface": true
    },
    "oai_ddi": {
      "displayName": "DDI",
      "mediaType": "application/xml",
      "isHarvestable": true,
      "isVisibleInUserInterface": false,
      "XMLNameSpace": "ddi:codebook:2_5",
      "XMLSchemaLocation": "https://ddialliance.org/Specification/DDI-Codebook/2.5/XMLSchema/codebook.xsd",
      "XMLSchemaVersion": "2.5"
    }
  }
}

New docs look like this:

julian-schneider · 2024-08-28T12:35:03Z

@pdurbin I believe I have addressed all the feedback you gave, have a nice day!

coveralls · 2024-08-31T11:50:45Z

coverage: 20.76% (+0.02%) from 20.741%
when pulling 3f81981 on julian-schneider:expose-export-formats
into 0d27957 on IQSS:develop.

johannes-darms · 2024-10-11T09:21:08Z

@pdurbin @GPortas could be get this API into the next release? This is a rather simple feature needed for the SPA to display the list of export formats.

pdurbin

I haven't run the code yet (or in a while, I forget) but I left a couple more suggestions.

Please also merge the latest from develop.

src/test/resources/json/export-formats.json

src/test/java/edu/harvard/iq/dataverse/api/InfoIT.java

pdurbin · 2024-10-15T18:21:17Z

@johannes-darms I do like the PR a lot. @cmbz @scolapasta what do you think? Can we put a 6.5 milestone on it or at least put it in "sprint ready"?

Also, I agree with Johannes that the SPA will need this some day. We might want to see what @ekraffmiller @ChengShi-1 @GPortas and @g-saracca think about the output, if they are happy with it.

cmbz · 2024-10-15T18:48:37Z

2024/10/15: Added to sprint ready after conversation with @pdurbin

pdurbin

@julian-schneider this is looking really good. I left you a few comments and a PR-for-this-PR ( julian-schneider#1 ) to consider. Also, can you please merge the latest from "develop" into your branch? Thanks!

doc/release-notes/expose-export-formats.md

doc/sphinx-guides/source/api/native-api.rst

pdurbin · 2024-11-12T19:12:30Z

src/main/java/edu/harvard/iq/dataverse/api/Info.java

+        for (String[] labels : instance.getExportersLabels()) {
+            try {
+                Exporter exporter = instance.getExporter(labels[1]);


This lookup seems a bit awkward but it's not your fault. I wonder if we should add a method to ExportService like this...

public Collection<Exporter> getExporters() { return exporterMap.values(); }

... so that it can be used like this in places like Info.java:

for (Exporter exporter : instance.getExporters()) { //... }

But I dunno. Maybe there's a reason why we don't offer a "getExporters" method like this.

I agree that the steps I had to use to get the exporters were a bit awkward. If you think it would be in scope, I can add a method like that to this PR.

No, let's think about it outside this PR. You're welcome to create an issue, though, if you like!

src/test/java/edu/harvard/iq/dataverse/api/InfoIT.java

src/main/java/edu/harvard/iq/dataverse/api/Info.java

remove Gson and test based on content, not string match IQSS#10739

Co-authored-by: Philip Durbin <philipdurbin@gmail.com>

pdurbin · 2024-11-20T13:41:58Z

@julian-schneider tests are failing in InfoIT.testGetExportFormats. Can you please take a look?

Error
data.Datacite.XMLNameSpace
Expected: http://datacite.org/schema/kernel-3
     got: http://datacite.org/schema/kernel-4
 ; data.Datacite.XMLSchemaLocation
Expected: http://datacite.org/schema/kernel-3 http://schema.datacite.org/meta/kernel-3/metadata.xsd
     got: http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4.5/metadata.xsd
 ; data.Datacite.XMLSchemaVersion
Expected: 3.0
     got: 4.5
 ; data.ddi.displayName
Expected: DDI
     got: DDI Codebook v2
 ; data.oai_ddi.displayName
Expected: DDI
     got: DDI Codebook v2
Stacktrace
java.lang.AssertionError: 
data.Datacite.XMLNameSpace
Expected: http://datacite.org/schema/kernel-3
     got: http://datacite.org/schema/kernel-4
 ; data.Datacite.XMLSchemaLocation
Expected: http://datacite.org/schema/kernel-3 http://schema.datacite.org/meta/kernel-3/metadata.xsd
     got: http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4.5/metadata.xsd
 ; data.Datacite.XMLSchemaVersion
Expected: 3.0
     got: 4.5
 ; data.ddi.displayName
Expected: DDI
     got: DDI Codebook v2
 ; data.oai_ddi.displayName
Expected: DDI
     got: DDI Codebook v2
	at org.skyscreamer.jsonassert.JSONAssert.assertEquals(JSONAssert.java:417)
	at org.skyscreamer.jsonassert.JSONAssert.assertEquals(JSONAssert.java:394)
	at org.skyscreamer.jsonassert.JSONAssert.assertEquals(JSONAssert.java:336)
	at edu.harvard.iq.dataverse.api.InfoIT.testGetExportFormats(InfoIT.java:99)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Standard Output
{
    "status": "OK",
    "data": {
        "OAI_ORE": {
            "displayName": "OAI_ORE",
            "mediaType": "application/json",
            "isHarvestable": false,
            "isVisibleInUserInterface": true
        },
        "Datacite": {
            "displayName": "DataCite",
            "mediaType": "application/xml",
            "isHarvestable": true,
            "isVisibleInUserInterface": true,
            "XMLNameSpace": "http://datacite.org/schema/kernel-4",
            "XMLSchemaLocation": "http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4.5/metadata.xsd",
            "XMLSchemaVersion": "4.5"
        },
        "oai_dc": {
            "displayName": "Dublin Core",
            "mediaType": "application/xml",
            "isHarvestable": true,
            "isVisibleInUserInterface": false,
            "XMLNameSpace": "http://www.openarchives.org/OAI/2.0/oai_dc/",
            "XMLSchemaLocation": "http://www.openarchives.org/OAI/2.0/oai_dc.xsd",
            "XMLSchemaVersion": "2.0"
        },
        "oai_datacite": {
            "displayName": "OpenAIRE",
            "mediaType": "application/xml",
            "isHarvestable": true,
            "isVisibleInUserInterface": true,
            "XMLNameSpace": "http://datacite.org/schema/kernel-4",
            "XMLSchemaLocation": "http://schema.datacite.org/meta/kernel-4.1/metadata.xsd",
            "XMLSchemaVersion": "4.1"
        },
        "schema.org": {
            "displayName": "Schema.org JSON-LD",
            "mediaType": "application/json",
            "isHarvestable": false,
            "isVisibleInUserInterface": true
        },
        "ddi": {
            "displayName": "DDI Codebook v2",
            "mediaType": "application/xml",
            "isHarvestable": false,
            "isVisibleInUserInterface": true,
            "XMLNameSpace": "ddi:codebook:2_5",
            "XMLSchemaLocation": "https://ddialliance.org/Specification/DDI-Codebook/2.5/XMLSchema/codebook.xsd",
            "XMLSchemaVersion": "2.5"
        },
        "dcterms": {
            "displayName": "Dublin Core",
            "mediaType": "application/xml",
            "isHarvestable": false,
            "isVisibleInUserInterface": true,
            "XMLNameSpace": "http://purl.org/dc/terms/",
            "XMLSchemaLocation": "http://dublincore.org/schemas/xmls/qdc/dcterms.xsd",
            "XMLSchemaVersion": "2.0"
        },
        "html": {
            "displayName": "DDI HTML Codebook",
            "mediaType": "text/html",
            "isHarvestable": false,
            "isVisibleInUserInterface": true
        },
        "dataverse_json": {
            "displayName": "JSON",
            "mediaType": "application/json",
            "isHarvestable": true,
            "isVisibleInUserInterface": true
        },
        "oai_ddi": {
            "displayName": "DDI Codebook v2",
            "mediaType": "application/xml",
            "isHarvestable": true,
            "isVisibleInUserInterface": false,
            "XMLNameSpace": "ddi:codebook:2_5",
            "XMLSchemaLocation": "https://ddialliance.org/Specification/DDI-Codebook/2.5/XMLSchema/codebook.xsd",
            "XMLSchemaVersion": "2.5"
        }
    }
}

pdurbin

As of f3b72c6 tests are passing: https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/job/PR-10739/9/testReport/

Thanks, @julian-schneider! Approved.

In DMs we did discuss how if there's a lot of churn in our formats we can do some mocking. Still, it's valuable to actually exercise the API endpoint itself, even if we make the assertions less strict in the future.

ofahimIQSS · 2024-11-22T18:51:15Z

No issues with PR discovered.
Additional tests were done in internal before and after installing Croissant. As expected, Croissant did appear in /api/info/exportFormats after installation
Testing of 10739.docx

julian-schneider added 2 commits August 2, 2024 13:50

Expose list of export formats in Info API

5d0352a

Document exportFormats method in native API guide

49dd83b

pdurbin added Size: 3 A percentage of a sprint. 2.1 hours. Type: Feature a feature request labels Aug 2, 2024

pdurbin reviewed Aug 2, 2024

View reviewed changes

julian-schneider added 2 commits August 28, 2024 10:47

Update api/info/exportFormats to give more available info, restructur…

cc99729

…ing response as JSON object

Document updated exportFormats method in native API guide

3b2c8e5

julian-schneider added 2 commits August 28, 2024 14:08

Add API test for api/info/exportFormats

95d34c8

Add release note snippet for exportFormats API method release

3f81981

pdurbin reviewed Oct 15, 2024

View reviewed changes

src/test/resources/json/export-formats.json Outdated Show resolved Hide resolved

src/test/java/edu/harvard/iq/dataverse/api/InfoIT.java Outdated Show resolved Hide resolved

pdurbin added the Champion: pdurbin label Oct 15, 2024

cmbz added this to the 6.5 milestone Oct 15, 2024

cmbz added the FY25 Sprint 10 FY25 Sprint 10 (2024-11-06 - 2024-11-20) label Nov 7, 2024

pdurbin removed the Champion: pdurbin label Nov 12, 2024

remove Gson and test based on content, not string match IQSS#10739

594acb7

pdurbin mentioned this pull request Nov 12, 2024

Remove Gson, Jackson and Everit JSON Schema Validation code dependencies #6810

Open

pdurbin reviewed Nov 12, 2024

View reviewed changes

pdurbin assigned pdurbin and julian-schneider Nov 12, 2024

pdurbin added the Status: Needs Input Applied to issues in need of input from someone currently unavailable label Nov 19, 2024

julian-schneider and others added 3 commits November 20, 2024 11:46

Merge pull request #1 from IQSS/10739-exportFormats

dd4b4d7

remove Gson and test based on content, not string match IQSS#10739

Merge branch 'develop' into expose-export-formats

6a033df

Add PR number to doc/release-notes/expose-export-formats.md

48d04e8

Co-authored-by: Philip Durbin <philipdurbin@gmail.com>

pdurbin unassigned julian-schneider Nov 20, 2024

pdurbin removed the Status: Needs Input Applied to issues in need of input from someone currently unavailable label Nov 20, 2024

pdurbin assigned julian-schneider Nov 20, 2024

Update expected JSON for api/info/exportFormats IT Test

f3b72c6

pdurbin approved these changes Nov 21, 2024

View reviewed changes

pdurbin unassigned pdurbin and julian-schneider Nov 21, 2024

cmbz added the FY25 Sprint 11 FY25 Sprint 11 (2024-11-20 - 2024-12-04) label Nov 21, 2024

ofahimIQSS self-assigned this Nov 22, 2024

ofahimIQSS merged commit b8e4758 into IQSS:develop Nov 22, 2024

ofahimIQSS removed their assignment Nov 22, 2024

pdurbin mentioned this pull request Nov 22, 2024

expose links to all export formats via Signposting #11045

Merged

pdurbin mentioned this pull request Dec 17, 2024

add new non-admin APIs for external tools registration (for marketplace) #11079

Merged

pdurbin mentioned this pull request Feb 13, 2025

enable the Croissant exporter by default (move code to main repo) #11254

Closed

pdurbin mentioned this pull request Mar 24, 2025

Add Export Metadata Dropdown IQSS/dataverse-frontend#189

Closed

Conversation

julian-schneider commented Aug 2, 2024 • edited by pdurbin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pdurbin left a comment

Choose a reason for hiding this comment

Uh oh!

pdurbin Aug 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julian-schneider Aug 28, 2024

Choose a reason for hiding this comment

Uh oh!

pdurbin Apr 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pdurbin Aug 2, 2024

Choose a reason for hiding this comment

Uh oh!

julian-schneider Aug 28, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

julian-schneider commented Aug 2, 2024

Uh oh!

julian-schneider commented Aug 28, 2024

Uh oh!

julian-schneider commented Aug 28, 2024

Uh oh!

coveralls commented Aug 31, 2024

Uh oh!

johannes-darms commented Oct 11, 2024

Uh oh!

pdurbin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pdurbin commented Oct 15, 2024

Uh oh!

cmbz commented Oct 15, 2024

Uh oh!

pdurbin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pdurbin Nov 12, 2024

Choose a reason for hiding this comment

Uh oh!

julian-schneider Nov 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pdurbin Nov 20, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pdurbin commented Nov 20, 2024

Uh oh!

pdurbin left a comment

Choose a reason for hiding this comment

Uh oh!

ofahimIQSS commented Nov 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

julian-schneider commented Aug 2, 2024 •

edited by pdurbin

Loading

pdurbin Aug 2, 2024 •

edited

Loading

pdurbin left a comment •

edited

Loading

julian-schneider Nov 20, 2024 •

edited

Loading