Spike: Inventory and prioritize all existing Harvesting related issues

[mreekie]

This is in support of:

• an NIH grant "The Harvard Dataverse repository: A generalist repository integrated with a Data Commons",
  • Aim 4: Improve harvesting and packaging standards to share metadata and data across repositories,

The first step is to figure out what has already been done by the dataverse team and by the community towards this aim and what still remains to be done.

For example:

• 8139 Can't harvest when Dublin core field language is set

And then to prioritize which issues are to be fixed.

Def of done

As completely as is reasonably possible in a 2 week period (sprint):

• Search out previous related issues that are problems with the current implementation. Take an inventory.
• Search out previous work done within the dataverse community as well.
• prioritize which of the issues/PRs that should be moved forward.

We need to keep in mind that to harvest something from a particular source requires that that source be bug free. Identify which sources have which bugs so that bugs for a particular source can be targeted. for example: ICPSR as an example. Zenodo is another.

More information:

There is a lot packaged into Aim 4

1. Improved Harvesting via the OAI-PMH standard
2. Improved support for Bagit
3. Improved support for Signposting

The scope for this issue is Harvesting via the OAI-PMH standard

Aim 4:

Improve harvesting and packaging standards to share metadata and data across repositories

Our proposed project will significantly improve the widely-used Harvard Dataverse repository to better support NIH-funded research.

A critical measure of the GREI program’s success is to standardize the discoverability across generalist repositories.

To help with this, **we propose to improve the existing harvesting functionality in the Dataverse software based on the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) standard, and coordinate with other repository packaging standards to share or move metadata and data. **

Dataverse already supports the Bags as defined by the Research Data Alliance (RDA) Research Data Repository Interoperability Working Group. Here we proposed to improve the support for Bags, test it for NIH-funded datasets, and explore and define the appropriate standard to use to move the metadata and data across generalist repositories. This will help with a sustainable and succession plan - if one repository cannot support anymore a specific dataset, it will allow to easily move the dataset to another repository without losing any information about the dataset.

Additionally we propose to implement Signposting in the Dataverse software. By adding additional http link headers throughout the application, we can more easily support automated metadata and data discovery in the repository, and allow for other applications and services to more accurately and completely represent the content in the Harvard Dataverse repository.

Related documents

• Notes on Dataverse Deliverablas for NIH OTA
• NIH OTA Progress Notes
• NIH OTA
• Exposing and harvesting metadata using the OAI metadata harvesting protocol: A tutoria (2001)
• Getting Started with BagIt in 2018
• NIH OTA
• bagit from Library of Congress video

[pdurbin]

The definition of done for this issue includes three items:

• Search out previous related issues that are problems with the current implementation. Take an inventory.
• Search out previous work done within the Dataverse community as well.
• Prioritize which of the issues/PRs that should be moved forward.

I'm going to provide lists below for the first two items (open issues and merged pull requests). Please note that PR IQSS/dataverse#7053 is an open pull request that addresses a harvesting issue. I'm not sure which list it belongs in but it closes IQSS/dataverse#7502 which is in the open issues list.

The third item is about prioritizing what to work on. I'm leaving this for the prioritization group (@lenwiz, @scolapasta, @TaniaSchlatter, @sbarbosadataverse, and @mreekie). At standup today Gustavo did say that developers like me are welcome to give opinions on what we think should be the priority. From a quick look I'd say we should work on IQSS/dataverse#5840 and IQSS/dataverse#8267 because they are both small documentation updates. Otherwise, it's hard for me to judge the priority. I'm always for fixing bugs and there are plenty of small bugs.

I did reach out to the community to encourage them to open new issues about harvesting if they know of any:

• https://groups.google.com/g/dataverse-community/c/bDaOH8a4Hmc/m/XfzYL-BlAQAJ
• https://matrix.to/#/!AmypvmJtUjBesRrnLM:matrix.org/$xr-LkLmDvTolW0O4FoWZivjegovdGttxu3pBbnGisxk
• https://dataversecommunity.slack.com/archives/C5V66TV6Y/p1649865300678319

Here are the lists.

Search out previous related issues that are problems with the current implementation. Take an inventory.

Here's the oneliner I used to get the following list of open issues from the GitHub Search API:

curl -H 'Accept: application/vnd.github.v3.text-match+json' 'https://api.github.com/search/issues?q=is:issue%20is:open%20label:%22Feature:%20Harvesting%22%20repo:IQSS/dataverse&per_page=100' | jq '.items[] | "- #\(.number)"' -r

• Figure out whether, or how to support the extended ISO 639-3 list of languages dataverse#8578
• deprecated Spike: Initiate or participate in a working group in support of NIH effort to improve harvesting capabilities (Deprecated) dataverse#8575
• Spike: Inventory and prioritize all existing Harvesting related issues #24
• Feature Request/Idea: Standardize standard license configuration dataverse#8512
• oai_dc export fails, OAI-PMH error dataverse#8306
• Trying to set up or complete a harvesting client through the API crashes Dataverse dataverse#8290
• The "ListSets" command fails during the creation of a harvesting client for Zenodo  dataverse#8289
• Feature Request/Idea: Documentation for the API to create and edit harvesting clients dataverse#8267
• Can't harvest when Dublin core field language is set dataverse#8139
• Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting dataverse#8129
• Year in citations of harvested datasets is always current year dataverse#8103
• Harvesting: OAI sets are not updated when datasets are deleted. dataverse#8005
• [feature request] stop an harvest job in progress dataverse#7940
• Fix handling of storageidentifiers in dataverse_json harvests dataverse#7736
• Add import for oai_datacite ("OpenAire") format (this will allow Dataverse to harvest this format too) dataverse#7727
• Unable to create harvesting client for SRDA repository dataverse#7624
• Harvesting : message "javax.ejb.EJBTransactionRolledbackException, Exception thrown from bean: java.lang.NullPointerException" dataverse#7546
• Saved Search inconsistencies around unpublished datasets dataverse#7453
• Feature request: create a harvested dataset via the API dataverse#7330
• Dataverse should use ISO codes for languages when exporting and importing metadata. dataverse#7243
• Some elements in the DDI XML response have an incorrect sequence dataverse#7127
• Should the Geographic Bounding Box allow optional coordinates and multiple boxes in the XML? dataverse#7108
• Harvest: Harvesting appears to just die silently intermittently, using Princeton set in prod dataverse#7074
• Harvesting: Get rid of this seemingly meaningless and noisy log entry dataverse#7073
• Harvesting run infinitely stays at IN PROGRESS and DELETE IN PROGRESS dataverse#7052
• Add DB Constraint for duplicate file names dataverse#7021
• Typeahead widget on Link Dataset popup is slow and doesn't display all options as expected dataverse#6772
• Carriage return in metadata blocking OAI-PMH Export dataverse#6275
• Add support for exporting more metadata formats via XSLT crosswalks dataverse#5960
• In OpenAIRE metadata export, access rights metadata about research objects, such as datasets, is being misapplied dataverse#5920
• Align or merge DataCite metadata exports dataverse#5889
• OAI server guides documentation - requires editing dataverse#5840
• Are Dataverse installations' harvesting sets exporting regularly? dataverse#5392
• Exclude deaccessioned dataset from being harvested dataverse#5112
• Spike: Investigate how Dataverse stakeholders and users need to collect and use funder metadata dataverse#4859
• Harvest: OAI client should follow http->https redirect since oai server validation allows http. dataverse#4831
• As a data repository, I need to harvest additional metadata in OAI_DC records dataverse#4176
• As a Dataverse collection admin, I want to use the UI to link harvested datasets into my collection dataverse#4025
• OAI-PMH responses indicating errors should be processable by OAI-PMH clients dataverse#3797
• OAI server: metadataPrefix unknown: Internal server error dataverse#3741
• Dataverse should return machine-readable metadata to requesting clients/servers (content negotiation) dataverse#3699
• Make Dataverse produce valid DDI codebook 2.5 XML dataverse#3648
• invalid schema and metadataNamespace fields in OAI-PMH ListMetadataFormats response dataverse#3621
• Harvest: Set export can temporarily be adversely affected by full (clean) reindex. dataverse#3437
• Harvest: Do not list a set as available in ListSets until it has been successfully exported at least once. dataverse#3322
• Harvesting: Manage Sets page goes deaf after a while, making create set popup not save, does nothing until refresh page. dataverse#3316
• Harvesting: Manage Sets, sets defined with dataverse scope become less valid when subdv's are added. dataverse#3315
• Harvesting: Manage sets, entering a longish string into set name throws exception, throws away set. dataverse#3314
• Harvest: Harvest all from IQSS in JSON format had 68 failures out of 19726 dataverse#3298
• Harvesting - Data Citation Values Missing dataverse#3290
• Harvest: Exporting all IQSS datasets has several failures. dataverse#3276
• Support MARC-XML for Export and Harvest dataverse#2721
• Dataset Citation: provide flexible options for information displayed in citation metadata dataverse#2297
• Crosswalk for Importing Datasets From NARA Into Dataverse dataverse#1814
• Support for Importing and Exporting DCAT Metadata dataverse#1592

Search out previous work done within the Dataverse community as well.

Here's the oneliner I used to get the following list of merged pull requests from the GitHub Search API:

curl -H 'Accept: application/vnd.github.v3.text-match+json' 'https://api.github.com/search/issues?q=is:pr%20is:merged%20label:%22Feature:%20Harvesting%22%20repo:IQSS/dataverse&per_page=100' | jq '.items[] | "- #\(.number)"' -r

• doc updates to better explain harvest server query format for subcollections dataverse#8197
• 41 add auth to harvest api dataverse#8106
• SPO/7786 Add holdings element to DDI and use PID URIs dataverse#7984
• 7431 fixing invalid oai pmh response dataverse#7974
• fix for the null pointer in isHarvested() method. (#7849) dataverse#7868
• 7451 dvobject constraint, storageidentifier dataverse#7741
• Support for ISO-639-3 language codes dataverse#7690
• 5050 Parse all dc identifier elements and allow identifiers that don'… dataverse#7214
• 4597 Fix invalid identify response. dataverse#7062
• 7056 Skip unknown types in a dataset which cause the whole dataset to… dataverse#7057
• #7023 Replace xml response substring in logging with URL parameters. … dataverse#7024
• #7009 Removed one of the redundant encoding functions. dataverse#7010
• #7003 Fix log formatting and in particular the OAI URL. dataverse#7004
• #7001 Fix INPROGRESS typo. dataverse#7002
• Remove typo after running a harvesting client from Bundle.properties #6987. dataverse#6988
• 5886 link harvested datasets via api dataverse#6935
• 4964 harvesting issues dataverse#6686
• 6650 export import mismatch dataverse#6669
• avoid null pointer when getting email for OAI-PMH #3619 dataverse#6538
• only show email portion system email via Harvesting #3619 dataverse#6513
• #6372 document harvest client log location, recommended Dataverse ver… dataverse#6376
• Harvester improvements. dataverse#6087
• 5735 fix manage harvest client page dataverse#5926
• 4203 - Add documentation about OAI-PMH endpoint and details dataverse#5834
• 5254 - don't index from harvested datasets dataverse#5255
• Auto reload harvest clients page dataverse#5248
• #5245 - fix global id url parsing when parsing OAI dataverse#5247
• A quick fix for the search card for a harvested file: it should not t… dataverse#4787
• Fixed ServletException #4548 dataverse#4691
• OpenAIRE guidelines for Data Archives dataverse#4664
• #4398 - Fix display of Harvest Client in edit mode dataverse#4432
• API for creating OAI-PMH sets dataverse#4096
• 3985 - added an else to flip harvesting to off if disabled in UI dataverse#4018
• 4.6 tweaks for harvesting from partners dataverse#3521
• OAI-PMH dc elements dataverse#3458
• 4.5 export harvest dataverse#3308
• Harvested File card render logic to hide empty metadata fields [ref #2221] dataverse#2938
• 2686 2907 bugfixpatch 424 dataverse#2933

[jggautier]

There are a few issues in the Harvard Dataverse Repository GitHub repo about harvesting:

• Re-harvesting ICPSR datasets dataverse.harvard.edu#63
• Special characters in dataset metadata prevent Dataverse repositories from harvesting from Harvard Dataverse dataverse.harvard.edu#72
• Spike: Review if inability to harvest all records from certain repositories will be or has been resolved when other harvesting-related issues are addressed dataverse.harvard.edu#92
• Unable to remove datasets that Harvard Dataverse Repository harvested from Swedish National Data Service  dataverse.harvard.edu#142

And some support emails saved in the RT system about harvesting issues in the Harvard repository:

• https://help.hmdc.harvard.edu/Ticket/Display.html?id=287243
• https://help.hmdc.harvard.edu/Ticket/Display.html?id=295542
• https://help.hmdc.harvard.edu/Ticket/Display.html?id=311383
• https://help.hmdc.harvard.edu/Ticket/Display.html?id=314258

[landreev]

(the lists below are work in progress, I'm actively working on them!)

I do believe that the third item under the "definition of done" - "prioritize" - was the actual important part of this spike. I also believe that most of that effort of prioritizing what's important can only be done within the dev. team. I can't think of how anyone outside of it could be more qualified to make these calls. So I'm going to make such an attempt.
(Note that I'm interpreting the word "prioritizing" as assigning some order of importance to these issues and bugs, what makes sense to fix first and/or what's ready to be worked on vs. what needs more discussion; not as scheduling them for specific sprints, etc.!)

The single most important harvesting issue: (ok, maybe not the most important - but seriously, this should be the first step of any meaningful cleanup of our harvesting implementation; should be fairly easy to wrap up too)

• Update and reorganize the XOAI dependencies under local_lib dataverse#8372

The following issues are important in that fixing them will make harvesting more reliable and robust overall (for example, in the current implementation a single missing metadata export that's supposed to be cached is going to break the entire harvesting run). All of the issues on the list below are defined clearly enough that they are ready to be worked on and fixed, without needing to conduct any extra research first. Some of them may be VERY OLD; but they look like something we should fix.

• Failure to export dataset not detected, dataset marked as exported dataverse#8484 [backend]
• [feature request] stop an harvest job in progress dataverse#7940 [backend, api]
• Harvesting: OAI sets are not updated when datasets are deleted. dataverse#8005 [backend]
• Fix handling of storageidentifiers in dataverse_json harvests dataverse#7736 [backend, import]
• Harvest: Set export can temporarily be adversely affected by full (clean) reindex. dataverse#3437 [backend]
• Harvest: Do not list a set as available in ListSets until it has been successfully exported at least once. dataverse#3322 [backend]
• Trying to set up or complete a harvesting client through the API crashes Dataverse dataverse#8290 [api]
• The "ListSets" command fails during the creation of a harvesting client for Zenodo  dataverse#8289 [backend, ui]

the following 3 issues are basically the same thing - people requesting extra ISO language codes to be added as legitimate controlled vocab. values (this is just a matter of adding extra values to citation.tsv); these are NOT duplicates, different things are being requested to be added in the issues below, but makes sense to get all 3 out of the way at the same time:

• Figure out whether, or how to support the extended ISO 639-3 list of languages dataverse#8578 [metadata, import]
• Can't harvest when Dublin core field language is set dataverse#8139 [metadata, import]
• Feature Request/Idea: Sanitize languages controlled vocabulary values dataverse#8243 [metadata, import]

The following issues are about the DDI exporter producing XML that is not valid under the schema. I would consider creating a wider scope umbrella issue, something like "Make sure our DDI is valid against the schema (and maybe add a real time validation step to the export?)
The first one is ready to be worked on, I believe:

• Some elements in the DDI XML response have an incorrect sequence dataverse#7127 [export]
  These may need some discussion:
• Make Dataverse produce valid DDI codebook 2.5 XML dataverse#3648 [export]
• Should the Geographic Bounding Box allow optional coordinates and multiple boxes in the XML? dataverse#7108 [export]

Similarly, the following issues are requests for changes in how we export DC; I believe these need to be reviewed/discussed, perhaps together?

• Change Dataverse / Dublin Core mapping to improve OAI-PMH harvesting dataverse#8129 [export]
• Dataverse should use ISO codes for languages when exporting and importing metadata. dataverse#7243 [export]

The following issues are proposed changes to the design of the harvesting framework and/or metadata exports. Meaning this is something we probably need to discuss as a team, before we decide that these are good ideas and proceed to implement them. But IMO they are (I opened all of them 😄):

• Consider making export time stamps more granular dataverse#8630 [backend]
• Revisit/reimplement the concept of a "Harvested file".  dataverse#8629 [backend]
• Add support for exporting more metadata formats via XSLT crosswalks dataverse#5960 [backend]
• Consider reorganizing metadata import code dataverse#8631 [backend]
  (ordered by importance, IMO)

There is of course this issue that was opened for figuring out what needs to be added specifically for the NIH/GREI grant:

• deprecated Spike: Initiate or participate in a working group in support of NIH effort to improve harvesting capabilities (Deprecated) dataverse#8575
  It is obviously super important and should be prioritized too; but it's more of a design discussion and research, rather than something we already can start coding.

The list above is by no means complete. If an issue is not listed, it does not necessarily mean that it's not important. But the ones that are listed above should be a good subset to start with.

[landreev]

The list(s) above should allow us to start working on cleaning up and improving harvesting.
There are still issues labeled "Feature: Harvesting" that need to be triaged (there are some that may have been resolved since they were opened; there are some that may not need to be addressed in the first place - but then we should close them). For the next stage of this work however, once the above issues are addressed, it'll probably be more important to figure out what needs to be resolved/what features need to be added specifically for the NIH/GREI project (the spike IQSS/dataverse#8575 on the list above).

[scolapasta]

I'm going to go ahead and close this issue as the spike work of identifying and prioritizing is done above. We've started adding a few of these issues onto the board (in Next Sprint) and we'll continue to use this as a reference to add more as that work gets completed. But no need to keep the actual spike open.