Publish Dataset - Fails when metadata contains HTML entities w/special characters such as  

[rmo-cdsp]

I added datasets in a Dataverse using Datacite for DOI's. Those datasets were added with python scripts, using the dataverse python api to first create a "simple" (only the required elements) dataset, then update its metadatas with a json (made by extracting datas from xlsx documents).

The problem here seems to come when I try to publish the dataset. The publish fails and there is the error message : "Error – This dataset may not be published because the DataCite Service is currently inaccessible. Please try again. Does the issue continue to persist? Please contact Dataverse Support for assistance. "

Logs are here :dataverse_event_published_error_log.txt

We can see a message : [xml] xml error: The entity "nbsp" was referenced, but not declared.]
So I tried to use the datacite api to try with "custom" xml files, and there was something: if I had a "&nbsp" element, I receive the same error. When I remove it, everything is fine. I tried by adding an entity element ( ]>), (it's a &# and 160 between the "") and the nbsp was replaced and everything worked like a charm. Except that it's by using the api. The problem here is that dataverse doesn't handle nbsp thing (coming from my imports maybe).

Here is a json example asked for doing test on a test server using datacite :
json.txt

[rmo-cdsp]

By looking more precisely to my datas, some utf-8 "No-Break Space", aka \xa0, aka & nbsp, were present. So I purged them, and the error disappeared. I had a second error after that, an argument, optionnal for Dataverse, but mandatory for Datacite it seems:
Response code: 400, [xml] xml error: cvc-minLength-valid: Value '' with length = '0' is not facet-valid with respect to minLength '1' for type '#AnonType_contributorNamecontributorcontributorsresource'.

Here, the contributorName is not part of the contributor field of a Dataset on Dataverse, but it concerns the dataset contact Name, that is optionnal for a dataset, but create an error if you use Datacite, because it's mandatory but not asked anywhere (except in the error message).

After adding a Name to my datasetContact field, I could finally properly Publish my imported Dataset :)

So to resume, there is 1 major problem, and 1 optionnal problem:

• Major : datasetContactName IS mandatory for Datacite publishing
• Optionnal : handle nbsp items entity in the xml sent to Datacite for publishing, or indicate an import problem, or even print in the dataset publishing error message ?

[pdurbin]

See also details at http://irclog.iq.harvard.edu/dataverse/2016-09-02#i_40373

[djbrooke]

We've had another report of the "optional" item from this issue regarding “<em>” tags in the metadata. These can be populated through the Dataverse UI, but they do not play well with Datacite.

[pdurbin]

@djbrooke basic includes: a, b, blockquote, br, cite, code, dd, dl, dt, em according to https://github.com/IQSS/dataverse/blob/v4.5.1/src/main/java/edu/harvard/iq/dataverse/util/MarkupChecker.java#L32 . It sounds like for now we should blacklist em.

[pdurbin]

I believe there's another case of this problem reported here today: https://groups.google.com/d/msg/dataverse-community/v6jgyewGqlI/peTMCoW-DAAJ

[pdurbin]

In pull request #4075 for issue #3845 I worked on the problem of how invalid XML was being sent to DataCite. I'm going to put this is QA so we can at least talk about it at standup.

[djbrooke]

We talked about this in standup. The expectation is that code changes that were already made had fixed this, so we should retest.

[kcondon]

So, to test this I need to:

1. Configure dataverse to use DataCite
2. Identify which metadata fields are sent to DataCite
   Title, Author, Description allow HTML
3. Identify which html tags are unsupported by DataCite
   DataCite doesn't like any tags (<br>) or entities (&,  ) so try each of these.
4. Create a dataset (via API?) which uses unsupported html tags in fields sent to DataCite
   API not needed. Previous code still renders HTML on dataverse but strips tags, characters from metadata sent to DataCite, allowing publication