API endpoint to update variable level metadata not working

[ingridpacheco]

Hey guys, I hope you can help me out with this problem. I tried using the API endpoint (#5971) to update my variable level metadata, but it turns out that it didn't really update it and nothing happened.

I followed exactly the steps in the documentation and performed a curl updating my file with an XML file:

curl -H "X-Dataverse-key:xxxxxx" -X PUT https://gofairbr.rnp.br/api/edit/118 --upload-file dct.xml

As a response, I received an empty object on my terminal screen and when I went to check my file on Dataverse, its variable metadata was exactly the same, as if I have never updated it.

Do you have any thoughts on what could have happened? Do I need special permission to do this or only being the owner of my dataset is enough? Does this endpoint need any special input that I am not aware of?

Thanks in advance,
Ingrid

[lubitchv]

Hello @ingridpacheco,

How do you get the dct.xml file?

The best way to get it is to get first variable metadata for specific file:
curl -H "X-Dataverse-key:XXXX" -X GET https://$SERVER/api/access/datafile/$FileID/metadata?fileMetadataId=$ID where $ID is file metadata id for that specific file and $FileID is DataFile id of that file. You also can get it from UI from Download->Variable Metadata. Then update the needed fields, for example label, or add additional fields. It maybe that XML is wrongly formatted or fields that are not allowed were added or edited.

Do you update publish version or Draft version? If it is published version, the new Draft version should be created.
How do you check that variable metadata remaines the same?

Another way to update variable level metadata is to use "Data Curation Tool". If your installation has it, you should see "Configure" drop-down on the right side of files listing. You can update variable metadata using that tool.
Also with "Data Curation Tool" you can see the variable level metadata.

It would be useful if you could attach original xml and updated xml (dct.xml in your example).

[pdurbin]

@ingridpacheco another (related) thought is to try the dct.xml file that's provided in the documentation, just as a test.

https://guides.dataverse.org/en/5.8/_downloads/d126c03c90aebaa09467b38513728564/dct.xml is the direct link from https://guides.dataverse.org/en/5.8/api/native-api.html#editing-variable-level-metadata

The example file is also available from https://github.com/IQSS/dataverse/blob/v5.8/src/test/resources/xml/dct.xml

Finally, if you want to see a full example of using this API (including uploading a file first), you can look at https://github.com/IQSS/dataverse/blob/v5.8/src/test/java/edu/harvard/iq/dataverse/api/EditDDIIT.java

[ingridpacheco]

Thanks so much for the attention!

In regards to the points mentioned by @lubitchv I was first creating my own XML file based on the current variable metadata my file had, but as soon as I realized the update was not happening I used the dct.xml file provided in the documentation (as suggested by @pdurbin) but it also didn't work for me.

To explain the steps in detail, I created two datasets with one file each. My research project is about creating variable metadata for RDF files, so in the first dataset I was analyzing a turtle (.ttl) file, extracting relevant information, and based on its current DDI file I was creating a new one adding the variables and variable groups just like it was in the dct.xml file, and then updating it using the API with the curl I posted above.

curl -H "X-Dataverse-key:xxxxxx" -X PUT https://gofairbr.rnp.br/api/edit/109 --upload-file dct.xml

To check if it really changed, I was exporting the DDI metadata directly from the file inside Dataverse, and I couldn't see any changes:

<codeBook xmlns="ddi:codebook:2_5" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="ddi:codebook:2_5 https://ddialliance.org/Specification/DDI-Codebook/2.5/XMLSchema/codebook.xsd" version="2.5"> <docDscr> <citation> <titlStmt> <titl>RDFs Usados em aula</titl> <IDNo agency="DOI">doi:10.80102/rnp.I7ETGH</IDNo> </titlStmt> <distStmt> <distrbtr source="archive">SERVIÇO EXPERIMENTAL DE P&D&I Dataverse GoFAIR.br para DADOS de PESQUISA --- DOIs para TESTES apenas</distrbtr> <distDate>2021-11-01</distDate> </distStmt> <verStmt source="archive"> <version date="2021-11-10" type="RELEASED">2</version> </verStmt> <biblCit>Pacheco, Ingrid, 2021, "RDFs Usados em aula", https://doi.org/10.80102/rnp.I7ETGH, SERVIÇO EXPERIMENTAL DE P&D&I Dataverse GoFAIR.br para DADOS de PESQUISA --- DOIs para TESTES apenas, V2</biblCit> </citation> </docDscr> <stdyDscr> <citation> <titlStmt> <titl>RDFs Usados em aula</titl> <IDNo agency="DOI">doi:10.80102/rnp.I7ETGH</IDNo> </titlStmt> <rspStmt> <AuthEnty affiliation="UFRJ">Pacheco, Ingrid</AuthEnty> </rspStmt> <prodStmt/> <distStmt> <distrbtr source="archive">SERVIÇO EXPERIMENTAL DE P&D&I Dataverse GoFAIR.br para DADOS de PESQUISA --- DOIs para TESTES apenas</distrbtr> <contact affiliation="UFRJ" email="iq.pacheco23@gmail.com">Pacheco, Ingrid</contact> <depositr>Pacheco, Ingrid</depositr> <depDate>2021-10-29</depDate> </distStmt> </citation> <stdyInfo> <subject> <keyword>Other</keyword> </subject> <abstract date="2021-06-23">Dataset com arquivos RDFs usados em aula</abstract> <sumDscr/> </stdyInfo> <method> <dataColl> <sources/> </dataColl> <anlyInfo/> </method> <dataAccs> <notes type="DVN:TOU" level="dv">CC0 Waiver</notes> <setAvail/> <useStmt/> </dataAccs> <othrStdyMat/> </stdyDscr> <otherMat ID="f110" URI="https://doi.org/10.80102/rnp.I7ETGH/QG7JR6" level="datafile"> <labl>Dan_Bric.rdf</labl> <notes level="file" type="DATAVERSE:CONTENTTYPE" subject="Content/MIME Type">application/rdf+xml</notes> </otherMat> <otherMat ID="f109" URI="https://doi.org/10.80102/rnp.I7ETGH/WUNRXA" level="datafile"> <labl>FOAF_Dan_Brickley.ttl</labl> <notes level="file" type="DATAVERSE:CONTENTTYPE" subject="Content/MIME Type">text/turtle</notes> </otherMat> </codeBook>

I had already checked my XML file in a parser but even though it theoretically didn't have any error, I tried the same process with the dct.xml file provided in the documentation, but, once again, nothing changed.

I started to wonder if it had something to do with the file extension as variables are inserted in the DDI file during the ingestion only in tabular files, and so I created my second dataset with a tabular file in it.

Finally, I tried both creating my own DDI file based on the current variables and using the one provided but nothing seemed to change in the DDI file I got after trying to export it directly from the Dataverse file.

<codeBook xmlns="ddi:codebook:2_5" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="ddi:codebook:2_5 https://ddialliance.org/Specification/DDI-Codebook/2.5/XMLSchema/codebook.xsd" version="2.5"> <docDscr> <citation> <titlStmt> <titl>COVID Interventions</titl> <IDNo agency="DOI">doi:10.80102/rnp.GJ3IOT</IDNo> </titlStmt> <distStmt> <distrbtr source="archive">SERVIÇO EXPERIMENTAL DE P&D&I Dataverse GoFAIR.br para DADOS de PESQUISA --- DOIs para TESTES apenas</distrbtr> <distDate>2021-11-10</distDate> </distStmt> <verStmt source="archive"> <version date="2021-11-10" type="RELEASED">1</version> </verStmt> <biblCit>Pacheco, Ingrid, 2021, "COVID Interventions", https://doi.org/10.80102/rnp.GJ3IOT, SERVIÇO EXPERIMENTAL DE P&D&I Dataverse GoFAIR.br para DADOS de PESQUISA --- DOIs para TESTES apenas, V1, UNF:6:drGTHvUUUko2Yhdnk3LeGA== [fileUNF]</biblCit> </citation> </docDscr> <stdyDscr> <citation> <titlStmt> <titl>COVID Interventions</titl> <IDNo agency="DOI">doi:10.80102/rnp.GJ3IOT</IDNo> </titlStmt> <rspStmt> <AuthEnty affiliation="UFRJ">Pacheco, Ingrid</AuthEnty> </rspStmt> <prodStmt/> <distStmt> <distrbtr source="archive">SERVIÇO EXPERIMENTAL DE P&D&I Dataverse GoFAIR.br para DADOS de PESQUISA --- DOIs para TESTES apenas</distrbtr> <contact affiliation="UFRJ" email="iq.pacheco23@gmail.com">Pacheco, Ingrid</contact> <depositr>Pacheco, Ingrid</depositr> <depDate>2021-11-10</depDate> </distStmt> </citation> <stdyInfo> <subject> <keyword>Medicine, Health and Life Sciences</keyword> </subject> <abstract>Dataset containing used and generated files in COVID Interventions project</abstract> <sumDscr/> </stdyInfo> <method> <dataColl> <sources/> </dataColl> <anlyInfo/> </method> <dataAccs> <notes type="DVN:TOU" level="dv">CC0 Waiver</notes> <setAvail/> <useStmt/> </dataAccs> <othrStdyMat/> </stdyDscr> <fileDscr ID="f118" URI="http://gofairbr.rnp.br/api/access/datafile/118"> <fileTxt> <fileName>COVID19_non-pharmaceutical-interventions_version2_utf8.tab</fileName> <dimensns> <caseQnty>11512</caseQnty> <varQnty>13</varQnty> </dimensns> <fileType>text/tab-separated-values</fileType> </fileTxt> <notes level="file" type="VDC:UNF" subject="Universal Numeric Fingerprint">UNF:6:drGTHvUUUko2Yhdnk3LeGA==</notes> </fileDscr> <dataDscr> <var ID="v147" name="id" intrvl="discrete"> <location fileid="f118"/> <labl level="variable">id</labl> <sumStat type="mean">5756.5</sumStat> <sumStat type="stdev">3323.372483888417</sumStat> <sumStat type="min">1.0</sumStat> <sumStat type="invd">0.0</sumStat> <sumStat type="vald">11512.0</sumStat> <sumStat type="medn">5756.5</sumStat> <sumStat type="mode">.</sumStat> <sumStat type="max">11512.0</sumStat> <varFormat type="numeric"/> <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:TvcSpMthbm1zmVMhSP17Tg==</notes> </var> <var ID="v145" name="Country" intrvl="discrete"> <location fileid="f118"/> <labl level="variable">Country</labl> <varFormat type="character"/> <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:fxE6LWi+KW1lkV00Di8KCQ==</notes> </var> <var ID="v154" name="iso3" intrvl="discrete"> <location fileid="f118"/> <labl level="variable">iso3</labl> <varFormat type="character"/> <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:sqV25N0J20l/I3E9FmoJ+w==</notes> </var> <var ID="v152" name="State" intrvl="discrete"> <location fileid="f118"/> <labl level="variable">State</labl> <varFormat type="character"/> <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:M0BvzPSE1DJ088SchAaPdw==</notes> </var> <var ID="v153" name="Region" intrvl="discrete"> <location fileid="f118"/> <labl level="variable">Region</labl> <varFormat type="character"/> <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:76qfEwt39P0rrifVjSFS6g==</notes> </var> <var ID="v148" name="Date" intrvl="discrete"> <location fileid="f118"/> <labl level="variable">Date</labl> <varFormat type="character" formatname="yyyy-MM-dd" category="date"/> <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:FDXcZ9jWJL0pYRtoySy0aw==</notes> </var> <var ID="v149" name="Measure_L1" intrvl="discrete"> <location fileid="f118"/> <labl level="variable">Measure_L1</labl> <varFormat type="character"/> <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:lBXb+czCJ6SVU21lVtpe5w==</notes> </var> <var ID="v157" name="Measure_L2" intrvl="discrete"> <location fileid="f118"/> <labl level="variable">Measure_L2</labl> <varFormat type="character"/> <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:Sx6iP6fEoBO8KK5c+ZvIxQ==</notes> </var> <var ID="v156" name="Measure_L3" intrvl="discrete"> <location fileid="f118"/> <labl level="variable">Measure_L3</labl> <varFormat type="character"/> <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:3mN6NU/kRQ2LyLWZ8sX9ug==</notes> </var> <var ID="v146" name="Measure_L4" intrvl="discrete"> <location fileid="f118"/> <labl level="variable">Measure_L4</labl> <varFormat type="character"/> <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:673H+fysUmITNhY0T0Nn9A==</notes> </var> <var ID="v151" name="Status" intrvl="discrete"> <location fileid="f118"/> <labl level="variable">Status</labl> <varFormat type="character"/> <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:cDVmDHuSEefsuXLslmcZJg==</notes> </var> <var ID="v150" name="Comment" intrvl="discrete"> <location fileid="f118"/> <labl level="variable">Comment</labl> <varFormat type="character"/> <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:HNCRIORO2MJLBDvwxRLMpQ==</notes> </var> <var ID="v155" name="Source" intrvl="discrete"> <location fileid="f118"/> <labl level="variable">Source</labl> <varFormat type="character"/> <notes subject="Universal Numeric Fingerprint" level="variable" type="Dataverse:UNF">UNF:6:mNNUB5b58bom3XwtH0PaWQ==</notes> </var> </dataDscr> </codeBook>

Both the XML I wrote here should have the variables from the file provided in the documentation, but they did not change from what they got during the ingestion.

Another additional information: Both files and respective datasets are published and no Draft version was created after the curl. Therefore, it leads me to certain questions as if this update is only available to happen with tabular files and if there is a better way to check if the variables were changed or not.

I truly appreciate your help!

If you need any additional info let me know :)

[lubitchv]

@ingridpacheco , Yes data curation for variable level metadata only works for tabular files, i.e those files (SPSS, excel, etc) that were ingested in dataverse and tab files were created. It does not work for ttl or rdf. Hence it will not work for files in this dataset https://gofairbr.rnp.br/dataset.xhtml?persistentId=doi:10.80102/rnp.I7ETGH (I assume this is your dataset)

For tab files it should work. So it should work for your second dataset https://gofairbr.rnp.br/dataset.xhtml?persistentId=doi:10.80102/rnp.GJ3IOT
By xml provided it seems to be in a wrong format. I suspect that you are exporting metadata for whole dataset (it should not be in "Export Metadata->DDI" but should be on the right side of files listing "Download" -> "Variable Metadata" or on file page "Access File -> Variable Metadata"

You can see your variable metadata using "Data Curation Tool" https://scholarsportal.github.io/Dataverse-Data-Curation-Tool/?dfId=118&siteUrl=https://gofairbr.rnp.br/
You can edit variable level metadata by selecting "Edit" (pencil icon on the right side of each variable). There will be pop-up window. Edit and press update. Do it for the variables you want. Then press Download on the upper right side. Do not press "Save to Dataverse", since this is demo and is not connected to any dataverse installation.
The "Download" will download the updated xml in proper format (dct.xml). Then you can feed that xml into your curl command:

curl -H "X-Dataverse-key:xxxxxx" -X PUT https://gofairbr.rnp.br/api/edit/118 --upload-file dct.xml

Draft version should be created. You can see the changes either downloading "Download-> "Variable metadata" or on file page "Access File -> Variable Metadata" or opening it again in Data Curation Tool.

[ingridpacheco]

Thanks, @lubitchv, it helped me a lot. I was able to update the variable metadata for my tabular file. Although my issue was closed, I still have a question about data curation for variable level metadata in RDF files. Is it only available for tabular data because of business decisions or due to coding limitations? Could I still open a pull request to make it available for RDF files as well or wouldn't this even be considered? Hope you can clear this out and, once again, thank you for the attention dismissed.

[lubitchv]

@ingridpacheco, tabular data is created in Dataverse through ingest process from SPSS, Stata, Excel, R etc files. You can read here which formats are supported by ingest https://guides.dataverse.org/en/latest/user/tabulardataingest/index.html

The idea was to convert proprietary formats to tab format for archival purposes and for separating data from metadata. Extracted metadata is saved in database and data is saved in tab on disk storage (file system, S3 etc). As you see these formats are from different statistical software. Extracted metadata can be used in dataverse external tools such as Two ravens, Data Curation Tool, Data Explorer for visualizing and editing and statistical analysis. I guess you do not have these external tools installed in installation that you use.
List of external tools here: https://guides.dataverse.org/en/latest/admin/external-tools.html

I am not versed in RDF format. It seems to me to be different from above statistical formats. It looks to me more like metadata. But I think you can open an issue asking about tabular ingets of rdf or maybe conversion of rdf to xml and editing tool similar to Data Curation Tool. As I said I am not very knowledgeable about RDF and maybe people who know more about it may answer.