DDI Metadata Enrichment (Adding weights to variables) aka The Data Curation Tool

[kevinworthington]

Having the ability to modify the DDI Metadata would be hugely beneficial for our community.
Primarily, being able to assign which variable is the weight and what variables use this weight, noting that there may be multiple weights within a dataset.

Here's how the DDI expresses weights:

Weight assignment
<var ID="V22" name="WeightForAllQs" wgt="wgt" files="F1" dcml="12" intrvl="contin">

Weight assignment to variable
<var ID="V7" name="Q7" wgt-var="V22" files="F1" dcml="2" intrvl="discrete">

Calculate weighted frequency (done automatically upon assignment

_<catStat type="freq">
500
</catStat>
<catStat wgtd="wgtd" type="freq">
479.707_

[landreev]

So, we are talking about adding an editor for the variable-level metadata, for tabular files - correct?
Yes, I agree, this could be very useful. (Just being able to edit/expand variable-level labels would be great...)

[kevinworthington]

Other variables to consider as we think about expanding the scope of this project have been outlined here

[pdurbin]

@kevinworthington @amberleahey @djbrooke @landreev @jggautier and I met over Skype on Friday and I took these notes:

• Related GitHub issue: DDI Metadata Enrichment (Adding weights to variables) aka The Data Curation Tool: DDI Metadata Enrichment (Adding weights to variables) aka The Data Curation Tool #4174
• Related RT ticket: https://help.hmdc.harvard.edu/Ticket/Display.html?id=257489
• Dataverse 4.0+ADA+Nesstar Metadata Crosswalk-DDI: https://docs.google.com/spreadsheets/d/11Fiy0huoEKzOmJS-hgB-Sq8KOCrsyHY075Yrdkg3U8M/edit?usp=sharing
• (Amber) Chatted with ADA (Steve and Marina) about how to move away from their Nesstar platform. Missing functionality is file-level DDI manipulation. A tool used for curation. Annotate the variables. Clear need for annotating the data for social science surveys. Shouldn't go into Data Explorer. Maybe a new tool. We put together a proposal for IASSIST. DANS might also be interested.
• (Leonid) I would also support starting with the DDI rather than a multi-disciplinary solution. Let's focus on what can be encoded in the DDI format. We know that we don't allow people to modify that information currently, the type of the variable. CSV files, for example, don't have any metadata. Researchers will want to say "number in seconds" or something. There is no API to received updated DDI a the variable level, but it could be added, in theory. It's an interesting approach that would probably work.
• (Kevin) How would versioning work?
• (Amber) We assume a new version would be triggered. It would be nice to capture that process as a meaningful event in the life of that dataset. As long as the edits tie back to the dataset-level versions.
• (Leonid) The variable metadata is currently immutable. The first step would be to make it versionable. We would replicate what we do with dataset level metadata, to tie it to a dataset version. Making any change to a file, such as fixing a typo in the description of a file, a new version is triggered. Once we make the variable metadata editable, it would use the same model.
• (Amber) How related is this to DataTags?
• (Danny) We were planning on using the PSI tool to provide differentially private summary statistics.
• (Leonid) I don't think PSI makes changes to the variable metadata.
• (Amber) If you're working on a tool to identify sensitive data in a dataset, maybe it could be used for editing. Are the workflows similar?
• (Leonid) There are definitely similarities. They don't use DDI. They use a proprietary (but open) JSON format that mirrors DDI in many ways. We don't parse the JSON that PSI sends back. We just store it.
• (Phil) Checking out the PSI demo’s might help get a sense of a the similarities: https://beta.dataverse.org/custom/DifferentialPrivacyPrototype/
• (Leonid) TwoRavens operates on a proprietary JSON format that the TwoRavens team invented while developing TwoRavens. It's created by some R code and Dataverse serves it up but treats it like a black box. Dataverse doesn't really know or care what's in that JSON. It doesn't parse it. It just serves it up.
• (Amber) There's no way to interact with the Dataverse system to do curation work. Curators want to be able to edit and annotate variable-level data.
• (Leonid) There's still a potential solution of building this functionality of editing and annotating variable-level metadata into Dataverse itself.
• (Kevin) Once the Dataverse APIs to allow for edits of variable metadata are in place, I'm volunteering to create the GUI tool to send the edited DDI file to Dataverse.
• (Danny) Two issues should be created. Changes to allow versioning and editing of variable metadata. API endpoints.
• (Julian) You can add these weights in Stata already. Right now Dataverse ignores it? So if someone uploads a file that has variable metadata from a tool they’ve used to add weights, is that metadata is ignored, and will the depositors have to re-add that metadata in this external tool?
• (Leonid) We honor a lot of it.
• (Amber) "Weight" is not honored by Dataverse ingest of Stata files.
• (Leonid) I'm pretty sure it does. Is anyone using those weights for practical purposes?
• (Danny) What's the timing on this?
• (Kevin) Having the APIs and versioning in place by summer would be great.
• (Amber) Demo release for fall/winter would be nice. Not a huge priority but would be nice to be able to talk about it as something that's coming in the fall/winter.
• (Kevin) Are you still having community calls?
• (Danny) Yes, but our website has been read only for a couple weeks so I haven’t been able to update the community calls page.

[kevinworthington]

Thanks for capturing and posting the meeting notes @pdurbin.
This tool will be a huge benefit for us and others looking to curate data for enhanced clarity and reuse.

[pdurbin]

@kevinworthington sure. Please do check out the issue that @landreev just created at #4448 (API and internal framework for modifying variable-level metadata) for some of the backend changes that would be required for curation tool you'd like to build. At some point we should figure out what the scope is of this issue.

[pdurbin]

@kevinworthington should this issue still be assigned to you?

[kevinworthington]

Hi @pdurbin,
We don't have a replacement at this point so I'm happy to stay assigned to this until that happens.
Thanks for prompt and for all your help with everything.

[pdurbin]

@kevinworthington cool. Just as a heads up I did go ahead and replace you with @amberleahey over at the "Dev Efforts by the Dataverse Community" spreadsheet at https://docs.google.com/spreadsheets/d/1pl9U0_CtWQ3oz6ZllvSHeyB0EG1M_vZEC_aZ7hREnhE/edit?usp=sharing but if there's a better contact, please let me or Amber (who has edit access) know. On a related note, there's a new column in our kanban board called "Community Dev" for stuff that's actively being worked on: https://groups.google.com/d/msg/dataverse-community/X2diSWYll0w/A7BkeGewCAAJ . I wasn't sure if this issue should be in there or not.