Citation: Remove MD5s, if you have UNF

[sbarbosadataverse]

Gary's sent the following:

why are there MD5's? these I think should all be removed. we have UNFs
instead.

[pdurbin]

MD5s are commonly used to verify that files were not corrupted during download. Every Mac and and Linux box has the native ability to calculate an MD5 of a file. For Windows it's a supported addon: https://support.microsoft.com/en-us/kb/841290

[mercecrosas]

This is issue is not well defined as it is. @thegaryking does this question apply to subsettable Tabular files? why is it bad to have both UNF and MD5? as @pdurbin comments, MD5 are the most commonly used by preservation groups and libraries, and can be useful in addition to UNF to verify the original deposited file.

[thegaryking]

we're first and foremost trying to communicate with users, almost none of
which know about either md5 or unfs. we have taken them another step and
told them about unfs; we have a page describing them, and when we put in
enough effort they get what they are. there's no reason to introduce
another new concept. let's just use another degree of indirection. if the
librarians want something for files for which we don't have unfs, then we
create a broader notion of what a unf is and we always have a unf for every
file. the plan would be quite like the idea of dv to begin with, which is
that the more it knows about the files which are uploaded, the more
services we provide. for unfs, we can do the same thing: if the file is in
a format we know what to do with (R, sas, spss, table, etc.) we can compute
a format-independent UNF. If it is anything else we can create a
format-dependent UNF, which if you want would be exactly a MD5, but would
be displayed as a UNF. we can also add another service if librarians want,
somewhere far out of the way of most users, that lets people type in a unf
and have dv tell them exactly what it is and how it was calculated,
including the full algorithm, an MD5 if it is in there, and anything else.

then from a UI/user understanding point of view, there will be only one
thing to understand; they can ignore the details and trust us if they want;
they can get the details if they like; and we can continue to innovate what
a UNF is since there's a version number embedded in it (and i agree that we
should do the latter; people tell me that videos and photos wouldn't be
hard and we could clearly expand to more forms of data. this, however, is
a separate project that we could perhaps seek funding for and do then).

Gary

Gary King - Albert J. Weatherhead III University Professor - Director,
IQSS http://iq.harvard.edu/\- Harvard University
GaryKing.org - King@Harvard.edu - @KingGary https://twitter.com/kinggary -
617-500-7570 - fax 812-8581 - Assistant king-assist@iq.harvard.edu:
495-9271

On Fri, May 22, 2015 at 2:06 PM, Merce Crosas notifications@github.com
wrote:

This is issue is not well defined as it is. @thegaryking
https://github.com/thegaryking does this question apply to subsettable
Tabular files? why is it bad to have both UNF and MD5? as @pdurbin
https://github.com/pdurbin comments, MD5 are the most commonly used by
preservation groups and libraries, and can be useful in addition to UNF to
verify the original deposited file.

—
Reply to this email directly or view it on GitHub
#2192 (comment).

[mercecrosas]

Yes, I agree with the general approach, but we need to do some research to
implement this well. Here are the three main issues:

• If we only provide a UNF for tabular files (spss, stata, r, etc), and not
  an MD5 for the original format, then we don't have a way to verify the
  original deposited file, which is important in some cases when we need to
  recalculate the UNF or there is some issues or uncertainties with
  reformatting, or simply for standard archival verifications that
  repositories should do. This is a request from some groups that want to
  make sure we support preservation good practices. There are some
  preservation certificates that Dataverse could not get without this.
• UNF is a fantastic concept, but it has some practical limitations and
  issues in the way that is currently defined. Given that each format treats
  some data types differently (time, binary and categorical variables,
  rounding), it could turn out that you convert from one format to another,
  to another and then to back to the first format, and end up with different
  UNFs (this is similar to the phenomenon of google translating from one
  language, to another, to another, back to the first and end up with
  different sentence). This has been improved considerably from the initial
  UNF version, but it's practically very difficult to include all the
  exceptions.
• But on of the main issue with UNF is that it doesn't include the
  metadata of the file, that is the variable name, for example, or data type.
  This mean that you might have a spss file with column A that corresponds to
  var1, but this was not correct and for some reason needs to be changed to
  var2, and this is not reflected in the UNF, while this is a critical
  critical change in the data file.

I agree it would be great, as you say, to generalize UNF and make it work
well across all cases, so we use and teach only one thing. But we need to
take these three issues (and I might be missing others that Leonid, Kevin
and others in the team might know) in consideration.

Mercè Crosas, Ph.D.
Director of Data Science, IQSS
Harvard University
http://scholar.harvard.edu/mercecrosas

On Sat, May 23, 2015 at 10:27 AM, Gary King notifications@github.com
wrote:

we're first and foremost trying to communicate with users, almost none of
which know about either md5 or unfs. we have taken them another step and
told them about unfs; we have a page describing them, and when we put in
enough effort they get what they are. there's no reason to introduce
another new concept. let's just use another degree of indirection. if the
librarians want something for files for which we don't have unfs, then we
create a broader notion of what a unf is and we always have a unf for every
file. the plan would be quite like the idea of dv to begin with, which is
that the more it knows about the files which are uploaded, the more
services we provide. for unfs, we can do the same thing: if the file is in
a format we know what to do with (R, sas, spss, table, etc.) we can compute
a format-independent UNF. If it is anything else we can create a
format-dependent UNF, which if you want would be exactly a MD5, but would
be displayed as a UNF. we can also add another service if librarians want,
somewhere far out of the way of most users, that lets people type in a unf
and have dv tell them exactly what it is and how it was calculated,
including the full algorithm, an MD5 if it is in there, and anything else.

then from a UI/user understanding point of view, there will be only one
thing to understand; they can ignore the details and trust us if they want;
they can get the details if they like; and we can continue to innovate what
a UNF is since there's a version number embedded in it (and i agree that we
should do the latter; people tell me that videos and photos wouldn't be
hard and we could clearly expand to more forms of data. this, however, is
a separate project that we could perhaps seek funding for and do then).

Gary

Gary King - Albert J. Weatherhead III University Professor - Director,
IQSS http://iq.harvard.edu/\- Harvard University

GaryKing.org - King@Harvard.edu - @KingGary https://twitter.com/kinggary

617-500-7570 - fax 812-8581 - Assistant king-assist@iq.harvard.edu:
495-9271

On Fri, May 22, 2015 at 2:06 PM, Merce Crosas notifications@github.com
wrote:

This is issue is not well defined as it is. @thegaryking
https://github.com/thegaryking does this question apply to subsettable
Tabular files? why is it bad to have both UNF and MD5? as @pdurbin
https://github.com/pdurbin comments, MD5 are the most commonly used by
preservation groups and libraries, and can be useful in addition to UNF
to
verify the original deposited file.

—
Reply to this email directly or view it on GitHub
#2192 (comment).

—
Reply to this email directly or view it on GitHub
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_IQSS_dataverse_issues_2192-23issuecomment-2D104902198&d=BQMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=n9HCCtgqDPssu5vpqjbO3q4h2g6vMeTOp0Ez7NsdVFM&m=n42TBHZeNCFjVWht9OJze2EDoPR2o7n87LpnMd0UIlQ&s=3H3Wxs075lnSQvTNGBQFfpirL4UfeRAgp1akLUnwL94&e=
.