File Upload - allow files with same MD5 (or other checksum) in a dataset

[jggautier]

If someone uploads a file in a dataset that Dataverse notices already has a file with the same content (both files have the same MD5), Dataverse shows an error and doesn't allow the "duplicate" file to be uploaded.

Issues with this feature have been discussed in another github issue (#2955, closed when File Replace was released in Dataverse 4.6.1), in Dataverse's Google Group here and here, and in a recent Dataverse support ticket, where a depositor wrote that "for uploading shape files for two different polygons but the same projection, it might be nice to be able to upload both at the same time." For this researcher, a common workaround, uploading the file in different double-zipped archived files (7-Zip, tar file, etc) won't work because the journal policy doesn't allow depositors to upload archived files.

[oscardssmith]

Is the issue here a hash collision or something else? Otherwise, wouldn't the different polygons have different hashes?

[jggautier]

I'm not sure. The files in question for that support ticket are .prj files, defined here as:

an optional file that contains the metadata associated with the shapefiles coordinate and projection system.

So I think the content is the same, but they're meant for different polygons. (It looks like the journal allowed the author to upload the second file in a zip file.)

[djbrooke]

• Since we have file hierarchy now (and code deposit in the future), we should re-evaluate this.

[pdurbin]

There's some related discussion in the "Add a checkbox to disable unzipping" issue at #3439 (comment) and below.

Also, as we think about pulling in files from GitHub (#2739 and #5372), we should consider that identical files are somewhat common. There could be identical .gitignore files, for example. It's also common in Python projects to have empty __init__.py files (which all have the same checksum, of course) all over: https://docs.python.org/3/tutorial/modules.html#packages . Geoconnect has a bunch of these:

[shlake]

Here's another example of the need to have two files with same content, but different filenames (3D files at UVA):

I have 3 different 3D files that I need to upload. The files include dependencies, for example, the OBJ format includes a .obj (geometry), .jpg (texture) and .mtl (material file that connects the texture with the geometry). I have 2 different OBJ files I need to upload to my dataset. Each has a .jpg texture file that is the same file but named differently. This is necessary for the OBJ file to be able to connect to its specific texture file. DV won't let me upload both files because it thinks the other is the same file even though it is named differently. I need to have both files uploaded.

[pdurbin]

@shlake great real world, non-code example. Thanks!