Feature Request/Idea: BagIt Support - Add automatic checksum validation on upload

[abujeda]

Overview of the Feature Request
Add automatic checksum validation if the file uploaded to Dataverse is in BagIt format.
Upon file upload, we will detect if it is a Bag and perform file checksums against the Bag manifest.
If there are checksum errors, we will display a warning message to the end-user.

UI changes still to be discussed.

What kind of user is the feature intended for?
Curator and Depositor

What inspired the request?
As part of the Harvard Data Commons project there is a requirement to add support for BagIt. The first iteration is to simply inspect the manifest files and perform a file validation when a new deposit is made.

What existing behavior do you want changed?
We want to enhanced the upload functionality to add the file validation based on the BagIt manifest.

Any brand new behavior do you want to add to Dataverse?
No

Any related open or closed issues to this feature request?
N/A

[shlake]

To support this, this issue needs to be fixed: #8449
#8449

As the checksum in the manifest is for the original uploaded file, but the ingested "tab" file is what is in the bag.

[qqmyers]

@shlake - FWIW: my understanding right now is that the import is for vanilla Bags (in particular, no OAI-ORE file describing the dataset), not ones generated by Dataverse (which have the issue you describe and also contain file metadata that is not in a vanilla Bag.)

[abujeda]

We had a first attempt at the UI to provide feedback to the users when uploading BagIt packages.
Summary:

• An error message will be shown at the top of the page.
• At most, we will show 5 errors to the end user.
• The BE will stop the file validation if 5 or more errors are found.
• If errors are found, we will not allow the file/files to be uploaded.

Here is an screenshot for when we find errors in the checksums:

[abujeda]

Created a draft PR to start the conversation about the implementation.

• BagIt Support - Add automatic checksum validation on upload #8677

[abujeda]

A demo of the functionality and a AWS environment has been shared with a group from the Instituto Brasileiro de Informação em Ciência e Tecnologia

[abujeda]

The final implementation is ready for review. PR: #8677