Dataverse -> DCM APIs

[pameyer]

Distinct from #3353 (roughly DCM -> DV APIs, in this context). This is a dependency for DCM UI implementation, but doesn't include UI.

Create dataset function/command needs to:

• send a message to the DCM requesting setup for a data deposition (for dcm/rsync+ssh, creation of either a transfer script or auth payload)
• asynchronously check if the setup has been done, and if so return/present the script/auth payload to the depositor

Some configuration (DCM url, etc) will be needed.

[pdurbin]

See #3352 for links to code that's already been written. I'm still partial to https://github.com/pdurbin/dataverse/tree/3145-dcm which has RequestRsyncScriptCommand and other goodies.

[pameyer]

failure modes / open questions:

• DCM down (create dataset command fail?)
• upload script expires (script tells depositor to contact support via email)
• possible interactions with workflows (RSAL)
• DV may not need to store script persistently; can hand users off to DCM.

[pdurbin]

@sekmiller and I are talking about this. I'm sure the screenshots in "2017-05-02 Review rsync prototype (Bill's UI changes)" at https://docs.google.com/a/harvard.edu/document/d/1Mi7I2w2FVYbN1Qb9oWLik3UsU2oPY3Wn9Z6_5JXrEAY/edit?usp=sharing will be helpful. They're oriented toward building a UI some day which is not in scope for this issue but will help give some background. I believe this meeting with @pameyer was even recorded.

[pdurbin]

Here's some whiteboarding I just did for @sekmiller .

I tried to focus on what the end user will see, even without the fancy UI work that will come in a future issue (if anyone know the issue number please advise):

• create dataset
• download rsync script
• run script
• look for single "data package" file

On the back end, here's roughly what we want to happen:

• On dataset creation, Dataverse asks DCM for an rsync script.
• DCM replies with the rsync script (original prototype) or a URL to the script (proposed, but be careful of security since the script contains credentials).
• User downloads the rsync script.
• User runs the rsync script, which creates a manfest file of checksums and then transfers the manifest and the files themselves to the DCM.
• The DCM verifies the checksum and notifies Dataverse that they're fine, passing JSON with the dataset ID and "validation passed".
• Dataverse kicks off the importer/crawler developed in File Import Batch job in support of rsync #3353 (pull request 3353 batch job import #3497) which will create a single DataPackage file.

That's the happy path. If the DCM tells Dataverse that the checksum fail, Dataverse sends a notification to the user via the normal Dataverse notification system.

Here's the (fugly) whiteboarding of the above:

[pdurbin]

@raprasad @sekmiller to get ready for development, please install and run the Data Capture Module (DCM) on your laptops. Instructions are at https://github.com/sbgrid/data-capture-module and @pameyer can help field questions since he wrote the code! 😄