Data Capture Module (rsync support)

[pdurbin]

@pameyer @bmckinney and I met yesterday to discuss what we're calling a "Data Capture Module" or "DCM" for short. http://guides.dataverse.org/en/4.3.1/installation/prep.html#architecture-and-components lists a number of optional components for Dataverse (Shibboleth, rApache, Rserve, Geoconnect, etc.) and "Data Capture Module" will be added to the list. The DCM's main role in the architecture is facilitating large file transfer (#952), especially via non-HTTP mechanism such as rsync.

The Minimum Viable Product (MVP) for the Data Capture Module includes support for rsync (#2960) but other mechanisms are under consideration such as Globus (#2728, #952), Aspera, and SFTP. https://data.sbgrid.org already supports rsync and we expect to be reusing code from that service, cleaning it up and generalizing it.

The task list for the Data Capture Module is still very much in flux but I'm creating this issue so that I have an issue number to associate a branch with as I start committing some code on the Dataverse side, especially API endpoints and the ability for Dataverse to talk to the DCM.

[pameyer]

For the DCM (and data uploads generally), "rsync" is short-hand for rsync over ssh. Client-side checksums are also part of the DCM: we'll have to decide how we want to handle the difference in hash functions (switching hashes or multi-hash support in Dataverse). Sorting that out might be out of scope for DCM MVP.