Enable Upload of larger files

[mercecrosas]

From Michael B-S:

Here's how vimeo allows uploading of arbitrary large files (in their case, HD videos):

http://developer.vimeo.com/api/upload#http-put-uploading

In short, they create an upload ticket with its own url, have the user perform multiple PUT requests with binary data to it, and then finish the upload with a DELETE request on the ticket. There are also simple ways of validating the uploaded file.

There are some improvements that can be done over this, mainly allowing parallel uploads and having an md5 sum to validate not just the amount of files but also their correctness. Still, an interesting approach we could possibly adopt.

[michbarsinai]

Possible improvements:

1. Allow parallel uploads by PUTting the bits into their sequence number and not serially: e.g. PUT a binary chunk of a file to dataverse-root/api/uploadTickets/<id>/<chunk number>. on end, concatenate the chunks in sequence.
2. The PUT response can return an md5 sum of the binary payload. Same for the DELETE response.

[michbarsinai]

Here's how Amazon S3 handles large files:
http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html

[posixeleni]

@mcrosas @scolapasta since this relates to the helmsley grant funded project we are working on that will need FRD work to start in early January who should this ticket be assigned to and which milestone should we add it to?

cc/ @pdurbin @eaquigley

[pdurbin]

I'm fine with using this issue to track work for @sbgrid (the Helmsley grant). If so, it can be assigned to me. Or we can create an issue dedicated to that work with specific requirements.

I'd also like to point out some related issues:

• Object Storage for Dataverse #1347 opened by @spalan mostly about S3 and Swift support (interest by @CCI-MOC)
• Linking to large external datasets #2728 opened by @mutanthumb mostly about storing the files elsewhere (i.e. Globus)

For storage abstraction in general @akio-sone has been working on some branches related to @irods support. I'm not sure what the status is.

Also @raprasad will be working in this space (storage, generally). We should see how much overlap there is on requirements.

Finally, both @sbgrid and @CCI-MOC are interested in computing on the data. I think they both have in mind that they will have direct access to the filesystem, which is different than how TwoRavens/Zelig works, from what I understand. I'd like to check in with @landreev on this.

[pameyer]

Does it make sense to include local checksumming (md5, sha, something else) in with the large file uploads, or should that be split off into something separate (since it could potentially be of use for small file uploads too)?

[pdurbin]

@pameyer good point about checksums. This certainly deserves its own issue so I just created one: #2788

[spalan]

I think the checksumming is critical in any context where a researcher needs to be sure of the integrity of the file he/she is working with. It is also critical to the data archivist for proactive management of content he/she is curating, so that periodically checksums can be validated and corruption due to bit rot or other issues can be flagged early on.

Alan

On Dec 1, 2015, at 2:45 PM, pameyer <notifications@github.commailto:notifications@github.com> wrote:

Does it make sense to include local checksumming (md5, sha, something else) in with the large file uploads, or should that be split off into something separate (since it could potentially be of use for small file uploads too)?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/952#issuecomment-161075150.