Download Manager related help text

[posixeleni]

A user recently reported (here https://help.hmdc.harvard.edu/Ticket/Display.html?id=232704) that they have had several people report to them that they were unable to download the files from their dataset. The file would start to download ok and then hang up, resulting in them having to re-start the whole process which for a larger file can take a lot of time.

I spoke with @landreev about this yesterday and we should look into a way to support the ability to resume a download which got interrupted.

[pdurbin]

@pameyer and I talked about this a bit at http://irclog.iq.harvard.edu/dataverse/2016-01-08#i_29469 and it's definitely something that's of interest for @sbgrid . Their in-house system uses rsync for this. I'm not sure the best way to implement this in Dataverse. This feature also seems strongly related to the Data IO Component BRD @raprasad is working on so I left a comment on it to check if it's in scope or not.

[pdurbin]

@bmckinney I'm giving you this issue to think about and discuss with @pameyer with regard to the needs of @sbgrid. Definitely open to suggestions. 😄

[bmckinney]

Thanks, I definitely have some thoughts on this. I'll plan to discuss with @pameyer and share a google doc.

[bmckinney]

See: https://docs.google.com/document/d/1JDCKh8UzB2fQoRkmlCU8BrY0iwH0dEGnOAgJhRZVepM/edit?usp=sharing

[pdurbin]

@bmckinney nice! Please see also the diagram by @michbarsinai at #2728 (comment) and the BRD: Data IO Component doc @raprasad has been working on.

[pdurbin]

@bmckinney I just mentioned your desktop app idea to @landreev and a standup and he agrees that you need something on the client side to coordinate resumption of uploads and downloads. Also, I'm reminded of a demo @astrofrog gave to me, @mcrosas (and others?) in May 2015. I just looked at http://irclog.iq.harvard.edu/search.pl?channel=dataverse&q=trobitaille but I can't find a reference to any code anywhere but it was written in Python. I do see a mention of Dataverse at https://github.com/glue-viz/glue/blob/16bc539736cb580ac4d42ccf5f6741ada553fde1/glue/main.py#L257 and there was a little discussion at dvn/swordpoc#3 . Anyway, if we do decide to pursue some sort of desktop client for Dataverse, let's create a new GitHub issue to track it.

[bmckinney]

before i forget, this project/protocol is somewhat relevant: http://tus.io/ caveats: only deals with uploads, currently no java server implementation. it could be a nice option for "somewhat large" file uploads

[pdurbin]

"Resumable upload supports uploading videos that are up to 1.75GB and 45 minutes long." https://developers.facebook.com/docs/graph-api/video-uploads#resumable via http://socialwg.indiewebcamp.com/irc/social/2016-06-10/line/1465619059043

Too small for our requirements but potentially interesting.