Linking to large external datasets

[mutanthumb]

We have a server for archived data (large files) and the plan is to use a Globus link to connect our data repository (currently DSpace) to the file. The majority of our datasets will be a reasonable size and can exist within a system such as Dataverse but some will be larger than Dataverse (or DSpace) will allow. For these large files ideally the metadata for the data file would exist within the Dataverse interface but the file will actually reside elsewhere and will be linked via a Globus link. So rather than uploading a file to Dataverse when the file metadata is entered a link would be entered instead.

We will be running our instance of Dataverse.

[pdurbin]

@mutanthumb thanks for opening this issue! I thought I'd link the discussion on the mailing list for some additional context: https://groups.google.com/forum/#!topic/dataverse-community/Pe6znGA000Y

In that thread @eugene-barsky mentioned https://www.computecanada.ca/research-portal/national-services/globus-portal/ and said he'll be able to report back to the Dataverse community with more information.

There are actually a lot of projects in the air related to large files such as https://hms.harvard.edu/news/dropbox-structural-biologists

I also can't resist uploading a quick doodle (just an idea!) by @michbarsinai about large files:

[eugene-barsky]

Yep, I plan to work on Globus/Dataverse integration maybe in Q1 2016 with
Compute Canada

E

On Fri, Nov 13, 2015 at 9:12 AM, Philip Durbin notifications@github.com
wrote:

@mutanthumb https://github.com/mutanthumb thanks for opening this
issue! I thought I'd link the discussion on the mailing list for some
additional context:
https://groups.google.com/forum/#!topic/dataverse-community/Pe6znGA000Y

In that thread @eugene-barsky https://github.com/eugene-barsky
mentioned
https://www.computecanada.ca/research-portal/national-services/globus-portal/
and said he'll be able to report back to the Dataverse community with more
information.

There are actually a lot of projects in the air related to large files
such as https://hms.harvard.edu/news/dropbox-structural-biologists

I also can't resist uploading a quick doodle (just an idea!) by
@michbarsinai https://github.com/michbarsinai about large files:

[image: largefileservice]
https://cloud.githubusercontent.com/assets/21006/11152310/c0d357da-89ff-11e5-83ae-cf556eb13e77.png

—
Reply to this email directly or view it on GitHub
#2728 (comment).

[posixeleni]

Just in case we want to consolidate this ticket this appears to be related to #1347

[pdurbin]

This issue is at least somewhat related to "Support The Ability To Resume Disrupted File Downloads" at #2960. Mostly because of the LargeFileService diagram that appears here. @michbarsinai when you get a chance can you please attach the plantuml file for that diagram?