MOC collaboration phase 1: Create a Swift/OpenStack storage driver

[landreev]

Creating this GitHub issue, so that I could make a branch for the code I'm writing. And to serve as a starting point for other developers who may get involved in this project.

Buzzword Glossary:

MOC - Massachusetts Open Cloud project;
OpenStack - open source OS for cloud computing; used by the MOC.
Swift aka OpenStack Object Storage - scalable/redundant storage system used in OpenStack;
JOSS/javaswift - Java client for Swift
Ceph - distributed storage platform; in our specific case, Ceph implements block access as Swift reads and writes data files.

The first, demo iteration of this storage driver will be using JOSS to read objects from a Swift endpoint. Thus Dataverse will be able to serve files physically stored on/by the MOC, transparently to the user. I.e. the file will be represented by a local DataFile object in the Dataverse database; and have its corresponding byte stream stored in the cloud and referenced by the storage identifier attribute of the DataFile. The code for this driver is based on the examples provided in javaswift/tutorial-joss-quickstart.

An instruction on how to access the MOC Swift "endpoint" and the Ceph storage "tenant" created for this collaboration is provided by Ata Turk from bu.edu here: https://www.dropbox.com/s/p053agfx31oxf6l/Readme.txt?dl=0 (ask me for the password).

The Dataverse instance for this project will be installed on a virtual server on the MOC. We'll have an account there that will allow us to launch a new VM from a CentOS image - should be trivial. (https://github.com/CCI-MOC/moc-public/wiki/Getting-started) I will add more info on that plus the address of the Dataverse node, etc.

[landreev]

Launched a brand-new CentOS node on the MOC, at 129.10.3.145.
(ask me for ssh credentials if you need to log in).
To get your own MOC account: fill the form at https://github.com/CCI-MOC/moc-public/wiki/MOC-Production-(Kaizen)-User-Account-Requests
Once you have received the password for the account:
The "getting-started" instructions, at https://github.com/CCI-MOC/moc-public/wiki/Getting-started appear to be missing a couple of important steps (as of today). But video tutorials are provided, where everything is explained:

1. how to configure and launch a VM:
   skip to 3:15!
   https://www.youtube.com/watch?v=9_PbcPV_jEU
2. once the VM is running, how to ssh to it from the outside:
   https://www.youtube.com/watch?v=ZjdrVHPjltI

[pdurbin]

@pameyer from @sbgrid expressed interest in object storage (S3) at http://irclog.iq.harvard.edu/dataverse/2016-02-25#i_31584 . e672380 is a good commit to look at (heads up @bmckinney ). Also, this issue is related to the more general "object storage" issue at #1347.

[spalan]

Scholars Portal in Ontario is both a Dataverse and OpenStack Swift user -- we have a 350TB Swift object store in production. Would be happy to help with testing as this develops.

[djbrooke]

Hey @landreev, @anuj-rt - is this moving forward with the MOC work? If it's done or being tracked elsewhere can we close it? Thanks!

[spalan]

I was wondering which branch I could use to get the version of DV that supports Swift. I would be interested in trying it with our local cluster.

Alan

On Jul 29, 2016, at 2:38 PM, Danny Brooke <notifications@github.commailto:notifications@github.com> wrote:

Hey @landreevhttps://github.com/landreev, @anuj-rthttps://github.com/anuj-rt - is this moving forward with the MOC work? If it's done or being tracked elsewhere can we close it? Thanks!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com//issues/2909#issuecomment-236260233, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABzXrNBS6xFFcvtpy0n9Wj8eARR_AeUhks5qakiagaJpZM4HSrLr.

[pdurbin]

@spalan pull request #3239 contains some Swift code by @anuj-rt but I'm not sure exactly what it does.

[anuj-rt]

@spalan @pdurbin @landreev pull request #3239 contains swift code with keystone authentication. Instead of getting a token from Swift service endpoint we get a token from Keystone identity service. This was needed moving forward with the use of MOC's OpenStack production environment.

[pdurbin]

@anuj-rt ok, sorry. I guess I got excited by stuff like this.getDataFile().setStorageIdentifier(swiftFileObject.getPublicURL()). Carry on!

[anuj-rt]

@pdurbin Yes, that line pulls the URL of the file stored on Swift but the authentication method is different.

Actually, @landreev does have a separate swift branch which works for Swift users and if I understand the above scenario correctly that branch will work for its purpose. Even this branch can be made to work with both keystone and swift users, some changes in the code base will be required.

[spalan]

That is interesting. What version is that branch at? We are running 4.5.1 now and would love to have the option of having data files stored on our OpenStack Swift storage cluster — for replication, to eliminate file storage size limits, etc…

Alan

On Oct 24, 2016, at 4:26 PM, Anuj Thakur <notifications@github.commailto:notifications@github.com> wrote:

@pdurbinhttps://github.com/pdurbin Actually, @landreevhttps://github.com/landreev does have a separate swift branch which works for Swift users and if I understand the above scenario correctly that branch will work for its purpose. Even this branch can be made to work with both keystone and swift users, some changes in the code base will be required.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/2909#issuecomment-255855811, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABzXrPwqwqq3H4R0N77OWe_P6hcGb4Sfks5q3RSOgaJpZM4HSrLr.