Get Dataverse running on OpenShift (Docker and Kubernetes)

[pdurbin]

Yesterday I met with @portante and @danmcp and talked a fair amount about the possibility of getting Dataverse running on OpenShift. There are multiple reasons why I'm interested in this:

• Getting Dataverse running on OpenShift is prerequisite for @portante at Red Hat running it. I'm excited about a potential first non-edu customer and was happy to meet Peter at the 2017 Dataverse Community Meeting!
• Getting Dataverse running on OpenShift would greatly improve our "kick the tires on Dataverse" story, which involves running Dataverse on Vagrant on your laptop. You could walk your laptop over to your boss but it would be much nicer to send her a link to Dataverse running in the cloud (such as on OpenShift) so she can kick the tires herself.
• Getting Dataverse running on OpenShift would help move Provide Docker file to create a dev environment #3938 forward because we would very likely build on the outstanding work by @craig-willis at @nds-org to Dockerize Dataverse to be part of NDS Labs Workbench. Related to this is that if we get Dataverse running on OpenShift, it should be fairly straightforward to get Dataverse running on other Kubernetes-based cloud offerings such as Google Container Engine (GKE).
• Getting Dataverse running on OpenShift might help us improve the "make it possible to hack on Dataverse from a Windows computer" in How to get Dataverse running on a Windows computer for development #3927 because I believe @danmcp mentioned that OpenShift can be used in a development context. It reminds me a bit of how @ferrys says she doesn't install Glassfish, PostgreSQL, and Solr directly on her Mac but rather pushes code to an instance she has running on @CCI-MOC. Our current solution for Windows developers wanting to hack on Dataverse is to use Vagrant, which works but is slow and problematic.
• Getting Dataverse running on OpenShift would open up the possibility of running https://dataverse.harvard.edu (or other installations of Dataverse) on AWS using something called OpenShift Dedicated, which supports both AWS and GKE (and Azure in the future). Related to this is that the "kick the tires" story could more easily transition into "go live" if OpenShift is used along the way.

Getting Dataverse running on Openshift isn't on our roadmap so I've created this issue so we can estimate it in sprint planning or backlog grooming. Anyone reading this is very welcome to leave comments or ask questions!

[pdurbin]

I mentioned this issue to @bjonnh yesterday in chat because I've tagged him as the primary contact for the dev effort by the community to work on Docker support in #3938. He said that free OpenShift accounts would be helpful for him, @donsizemore, and anyone else who wants to help with the effort to get Dataverse running on OpenShift.

[danmcp]

You can get a free account with the starter tier:

https://www.openshift.com/pricing/index.html

That gets you 1GB of free memory. You can also use oc cluster up

https://github.com/openshift/origin/blob/master/docs/cluster_up_down.md

or minishift:

https://www.openshift.org/minishift/

to run OpenShift on your laptop.

[danmcp]

Rules for writing good images:

https://docs.openshift.org/latest/creating_images/guidelines.html

How to set the memory based on the cgroup:

There is an example in here:

https://blog.openshift.com/managing-compute-resources-openshiftkubernetes/

Look under "Writing Applications". And here is an example from mysql:

https://github.com/sclorg/mysql-container/blob/master/5.5/root/usr/bin/cgroup-limits#L48

Our postgresql image:

https://hub.docker.com/r/openshift/postgresql-92-centos7/

Example templates:

https://github.com/openshift/origin/tree/master/examples

hello-openshift is a fine place to start and move on to sample-app

[pdurbin]

@danmcp thanks!

@portante @danmcp @landreev @scolapasta and I had a great meeting today. Here's a picture of the whiteboard:

@danmcp already did his to do list items and my todo list item is to take the latest images from https://hub.docker.com/r/ndslabs/dataverse/ and reference them in a new file at conf/openshift/openshift.yaml ( @danmcp I'm seeing a YAML example at https://docs.openshift.org/latest/dev_guide/templates.html#writing-templates but not in https://github.com/openshift/origin/tree/master/examples/hello-openshift )

Basically, we'll be trying to see what breaks when we try to deploy the NDS Labs images "as is" to OpenShift. From the whiteboard, we'll need to dig into these questions about the DNS images in order to make sure they run on OpenShift:

• Do the images run as root?
• Is the storage abstracted?
• Can we use the CentOS PosgreSQL image where we already know that the image doesn't run as root and the storage is already abstracted?
• Have you tuned the memory on Java apps (Solr and Glassfish) based on running inside a container with cgroups?

We are deferring the following concerns until the future:

• clustering PostgreSQL
• clustering Solr
• running multiple Glassfish servers
• a pipeline for updating Dataverse within OpenShift (new Dataverse release, updated config for Dataverse, etc.)

Basically, the definition of done for this issue is that someone interested in kicking the tires on Dataverse for non-production use will be able to spin it up for the free 1 GB Openshift "starter" plan. The whiteboard drawing offers some clues on what the pull request might look like. In our conf directory, we'll have a Dockerfile each for Solr, PosgreSQL, and Dataverse+Glassfish. We'll have a build script to create the images and push them to DockerHub (I'll create an account for IQSS). We'll have the Openshift YAML file I mentioned above. We'll have some docs for people who want to kick the tires on Dataverse.

@danmcp I swung by @djbrooke 's office and we'd like to figure out when a good time to put this into a sprint would be. The first available would start next Wednesday, Sep 13 and go for two weeks. Let's not pick a time when you're on vacation! 😄

[danmcp]

@pdurbin Many of the examples are in json. The templates can be either.

I should be around most of the time over the next few weeks.