Docker for production

[omaralsoudanii]

Hello ,
Is there official production ready docker container for data verse ? and if available where can i find the docs?
Thank you

[pdurbin]

@omaralsoudanii thanks for opening this issue! It's been nice talking to you over at nds-org/ndslabs-dataverse#8 but this main issue tracker for Dataverse is a better place to discuss the state of Docker support. I can try to explain.

The short answer is that unfortunately, there is no production-ready Docker image for Dataverse as of this writing. I'm still learning Docker and its ecosystem so I'd like to know from you and others who are interested in Docker support how you expect it to work. Do you expect all of Dataverse to be running in a single container? Do you expect some of the components such as PostgreSQL and Solr to be running in different containers? Do you expect to use docker-compose? Do you expect to run Docker images in an orchestration platform such as Kubernetes?

If you look at the source tree, this is what you will find today:

• conf/docker: Docker images intended to be run in OpenShift, which is a distribution of Kubernetes. See Get Dataverse running on OpenShift (Docker and Kubernetes) #4040 and http://guides.dataverse.org/en/4.8.6/developers/containers.html#future-production-use-on-minishift-openshift-kubernetes
• conf/docker-aio: A single "all in one" Docker image that some of use to run the API test suite. See http://guides.dataverse.org/en/4.8.6/developers/containers.html#all-in-one-docker-images-for-testing

I hope this helps. If you and others interested in Docker support can answer the questions above or other questions I don't even know to ask, it would be very helpful! Thanks!

[omaralsoudanii]

@pdurbin Thanks for this ,
It would be nice to have 3 containers :
1- Data verse repo container
2- Postgres container
3- Solr container

And a docker-composer.yml file that will build 3 images / 3 containers for those automatically ,
Also it would be good to include an entry point to manage update scripts to date verse and Postgres sql changes , See https://docs.docker.com/engine/reference/builder/#entrypoint .

Because the problem we're having now is in order to update and maintain data verse we need to adjust DockerFile manually (change data verse instance version) and do the update process manually (run each Postgress schema changes manually ) , So some automated script that takes cares of those would be nice .

Another suggestion would be that the data verse team adjusts the DockerFile configuration each time an update to data verse is released so that we can simply pull docker changes from data verse repo and do :

docker-compose down

docker-compose up --build

For our current version (4.7) though we're trying to update to the latest version (4.8.6) in order to use the license and restrict files native API but i need a fresh data verse Postgres database , Is there a script or sql file for that?

Thank you.

[pameyer]

@omaralsoudanii Any thoughts on an additional container for apache? My understanding is that it's preferable to not expose glassfish/dataverse directly to the outside world, so it might make sense to deal with it at this level of abstraction (but I'm not currently planning on using dockerized dataverse in production, so input from folks that are would be interesting).

[vsoch]

hey everyone! I think I can definitely help with this, and I'd like to suggest starting with a simple docker-compose setup, and then using kompose to transition to kubernetes when the time is right! I'll start poking around this weekend (I'm not familiar with the code base) and then we can loop back next week (when the RedHat intern joins up?) How would you like to have our mentoring / discussion? Github? Slack? Other?

[pdurbin]

@vsoch thanks for the offer to help with this effort! I haven't even met the intern yet but @danmcp mentioned internship goals the other day at http://irclog.iq.harvard.edu/dataverse/2018-05-22#i_67516 . I believe @djbrooke said this individual might be starting as soon as next week and I said I'd be happy to mentor him or her on all things related to Dataverse. I'm happy to help you and others dive in as well, of course!

Perhaps a timeline of events (from my perspective anyway) would help ground us:

• 2016-03: @craig-willis from @nds-org dockerizes Dataverse and gets it running on Kubernetes for NDS Labs Workbench , which is described from the Dataverse perspective at http://guides.dataverse.org/en/4.8.6/installation/prep.html#nds-labs-workbench-for-testing-only . The code can be found at https://github.com/nds-org/ndslabs-dataverse
• 2017-06 @portante from Red Hat attends the third annual Dataverse Community Meeting and explains that Red Hat might be interested in running Dataverse if it runs on OpenShift, a distribution of Kubernetes. Get Dataverse running on OpenShift (Docker and Kubernetes) #4040 is opened to track this.
• 2017-09 @pdurbin uses the Docker/Kubernetes work by @craig-willis as a starting point to create pull request 4040 docker openshift #4168 with help from @danmcp to get Dataverse runnig on OpenShift but not for production use. A list of shortcomings appears at http://guides.dataverse.org/en/4.8.6/developers/containers.html#known-issues-with-dataverse-images-on-docker-hub
• 2017-12 @pameyer adds an "all in one" Docker config in pull request docker all-in-one for integration tests #4370 that can be easily spun up for the purpose of integration testing.
• 2018-02 @joelmarkanderson says, "We are starting to actively develop on the Terraform -> Kubernetes path, and I've begun reconfiguring Docker containers to reflect current dataverse versions": The name and future of this git repo dataverse-aws#11 (comment)
• 2018-03 @aculich expresses interest in running Dataverse on Docker at http://irclog.iq.harvard.edu/dataverse/2018-03-02#i_64000 and follows up with Binderverse: integrating Binderhub with Dataverse (using docker+kubernetes) #4714.
• 2018-03 @4tikhonov provides a link to the docker-compose.yml file he's been working on: https://github.com/Dans-labs/dataverse/blob/4.8.5-compose/conf/docker-eu/docker-compose.yml
• 2018-04 @danmcp @pdurbin and @DirectXMan12 wrap up a semester of mentoring BU students Patrick Dillon, Michael Clifford, Ashwin Pillai, and Ryan Morano to start the work of making it possible to scale up the number of Glassfish (Adding Glassfish Statefulsets for OpenShift Deployment #4626) and PostgreSQL (Add Postgresql statefulsets with master/slave replication on OpenShift. #4599) servers in use within OpenShift. A summary and link to the final class video can be found at Get Dataverse running on OpenShift (Docker and Kubernetes) #4040 (comment)
• 2018-05 @omaralsoudanii creates this issue, indicating interest in running Dataverse in production on Docker. @xibriz sounds interested, based on a comment at Get Dataverse running on OpenShift (Docker and Kubernetes) #4040 (comment) .

I'm probably forgetting people and events but it's a start. In a future comment I'd like to write more about how I can imagine the work for this issue being broken up into smaller chunks.

[pameyer]

@vsoch The idea of starting with docker-compose sounds like a reasonable place to start. There's an example of docker-compose in this repository (although intended for a different purpose) in conf/docker-dcm.

One thing that may be a factor for production-izing Docker/Dataverse (and that @pdurbin and I have talked about) is how to handle the intersection of the docker entry point and the dataverse installer. It's very likely that the approach I've been taking as been sub-optimal for production usage (and so wouldn't make a good prototype for production).

[4tikhonov]

Probably our docker-compose from DataverseEU project can be interesting to try as well: https://github.com/Dans-labs/dataverse-docker/blob/master/docker-compose.yml

[vsoch]

@pameyer agreed I would not have the command coincide with the entire install, so the container can be restarted without prompting it done again!

@4tikhonov the work you've done for DataverseEU looks great! It's 99% there! Is it just a matter of combining the two repos into a consolidated thing with a good set of docs? With your blessings I can give a first shot at this.

[4tikhonov]

Thanks! We're running it with Kubernetes on Google Cloud already http://dataverse-dev.cessda.eu
Some documentation for Docker module is available here: https://github.com/Dans-labs/dataverse-docker

[vsoch]

So what needs to be done then?

[4tikhonov]

We're still working on the proxy implementation for multilingual support to get different languages on selected path (/fr for French, for example). First we tried with apache but Google Cloud already has nginx as integrated service running inside of Kubernetes, it seems to be better solution.

[pameyer]

@4tikhonov How did you end up handling the entrypoint / installer issue (run installer in entrypoint, something else)?

[4tikhonov]

@pameyer, I remember that you advised me to do health check before running installer second time:
size=$(curl -sI http://localhost:8080/api/info/version | grep Content-Length|awk '{print $2}')

It's in entrypoint indeed, if size > 0 then Dataverse is already installed and further installation steps can be skipped.