Add Postgresql Statefulsets with Replication to OpenShift/Kubernetes

[patrickdillon]

As students in Boston University's EC528 Cloud Computing Course, my team has been working with @pdurbin @danmcp & @DirectXMan12 to further work on #4040 & #4168.

I have been working on scaling the postgres pods and am ready to open a pull request. My solution entails creating a statefulset and providing a new command at startup for the centos/postgres image.

[patrickdillon]

Pull request: #4599

For context, our project description is available in the class repo, which I just switched from private to public:
https://github.com/BU-NU-CLOUD-SP18/Dataverse-Scaling

[pdurbin]

@patrickdillon thanks for the pull request! As we discussed in today's call ( https://bluejeans.com/s/7dg33/ ) it sounds like there are a few tweaks you might want to make (possibly stetting replicas to 1?) and some additional testing you might want to do before we pass this to QA. If there's anything you need, please let us know! Thanks!

[pdurbin]

@patrickdillon I assigned this issue to you and at the moment it's in code review at our kanban board at https://waffle.io/IQSS/dataverse . I can move it back to the "development" column if you'd like. Please just keep me posted when you'd like more code review or if you're done making commits and want to move it to QA. Thanks!

[pdurbin]

I spoke with @djbrooke about this issue and pull request this morning and the plan is for me to QA it. I'm in the middle of other coding but I'll try to get to it soon. @patrickdillon made some changes which are reflected in the pull request and which we just discussed at https://bluejeans.com/s/ygEhi

[pdurbin]

The change to default.config is going to require me to push a new Dataverse/Glassfish image to Docker Hub. I've been talking to @MichaelClifford about my process for testing and to make it concrete, I'll explain below.

First, I add a remote and switch to the branch behind the pull request:

git fetch EC528-Dataverse-Scaling
git checkout 4598-openshift-postgresql

Then, I edit openshift.json to change the tag from latest to the name of the branch, but I won't be committing this change because when we merge the branch, I want it to still be latest:

murphy:dataverse pdurbin$ vi conf/openshift/openshift.json
murphy:dataverse pdurbin$ git diff conf/openshift/openshift.json
diff --git a/conf/openshift/openshift.json b/conf/openshift/openshift.json
index bf74631..a6c2089 100644
--- a/conf/openshift/openshift.json
+++ b/conf/openshift/openshift.json
@@ -212,7 +212,7 @@
               ],
               "from": {
                 "kind": "ImageStreamTag",
-                "name": "dataverse-plus-glassfish:latest"
+                "name": "dataverse-plus-glassfish:4598-openshift-postgresql"
               }
             }
           },
murphy:dataverse pdurbin$ 

To get ready to push images to DockerHub, I clean out the war file and the installer:

murphy:dataverse pdurbin$ mvn clean
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building dataverse 4.8.6
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ dataverse ---
[INFO] Deleting /Users/pdurbin/github/iqss/dataverse/target
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.808 s
[INFO] Finished at: 2018-04-24T08:53:46-04:00
[INFO] Final Memory: 7M/245M
[INFO] ------------------------------------------------------------------------
murphy:dataverse pdurbin$ cd scripts/installer/
murphy:installer pdurbin$ make clean
/bin/rm -rf dvinstall dvinstall.zip
murphy:installer pdurbin$ 
murphy:installer pdurbin$ cd ../..
murphy:dataverse pdurbin$ 

I take a look at https://hub.docker.com/r/iqss/dataverse-glassfish/tags/ to make sure I'm not going to overwrite someone else's tag.

Then, I run the build script, pushing to the branch name:

murphy:docker pdurbin$ ./build.sh 
No argument supplied. Please specify "branch" or "custom my-custom-tag" for experiments or "stable" if your change won't break anything.
murphy:docker pdurbin$ ./build.sh branch
We'll push a tag to the branch you're on.
Images will be pushed to Docker Hub with the tag "4598-openshift-postgresql".
...

This is the part that takes forever. Stay tuned!

[pdurbin]

@patrickdillon I'm afraid I'm struggling a bit to test pull request #4599.

"dataverse-glassfish" is saying "No deployments. A new deployment will start automatically when an image is pushed to project1/dataverse-plus-glassfish:4598-openshift-postgresql." But I've already pushed the tag to Docker Hub. I'll included screenshots below. I'll also attach my openshift.json file, which has the edit above to change latest to 4598-openshift-postgresql, which is the tag I pushed.

Here's my openshift.json file, renamed with the commit in it so I remember which commit I'm on and with .txt added so I can upload it to GitHub: openshift.json-51cecfc.txt

@danmcp @DirectXMan12 if you have any ideas for me, please let me know. I hope it's just that I'm forgetting to do something simple.

@patrickdillon maybe you or one of the other students can try the image I pushed to the 4598-openshift-postgresql tag at https://hub.docker.com/r/iqss/dataverse-glassfish/tags/

[patrickdillon]

@pdurbin Unfortunately I never really figured out tags. I spent some time this morning trying to look at this in particular but I would need more time. Perhaps @danmcp or @DirectXMan12 could fix it the right way or show best practices.

I could never get any tags besides latest to work so my workflow was always to copy the image to my own personal repo with the latest tag. So it doesn't solve your exact problem but you could copy the image with a different name and the latest tag to your repo or the IQSS repo. That might let you test until we resolve the tag issue. Let me know if that doesn't make sense.

[danmcp]

@pdurbin To be clear, if you manually hit the deploy button, does it work?