Add Glassfish Statefulsets to OpenShift/Kubernetes

[MichaelClifford]

As students in Boston University's EC528 Cloud Computing Course, my team has been working with @pdurbin @danmcp & @DirectXMan12 to further work on #4040 & #4168.

I have been working on scaling the Glassfish pods and am ready to open a pull request. My solution entails creating a statefulset and providing updates to the Glassfish installation script to differentiate between the initial master pod, dataverse-glassfish-0, and any subsequently deployed pods.

[pdurbin]

@MichaelClifford thanks for opening this issue and working on this project! You're editingopenshift.json, right? I'm asking because pull request #4599 by @patrickdillon has edits to that file and I'm close to merging that pull request. I don't mean to complicate your life but you might want to branch from his branch before you make your pull request. That way, we can test both changes together.

[MichaelClifford]

@pdurbin <https://github.com/pdurbin>, yes, that's correct. I have made some updates to the openshift.json configuration file. I was just about to submit my pull request in the next hour or so here (doing some final checks), but if you want me to hold off until #4599 is merged, that's fine. Just let me know.

…

On Wed, Apr 25, 2018 at 6:38 PM, Philip Durbin ***@***.***> wrote: @MichaelClifford <https://github.com/MichaelClifford> thanks for opening this issue and working on this project! You're editingopenshift.json, right? I'm asking because pull request #4599 <#4599> by @patrickdillon <https://github.com/patrickdillon> has edits to that file and I'm close to merging that pull request. I don't mean to complicate your life but you might want to branch from his branch before you make your pull request. That way, we can test both changes together. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4617 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AESWOq0Pq0Ap2-HRzB5Ycn6UISRdD1yFks5tsPr5gaJpZM4TkETi> .

[pdurbin]

@MichaelClifford right, it's the final checks that I don't want you to waste your time on if I'm going to ask you to retest with the combined solution. Does that make sense? I'm also leaving comments for you in #4598 just to make this extra hard. 😉

[pdurbin]

@MichaelClifford ok I just merged #4599 so please makes sure your (future) pull request has those changes in it before you spend a lot of time testing. @patrickdillon and I think his changes and your changes to openshift.json should be compatible. Please let me know if you have questions. Thanks!

[pdurbin]

@MichaelClifford thanks for making pull request #4626! I requested a few minor changes and I'm going to ask @landreev to take a look at changes you made to glassfish-setup.sh.

If you have any questions, please let me know!

[pdurbin]

@MichaelClifford how much memory do you recommend for minishift? I'm been using 4GB like this:

minishift start --vm-driver=virtualbox --memory=4GB

But as of your latest fixes (ae115fd) I'm getting "insufficient memory" errors:

[MichaelClifford]

@pdurbin I would recommend changing line 156 of the openshift.json file "resources": {"limits": {"memory": "3072Mi"} to "resources": {"limits": {"memory": "512Mi"} when deploying locally.

This seems to work on minishift, but when deployed to the MOC we needed additional resources, about 3Gb, to successful run the glassfish pods. Let me know if that helps.

[pdurbin]

@MichaelClifford Whoops! I wrote 8GB and 4GB above but I'm really using 4GB, I just edited my comment above to reflect reality.

Does that change anything? It seems a bit counterintuitive to give it less memory (512Mi instead of 3072Mi), but I'll give it a shot!

[pdurbin]

Ok, I tried a few things. I switched to 8GB like this:

minishift start --vm-driver=virtualbox --memory=8GB

I'm not messing with memory in openshift.json. The only changes I made locally are to switch to my Docker Hub account ("pdurbin") rather than "iqss". I won't be pushing this change, of course:

murphy:dataverse pdurbin$ git diff conf/openshift/openshift.json
diff --git a/conf/openshift/openshift.json b/conf/openshift/openshift.json
index b49f7fe..684b1d1 100644
--- a/conf/openshift/openshift.json
+++ b/conf/openshift/openshift.json
@@ -100,7 +100,7 @@
         "name": "dataverse-plus-glassfish"
       },
       "spec": {
-        "dockerImageRepository": "iqss/dataverse-glassfish"
+        "dockerImageRepository": "pdurbin/dataverse-glassfish"
       }
     },
     {
@@ -120,7 +120,7 @@
         "name": "iqss-dataverse-solr"
       },
       "spec": {
-        "dockerImageRepository": "iqss/dataverse-solr"
+        "dockerImageRepository": "pdurbin/dataverse-solr"
       }
     },
     {
@@ -146,7 +146,7 @@
             "containers": [
               {
                 "name": "dataverse-plus-glassfish",
-                "image": "iqss/dataverse-glassfish:latest",
+                "image": "pdurbin/dataverse-glassfish:latest",
                 "ports": [
                   {
                     "containerPort": 8080,
murphy:dataverse pdurbin$ 

The good news is that oc new-app conf/openshift/openshift.json "just worked" in the sense that I got a working Dataverse installation, was able to log in, create a dataverse, see that it was indexed into Solr, etc. Then I thought I'd try a similar trick as at #4598 (comment) to scale the number of replicas to 3:

oc scale statefulset/dataverse-glassfish --replicas=3

For some reason, I'm getting "The pod has been stuck in the pending state for more than five minutes." Here are screenshots:

Before (1 replica)

After (immediately)

After (five minutes later)

I'm not really sure what's going on. Dataverse is still up, running off dataverse-glassfish-0, but I'm not sure how to scale up to additional replicas.

Any thoughts for me @MichaelClifford @danmcp @DirectXMan12 @patrickdillon @rockash? I tried looking at the log for dataverse-glassfish-1 but this pod seems not well:

I'm on commit ae115fd

[danmcp]

Anything going on in events?

[MichaelClifford]

@pdurbin, Not sure if you tired this yet or not, but did you try using my glassfish image jcliffbu/glassboat:test2 or did you rebuild the entire application and push it to your docker hub?