Skip to content

Add Postgresql statefulsets with master/slave replication on OpenShift. #4599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pdurbin merged 2 commits into from
Apr 25, 2018
Merged

Add Postgresql statefulsets with master/slave replication on OpenShift. #4599

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
51cecfc
Add Postgresql statefulsets with master/slave replication on OpenShift.
patrickdillon Mar 8, 2018
e7a56c7
add `huborg <USERNAME/ORG>` for non-iqss Docker Hub #4599
pdurbin Apr 25, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 25 additions & 11 deletions conf/docker/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,15 @@
#!/bin/sh
# Creates images and pushes them to Docker Hub.
# The "latest" tag should be relatively stable. Don't push breaking changes there.
# The "latest" tag under "iqss" should be relatively stable. Don't push breaking changes there.
# None of the tags are suitable for production use. See https://github.com/IQSS/dataverse/issues/4040
# Push to custom tags or tags based on branch names to iterate on the images.
# To interate on images, push to custom tags or tags based on branch names or a non-iqss Docker Hub org/username.
# Docker Hub organization or username
HUBORG=iqss
# The most stable tag we have.
STABLE=latest
#FIXME: Use a real flag/argument parser. download-files.sh uses "getopts" for example.
if [ -z "$1" ]; then
echo "No argument supplied. Please specify \"branch\" or \"custom my-custom-tag\" for experiments or \"stable\" if your change won't break anything."
echo "No argument supplied. For experiments, specify \"branch\" or \"custom my-custom-tag\" or \"huborg <USERNAME/ORG>\". Specify \"stable\" to push to the \"$STABLE\" tag under \"$HUBORG\" if your change won't break anything."
exit 1
fi

Expand All @@ -14,23 +19,32 @@ if [ "$1" == 'branch' ]; then
TAG=$GIT_BRANCH
elif [ "$1" == 'stable' ]; then
echo "We'll push a tag to the most stable tag (which isn't saying much!)."
TAG=kick-the-tires
TAG=$STABLE
elif [ "$1" == 'custom' ]; then
if [ -z "$1" ]; then
echo "You must provide a custom tag as the second argument."
if [ -z "$2" ]; then
echo "You must provide a custom tag as the second argument. Something other than \"$STABLE\"."
exit 1
else
echo "We'll push a custom tag."
TAG=$2
fi
elif [ "$1" == 'huborg' ]; then
if [ -z "$2" ]; then
echo "You must provide your Docker Hub organization or username as the second argument. \"$USER\" or whatever."
exit 1
else
HUBORG=$2
TAG=$STABLE
echo "We'll push to the Docker Hub organization or username you specified: $HUBORG."
fi
else
echo "Unexpected argument: $1. Exiting. Run with no arguments for help."
exit 1
fi
echo Images will be pushed to Docker Hub with the tag \"$TAG\".
echo Images will be pushed to Docker Hub org/username \"$HUBORG\" with the tag \"$TAG\".
# Use "conf" directory as context so we can copy schema.xml into Solr image.
docker build -t iqss/dataverse-solr:$TAG -f solr/Dockerfile ../../conf
docker push iqss/dataverse-solr:$TAG
docker build -t $HUBORG/dataverse-solr:$TAG -f solr/Dockerfile ../../conf
docker push $HUBORG/dataverse-solr:$TAG
# TODO: Think about if we really need dataverse.war because it's in dvinstall.zip.
cd ../..
mvn clean
Expand Down Expand Up @@ -58,6 +72,6 @@ if [[ "$?" -ne 0 ]]; then
fi
# We'll assume at this point that the download script has been run.
cp ../../downloads/weld-osgi-bundle-2.2.10.Final-glassfish4.jar dataverse-glassfish
docker build -t iqss/dataverse-glassfish:$TAG dataverse-glassfish
docker build -t $HUBORG/dataverse-glassfish:$TAG dataverse-glassfish
# FIXME: Check the output of `docker build` and only push on success.
docker push iqss/dataverse-glassfish:$TAG
docker push $HUBORG/dataverse-glassfish:$TAG
2 changes: 1 addition & 1 deletion conf/docker/dataverse-glassfish/default.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ GLASSFISH_DIRECTORY /usr/local/glassfish4
ADMIN_EMAIL
MAIL_SERVER mail.hmdc.harvard.edu
POSTGRES_ADMIN_PASSWORD secret
POSTGRES_SERVER dataverse-postgresql-service
POSTGRES_SERVER dataverse-postgresql-0.dataverse-postgresql-service
POSTGRES_PORT 5432
POSTGRES_DATABASE dvndb
POSTGRES_USER dvnapp
Expand Down
49 changes: 43 additions & 6 deletions conf/openshift/openshift.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
"selector": {
"name": "iqss-dataverse-postgresql"
},
"clusterIP": "None",
"ports": [
{
"name": "database",
Expand Down Expand Up @@ -150,6 +151,10 @@
}
],
"env": [
{
"name": "POSTGRES_SERVER",
"value": "dataverse-postgresql-0"
},
{
"name": "POSTGRES_SERVICE_HOST",
"value": "dataverse-postgresql-service"
Expand Down Expand Up @@ -222,15 +227,17 @@
}
},
{
"kind": "DeploymentConfig",
"apiVersion": "v1",
"kind": "StatefulSet",
"apiVersion": "apps/v1beta1",
Copy link
Member

@pdurbin pdurbin Apr 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does v1beta1 require a beta version of minishift? Or can I just use the most recent regular release of minishift?

Copy link
Contributor Author

@patrickdillon patrickdillon Apr 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regular release is fine. I was running on v1.14

Copy link
Member

@pdurbin pdurbin Apr 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, great. Thanks.

"metadata": {
"name": "dataverse-postgresql",
"annotations": {
"template.alpha.openshift.io/wait-for-ready": "true"
}
},
"spec": {
"serviceName" : "dataverse-postgresql-service",
"replicas" : 1,
Copy link
Contributor Author

@patrickdillon patrickdillon Apr 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@danmcp raised the question of whether running a setup with only a master and no slaves would have any side effects. I think there would be no side effects and it would be similar to just running a single instance Postgres setup.

Our code is based off this example, which states:

Once the master is started, it works as a standalone database server, fully independent of the slaves.

I dug in a little more to the image to see what is happening:
For the postgres image, the only side effect for running as master is that these configuration settings are copied into the main postgres config file: https://github.com/sclorg/postgresql-container/blob/master/src/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql-replication.conf.template

To me, the above settings look innocuous.

For further context, here is the master-binary: https://github.com/sclorg/postgresql-container/blob/master/src/root/usr/bin/run-postgresql-master

And that environment variable is called in this common script, which just copies the first file above into the main config settings: https://github.com/sclorg/postgresql-container/blob/b737ad26db4f8769e775ca165284f62c7f2a66db/src/root/usr/share/container-scripts/postgresql/common.sh

Copy link
Member

@pdurbin pdurbin Apr 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@patrickdillon thanks, I just moved this to QA.

"template": {
"metadata": {
"labels": {
Expand All @@ -241,7 +248,10 @@
"containers": [
{
"name": "centos-postgresql-94-centos7",
"image": "centos-postgresql-94-centos7",
"image": "centos/postgresql-94-centos7",
Copy link
Contributor Author

@patrickdillon patrickdillon Apr 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By using centos/postgresql-94-centos7 instead of centos-postgresql-94-centos7 we are pointing directly to the Docker Hub repo rather than the ImageStream. The reason I changed this was because it started throwing an error after changing from a DeploymentConfig to a StatefulSet v1beta1. I just tried testing again with the image stream and it still is getting the error. Once OpenShift is updated to a newer version we can try changing it back or if there are any suggestions let me know.

"command": [
"sh", "-c", "echo 'Setting up Postgres Master/Slave replication...'; [[ `hostname` =~ -([0-9]+)$ ]] || exit 1; ordinal=${BASH_REMATCH[1]}; if [[ $ordinal -eq 0 ]]; then run-postgresql-master; else run-postgresql-slave; fi;"
],
"ports": [
{
"containerPort": 5432,
Expand All @@ -253,10 +263,34 @@
"name": "POSTGRESQL_USER",
"value": "dvnapp"
},
{
"name": "POSTGRESQL_MASTER_USER",
"value": "master"
},
{
"name": "POSTGRESQL_PASSWORD",
"value": "secret"
},
{
"name": "POSTGRESQL_MASTER_PASSWORD",
"value": "master"
},
{
"name": "POSTGRESQL_MASTER_SERVICE_NAME",
"value": "dataverse-postgresql-service"
},
{
"name": "POSTGRESQL_MASTER_IP",
"value": "dataverse-postgresql-0.dataverse-postgresql-service"
},
{
"name": "postgresql_master_addr",
"value": "dataverse-postgresql-0.dataverse-postgresql-service"
},
{
"name": "master_fqdn",
"value": "dataverse-postgresql-0.dataverse-postgresql-service"
},
{
"name": "POSTGRESQL_DATABASE",
"value": "dvndb"
Expand All @@ -281,6 +315,7 @@
]
}
},

"strategy": {
"type": "Rolling",
"rollingParams": {
Expand All @@ -300,17 +335,19 @@
],
"from": {
"kind": "ImageStreamTag",
"name": "centos-postgresql-94-centos7:latest"
"name": "centos/postgresql-94-centos7:latest"
}
}
},
{
"type": "ConfigChange"
}
],
"replicas": 1,
"selector": {
"name": "iqss-dataverse-postgresql"
"name": "iqss-dataverse-postgresql",
"matchLabels" : {
"name" : "iqss-dataverse-postgresql"
}
}
}
},
Expand Down