From b153aa85a538494bee6f6f2ca46863c3f019c34e Mon Sep 17 00:00:00 2001 From: jeff vance Date: Tue, 22 Sep 2015 20:37:35 -0400 Subject: [PATCH 1/9] ceph examples --- examples/README.md | 1 + examples/storage-examples/README.md | 10 + .../storage-examples/ceph-examples/ENV.md | 54 ++++ .../storage-examples/ceph-examples/MYSQL.md | 124 +++++++++ .../storage-examples/ceph-examples/OSE.md | 180 +++++++++++++ .../storage-examples/ceph-examples/README.md | 27 ++ .../ceph-examples/ceph-secret.yaml | 6 + .../ceph-examples/mysql_ceph_host/README.md | 176 +++++++++++++ .../ceph-examples/mysql_ceph_host/mysql.yaml | 20 ++ .../ceph-examples/mysql_ceph_plugin/README.md | 236 ++++++++++++++++++ .../mysql_ceph_plugin/mysql-ceph-pod.yaml | 31 +++ .../ceph-examples/mysql_ceph_pvc/README.md | 182 ++++++++++++++ .../mysql_ceph_pvc/ceph-claim.yaml | 10 + .../mysql_ceph_pvc/ceph-mysql-pvc-pod.yaml | 25 ++ .../ceph-examples/mysql_ceph_pvc/ceph-pv.yaml | 20 ++ .../mysql_ceph_template/README.md | 203 +++++++++++++++ .../ceph-mysql-template.yaml | 43 ++++ 17 files changed, 1348 insertions(+) create mode 100644 examples/storage-examples/README.md create mode 100644 examples/storage-examples/ceph-examples/ENV.md create mode 100644 examples/storage-examples/ceph-examples/MYSQL.md create mode 100644 examples/storage-examples/ceph-examples/OSE.md create mode 100644 examples/storage-examples/ceph-examples/README.md create mode 100644 examples/storage-examples/ceph-examples/ceph-secret.yaml create mode 100644 examples/storage-examples/ceph-examples/mysql_ceph_host/README.md create mode 100644 examples/storage-examples/ceph-examples/mysql_ceph_host/mysql.yaml create mode 100644 examples/storage-examples/ceph-examples/mysql_ceph_plugin/README.md create mode 100644 examples/storage-examples/ceph-examples/mysql_ceph_plugin/mysql-ceph-pod.yaml create mode 100644 examples/storage-examples/ceph-examples/mysql_ceph_pvc/README.md create mode 100644 examples/storage-examples/ceph-examples/mysql_ceph_pvc/ceph-claim.yaml create mode 100644 examples/storage-examples/ceph-examples/mysql_ceph_pvc/ceph-mysql-pvc-pod.yaml create mode 100644 examples/storage-examples/ceph-examples/mysql_ceph_pvc/ceph-pv.yaml create mode 100644 examples/storage-examples/ceph-examples/mysql_ceph_template/README.md create mode 100644 examples/storage-examples/ceph-examples/mysql_ceph_template/ceph-mysql-template.yaml diff --git a/examples/README.md b/examples/README.md index 24e2bafee347..6a0fb508f4e1 100644 --- a/examples/README.md +++ b/examples/README.md @@ -11,6 +11,7 @@ available on top of Kubernetes and Docker. * [Node.js echo Sample](https://github.com/openshift/nodejs-ex) highlights the simple workflow from creating project, new app from GitHub, building, deploying, running and updating. * [Project Quotas and Resource Limits](./project-quota) demonstrates how quota and resource limits can be applied to resources in an OpenShift project. * [Replicated Zookeper Template](./zookeeper) provides a template for an OpenShift service that exposes a simple set of primitives that distributed applications can build upon to implement higher level services for synchronization, configuration maintenance, and groups and naming. +* [Storage Examples](./storage-examples) provides a high level tutorial and templates for local and persistent storage on OpenShift using simple nginx applications. * [Database Templates](./db-templates) provide templates for ephemeral and persistent storage on OpenShift using MongoDB, MySQL, and PostgreSQL. * [Clustered Etcd Template](./etcd) provides a template for setting up a clustered instance of the [Etcd](https://github.com/coreos/etcd) key-value store as a service on OpenShift. * [Configurable Git Server](./gitserver) sets up a serivce capable of automatic mirroring of Git repositories, intended for use within a container or Kubernetes pod. diff --git a/examples/storage-examples/README.md b/examples/storage-examples/README.md new file mode 100644 index 000000000000..ee4ccdf60520 --- /dev/null +++ b/examples/storage-examples/README.md @@ -0,0 +1,10 @@ +# OpenShift Container Storage Examples [WIP] + +OpenShift Applications/Containers/Pods have the ability to use Persistent Local and Distributed Storage, below are some examples that will explore some of these scenarios: + +* [Local](./local-storage-examples) +* [GlusterFS Storage Examples](./gluster-examples) +* [Ceph Storage Examples](./ceph-examples) +* TBD - Cinder +* + diff --git a/examples/storage-examples/ceph-examples/ENV.md b/examples/storage-examples/ceph-examples/ENV.md new file mode 100644 index 000000000000..4b8c49cd94c5 --- /dev/null +++ b/examples/storage-examples/ceph-examples/ENV.md @@ -0,0 +1,54 @@ +OSE Environment +=============== + +* 2 RHEL-7 VMs for running the openshift enterprise (OSE) master and node hosts +* A working ceph cluster, which can be a bare metal cluster, one or more VMs, one or more containers, or a very simple all-in-one container. + +The RHEL-7 hosts running the OSE master and OSE nodes should have the following services enabled and running: +* selinux (*setenforce 1*) +* iptables (*systemctl start iptables*) +* firewalld (*systemctl start firewalld*) Note, if you cannot start firewalld due to the service being masked, you can do a *systemctl unmask firewalld* and then restart it +* all OSE nodes (master and workers) and the ceph host need to be running docker. Currently docker version 1.8 has storage setup issues, so see below on how to upgrade or downgrade the version of docker on your VMs. + +### Docker: +Docker may be on the OS running OSE, but make sure the docker version is 1.6 or 1.7 -- not 1.8. + +``` +$ docker --version +Docker version 1.6.0, build 350a636/1.6.0 +``` + +#### Downgrading Docker: +There seems to be a docker 1.8 problem with storage setup where the docker-metapool is created too small. So, on docker version 1.8 consider downgrading to 1.7 or 1.6. + +``` +$ yum --showduplicates list | grep ^docker + +#if you see docker 1.6 or 1.7 then... +$ yum install -y docker-1.6 #or docker-1.7 +``` + +If the above docker downgrade fails, reporting "Error: Nothing to do", then first attempt a yum clean: + +``` +$ yum clean all +# redo yum install from above... +``` + +If the downgrade still fails then the docker target version rpm can be downloaded directly from docker.com. As of the time of this writing this link worked: + +``` +https://get.docker.com/rpm/1.7.1/fedora-21/RPMS/x86_64/docker-engine-1.7.1-1.fc21.x86_64.rpm +``` + +#### Upgrading Docker: +If docker is lower than 1.6 then: + +``` +yum install -y docker-1.6 #or docker-1.7 +``` + + +### Other Installations: +1. [OSE installation](OSE.md) +2. [MySQL installation](MYSQL.md) diff --git a/examples/storage-examples/ceph-examples/MYSQL.md b/examples/storage-examples/ceph-examples/MYSQL.md new file mode 100644 index 000000000000..1fe9598d11f0 --- /dev/null +++ b/examples/storage-examples/ceph-examples/MYSQL.md @@ -0,0 +1,124 @@ +Setting Up and Validating Containerized MySQL +============================================= + +First, OSE's Security Context Constraints (SSC) need to be defined such that the *seLinuxContext* and *runAsUser* values are set to "RunAsAny". Selinux is still enabled/enforcing on the OSE-master and OSE-nodes, as decribed in the [OSE environment readme](ENV.md). + +The mysql example uses the official mysql image found [here](https://hub.docker.com/_/mysql/). First, pull down the image on *each* OSE-nodes: + +``` +#on *each* OSE-node: +$ docker pull mysql +``` + +Next, test to ensure that mysql can be run in a container: + +``` +$ docker run --name mysql -e MYSQL_ROOT_PASSWORD=foopass -d mysql +``` + +Note: to re-run the above first remove the archived container so that the container name, "mysql", can be reused: + +``` +$ docker ps -a +$ docker stop +$ docker rm + +#NOTE: to remove all containers: +$ docker rm $(docker ps -a) +``` + +Shell into the mysql container, run mysql -p and create a simple "us_states" database: + +``` +$ docker exec -it bash +bash# mysql -p +mysql> show databases; ++---------------------+ +| Database | ++---------------------+ +| information_schema | +| #mysql50#lost+found | +| mysql | +| performance_schema | ++---------------------+ +4 rows in set (0.12 sec) + +# create a simple database for us states: +mysql> CREATE DATABASE us_states; +Query OK, 1 row affected (0.03 sec) + +mysql> USE us_states; +Database changed + +mysql> CREATE TABLE states (id INT NOT NULL PRIMARY KEY AUTO_INCREMENT, state CHAR(25), population INT(9)); +Query OK, 0 rows affected (1.93 sec) + +mysql> INSERT INTO states (id, state, population) VALUES (NULL, 'Alabama', '4822023'); +Query OK, 1 row affected (0.17 sec) + +mysql> SELECT * FROM states; ++----+---------+------------+ +| id | state | population | ++----+---------+------------+ +| 1 | Alabama | 4822023 | ++----+---------+------------+ +1 row in set (0.00 sec) + +mysql> quit +bash# exit +``` + +Note: *-p* causes mysql to prompt for the user's password, even if the password has been provided via the MYSQL_ROOT_PASSWORD env variable. In some cases, as seen here, *-p* is required for the mysql command to succeed. Omitting *-p* (even when the root password is specified in the environment), can cause the error below: + +``` +#On the host running the mysql container we see the password is provided: +$ docker inspect 9501299ac215 | grep PASSWORD + "MYSQL_ROOT_PASSWORD=foopass", + +#inside the mysql container we get an error unless we use -p and supply the password: +root@mysql:/# mysql #no -p +ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO) +``` + +Delete the mysql container so that it can be re-created from a pod later: + +``` +$ docker stop +$ docker rm +``` + +### Security: +When using the ceph-rbd storage plugin, if the mysql container is not privileged then it will fail with this error: + +``` +chown: cannot read directory `/var/lib/mysql/': Permission denied +``` + +The above error is found by looking at the docker logs on the target OSE-node: + +``` +$ docker ps -a #need -a since the container start fails +$ docker logs +``` + +The "quick" solution is to set the mysql pod to be privileged, which is done with this yaml fragment: + +``` +... +spec: + containers: + - image: mysql + ... + securityContext: + capabilities: {} + privileged: true +... +``` + +Note that selinux is still enabled/enforcing. + +``` +$ getenforce +Enforcing +``` + diff --git a/examples/storage-examples/ceph-examples/OSE.md b/examples/storage-examples/ceph-examples/OSE.md new file mode 100644 index 000000000000..2f0a06db1678 --- /dev/null +++ b/examples/storage-examples/ceph-examples/OSE.md @@ -0,0 +1,180 @@ +Setup a Simple OpenShift Enterprise (OSE) Cluster +================================================= + +### Environment: +The enviromnent used for all of the ceph examples is described [here](ENV.md). It is assumed that ceph is already up and running, either on bare metal, in a VM, or containerized. + +### Setting up OSE: +The following OpenShift/origin guides should be followed when setting up OSE: +* https://docs.openshift.com/enterprise/3.0/admin_guide/install/prerequisites.html -- Prerequisites +* https://docs.openshift.org/latest/getting_started/administrators.html#running-in-a-docker-container -- Getting Started for Administrator +* https://docs.openshift.com/enterprise/3.0/admin_guide/install/quick_install.html -- **Quick Installation** +* https://docs.openshift.com/enterprise/3.0/admin_guide/configuring_authentication.html -- Configuring Authentication +* https://docs.openshift.com/enterprise/3.0/admin_guide/install/advanced_install.html -- Advanced Installation + +The examples in here use "Method 1: Running the Installation Utility From the Internet" described in the [Quick Installation](https://docs.openshift.com/enterprise/3.0/admin_guide/install/quick_install.html) Guide above. + +The following checks can be made to ensure that OSE is installed correctly and is running: +* is the OSE master server is running? On the OSE-master host, execute *systemctl status openshift-master*. Use *systemctl restart openshift-master* to start the master services on the OSE-master. +* are all OSE worker-nodes are running? On each OSE-node execute *systemctl status openshift-node* and, to restart the services, use *systemctl restart openshift-node*, done on each OSE-node. +* is the OSE Web Console accessible? Login to the OpenShift Console via the GUI at https://ose-master-host-name-or-ip:8443/console +* is the *oc* command available? On the OSE-master server login to OSE via the command line using *oc login -u admin*: +``` +$ oc login -u admin +Password: +Using project "default" + +$ oc get projects +NAME DISPLAY NAME STATUS +default Active +openshift Active +openshift-infra Active +``` + +### Ceph on each OSE-node: +Each schedulable OSE-node needs the ceph-common library installed, and for now, due to a current ceph packaging bug, also needs full ceph installed. + +Note: in order to install the full ceph, each OSE node may need certain ceph repos enabled. + +``` +$ subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-rhceph-1.3-installer-rpms \ + --enable=rhel-7-server-rhceph-1.3-mon-rpms --enable=rhel-7-server-rhceph-1.3-osd-rpms +``` + +Now install ceph on schedulable OSE-node: + +``` +$ yum install -y ceph-common + +#and due to the ceph packaging bug where ceph-rbdnamed() is missing +$ yum install -y ceph +``` + +### Ceph Secret: +The ceph-rbd storage plugin uses a ceph secret for authorization. This is a short yaml file which resides on the OSE-master host but gets its value from the ceph monitor host. + +``` +#on a ceph monitor server: +$ ceph auth get-key client.admin +AQDva7JVEuVJBBAAc8e1ZBWhqUB9K/zNZdOHoQ== + +$ echo "AQDva7JVEuVJBBAAc8e1ZBWhqUB9K/zNZdOHoQ=="| base64 +QVFEdmE3SlZFdVZKQkJBQWM4ZTFaQldocVVCOUsvek5aZE9Ib1E9PQo= +# copy the above output +``` + +Back on the OSE-master node, edit the [ceph-secret file](ceph-secret.yaml) pasting the base64 value above. Then *oc create* the ceph-secret object: + +``` +$ oc create -f ceph-secret.yaml +secrets/ceph-secret + +$ oc get secret +NAME TYPE DATA +ceph-secret Opaque 1 +... +``` + +### Security: +OSE Security Context Constraints (SCC) are described in this [OSE Authorization Guide](https://docs.openshift.com/enterprise/3.0/architecture/additional_concepts/authorization.html#security-context-constraints). The *privileged* and *restricted* SCCs are added as defaults by OSE and need to be modified in order for mysql and ceph to have sufficient privileges. See also [General OSE Authorization](https://docs.openshift.com/enterprise/3.0/architecture/additional_concepts/authorization.html). + +After logging-in to OSE as the *admin* user, edit the two SCC files, as shown below: + +``` +$ oc login -u admin +$ oc edit scc privilege +$ oc edit scc restricted +#change "MustRunAsRange" to "RunAsAny" + +$ oc get scc +NAME PRIV CAPS HOSTDIR SELINUX RUNASUSER +privileged true [] true RunAsAny RunAsAny +restricted false [] false RunAsAny RunAsAny +``` + +### MySQL: +For each OSE-node see the [setting up mysql doc](MYSQL.md). + +### Verification/Validation: + +On the OSE master host: + +``` +$ systemctl status openshift-master -l +openshift-master.service - OpenShift Master + Loaded: loaded (/usr/lib/systemd/system/openshift-master.service; enabled) + Active: active (running) since Thu 2015-09-03 16:35:56 EDT; 8h ago + Docs: https://github.com/openshift/origin + Main PID: 49702 (openshift) + CGroup: /system.slice/openshift-master.service + └─49702 /usr/bin/openshift start master --config=/etc/openshift/master/master-config.yaml --loglevel=4 +... +DeploymentConfigs for trigger on ImageStream openshift/mysql +Sep 04 00:46:02 rhel7-ose-1 openshift-master[49702]: I0904 00:46:02.803183 49702 controller.go:38] Detecting changed images for DeploymentConfig default/docker-registry:2 +... +Sep 04 00:46:03 rhel7-ose-1 openshift-master[49702]: I0904 00:46:03.001061 49702 controller.go:85] Ignoring DeploymentConfig change for default/docker-registry:2 (latestVersion=2); same as Deployment default/docker-registry-2 +``` + +And, still on the OSE-master: + +``` +$ oc get nodes +NAME LABELS STATUS +192.168.122.179 kubernetes.io/hostname=192.168.122.179 Ready,SchedulingDisabled +192.168.122.254 kubernetes.io/hostname=192.168.122.254 Ready + +``` + +On *each* OSE schedulable node: + +``` +$ rpm -qa|grep ceph +ceph-common-0.94.1-16.el7cp.x86_64 +ceph-0.94.1-16.el7cp.x86_64 +``` + +``` +$ systemctl status openshift-node -l +openshift-node.service - OpenShift Node + Loaded: loaded (/usr/lib/systemd/system/openshift-node.service; enabled) + Drop-In: /usr/lib/systemd/system/openshift-node.service.d + └─openshift-sdn-ovs.conf + Active: active (running) since Tue 2015-09-01 18:58:27 EDT; 2 days ago + Docs: https://github.com/openshift/origin + Main PID: 94526 (openshift) + CGroup: /system.slice/openshift-node.service + └─94526 /usr/bin/openshift start node --config=/etc/openshift/node/node-config.yaml --loglevel=4 + + +Sep 04 00:48:09 rhel7-ose-2 openshift-node[94526]: I0904 00:48:09.198143 94526 manager.go:1388] Pod infra container looks good, keep it "mysql_default" +Sep 04 00:48:09 rhel7-ose-2 openshift-node[94526]: I0904 00:48:09.198195 94526 manager.go:1411] pod "mysql_default" container "mysql" exists as 77f4af567e3dd3b10656ad5ee38a39600174a87c519d6a23735e96cf0ee4208a +Sep 04 00:48:09 rhel7-ose-2 openshift-node[94526]: I0904 00:48:09.198214 94526 prober.go:180] Readiness probe for "mysql_default:mysql" succeeded +Sep 04 00:48:09 rhel7-ose-2 openshift-node[94526]: I0904 00:48:09.198224 94526 manager.go:1442] probe success: "mysql" +Sep 04 00:48:09 rhel7-ose-2 openshift-node[94526]: I0904 00:48:09.198239 94526 manager.go:1515] Got container changes for pod "mysql_default": {StartInfraContainer:false InfraContainerId:dca749fa3530d552a643a836051d4b00ef4e3a69c69ebc8ede059848b3b27569 ContainersToStart:map[] ContainersToKeep:map[dca749fa3530d552a643a836051d4b00ef4e3a69c69ebc8ede059848b3b27569:-1 77f4af567e3dd3b10656ad5ee38a39600174a87c519d6a23735e96cf0ee4208a:0]} +Sep 04 00:48:09 rhel7-ose-2 openshift-node[94526]: I0904 00:48:09.198268 94526 kubelet.go:2245] Generating status for "mysql_default" +... +Sep 04 00:48:09 rhel7-ose-2 openshift-node[94526]: I0904 00:48:09.202366 94526 status_manager.go:129] Ignoring same status for pod "mysql_default", status: {Phase:Running Conditions:[{Type:Ready Status:True}] Message: Reason: HostIP:192.168.122.254 PodIP:10.1.0.41 StartTime:2015-09-03 19:10:07.598461546 -0400 EDT ContainerStatuses:[{Name:mysql State:{Waiting: Running:0xc213e6d0c0 Terminated:} LastTerminationState:{Waiting: Running: Terminated:} Ready:true RestartCount:0 Image:mysql ImageID:docker://7eee2d462c8f6ffacfb908cc930559e21778f60afdb2d7e9cf0f3025274d7ea8 ContainerID:docker://77f4af567e3dd3b10656ad5ee38a39600174a87c519d6a23735e96cf0ee4208a}]} +``` + +And some docker checks on *each* OSE node: + +``` +$ docker ps #make sure docker is running + +#if docker is not running then start it: +$ systemctl start docker +$ systemctl enable docker +$ systemctl status docker -l +``` + +### Log Files +*journalctl* and *systemctl status* are the main ways to view OSE log files. The *systemctl status* command is shown above. Here are some *journalctl* examples: + +``` +$ journalctl -xe -u openshift-master + +#and on the OSE node: +$ journalctl -xe -u openshift-node +``` + +It's often necessary to scroll right (right arrow) to see pertinent info. diff --git a/examples/storage-examples/ceph-examples/README.md b/examples/storage-examples/ceph-examples/README.md new file mode 100644 index 000000000000..a57927623e4d --- /dev/null +++ b/examples/storage-examples/ceph-examples/README.md @@ -0,0 +1,27 @@ +MySQL + Ceph Persistent Volume +============================== + +Here are examples showing how to run MySQL in the following environments: +1. in a container launched directly from docker (see the [mysql readme](MYSQL.md)) +2. via an openshift/kubernetes pod (examples [2](mysql_ceph_host) and [3](mysql_ceph_pvc)) +3. via an openshift application template (example [4](mysql_ceph_template)). + +Both [local OSE-node storage](mysql_ceph_host) and [ceph-rbd block storage](mysql_ceph_plugin) under an ext4 file system to persist the database. + +The next few sections are common across almost all of the examples and are also shown below: + +### Environment: +The basic enviromnent used for all of the examples is described [here](ENV.md). It is assumed that ceph is already up and running, either on bare metal, in a VM, or containerized. + +### Setting up Openshift Enterprise (OSE): +The steps needed to setup a simple OSE cluster with 1 master and 1 worker node are described [here](OSE.md). + +### Setting up MySQL: +Follow the instructions [here](MYSQL.md) to initialize and validate containerized mysql. + +### Specific Examples: +1. [mysql + local/host storage](mysql_ceph_host) - mysql database lives on the OSE host where the pod is scheduled +2. [mysql + ceph plugin](mysql_ceph_plugin) - mysql database resides in ceph, a rbd plugin is specfied +3. [mysql + ceph + pvc](mysql_ceph_pvc) - mysql database resides in ceph, a Persistent Volume (PV) and Persistent Volume Claim (PVC) are used +4. [mysql + ceph + template](mysql_ceph_template) -- same as the above example except the pod and pvc are defined in a single template file + diff --git a/examples/storage-examples/ceph-examples/ceph-secret.yaml b/examples/storage-examples/ceph-examples/ceph-secret.yaml new file mode 100644 index 000000000000..5052e366f0e6 --- /dev/null +++ b/examples/storage-examples/ceph-examples/ceph-secret.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: Secret +metadata: + name: ceph-secret +data: + key: QVFEdmE3SlZFdVZKQkJBQWM4ZTFaQldocVVCOUsvek5aZE9Ib1E9PQo= diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_host/README.md b/examples/storage-examples/ceph-examples/mysql_ceph_host/README.md new file mode 100644 index 000000000000..b3025fd9ca19 --- /dev/null +++ b/examples/storage-examples/ceph-examples/mysql_ceph_host/README.md @@ -0,0 +1,176 @@ +Example 1: Using Local (hostPath) Storage +========================================= + +This example uses the local storage directly on the OSE host where the mysql container is running. Make sure that any mysql pods created in other examples have been deleted before continuing with this example: + +``` +#on the OSE-master: +$ oc get pods + +#if you see a mysql pod above: +$ oc delete pod +``` + +### Environment: +If the steps to install the environment, OSE, and mysql have not already been completed, then follow the instuctions linked-to directly below: + +The enviromnent used for all of the examples here is described [here](../ENV.md). + +### Setting up Openshift Enterprise (OSE): +The steps needed to setup a simple OSE cluster with 1 master and 1 worker node are described [here](../OSE.md). + +### Setting up MySQL: +Follow the instructions [here](../MYSQL.md) to initialize and validate containerized mysql. + +### Mysql Pod Spec File: +The [pod spec](mysql.yaml) uses a mysql image, defines the password as an environment variable, and maps the container's volume (/var/lib/mysql) to the target OSE-node's volume (/opt/mysql), where the database resides. Since selinux is enabled/enforcing, before the pod can be created, the local directory (/opt/mysql) on *each* schedulable OSE-node needs a selinux context label set to permit mysql to access the directory (via docker's bind mount): + +``` +#on *each* OSE-node: +$ getenforce +$ setenforce 1 #if selinux is permissive + +$ mkdir -p /opt/mysql + +$ chcon -Rt svirt_sandbox_file_t /opt/mysql +$ ls -Zd /opt/mysql/ +drwxr-xr-x. polkitd ssh_keys system_u:object_r:svirt_sandbox_file_t:s0 /opt/mysql/ +``` + +On the OSE-master, create the mysql pod via *oc create*: + +``` +$ oc create -f mysql.yaml +pods/mysql + +$ oc get pod +NAME READY STATUS RESTARTS AGE +mysql 1/1 Running 0 18s +``` + +To see which OSE host the mysql pod has been scheduled on: + +``` +#on the OSE-master: +$ oc describe pod mysql +NAME READY STATUS RESTARTS AGE +docker-registry-2-223nv 0/1 Image: registry.access.redhat.com/openshift3/ose-docker-registry:v3.0.1.0 is not ready on the node 0 3d +mysql 1/1 Running 0 18s +[root@rhel7-ose-1 ceph]# oc describe pod mysql +Name: mysql +Namespace: default +Image(s): mysql +Node: 192.168.122.254/192.168.122.254 ## <--- the hostname is often shown in addition to ip +Labels: name=mysql +Status: Running +Reason: +Message: +IP: 10.1.0.41 +Replication Controllers: +Containers: + mysql: + Image: mysql + State: Running + Started: Thu, 03 Sep 2015 19:10:15 -0400 + Ready: True + Restart Count: 0 +Conditions: + Type Status + Ready True +Events: + FirstSeen LastSeen Count From SubobjectPath Reason Message + Thu, 03 Sep 2015 19:10:07 -0400 Thu, 03 Sep 2015 19:10:07 -0400 1 {scheduler } scheduled Successfully assigned mysql to 192.168.122.254 +... + Thu, 03 Sep 2015 19:10:15 -0400 Thu, 03 Sep 2015 19:10:15 -0400 1 {kubelet 192.168.122.254} spec.containers{mysql} startedStarted with docker id 77f4af567e3d +``` + +On the target (scheduled) OSE host, run docker to get information about the mysql container: + +``` +#on the target OSE-node: +$ docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +77f4af567e3d mysql "/entrypoint.sh mysq 5 minutes ago Up 5 minutes k8s_mysql.4977675e_mysql_default_ea9b64de-5290-11e5-b56b-52540039f12e_2257d0b5 +dca749fa3530 openshift3/ose-pod:v3.0.1.0 "/pod" 5 minutes ago Up 5 minutes k8s_POD.892ec37e_mysql_default_ea9b64de-5290-11e5-b56b-52540039f12e_aa534a81 + +$ docker inspect mysql +[{ + "Id": "7eee2d462c8f6ffacfb908cc930559e21778f60afdb2d7e9cf0f3025274d7ea8", +... + "ContainerConfig": { + "Hostname": "c6ebf900c860", +... + "ExposedPorts": { + "3306/tcp": {} + }, +... + "Env": [ + "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "MYSQL_MAJOR=5.6", + "MYSQL_VERSION=5.6.26" + ], + "Cmd": [ + "/bin/sh", + "-c", + "#(nop) CMD [\"mysqld\"]" + ], + "Image": "15a3cddfc178c4dbaa8f56142d4eebef6d22a3cd1842820844cf815992fe5a13", + "Volumes": { + "/var/lib/mysql": {} + }, + "VolumeDriver": "", + "WorkingDir": "", + "Entrypoint": [ + "/entrypoint.sh" + ], +... + }, + "DockerVersion": "1.7.1", +... + "Env": [ + "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "MYSQL_MAJOR=5.6", + "MYSQL_VERSION=5.6.26" + ], + "Cmd": [ + "mysqld" + ], + "Image": "15a3cddfc178c4dbaa8f56142d4eebef6d22a3cd1842820844cf815992fe5a13", + "Volumes": { + "/var/lib/mysql": {} + }, + "VolumeDriver": "", + "WorkingDir": "", + "Entrypoint": [ + "/entrypoint.sh" + ], +... + }, +... +}] + +$ docker logs 77f4af567e3d # <--- container ID +2015-09-03 23:10:17 0 [Note] mysqld (mysqld 5.6.26) starting as process 1 ... +... +2015-09-03 23:10:18 1 [Note] InnoDB: 5.6.26 started; log sequence number 1626017 +2015-09-03 23:10:18 1 [Note] Server hostname (bind-address): '*'; port: 3306 +... +2015-09-03 23:10:18 1 [Note] mysqld: ready for connections. +Version: '5.6.26' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL) +``` + +Finally, on the same OSE-node, run mysql inside the container: + +``` +$ docker exec -it 77f4af567e3d bash # <--- container ID again +root@mysql:/# mysql -p +Enter password: +Welcome to the MySQL monitor. Commands end with ; or \g. +... +Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. + +mysql> quit +Bye +root@mysql:/# exit +exit +``` diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_host/mysql.yaml b/examples/storage-examples/ceph-examples/mysql_ceph_host/mysql.yaml new file mode 100644 index 000000000000..98128b5ddd6c --- /dev/null +++ b/examples/storage-examples/ceph-examples/mysql_ceph_host/mysql.yaml @@ -0,0 +1,20 @@ +apiVersion: v1 +kind: Pod +metadata: + name: mysql + labels: + name: mysql +spec: + containers: + - image: mysql + name: mysql + volumeMounts: + - name: varlibmysql + mountPath: /var/lib/mysql + env: + - name: MYSQL_ROOT_PASSWORD + value: foopass + volumes: + - name: varlibmysql + hostPath: + path: /opt/mysql diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_plugin/README.md b/examples/storage-examples/ceph-examples/mysql_ceph_plugin/README.md new file mode 100644 index 000000000000..1607b9e487ca --- /dev/null +++ b/examples/storage-examples/ceph-examples/mysql_ceph_plugin/README.md @@ -0,0 +1,236 @@ +Example 2: Using the Ceph-RBD Plugin +==================================== + +This example uses the ceph/rbd plugin directly in the pod spec. Make sure that any mysql pods created in other examples have been deleted before continuing with this example: + +``` +#on the OSE-master: +$ oc get pods + +#if you see a mysql pod above: +$ oc delete pod +``` + +### Environment: +If the steps to install the environment, ceph, ose, and mysql have not already been completed, then follow the instuctions linked-to directly below: + +The enviromnent used for all of the examples is described [here](../ENV.md). It is assumed that ceph is already up and running, either on bare metal, in a VM, or containerized. + +### Setting up Openshift Enterprise (OSE): +The steps needed to setup a simple OSE cluster with 1 master and 1 worker node are described [here](../OSE.md). + +### Setting up MySQL: +Follow the instructions [here](../MYSQL.md) to initialize and validate containerized mysql. + +### Defining the Pod Spec File: +This example defines a simple [pod spec](mysql-ceph-pod.yaml) using the same mysql image, a container volume mount which references the ceph-rbd persistent storage plugin, and the ceph monitor's ip address. + +The pod created by this spec needs to be **privileged**, as seen in the container's "securityContext" section. This is necessary for the rbd mount to succeed. If the container is not privileged and if the OSE-node has selinux set to *enforcing* then this error will be seen: + +``` +#on the target OSE-node: +$ docker logs 45f023ef10ce #mysql container-ID +chown: cannot read directory `/var/lib/mysql/': Permission denied +``` + +By setting selinux to permissive on *each* OSE-node the above error is also aboided. However, running the mysql container privileged is likely preferred. + +On the OSE-master, create the mysql pod: + +``` +#on the OSE-master: +$ oc create -f mysql-ceph-pod.yaml +pods/mysql + +$ oc get pod +NAME READY STATUS RESTARTS AGE +mysql 1/1 Running 0 18s +``` + +To see volume information for this pod: + +``` +#on the OSE-master: +$ oc volume pod mysql --list +# pods mysql, volumes: +mysql-ceph +default-token-sqef4 + # container mysql, volume mounts: + mysql-ceph /var/lib/mysql + default-token-sqef4 /var/run/secrets/kubernetes.io/serviceaccount + +``` +"mysql-ceph" is the volume name defined in the pod spec. + +To see which OSE host the mysql pod has been scheduled on: + +``` +#on the OSE-master: +$ oc describe pod mysql +Name: mysql +Namespace: default +Image(s): mysql +Node: 192.168.122.254/192.168.122.254 ## <--- the hostname is often shown here +Labels: name=mysql +Status: Running +Reason: +Message: +IP: 10.1.0.41 +Replication Controllers: +Containers: + mysql: + Image: mysql + State: Running + Started: Thu, 03 Sep 2015 19:10:15 -0400 + Ready: True + Restart Count: 0 +Conditions: + Type Status + Ready True +Events: + FirstSeen LastSeen Count From SubobjectPath Reason Message + Thu, 03 Sep 2015 19:10:07 -0400 Thu, 03 Sep 2015 19:10:07 -0400 1 {scheduler } scheduled Successfully assigned mysql to 192.168.122.254 +... + Thu, 03 Sep 2015 19:10:15 -0400 Thu, 03 Sep 2015 19:10:15 -0400 1 {kubelet 192.168.122.254} spec.containers{mysql} startedStarted with docker id 77f4af567e3d +``` + +On the target OSE-node use docker to get information about the mysql container: + +``` +$ docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +77f4af567e3d mysql "/entrypoint.sh mysq 5 minutes ago Up 5 minutes k8s_mysql.4977675e_mysql_default_ea9b64de-5290-11e5-b56b-52540039f12e_2257d0b5 +dca749fa3530 openshift3/ose-pod:v3.0.1.0 "/pod" 5 minutes ago Up 5 minutes k8s_POD.892ec37e_mysql_default_ea9b64de-5290-11e5-b56b-52540039f12e_aa534a81 + +$ docker inspect mysql +[{ + "Id": "7eee2d462c8f6ffacfb908cc930559e21778f60afdb2d7e9cf0f3025274d7ea8", + "Parent": "15a3cddfc178c4dbaa8f56142d4eebef6d22a3cd1842820844cf815992fe5a13", + "Comment": "", + "Created": "2015-08-24T21:55:13.704277966Z", + "Container": "113d16e2420bb0f5e17a74f5f8c85d70572efe1720451da83e0a84fe3fcd04fd", + "ContainerConfig": { + "Hostname": "c6ebf900c860", + "Domainname": "", + "User": "", + "AttachStdin": false, + "AttachStdout": false, + "AttachStderr": false, + "PortSpecs": null, + "ExposedPorts": { + "3306/tcp": {} + }, + "Tty": false, + "OpenStdin": false, + "StdinOnce": false, + "Env": [ + "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "MYSQL_MAJOR=5.6", + "MYSQL_VERSION=5.6.26" + ], + "Cmd": [ + "/bin/sh", + "-c", + "#(nop) CMD [\"mysqld\"]" + ], + "Image": "15a3cddfc178c4dbaa8f56142d4eebef6d22a3cd1842820844cf815992fe5a13", + "Volumes": { + "/var/lib/mysql": {} + }, + "VolumeDriver": "", + "WorkingDir": "", + "Entrypoint": [ + "/entrypoint.sh" + ], + "NetworkDisabled": false, + "MacAddress": "", + "OnBuild": [], + "Labels": {}, + "Init": "" + }, + "DockerVersion": "1.7.1", + "Author": "", + "Config": { + "Hostname": "c6ebf900c860", + "Domainname": "", + "User": "", + "AttachStdin": false, + "AttachStdout": false, + "AttachStderr": false, + "PortSpecs": null, + "ExposedPorts": { + "3306/tcp": {} + }, + "Tty": false, + "OpenStdin": false, + "StdinOnce": false, + "Env": [ + "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + "MYSQL_MAJOR=5.6", + "MYSQL_VERSION=5.6.26" + ], + "Cmd": [ + "mysqld" + ], + "Image": "15a3cddfc178c4dbaa8f56142d4eebef6d22a3cd1842820844cf815992fe5a13", + "Volumes": { + "/var/lib/mysql": {} + }, + "VolumeDriver": "", + "WorkingDir": "", + "Entrypoint": [ + "/entrypoint.sh" + ], + "NetworkDisabled": false, + "MacAddress": "", + "OnBuild": [], + "Labels": {}, + "Init": "" + }, + "Architecture": "amd64", + "Os": "linux", + "Size": 0, + "VirtualSize": 283575255 +}] + +$ docker logs 77f4af567e3d # <--- container ID +2015-09-03 23:10:17 0 [Note] mysqld (mysqld 5.6.26) starting as process 1 ... +2015-09-03 23:10:17 1 [Note] Plugin 'FEDERATED' is disabled. +2015-09-03 23:10:17 1 [Note] InnoDB: Using atomics to ref count buffer pool pages +2015-09-03 23:10:17 1 [Note] InnoDB: The InnoDB memory heap is disabled +2015-09-03 23:10:17 1 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins +2015-09-03 23:10:17 1 [Note] InnoDB: Memory barrier is not used +2015-09-03 23:10:17 1 [Note] InnoDB: Compressed tables use zlib 1.2.7 +2015-09-03 23:10:17 1 [Note] InnoDB: Using Linux native AIO +2015-09-03 23:10:17 1 [Note] InnoDB: Using CPU crc32 instructions +2015-09-03 23:10:17 1 [Note] InnoDB: Initializing buffer pool, size = 128.0M +2015-09-03 23:10:17 1 [Note] InnoDB: Completed initialization of buffer pool +2015-09-03 23:10:17 1 [Note] InnoDB: Highest supported file format is Barracuda. +2015-09-03 23:10:18 1 [Note] InnoDB: 128 rollback segment(s) are active. +2015-09-03 23:10:18 1 [Note] InnoDB: Waiting for purge to start +2015-09-03 23:10:18 1 [Note] InnoDB: 5.6.26 started; log sequence number 1626017 +2015-09-03 23:10:18 1 [Note] Server hostname (bind-address): '*'; port: 3306 +2015-09-03 23:10:18 1 [Note] IPv6 is available. +2015-09-03 23:10:18 1 [Note] - '::' resolves to '::'; +2015-09-03 23:10:18 1 [Note] Server socket created on IP: '::'. +2015-09-03 23:10:18 1 [Warning] 'proxies_priv' entry '@ root@mysql' ignored in --skip-name-resolve mode. +2015-09-03 23:10:18 1 [Note] Event Scheduler: Loaded 0 events +2015-09-03 23:10:18 1 [Note] mysqld: ready for connections. +Version: '5.6.26' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL) +``` + +Finally, on the same OSE-node, run mysql inside the container: + +``` +$ docker exec -it 77f4af567e3d bash # <--- mysql container ID +root@mysql:/# mysql -p +Enter password: +Welcome to the MySQL monitor. Commands end with ; or \g. +... +Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. + +mysql> quit +Bye +root@mysql:/# exit +exit +``` diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_plugin/mysql-ceph-pod.yaml b/examples/storage-examples/ceph-examples/mysql_ceph_plugin/mysql-ceph-pod.yaml new file mode 100644 index 000000000000..3521e237b755 --- /dev/null +++ b/examples/storage-examples/ceph-examples/mysql_ceph_plugin/mysql-ceph-pod.yaml @@ -0,0 +1,31 @@ +apiVersion: v1 +kind: Pod +metadata: + name: mysql + labels: + name: mysql +spec: + containers: + - image: mysql + name: mysql + volumeMounts: + - name: mysql-ceph + mountPath: /var/lib/mysql + env: + - name: MYSQL_ROOT_PASSWORD + value: foopass + securityContext: + capabilities: {} + privileged: true #privileged required for the rbd mount + volumes: + - name: mysql-ceph + rbd: + monitors: #ip addr of the ceph monitor + - 192.168.122.133:6789 + pool: rbd + image: foo + user: admin + secretRef: + name: ceph-secret + fsType: ext4 + readOnly: false diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_pvc/README.md b/examples/storage-examples/ceph-examples/mysql_ceph_pvc/README.md new file mode 100644 index 000000000000..76d71fa0620f --- /dev/null +++ b/examples/storage-examples/ceph-examples/mysql_ceph_pvc/README.md @@ -0,0 +1,182 @@ +Example 3: Using MySQL, Ceph Persistent Volume and Claim +======================================================== + +Here is an example of how to create the mysql application, using ceph rbd as the persistent store, where the ceph rbd is defined in a persistent volume (PV), and the pod uses a persistent volume claim (PVC), rather than defining the rbd volume inline. The kubernetetes PVClaimBinder matches the pod's claim against PVs and binds the claim to the PV that has the best match. Today, the matching criteria is very simple -- capacity and sharing attributes -- but, hopefully richer PV and PVC definitions will be coming in the future. See the [kubernetes persistent storage](https://github.com/kubernetes/kubernetes/blob/master/docs/design/persistent-storage.md) document for more information on PVs and PVCs. + +Make sure that any mysql pods created in other examples have been deleted before continuing with this example: + +``` +#on the OSE-master: +$ oc get pods + +#if you see a mysql pod above: +$ oc delete pod +``` + +### Environment: +If the steps to install the environment, ceph, ose and mysql have not already been completed, then follow the instuctions linked-to directly below: + +The enviromnent used for all of the examples is described [here](../ENV.md). It is assumed that ceph is already up and running, either on bare metal, in a VM, or containerized. + +### Setting up Openshift Enterprise (OSE): +The steps needed to seup a simple OSE cluster with 1 master and 1 worker node are described [here](../OSE.md). + +### Setting up MySQL: +Follow the instructions [here](../MYSQL.md) to initialize and validate containerized mysql. + +### Defining the PV and PVC Files: +A persistent volume is created from a file defining the name, capacity, and access methods for persistent storage. The PV spec used in this example is [here](ceph-pv.yaml), and the PVC is [here](ceph-claim.yaml). + +PVs are typically created by an OSE administrator, whereas PVCs will typically be created and requested by non-admins. The example here creates both the PV and claim separate from the pod. There is also a [template](../mysql_ceph_template) example which defines the PVC in the same file used to define the pod. + +### Creating the PV and PVC: +*oc create -f* is execute on the OSE-master to create almost all OSE objects and is used here to create the ceph PV and PVC. + +``` +#on the OSE-master: +$ oc create -f ceph-pv.yaml +persistentvolumes/ceph-pv + +$ oc get pv +NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON +ceph-pv 2147483648 RWX Available + +$ oc create -f ceph-claim.yaml +persistentvolumeclaims/ceph-claim + +$ oc get pvc +NAME LABELS STATUS VOLUME +ceph-claim map[] Bound ceph-pv +``` + +Notice that the claim has been bound to the "ceph-pv" persistent volume. + +### Creating the Pod: +The [pod spec](ceph-mysql-pvc-pod.yaml) references the same mysql image and defines the named claim to be used for persistent storage. As with [example 2](../mysql_ceph_plugin), the mysql container needs to run privileged. *oc create* is used to create the pod: + +``` +#on the OSE-master: +$ oc create -f ceph-mysql-pvc-pod.yaml +pods/ceph-mysql + +$ oc get pods +NAME READY STATUS RESTARTS AGE +ceph-mysql 1/1 Running +``` + +Volume information is also visible on the OSE-master: + +``` +#on the OSE-master: +$ oc volume pod ceph-mysql --list +# pods ceph-mysql, volumes: +mysql-pv +default-token-sqef4 + # container ceph-mysql, volume mounts: + mysql-pv /var/lib/mysql + default-token-sqef4 /var/run/secrets/kubernetes.io/serviceaccount +``` + +We execute *oc describe pod* to see which OSE host the pod is running on, and to see the pod's recent events: + +``` +#on the OSE-master: +$ oc describe pod ceph-mysql +Name: ceph-mysql +Namespace: default +Image(s): mysql +Node: 192.168.122.254/192.168.122.254 # <--- often both hostname and ip are shown here +Labels: +Status: Running +Reason: +Message: +IP: 10.1.0.43 +Replication Controllers: +Containers: + ceph-mysql: + Image: mysql + State: Running + Started: Fri, 04 Sep 2015 14:47:31 -0400 + Ready: True + Restart Count: 0 +Conditions: + Type Status + Ready True +Events: + FirstSeen LastSeen Count From SubobjectPath Reason Message + Fri, 04 Sep 2015 14:47:22 -0400 Fri, 04 Sep 2015 14:47:22 -0400 1 {scheduler } scheduled Successfully assigned ceph-mysql to 192.168.122.254 + Fri, 04 Sep 2015 14:47:24 -0400 Fri, 04 Sep 2015 14:47:24 -0400 1 {kubelet 192.168.122.254} implicitly required container POD pulled Pod container image "openshift3/ose-pod:v3.0.1.0" already present on machine + Fri, 04 Sep 2015 14:47:26 -0400 Fri, 04 Sep 2015 14:47:26 -0400 1 {kubelet 192.168.122.254} implicitly required container POD createdCreated with docker id acbee2db9a18 + Fri, 04 Sep 2015 14:47:27 -0400 Fri, 04 Sep 2015 14:47:27 -0400 1 {kubelet 192.168.122.254} implicitly required container POD startedStarted with docker id acbee2db9a18 + Fri, 04 Sep 2015 14:47:30 -0400 Fri, 04 Sep 2015 14:47:30 -0400 1 {kubelet 192.168.122.254} spec.containers{ceph-mysql} createdCreated with docker id 9a43017dbebf + Fri, 04 Sep 2015 14:47:31 -0400 Fri, 04 Sep 2015 14:47:31 -0400 1 {kubelet 192.168.122.254} spec.containers{ceph-mysql} startedStarted with docker id 9a43017dbebf +``` + +We see that the pod was scheduled on OSE host 192.168.122.254 (often there is a hostname visible too). On the target OSE node verify that the mysql container is running and that it's using the ceph-rbd volume: + +``` +#on the target/scheduled OSE-node: +$ docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +9a43017dbebf mysql "/entrypoint.sh mysq 4 minutes ago Up 4 minutes k8s_ceph-mysql.d8cb6e3e_ceph-mysql_default_608af6aa-5335-11e5-b56b-52540039f12e_b0163abc +acbee2db9a18 openshift3/ose-pod:v3.0.1.0 "/pod" 4 minutes ago Up 4 minutes k8s_POD.dbbbe7c7_ceph-mysql_default_608af6aa-5335-11e5-b56b-52540039f12e_d030f08f +``` + +The mysql container ID is 9a43017dbebf. More details on this container are availble via *docker inspect container-ID-or-name*. Log information is shown via *docker logs container-ID*, and via the *systemctl status openshift-node -l* and *journalctl -xe -u openshift-node docker* commands. + +The container's rbd mounts are visible directly from the host and from within the container itself. On the OSE host: + +``` +#on the target/scheduled OSE-node: +$ mount | grep rbd +/dev/rbd0 on /var/lib/openshift/openshift.local.volumes/plugins/kubernetes.io/rbd/rbd/rbd-image-foo type ext4 (rw,relatime,seclabel,stripe=1024,data=ordered) +/dev/rbd0 on /var/lib/openshift/openshift.local.volumes/pods/608af6aa-5335-11e5-b56b-52540039f12e/volumes/kubernetes.io~rbd/ceph-pv type ext4 (rw,relatime,seclabel,stripe=1024,data=ordered) +``` + +And, shelling into the container: + +``` +#on the target/scheduled OSE-node: +$ docker exec -it 9a43017dbebf bash +root@ceph-mysql:/# mount | grep rbd +/dev/rbd0 on /var/lib/mysql type ext4 (rw,relatime,seclabel,stripe=1024,data=ordered) +root@ceph-mysql:/# exit +exit +``` + +Mysql can also be run in the container as follows: + +``` +#on the target/scheduled OSE-node: +$ docker exec -it 9a43017dbebf bash +root@ceph-mysql:/# mysql +Welcome to the MySQL monitor. Commands end with ; or \g. +... +mysql> show databases; ++---------------------+ +| Database | ++---------------------+ +| information_schema | +| #mysql50#lost+found | +| mysql | +| performance_schema | +| us_states | ++---------------------+ +5 rows in set (0.00 sec) + +mysql> use us_states; +Reading table information for completion of table and column names +... +mysql> select * from states; ++----+---------+------------+ +| id | state | population | ++----+---------+------------+ +| 1 | Alabama | 4822023 | ++----+---------+------------+ +1 row in set (0.00 sec) + +mysql> quit +Bye +root@ceph-mysql:/# exit +exit +``` diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_pvc/ceph-claim.yaml b/examples/storage-examples/ceph-examples/mysql_ceph_pvc/ceph-claim.yaml new file mode 100644 index 000000000000..86d88bc92b8b --- /dev/null +++ b/examples/storage-examples/ceph-examples/mysql_ceph_pvc/ceph-claim.yaml @@ -0,0 +1,10 @@ +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: ceph-claim +spec: + accessModes: + - ReadWriteMany + resources: + requests: + storage: 1Gi diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_pvc/ceph-mysql-pvc-pod.yaml b/examples/storage-examples/ceph-examples/mysql_ceph_pvc/ceph-mysql-pvc-pod.yaml new file mode 100644 index 000000000000..e770add92268 --- /dev/null +++ b/examples/storage-examples/ceph-examples/mysql_ceph_pvc/ceph-mysql-pvc-pod.yaml @@ -0,0 +1,25 @@ +apiVersion: v1 +id: mysql +kind: Pod +metadata: + name: ceph-mysql +spec: + containers: + - name: ceph-mysql + image: mysql + ports: + - name: mysql-db + containerPort: 3306 + volumeMounts: + - name: mysql-pv + mountPath: /var/lib/mysql + env: + - name: MYSQL_ROOT_PASSWORD + value: foopass + securityContext: + capabilities: {} + privileged: true #privileged required for the rbd mount + volumes: + - name: mysql-pv + persistentVolumeClaim: + claimName: ceph-claim diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_pvc/ceph-pv.yaml b/examples/storage-examples/ceph-examples/mysql_ceph_pvc/ceph-pv.yaml new file mode 100644 index 000000000000..cdc0c0db4838 --- /dev/null +++ b/examples/storage-examples/ceph-examples/mysql_ceph_pvc/ceph-pv.yaml @@ -0,0 +1,20 @@ +--- + apiVersion: v1 + kind: PersistentVolume + metadata: + name: ceph-pv + spec: + capacity: + storage: 2Gi + accessModes: + - ReadWriteMany + rbd: + monitors: + - 192.168.122.133:6789 + pool: rbd + image: foo + user: admin + secretRef: + name: ceph-secret + fsType: ext4 + readOnly: false diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_template/README.md b/examples/storage-examples/ceph-examples/mysql_ceph_template/README.md new file mode 100644 index 000000000000..23b11ebc5b77 --- /dev/null +++ b/examples/storage-examples/ceph-examples/mysql_ceph_template/README.md @@ -0,0 +1,203 @@ +Example 4: MySQL + PVClaim in Single OSE Template +================================================= + +Here is an example showing how to create the mysql application using ceph rbd as the persistent store, where the pod and persistent volume claim (PVC) are both defined in one OSE template file. Make sure that any mysql pods created in other examples have been deleted before continuing with this example: + +``` +#on the OSE-master: +$ oc get pods + +#if you see a mysql pod above: +$ oc delete pod +``` + +### Environment: +If the steps to install the environment, ceph, ose and mysql have not already been completed, then follow the instuctions linked-to directly below: + +The enviromnent used for all of the examples is described [here](../ENV.md). It is assumed that ceph is already up and running, either on bare metal, in a VM, or containerized. + +### Setting up Openshift Enterprise (OSE): +The steps needed to seup a simple OSE cluster with 1 master and 1 worker node are described [here](../OSE.md). + +### Setting up MySQL: +Follow the instructions [here](../MYSQL.md) to initialize and validate containerized mysql. + +### Defining the Template File: +Here is the [template file](ceph-mysql-template.yaml) which defines both the persistent volume claim (PVC) and the pod. The actual persistent volume (PV) has already been created in [example 3](../mysql_ceph_pvc), and is verified here. Note, as in the other examples, the mysql pod/container needs to run privileged, which is specified in the "container:" portion of the template. + +``` +#on the OSE-master: +$ oc get pv +NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON +ceph-pv 2147483648 RWX Bound default/ceph-claim +``` + +### Create the Template Object: + +``` +#on the OSE-master: +$ oc create -f ceph-mysql-template.yaml +templates/ceph-mysql-template + +# oc get templates +NAME DESCRIPTION PARAMETERS OBJECTS +ceph-mysql-template mysql persistent ceph application template using inline PVC 0 (all set) 2 +``` + +Note: when re-using this template, if the pvc already exists, you'll see this error: *Error: persistentvolumeclaims "ceph-claim-template" already exists*. This error can be ignored since the pod is still started correctly. + +### PVC Cleanup: +Before the mysql app can be created from the above template, we need to make sure there is a PV available for the defined template claim. The *oc get pv* command above shows that the only PV we have is Bound to the "ceph-claim", which we created in [example 3](../mysql_ceph_pvc). Therefore, we first need to delete the current PVC: + +``` +#on the OSE-master: +$ oc delete pvc ceph-claim +persistentvolumeclaims/ceph-claim + +# oc get pvc +NAME LABELS STATUS VOLUME + +# oc get pv +NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON +ceph-pv 2147483648 RWX Released default/ceph-claim +``` + +Notice that the "ceph-claim" is gone and that the "ceph-pv" is Released. This may be a bug, but currently a new claim cannot bind to a Released PV; therefore, we have to also delete and recreate the PV before we can start mysql from the template. + +``` +#on the OSE-master: +$ oc delete pv +persistentvolumes/ceph-pv + +$ oc create -f ceph-pv.yaml +persistentvolumes/ceph-pv + +$ oc get pv +NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON +ceph-pv 2147483648 RWX Available + +``` +The pv file used above is defined [here](../mysql_ceph_pvc/ceph-pv.yaml). + +### Create the Mysql Pod: +The *oc new-app* command, which accepts a template object (or a template file can be specified), is used to create the mysql app from this template: + +``` +#on the OSE-master: +$ oc new-app ceph-mysql-template +persistentvolumeclaims/ceph-claim-template +pods/ceph-mysql-pod +Run 'oc status' to view your app. +``` + +Check on the PV, PVC and pod: + +``` +#on the OSE-master: +$ oc get pvc +NAME LABELS STATUS VOLUME +ceph-claim-template map[] Bound ceph-pv + +$ oc get pv +NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON +ceph-pv 2147483648 RWX Bound default/ceph-claim-template + +$ oc get pods +NAME READY STATUS RESTARTS AGE +ceph-mysql-pod 1/1 Running 0 25s + +$ oc describe pod ceph-mysql-pod +Name: ceph-mysql-pod +Namespace: default +Image(s): mysql +Node: 192.168.122.254/192.168.122.254 +Labels: +Status: Running +Reason: +Message: +IP: 10.1.0.44 +Replication Controllers: +Containers: + mysql-from-template: + Image: mysql + State: Running + Started: Fri, 04 Sep 2015 16:04:11 -0400 + Ready: True + Restart Count: 0 +Conditions: + Type Status + Ready True +Events: + FirstSeen LastSeen Count From SubobjectPath Reason Message + Fri, 04 Sep 2015 15:39:41 -0400 Fri, 04 Sep 2015 15:39:41 -0400 1 {scheduler } scheduled Successfully assigned ceph-mysql-pod to 192.168.122.254 + + Fri, 04 Sep 2015 16:04:04 -0400 Fri, 04 Sep 2015 16:04:04 -0400 1 {kubelet 192.168.122.254} implicitly required container POD pulled Pod container image "openshift3/ose-pod:v3.0.1.0" already present on machine + Fri, 04 Sep 2015 16:04:07 -0400 Fri, 04 Sep 2015 16:04:07 -0400 1 {kubelet 192.168.122.254} implicitly required container POD createdCreated with docker id 5d310cd5ddc2 + Fri, 04 Sep 2015 16:04:07 -0400 Fri, 04 Sep 2015 16:04:07 -0400 1 {kubelet 192.168.122.254} implicitly required container POD startedStarted with docker id 5d310cd5ddc2 + Fri, 04 Sep 2015 16:04:11 -0400 Fri, 04 Sep 2015 16:04:11 -0400 1 {kubelet 192.168.122.254} spec.containers{mysql-from-template} createdCreated with docker id 183be9a22a13 + Fri, 04 Sep 2015 16:04:11 -0400 Fri, 04 Sep 2015 16:04:11 -0400 1 {kubelet 192.168.122.254} spec.containers{mysql-from-template} startedStarted with docker id 183be9a22a13 +``` + +On the target OSE node we can verify that msql is working: + +``` +#on the scheduled/target OSE-node: +$ docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +183be9a22a13 mysql "/entrypoint.sh mysq 5 minutes ago Up 5 minutes k8s_mysql-from-template.b4384d92_ceph-mysql-pod_default_af388e69-533c-11e5-b56b-52540039f12e_faf1cdc0 +5d310cd5ddc2 openshift3/ose-pod:v3.0.1.0 "/pod" 5 minutes ago Up 5 minutes k8s_POD.892ec37e_ceph-mysql-pod_default_af388e69-533c-11e5-b56b-52540039f12e_ede13b32 +Using the above container ID, we can inspect the docker logs, and then run a shell inside this container to show the ceph/rbd mount and to access a simple database (previously created), as follows: +``` + +And, as in other examples, we can shell into the running mysql container and execute mysql: + +``` +$ docker exec -it 183be9a22a13 bash +root@ceph-mysql-pod:/# ls /var/lib/mysql/ +ib_logfile0 ibdata1 mysql us_states +ib_logfile1 lost+found performance_schema + +root@ceph-mysql-pod:/# mysql +Welcome to the MySQL monitor. Commands end with ; or \g. +Your MySQL connection id is 2 +Server version: 5.5.44-0ubuntu0.14.04.1 (Ubuntu) +... +Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. + +mysql> show databases; ++---------------------+ +| Database | ++---------------------+ +| information_schema | +| #mysql50#lost+found | +| mysql | +| performance_schema | +| us_states | ++---------------------+ +5 rows in set (0.00 sec) + +mysql> use us_states; +Reading table information for completion of table and column names +... + +mysql> show tables; ++---------------------+ +| Tables_in_us_states | ++---------------------+ +| states | ++---------------------+ +1 row in set (0.00 sec) + +mysql> select * from states; ++----+---------+------------+ +| id | state | population | ++----+---------+------------+ +| 1 | Alabama | 4822023 | ++----+---------+------------+ +1 row in set (0.00 sec) + +mysql> quit +Bye +root@ceph-mysql-pod:/# exit +exit +``` diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_template/ceph-mysql-template.yaml b/examples/storage-examples/ceph-examples/mysql_ceph_template/ceph-mysql-template.yaml new file mode 100644 index 000000000000..8003bed7e83d --- /dev/null +++ b/examples/storage-examples/ceph-examples/mysql_ceph_template/ceph-mysql-template.yaml @@ -0,0 +1,43 @@ +apiVersion: v1 +kind: Template +metadata: + name: ceph-mysql-template + annotations: + description: "mysql persistent ceph application template using inline PVC" + tags: "ceph mysql pvc" + +objects: +- apiVersion: v1 + id: ceph-claim-template + kind: PersistentVolumeClaim + metadata: + name: ceph-claim-template + spec: + accessModes: + - ReadWriteMany + resources: + requests: + storage: 1Gi + +- apiVersion: v1 + id: ceph-mysql-pod + kind: Pod + metadata: + name: ceph-mysql-pod + spec: + containers: + - image: mysql + name: mysql-from-template + volumeMounts: + - mountPath: /var/lib/mysql + name: mysql-pv + env: + - name: MYSQL_ROOT_PASSWORD + value: foopass + securityContext: + capabilities: {} + privileged: true #privileged required for the rbd mount + volumes: + - name: mysql-pv + persistentVolumeClaim: + claimName: ceph-claim-template From 9611ffb58a5cc5df013b882980dc92e41c140466 Mon Sep 17 00:00:00 2001 From: Jeff Vance Date: Wed, 23 Sep 2015 09:06:22 -0700 Subject: [PATCH 2/9] Update README.md --- examples/storage-examples/ceph-examples/README.md | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/examples/storage-examples/ceph-examples/README.md b/examples/storage-examples/ceph-examples/README.md index a57927623e4d..dbba1bfc20fc 100644 --- a/examples/storage-examples/ceph-examples/README.md +++ b/examples/storage-examples/ceph-examples/README.md @@ -1,20 +1,15 @@ MySQL + Ceph Persistent Volume ============================== -Here are examples showing how to run MySQL in the following environments: -1. in a container launched directly from docker (see the [mysql readme](MYSQL.md)) -2. via an openshift/kubernetes pod (examples [2](mysql_ceph_host) and [3](mysql_ceph_pvc)) -3. via an openshift application template (example [4](mysql_ceph_template)). +Here are examples showing how to run MySQL in openshift pods (examples [2](mysql_ceph_host) and [3](mysql_ceph_pvc)), and via an openshift application template (example [4](mysql_ceph_template)). Both [local OSE-node storage](mysql_ceph_host) and [ceph-rbd block storage](mysql_ceph_plugin) are used to persist the database. -Both [local OSE-node storage](mysql_ceph_host) and [ceph-rbd block storage](mysql_ceph_plugin) under an ext4 file system to persist the database. - -The next few sections are common across almost all of the examples and are also shown below: +The next few sections are common across all of the examples: ### Environment: The basic enviromnent used for all of the examples is described [here](ENV.md). It is assumed that ceph is already up and running, either on bare metal, in a VM, or containerized. -### Setting up Openshift Enterprise (OSE): -The steps needed to setup a simple OSE cluster with 1 master and 1 worker node are described [here](OSE.md). +### Setting up Openshift with Ceph: +The steps needed to setup a simple OSE cluster that works with ceph are described [here](OSE.md). ### Setting up MySQL: Follow the instructions [here](MYSQL.md) to initialize and validate containerized mysql. From 4ba19b9f0c4299151e0249ff1753f13703c8e08a Mon Sep 17 00:00:00 2001 From: Jeff Vance Date: Wed, 23 Sep 2015 09:18:35 -0700 Subject: [PATCH 3/9] Update ENV.md --- examples/storage-examples/ceph-examples/ENV.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/storage-examples/ceph-examples/ENV.md b/examples/storage-examples/ceph-examples/ENV.md index 4b8c49cd94c5..167ef3681de5 100644 --- a/examples/storage-examples/ceph-examples/ENV.md +++ b/examples/storage-examples/ceph-examples/ENV.md @@ -11,7 +11,7 @@ The RHEL-7 hosts running the OSE master and OSE nodes should have the following * all OSE nodes (master and workers) and the ceph host need to be running docker. Currently docker version 1.8 has storage setup issues, so see below on how to upgrade or downgrade the version of docker on your VMs. ### Docker: -Docker may be on the OS running OSE, but make sure the docker version is 1.6 or 1.7 -- not 1.8. +Make sure the docker version is 1.6 or 1.7 -- not 1.8. Docker versions later than 1.8 may work fine but there is currently a storage related issue with docker 1.8. The examples here have been tested with docker 1.6 and 1.7. ``` $ docker --version From 72d94888547692f91a4e7086022432aef531df17 Mon Sep 17 00:00:00 2001 From: Jeff Vance Date: Wed, 23 Sep 2015 09:21:01 -0700 Subject: [PATCH 4/9] Update OSE.md --- examples/storage-examples/ceph-examples/OSE.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/examples/storage-examples/ceph-examples/OSE.md b/examples/storage-examples/ceph-examples/OSE.md index 2f0a06db1678..a859f4558389 100644 --- a/examples/storage-examples/ceph-examples/OSE.md +++ b/examples/storage-examples/ceph-examples/OSE.md @@ -1,11 +1,11 @@ -Setup a Simple OpenShift Enterprise (OSE) Cluster -================================================= +Setup a Simple OpenShift Cluster Supporting Ceph +================================================ ### Environment: The enviromnent used for all of the ceph examples is described [here](ENV.md). It is assumed that ceph is already up and running, either on bare metal, in a VM, or containerized. ### Setting up OSE: -The following OpenShift/origin guides should be followed when setting up OSE: +The following OpenShift documents are pertinent to setting up OSE: * https://docs.openshift.com/enterprise/3.0/admin_guide/install/prerequisites.html -- Prerequisites * https://docs.openshift.org/latest/getting_started/administrators.html#running-in-a-docker-container -- Getting Started for Administrator * https://docs.openshift.com/enterprise/3.0/admin_guide/install/quick_install.html -- **Quick Installation** From 50ee251bc85b499594e06378358d24a20eb31964 Mon Sep 17 00:00:00 2001 From: Jeff Vance Date: Wed, 23 Sep 2015 09:22:05 -0700 Subject: [PATCH 5/9] Update ENV.md --- examples/storage-examples/ceph-examples/ENV.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/storage-examples/ceph-examples/ENV.md b/examples/storage-examples/ceph-examples/ENV.md index 167ef3681de5..70f1aa358729 100644 --- a/examples/storage-examples/ceph-examples/ENV.md +++ b/examples/storage-examples/ceph-examples/ENV.md @@ -11,7 +11,7 @@ The RHEL-7 hosts running the OSE master and OSE nodes should have the following * all OSE nodes (master and workers) and the ceph host need to be running docker. Currently docker version 1.8 has storage setup issues, so see below on how to upgrade or downgrade the version of docker on your VMs. ### Docker: -Make sure the docker version is 1.6 or 1.7 -- not 1.8. Docker versions later than 1.8 may work fine but there is currently a storage related issue with docker 1.8. The examples here have been tested with docker 1.6 and 1.7. +On each OSE node, make sure the docker version is 1.6 or 1.7 -- not 1.8. Docker versions later than 1.8 may work fine but there is currently a storage related issue with docker 1.8. The examples here have been tested with docker 1.6 and 1.7. ``` $ docker --version From f9508f14e283f0847a7d43edf909a072f8408c51 Mon Sep 17 00:00:00 2001 From: Jeff Vance Date: Wed, 23 Sep 2015 09:24:00 -0700 Subject: [PATCH 6/9] Update README.md --- .../storage-examples/ceph-examples/mysql_ceph_host/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_host/README.md b/examples/storage-examples/ceph-examples/mysql_ceph_host/README.md index b3025fd9ca19..cd6090923fda 100644 --- a/examples/storage-examples/ceph-examples/mysql_ceph_host/README.md +++ b/examples/storage-examples/ceph-examples/mysql_ceph_host/README.md @@ -17,7 +17,7 @@ If the steps to install the environment, OSE, and mysql have not already been co The enviromnent used for all of the examples here is described [here](../ENV.md). ### Setting up Openshift Enterprise (OSE): -The steps needed to setup a simple OSE cluster with 1 master and 1 worker node are described [here](../OSE.md). +The steps needed to setup a simple OSE cluster that supports ceph (even though ceph is not used in this example) are described [here](../OSE.md). ### Setting up MySQL: Follow the instructions [here](../MYSQL.md) to initialize and validate containerized mysql. From fdee4e8fe4b7a5249dcac95c6f7874aabd3b27d8 Mon Sep 17 00:00:00 2001 From: Jeff Vance Date: Wed, 23 Sep 2015 09:25:42 -0700 Subject: [PATCH 7/9] Update README.md --- .../ceph-examples/mysql_ceph_plugin/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_plugin/README.md b/examples/storage-examples/ceph-examples/mysql_ceph_plugin/README.md index 1607b9e487ca..c6d56e922d23 100644 --- a/examples/storage-examples/ceph-examples/mysql_ceph_plugin/README.md +++ b/examples/storage-examples/ceph-examples/mysql_ceph_plugin/README.md @@ -16,8 +16,8 @@ If the steps to install the environment, ceph, ose, and mysql have not already b The enviromnent used for all of the examples is described [here](../ENV.md). It is assumed that ceph is already up and running, either on bare metal, in a VM, or containerized. -### Setting up Openshift Enterprise (OSE): -The steps needed to setup a simple OSE cluster with 1 master and 1 worker node are described [here](../OSE.md). +### Setting up Openshift for Ceph: +The steps needed to setup a simple OSE cluster that supports ceph are described [here](../OSE.md). ### Setting up MySQL: Follow the instructions [here](../MYSQL.md) to initialize and validate containerized mysql. From 46ea8285f24cc2b6389ba4073e95ad78e51717b6 Mon Sep 17 00:00:00 2001 From: Jeff Vance Date: Wed, 23 Sep 2015 09:26:29 -0700 Subject: [PATCH 8/9] Update README.md --- .../storage-examples/ceph-examples/mysql_ceph_pvc/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_pvc/README.md b/examples/storage-examples/ceph-examples/mysql_ceph_pvc/README.md index 76d71fa0620f..9231e3f85a81 100644 --- a/examples/storage-examples/ceph-examples/mysql_ceph_pvc/README.md +++ b/examples/storage-examples/ceph-examples/mysql_ceph_pvc/README.md @@ -18,8 +18,8 @@ If the steps to install the environment, ceph, ose and mysql have not already be The enviromnent used for all of the examples is described [here](../ENV.md). It is assumed that ceph is already up and running, either on bare metal, in a VM, or containerized. -### Setting up Openshift Enterprise (OSE): -The steps needed to seup a simple OSE cluster with 1 master and 1 worker node are described [here](../OSE.md). +### Setting up Openshift for Ceph: +The steps needed to setup a simple OSE cluster that supports ceph are described [here](../OSE.md). ### Setting up MySQL: Follow the instructions [here](../MYSQL.md) to initialize and validate containerized mysql. From a852c40b67812461c5927be224e29a70a4f0f5eb Mon Sep 17 00:00:00 2001 From: Jeff Vance Date: Wed, 23 Sep 2015 09:27:00 -0700 Subject: [PATCH 9/9] Update README.md --- .../ceph-examples/mysql_ceph_template/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/storage-examples/ceph-examples/mysql_ceph_template/README.md b/examples/storage-examples/ceph-examples/mysql_ceph_template/README.md index 23b11ebc5b77..2e9c58f38458 100644 --- a/examples/storage-examples/ceph-examples/mysql_ceph_template/README.md +++ b/examples/storage-examples/ceph-examples/mysql_ceph_template/README.md @@ -16,8 +16,8 @@ If the steps to install the environment, ceph, ose and mysql have not already be The enviromnent used for all of the examples is described [here](../ENV.md). It is assumed that ceph is already up and running, either on bare metal, in a VM, or containerized. -### Setting up Openshift Enterprise (OSE): -The steps needed to seup a simple OSE cluster with 1 master and 1 worker node are described [here](../OSE.md). +### Setting up Openshift for Ceph: +The steps needed to setup a simple OSE cluster that supports ceph are described [here](../OSE.md). ### Setting up MySQL: Follow the instructions [here](../MYSQL.md) to initialize and validate containerized mysql.