diff --git a/_topic_map.yml b/_topic_map.yml index c88ff3647ce2..7145d2a5a88b 100644 --- a/_topic_map.yml +++ b/_topic_map.yml @@ -448,6 +448,8 @@ Topics: File: gluster_dynamic_example - Name: Dynamic Provisioning Example Using Dedicated GlusterFS File: dedicated_gluster_dynamic_example + - Name: Containerized Heketi for Managing Dedicated GlusterFS + File: containerized_heketi_with_dedicated_gluster - Name: Mounting Volumes To Privileged Pods File: privileged_pod_storage - Name: Backing Docker Registry with GlusterFS Storage diff --git a/install_config/storage_examples/containerized_heketi_with_dedicated_gluster.adoc b/install_config/storage_examples/containerized_heketi_with_dedicated_gluster.adoc new file mode 100644 index 000000000000..4e900b5081bb --- /dev/null +++ b/install_config/storage_examples/containerized_heketi_with_dedicated_gluster.adoc @@ -0,0 +1,386 @@ +[[install-config-storage-examples-containerized-heketi-dedicated-gluster]] += Example: Containerized Heketi for managing dedicated GlusterFS storage +{product-author} +{product-version} +:data-uri: +:icons: +:experimental: +:toc: macro +:toc-title: +:prewrap!: + +toc::[] + +[[containerized-heketi-dedicated-gluster-overview]] +== Overview +This example provides information about the integration, deployment, and management of GlusterFS containerized storage nodes by using Heketi running on OpenShift Container Platform. + +This example: + +* Shows how to install and configure a Heketi server on OpenShift to perform dynamic provisioning. +* Assumes you have familiarity with Kubernetes and the Kubernetes Persistent Storage model. +* Assumes you have access to an existing, dedicated GlusterFS cluster that has raw devices available for consumption and management by a Heketi server. If you do not have this, you can create a three node cluster using your virtual machine solution of choice. Ensure sure you create a few raw devices and give plenty of space (at least 100GB recommended). See +link:https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Installation_Guide/[Red Hat Gluster Storage Installation Guide]. + +[[containerized-heketi-dedicated-gluster-prerequisites]] +== Environment and Prerequisites +This example uses the following environment and prerequisites: + +* GlusterFS cluster running Red Hat Gluster Storage (RHGS) 3.1. Three nodes, each with at least two 100GB RAW devices: +** *gluster23.rhs* (192.168.1.200) +** *gluster24.rhs* (192.168.1.201) +** *gluster25.rhs* (192.168.1.202) +* This example uses an all-in-one {product-title} cluster (master and node on a single host), though it can work using a standard, multi-node cluster as well. +** *k8dev2.rhs* (192.168.1.208) + +[[containerized-heketi-dedicated-gluster-install-heketi]] +== Installing and Configuring Heketi +Heketi is used to manage the Gluster cluster storage (adding volumes, removing volumes, etc.). Download link:https://github.com/heketi/heketi/blob/master/extras/openshift/templates/deploy-heketi-template.json[`deploy-heketi-template`] to install Heketi on OpenShift. + +[NOTE] +==== +This template file places the database in an EmptyDir volume. Adjust the database accordingly for a reliable persistent storage. +==== + +. Create a new project: ++ +[source, bash] +---- +$ oc new-project +---- + +. Enable privileged containers in the new project: ++ +[source, bash] +---- +$ oc adm policy add-scc-to-user privileged -z default +---- + +. Register the `deploy-heketi` template: ++ +[source, bash] +---- +$ oc create -f /deploy-heketi-template +---- + +. Deploy the bootstrap Heketi container: ++ +[source, bash] +---- +$ oc process deploy-heketi -v \ + HEKETI_KUBE_NAMESPACE= \ + HEKETI_KUBE_APIHOST= \ + HEKETI_KUBE_INSECURE=y \ + HEKETI_KUBE_USER= \ + HEKETI_KUBE_PASSWORD= | oc create -f - +---- + +. Wait until the `deploy-heketi` pod starts and all services are running. Then get Heketi service details: ++ +[source, bash] +---- +$ oc get svc +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +deploy-heketi 172.30.96.173 8080/TCP 2m +---- + +. Check if Heketi services are running properly, it must return `Hello from Heketi`. ++ +[source, bash] +---- +$ curl http://:8080/hello +Hello from Heketi +---- + +. Set an environment variable for the Heketi server: ++ +[source, bash] +---- +$ export HEKETI_CLI_SERVER=http://:8080 +---- + +[[dedicated-glusterfs-dynamic-loading-topology]] +== Loading Topology + +Topology is used to tell Heketi about the environment and what nodes and devices +it will manage. + +[NOTE] +==== +Heketi is currently limited to managing raw devices only. If a device is already +a Gluster volume, it is skipped and ignored. +==== + +. Create and load the topology file. There is a sample file located in +*_/usr/share/heketi/toplogy-sample.json_* by default, or *_/etc/heketi_* +depending on how it was installed. ++ +[NOTE] +==== +Depending upon your method of installation this file may not exist. If it is missing, manually create the *_toplogy-sample.json_* file, as shown in the following example. +==== ++ +[source, json] +---- +{ + "clusters": [ + { + "nodes": [ + { + "node": { + "hostnames": { + "manage": [ + "gluster23.rhs" + ], + "storage": [ + "192.168.1.200" + ] + }, + "zone": 1 + }, + "devices": [ + "/dev/sde", + "/dev/sdf" + ] + }, + { + "node": { + "hostnames": { + "manage": [ + "gluster24.rhs" + ], + "storage": [ + "192.168.1.201" + ] + }, + "zone": 1 + }, + "devices": [ + "/dev/sde", + "/dev/sdf" + ] + }, + { + "node": { + "hostnames": { + "manage": [ + "gluster25.rhs" + ], + "storage": [ + "192.168.1.202" + ] + }, + "zone": 1 + }, + "devices": [ + "/dev/sde", + "/dev/sdf" + ] + } + ] + } + ] +} +---- + +. Run the following command to load the topology of your +environment. ++ +[source, bash] +---- +$ heketi-cli topology load --json=toplogy-sample.json + + Found node gluster23.rhs on cluster bdf9d8ca3fa269ff89854faf58f34b9a + Adding device /dev/sde ... OK + Adding device /dev/sdf ... OK + Creating node gluster24.rhs ... ID: 8e677d8bebe13a3f6846e78a67f07f30 + Adding device /dev/sde ... OK + Adding device /dev/sdf ... OK +... +---- + +. Create a Gluster volume to verify Heketi: ++ +[source, bash] +---- +$ heketi-cli volume create --size=50 +---- + +. View the volume information from one of the the Gluster nodes: ++ +[source, bash] +---- +$ gluster volume info + + Volume Name: vol_335d247ac57ecdf40ac616514cc6257f <1> + Type: Distributed-Replicate + Volume ID: 75be7940-9b09-4e7f-bfb0-a7eb24b411e3 + Status: Started +... +---- +<1> Volume created by `heketi-cli`. + +[[dedicated-glusterfs-dynamic-provision-volume]] +== Dynamically Provision a Volume +[NOTE] +==== +If you installed {product-title} by using the link:https://github.com/openshift/openshift-ansible/tree/master/inventory/byo[BYO (Bring your own) OpenShift Ansible inventory configuration files] for either link:https://github.com/openshift/openshift-ansible/blob/master/inventory/byo/hosts.byo.glusterfs.native.example[native] or link:https://github.com/openshift/openshift-ansible/blob/master/inventory/byo/hosts.byo.glusterfs.external.example[external] GlusterFS instance, the GlusterFS StorageClass automatically get created during the installation. For such cases you can skip the following storage class creation steps and directly proceed with creating persistent volume claim instruction. +==== + +. Create a `StorageClass` object definition. The following definition is based on the +minimum requirements needed for this example to work with {product-title}. See +xref:../../install_config/persistent_storage/dynamically_provisioning_pvs.adoc#install-config-persistent-storage-dynamically-provisioning-pvs[Dynamic +Provisioning and Creating Storage Classes] for additional parameters and +specification definitions. ++ +[source,yaml] +---- +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: gluster-dyn +provisioner: kubernetes.io/glusterfs +parameters: + resturl: "http://glusterclient2.rhs:8080" <1> + restauthenabled: "false" <2> +---- +<1> The Heketi server from the `HEKETI_CLI_SERVER` environment variable. +<2> Since authentication is not turned on in this example, set to `false`. + +. From the {product-title} master host, create the storage class: ++ +[source, bash] +---- +$ oc create -f glusterfs-storageclass1.yaml +storageclass "gluster-dyn" created +---- + +. Create a persistent volume claim (PVC), requesting the newly-created storage +class. For example: ++ +[source,yaml] +---- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: gluster-dyn-pvc +spec: + accessModes: + - ReadWriteMany + resources: + requests: + storage: 30Gi + storageClassName: gluster-dyn +---- + +. From the {product-title} master host, create the PVC: ++ +[source, bash] +---- +$ oc create -f glusterfs-pvc-storageclass.yaml +persistentvolumeclaim "gluster-dyn-pvc" created +---- + +. View the PVC to see that the volume was dynamically created and bound to the PVC: ++ +[source, bash] +---- +$ oc get pvc +NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE +gluster-dyn-pvc Bound pvc-78852230-d8e2-11e6-a3fa-0800279cf26f 30Gi RWX gluster-dyn 42s +---- + +. Verify and view the new volume on one of the Gluster nodes: ++ +[source, bash] +---- +$ gluster volume info + + Volume Name: vol_335d247ac57ecdf40ac616514cc6257f <1> + Type: Distributed-Replicate + Volume ID: 75be7940-9b09-4e7f-bfb0-a7eb24b411e3 + Status: Started + ... + Volume Name: vol_f1404b619e6be6ef673e2b29d58633be <2> + Type: Distributed-Replicate + Volume ID: 7dc234d0-462f-4c6c-add3-fb9bc7e8da5e + Status: Started + Number of Bricks: 2 x 2 = 4 + ... +---- +<1> Volume created by `heketi-cli`. +<2> New dynamically created volume triggered by Kubernetes and the storage class. + +[[dedicated-glusterfs-dynamic-nginx]] +== Creating a NGINX Pod That Uses the PVC + +At this point, you have a dynamically created GlusterFS volume bound to a PVC. +You can now now utilize this PVC in a pod. In this example, create a simple +NGINX pod. + +. Create the pod object definition: ++ +[source,yaml] +---- +apiVersion: v1 +kind: Pod +metadata: + name: gluster-pod1 + labels: + name: gluster-pod1 +spec: + containers: + - name: gluster-pod1 + image: gcr.io/google_containers/nginx-slim:0.8 + ports: + - name: web + containerPort: 80 + securityContext: + privileged: true + volumeMounts: + - name: gluster-vol1 + mountPath: /usr/share/nginx/html + volumes: + - name: gluster-vol1 + persistentVolumeClaim: + claimName: gluster-dyn-pvc <1> +---- +<1> The name of the PVC created in the previous step. + +. From the {product-title} master host, create the pod: ++ +[source, bash] +---- +$ oc create -f nginx-pod.yaml +pod "gluster-pod1" created +---- + +. View the pod. Give it a few minutes, as it might need to download the image if +it does not already exist: ++ +[source, bash] +---- +$ oc get pods -o wide +NAME READY STATUS RESTARTS AGE IP NODE +gluster-pod1 1/1 Running 0 9m 10.38.0.0 node1 +---- + +. Now remote into the container with `oc exec` and create an *_index.html_* file: ++ +[source, bash] +---- +$ oc exec -ti gluster-pod1 /bin/sh +$ cd /usr/share/nginx/html +$ echo 'Hello World from GlusterFS!!!' > index.html +$ ls +index.html +$ exit +---- + +. Now `curl` the URL of the pod: ++ +[source, bash] +---- +$ curl http://10.38.0.0 +Hello World from GlusterFS!!! +---- diff --git a/install_config/storage_examples/dedicated_gluster_dynamic_example.adoc b/install_config/storage_examples/dedicated_gluster_dynamic_example.adoc index a60f631d630e..5fe0b84ecfa2 100644 --- a/install_config/storage_examples/dedicated_gluster_dynamic_example.adoc +++ b/install_config/storage_examples/dedicated_gluster_dynamic_example.adoc @@ -210,7 +210,7 @@ depending on how it was installed. "/dev/sde", "/dev/sdf" ] - }, + } ] } ]