Allow mounting of NFS volumes instead of using st2packs for pack management

CHANGELOG.md

@@ -1,7 +1,7 @@
 # Changelog
 
 ## In Development
-
+* Add an option to mount NFS volumes instead of using the `st2packs` image (#118) (by @AngryDevelopper)
 
 ## v0.32.0
 * Fix a bug when datastore encrypted keys didn't work in scheduled rules. datastore_crypto_key is now shared with the ``st2scheduler`` pods (#148) (by @rahulshinde26)

README.md

@@ -191,14 +191,57 @@ StackStorm employs etcd as a distributed coordination backend, required for st2
 As any other Helm dependency, it's possible to further configure it for specific scaling needs via `values.yaml`.
 
 ## Install custom st2 packs in the cluster
-In distributed environment of the Kubernetes cluster `st2 pack install` won’t work.
-Instead, you need to bake the packs into a custom docker image, push it to a private or public docker registry and reference that image in Helm values.
+There are two ways of installing packs in the cluster. Using a dedicated packs image or using NFS volumes.
+
+### Using a docker image
+You need to bake the packs into a custom docker image, push it to a private or public docker registry and reference that image in Helm values.
 Helm chart will take it from there, sharing `/opt/stackstorm/{packs,virtualenvs}` via a sidecar container in pods which require access to the packs.
+In this mode the packs and virtualenvs volume are mounted read only, so `st2 pack install` won't work
 
-### Building st2packs image
+#### Building st2packs image
 For your convenience, we created a new `st2-pack-install <pack1> <pack2> <pack3>` utility and included it in a container that will help to install custom packs during the Docker build process without relying on live DB and MQ connection.
 Please see https://github.com/StackStorm/st2packs-dockerfiles/ for instructions on how to build your custom `st2packs` image.
 
+#### Pull st2packs from a private Docker registry
+If you need to pull your custom packs Docker image from a private repository, create a Kubernetes Docker registry secret and pass it to Helm values.
+See [K8s documentation](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) for more info.
+```
+# Create a Docker registry secret called 'st2packs-auth'
+kubectl create secret docker-registry st2packs-auth --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-password>
+```
+Once secret created, reference its name in helm value: `st2.packs.image.pullSecret`.
+
+### Using an NFS mount
+You can also mount an NFS export inside the cluster, using the provided `st2.packs.nfs.*` configurations variables.
+A very simple example would be to create two exports as follow on your NFS server:
+```shell script
+$ cat /etc/exports
+/var/nfsshare/packs    *(rw,sync,no_root_squash,no_all_squash)
+/var/nfsshare/virtualenvs    *(rw,sync,no_root_squash,no_all_squash)
+```
+
+Then setup stackstorm-ha to mount those exports:
+```yaml
+st2:
+  packs:
+    nfs: YOUR_NFS_SERVER_ADDRESS
+    virtualenvsPath: /var/nfsshare/virtualenvs
+    packsPath: /var/nfsshare/packs
+```
+You can now either copy your custom packs directly inside the exported `/var/nfsshare/packs` directory, or use stackstorm's
+API to install packs from Stackstorm Exchange
+
+#### NFS caveats
+Manually copied packs are not automatically registered nor installed, you'll need to trigger the process through the st2's API using
+the following endpoints: [install](https://api.stackstorm.com/api/v1/packs/#/packs_controller.install.post),
+[registration](https://api.stackstorm.com/api/v1/packs/#/packs_controller.register.post)
+
+You will have to repeat the process each time the packs code is modified.
+
+On first provisionning of the Stackstorm deployment, the system packs will be copied and registered. 
+
+
+
 ### How to provide custom pack configs
 Update the `st2.packs.configs` section of Helm values:
 
@@ -215,16 +258,6 @@ For example:
 ```
 Don't forget running Helm upgrade to apply new changes.
 
-### Pull st2packs from a private Docker registry
-If you need to pull your custom packs Docker image from a private repository, create a Kubernetes Docker registry secret and pass it to Helm values.
-See [K8s documentation](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) for more info.
-```
-# Create a Docker registry secret called 'st2packs-auth'
-kubectl create secret docker-registry st2packs-auth --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-password>
-```
-Once secret created, reference its name in helm value: `st2.packs.image.pullSecret`.
-
-
 ## Tips & Tricks
 Grab all logs for entire StackStorm cluster with dependent services in Helm release:
 ```

templates/_helpers.tpl

@@ -71,3 +71,21 @@ Create the name of the stackstorm-ha service account to use
 {{ $mongo_fullname }}-{{ $index0 }}.{{ $mongo_fullname }}{{ if ne $index1 $replicas }},{{ end }}
   {{- end -}}
 {{- end -}}
+
+{{- define "packs-volumes" -}}
+{{- if .Values.st2.packs.image.repository }}
+- name: st2-packs-vol
+  emptyDir: {}
+- name: st2-virtualenvs-vol
+  emptyDir: {}
+{{- else if .Values.st2.packs.nfs.server }}
+- name: st2-packs-vol
+  nfs:
+    server: {{ .Values.st2.packs.nfs.server }}
+    path: {{ .Values.st2.packs.nfs.packsPath }}
+- name: st2-virtualenvs-vol
+  nfs:
+    server: {{ .Values.st2.packs.nfs.server }}
+    path: {{ .Values.st2.packs.nfs.virtualenvsPath }}
+{{- end }}
+{{- end -}}

templates/deployments.yaml

@@ -217,6 +217,11 @@ spec:
         - name: st2-virtualenvs-vol
           mountPath: /opt/stackstorm/virtualenvs
           readOnly: true
+        {{- else if .Values.st2.packs.nfs.server }}
+        - name: st2-packs-vol
+          mountPath: /opt/stackstorm/packs
+        - name: st2-virtualenvs-vol
+          mountPath: /opt/stackstorm/virtualenvs
         {{- end }}
         resources:
 {{ toYaml .Values.st2api.resources | indent 10 }}
@@ -235,12 +240,7 @@ spec:
         - name: st2-config-vol
           configMap:
             name: {{ .Release.Name }}-st2-config
-        {{- if .Values.st2.packs.image.repository }}
-        - name: st2-packs-vol
-          emptyDir: {}
-        - name: st2-virtualenvs-vol
-          emptyDir: {}
-        {{- end }}
+        {{- include "packs-volumes" . | indent 8 }}
     {{- with .Values.st2api.nodeSelector }}
       nodeSelector:
 {{ toYaml . | indent 8 }}
@@ -964,6 +964,11 @@ spec:
         - name: st2-virtualenvs-vol
           mountPath: /opt/stackstorm/virtualenvs
           readOnly: true
+        {{- else if $.Values.st2.packs.nfs.server }}
+        - name: st2-packs-vol
+          mountPath: /opt/stackstorm/packs
+        - name: st2-virtualenvs-vol
+          mountPath: /opt/stackstorm/virtualenvs
         {{- end }}
         {{- if $.Values.secrets.st2.datastore_crypto_key }}
         - name: st2-encryption-key-vol
@@ -987,13 +992,8 @@ spec:
         - name: st2-config-vol
           configMap:
             name: {{ $.Release.Name }}-st2-config
-        {{- if $.Values.st2.packs.image.repository }}
-        - name: st2-packs-vol
-          emptyDir: {}
-        - name: st2-virtualenvs-vol
-          emptyDir: {}
-        {{- end }}
-    {{- with .nodeSelector }}
+        {{- include "packs-volumes" $ | indent 8}}
+        {{- with .nodeSelector }}
       nodeSelector:
 {{ toYaml . | indent 8 }}
     {{- end }}
@@ -1123,6 +1123,11 @@ spec:
         - name: st2-virtualenvs-vol
           mountPath: /opt/stackstorm/virtualenvs
           readOnly: true
+        {{- else if .Values.st2.packs.nfs.server }}
+        - name: st2-packs-vol
+          mountPath: /opt/stackstorm/packs
+        - name: st2-virtualenvs-vol
+          mountPath: /opt/stackstorm/virtualenvs
         {{- end }}
         resources:
 {{ toYaml .Values.st2actionrunner.resources | indent 10 }}
@@ -1149,12 +1154,7 @@ spec:
               path: stanley_rsa
               # 0400 file permission
               mode: 256
-        {{- if .Values.st2.packs.image.repository }}
-        - name: st2-packs-vol
-          emptyDir: {}
-        - name: st2-virtualenvs-vol
-          emptyDir: {}
-        {{- end }}
+        {{- include "packs-volumes" . | indent 8 }}
     {{- with .Values.st2actionrunner.nodeSelector }}
       nodeSelector:
 {{ toYaml . | indent 8 }}
@@ -1400,6 +1400,11 @@ spec:
         - name: st2-virtualenvs-vol
           mountPath: /opt/stackstorm/virtualenvs
           readOnly: true
+        {{- else if .Values.st2.packs.nfs.server }}
+        - name: st2-packs-vol
+          mountPath: /opt/stackstorm/packs
+        - name: st2-virtualenvs-vol
+          mountPath: /opt/stackstorm/virtualenvs
         {{- end }}
         command:
           - 'bash'
@@ -1446,12 +1451,7 @@ spec:
               path: stanley_rsa
               # 0400 file permission
               mode: 256
-        {{- if .Values.st2.packs.image.repository }}
-        - name: st2-packs-vol
-          emptyDir: {}
-        - name: st2-virtualenvs-vol
-          emptyDir: {}
-        {{- end }}
+        {{- include "packs-volumes" . | indent 8 }}
 
 {{ if .Values.st2chatops.enabled -}}
 ---

templates/jobs.yaml

@@ -282,7 +282,6 @@ spec:
           secret:
             secretName: {{ .Release.Name }}-st2-kv
       restartPolicy: OnFailure
-
 ---
 apiVersion: batch/v1
 kind: Job
@@ -324,8 +323,8 @@ spec:
       {{- if .Values.st2.packs.image.pullSecret }}
       - name: {{ .Values.st2.packs.image.pullSecret }}
       {{- end }}
-      {{- if .Values.st2.packs.image.repository }}
       initContainers:
+      {{- if .Values.st2.packs.image.repository }}
       # Merge packs and virtualenvs from st2actionrunner with those from the st2.packs image
       # Custom packs
       - name: st2-custom-packs
@@ -342,6 +341,7 @@ spec:
           - |
             /bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
             /bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
+      {{ end }}
       # System packs
       - name: st2-system-packs
         image: "{{ template "imageRepository" . }}/st2actionrunner{{ template "enterpriseSuffix" . }}:{{ .Chart.AppVersion }}"
@@ -355,7 +355,6 @@ spec:
           - |
             /bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
             /bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
-      {{ end }}
       containers:
       - name: st2-register-content
         image: "{{ template "imageRepository" . }}/st2actionrunner{{ template "enterpriseSuffix" . }}:{{ .Chart.AppVersion }}"
@@ -378,9 +377,16 @@ spec:
           mountPath: /opt/stackstorm/configs/
         {{- if .Values.st2.packs.image.repository }}
         - name: st2-packs-vol
-          mountPath: /opt/stackstorm/packs/
+          mountPath: /opt/stackstorm/packs
+          readOnly: true
         - name: st2-virtualenvs-vol
-          mountPath: /opt/stackstorm/virtualenvs/
+          mountPath: /opt/stackstorm/virtualenvs
+          readOnly: true
+        {{- else if .Values.st2.packs.nfs.server }}
+        - name: st2-packs-vol
+          mountPath: /opt/stackstorm/packs
+        - name: st2-virtualenvs-vol
+          mountPath: /opt/stackstorm/virtualenvs
         {{- end }}
         # TODO: Find out default resource limits for this specific service (#5)
         #resources:
@@ -391,10 +397,5 @@ spec:
         - name: st2-pack-configs-vol
           configMap:
             name: {{ .Release.Name }}-st2-pack-configs
-        {{- if .Values.st2.packs.image.repository }}
-        - name: st2-packs-vol
-          emptyDir: {}
-        - name: st2-virtualenvs-vol
-          emptyDir: {}
-        {{- end }}
+        {{- include "packs-volumes" . | indent 8 }}
       restartPolicy: OnFailure

values.yaml

@@ -103,6 +103,18 @@ st2:
       # Optional name of the imagePullSecret if your custom packs image is hosted by a private Docker registry behind the auth
       #pullSecret: st2packs-auth
 
+    # Custom packs and virtualenvs can be mounted using NFS.
+    # Disclaimer: if you chose to use an NFS mount, be advised that the responsibility of installing and registering the
+    # packs is deferred to you.
+    # If both st2.packs.image.repository and st2.packs.nfs.server are set, the st2packs image will be used
+    nfs:
+      # Uncommend the following block to enabled NFS mounting of the packs
+      #server: 0.0.0.0
+      # Required. This should point to a directory containing all the packs folders
+      virtualenvsPath: /var/nfsshare/virtualenvs
+      # Required. This should point to an empty directoy, or a directory containing pre built venvs for the packs
+      packsPath: /var/nfsshare/packs
+
     # https://docs.stackstorm.com/reference/ha.html#st2sensorcontainer
     # It is possible to run st2sensorcontainer in HA mode by running one process on each compute instance.
     # Each sensor node needs to be provided with proper partition information to share work with other sensor