diff --git a/CHANGELOG.md b/CHANGELOG.md index 544dea68..b7c1769b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,7 @@ # Changelog ## In Development - +* Add an option to mount NFS volumes instead of using the `st2packs` image (#118) (by @AngryDevelopper) ## v0.32.0 * Fix a bug when datastore encrypted keys didn't work in scheduled rules. datastore_crypto_key is now shared with the ``st2scheduler`` pods (#148) (by @rahulshinde26) diff --git a/README.md b/README.md index b6adab45..013a9dd2 100644 --- a/README.md +++ b/README.md @@ -191,14 +191,57 @@ StackStorm employs etcd as a distributed coordination backend, required for st2 As any other Helm dependency, it's possible to further configure it for specific scaling needs via `values.yaml`. ## Install custom st2 packs in the cluster -In distributed environment of the Kubernetes cluster `st2 pack install` won’t work. -Instead, you need to bake the packs into a custom docker image, push it to a private or public docker registry and reference that image in Helm values. +There are two ways of installing packs in the cluster. Using a dedicated packs image or using NFS volumes. + +### Using a docker image +You need to bake the packs into a custom docker image, push it to a private or public docker registry and reference that image in Helm values. Helm chart will take it from there, sharing `/opt/stackstorm/{packs,virtualenvs}` via a sidecar container in pods which require access to the packs. +In this mode the packs and virtualenvs volume are mounted read only, so `st2 pack install` won't work -### Building st2packs image +#### Building st2packs image For your convenience, we created a new `st2-pack-install ` utility and included it in a container that will help to install custom packs during the Docker build process without relying on live DB and MQ connection. Please see https://github.com/StackStorm/st2packs-dockerfiles/ for instructions on how to build your custom `st2packs` image. +#### Pull st2packs from a private Docker registry +If you need to pull your custom packs Docker image from a private repository, create a Kubernetes Docker registry secret and pass it to Helm values. +See [K8s documentation](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) for more info. +``` +# Create a Docker registry secret called 'st2packs-auth' +kubectl create secret docker-registry st2packs-auth --docker-server= --docker-username= --docker-password= +``` +Once secret created, reference its name in helm value: `st2.packs.image.pullSecret`. + +### Using an NFS mount +You can also mount an NFS export inside the cluster, using the provided `st2.packs.nfs.*` configurations variables. +A very simple example would be to create two exports as follow on your NFS server: +```shell script +$ cat /etc/exports +/var/nfsshare/packs *(rw,sync,no_root_squash,no_all_squash) +/var/nfsshare/virtualenvs *(rw,sync,no_root_squash,no_all_squash) +``` + +Then setup stackstorm-ha to mount those exports: +```yaml +st2: + packs: + nfs: YOUR_NFS_SERVER_ADDRESS + virtualenvsPath: /var/nfsshare/virtualenvs + packsPath: /var/nfsshare/packs +``` +You can now either copy your custom packs directly inside the exported `/var/nfsshare/packs` directory, or use stackstorm's +API to install packs from Stackstorm Exchange + +#### NFS caveats +Manually copied packs are not automatically registered nor installed, you'll need to trigger the process through the st2's API using +the following endpoints: [install](https://api.stackstorm.com/api/v1/packs/#/packs_controller.install.post), +[registration](https://api.stackstorm.com/api/v1/packs/#/packs_controller.register.post) + +You will have to repeat the process each time the packs code is modified. + +On first provisionning of the Stackstorm deployment, the system packs will be copied and registered. + + + ### How to provide custom pack configs Update the `st2.packs.configs` section of Helm values: @@ -215,16 +258,6 @@ For example: ``` Don't forget running Helm upgrade to apply new changes. -### Pull st2packs from a private Docker registry -If you need to pull your custom packs Docker image from a private repository, create a Kubernetes Docker registry secret and pass it to Helm values. -See [K8s documentation](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) for more info. -``` -# Create a Docker registry secret called 'st2packs-auth' -kubectl create secret docker-registry st2packs-auth --docker-server= --docker-username= --docker-password= -``` -Once secret created, reference its name in helm value: `st2.packs.image.pullSecret`. - - ## Tips & Tricks Grab all logs for entire StackStorm cluster with dependent services in Helm release: ``` diff --git a/templates/_helpers.tpl b/templates/_helpers.tpl index 9374d825..e0c16982 100644 --- a/templates/_helpers.tpl +++ b/templates/_helpers.tpl @@ -71,3 +71,21 @@ Create the name of the stackstorm-ha service account to use {{ $mongo_fullname }}-{{ $index0 }}.{{ $mongo_fullname }}{{ if ne $index1 $replicas }},{{ end }} {{- end -}} {{- end -}} + +{{- define "packs-volumes" -}} +{{- if .Values.st2.packs.image.repository }} +- name: st2-packs-vol + emptyDir: {} +- name: st2-virtualenvs-vol + emptyDir: {} +{{- else if .Values.st2.packs.nfs.server }} +- name: st2-packs-vol + nfs: + server: {{ .Values.st2.packs.nfs.server }} + path: {{ .Values.st2.packs.nfs.packsPath }} +- name: st2-virtualenvs-vol + nfs: + server: {{ .Values.st2.packs.nfs.server }} + path: {{ .Values.st2.packs.nfs.virtualenvsPath }} +{{- end }} +{{- end -}} diff --git a/templates/deployments.yaml b/templates/deployments.yaml index 1758cfc4..e0a8343b 100644 --- a/templates/deployments.yaml +++ b/templates/deployments.yaml @@ -217,6 +217,11 @@ spec: - name: st2-virtualenvs-vol mountPath: /opt/stackstorm/virtualenvs readOnly: true + {{- else if .Values.st2.packs.nfs.server }} + - name: st2-packs-vol + mountPath: /opt/stackstorm/packs + - name: st2-virtualenvs-vol + mountPath: /opt/stackstorm/virtualenvs {{- end }} resources: {{ toYaml .Values.st2api.resources | indent 10 }} @@ -235,12 +240,7 @@ spec: - name: st2-config-vol configMap: name: {{ .Release.Name }}-st2-config - {{- if .Values.st2.packs.image.repository }} - - name: st2-packs-vol - emptyDir: {} - - name: st2-virtualenvs-vol - emptyDir: {} - {{- end }} + {{- include "packs-volumes" . | indent 8 }} {{- with .Values.st2api.nodeSelector }} nodeSelector: {{ toYaml . | indent 8 }} @@ -964,6 +964,11 @@ spec: - name: st2-virtualenvs-vol mountPath: /opt/stackstorm/virtualenvs readOnly: true + {{- else if $.Values.st2.packs.nfs.server }} + - name: st2-packs-vol + mountPath: /opt/stackstorm/packs + - name: st2-virtualenvs-vol + mountPath: /opt/stackstorm/virtualenvs {{- end }} {{- if $.Values.secrets.st2.datastore_crypto_key }} - name: st2-encryption-key-vol @@ -987,13 +992,8 @@ spec: - name: st2-config-vol configMap: name: {{ $.Release.Name }}-st2-config - {{- if $.Values.st2.packs.image.repository }} - - name: st2-packs-vol - emptyDir: {} - - name: st2-virtualenvs-vol - emptyDir: {} - {{- end }} - {{- with .nodeSelector }} + {{- include "packs-volumes" $ | indent 8}} + {{- with .nodeSelector }} nodeSelector: {{ toYaml . | indent 8 }} {{- end }} @@ -1123,6 +1123,11 @@ spec: - name: st2-virtualenvs-vol mountPath: /opt/stackstorm/virtualenvs readOnly: true + {{- else if .Values.st2.packs.nfs.server }} + - name: st2-packs-vol + mountPath: /opt/stackstorm/packs + - name: st2-virtualenvs-vol + mountPath: /opt/stackstorm/virtualenvs {{- end }} resources: {{ toYaml .Values.st2actionrunner.resources | indent 10 }} @@ -1149,12 +1154,7 @@ spec: path: stanley_rsa # 0400 file permission mode: 256 - {{- if .Values.st2.packs.image.repository }} - - name: st2-packs-vol - emptyDir: {} - - name: st2-virtualenvs-vol - emptyDir: {} - {{- end }} + {{- include "packs-volumes" . | indent 8 }} {{- with .Values.st2actionrunner.nodeSelector }} nodeSelector: {{ toYaml . | indent 8 }} @@ -1400,6 +1400,11 @@ spec: - name: st2-virtualenvs-vol mountPath: /opt/stackstorm/virtualenvs readOnly: true + {{- else if .Values.st2.packs.nfs.server }} + - name: st2-packs-vol + mountPath: /opt/stackstorm/packs + - name: st2-virtualenvs-vol + mountPath: /opt/stackstorm/virtualenvs {{- end }} command: - 'bash' @@ -1446,12 +1451,7 @@ spec: path: stanley_rsa # 0400 file permission mode: 256 - {{- if .Values.st2.packs.image.repository }} - - name: st2-packs-vol - emptyDir: {} - - name: st2-virtualenvs-vol - emptyDir: {} - {{- end }} + {{- include "packs-volumes" . | indent 8 }} {{ if .Values.st2chatops.enabled -}} --- diff --git a/templates/jobs.yaml b/templates/jobs.yaml index 695a7b12..69fbb173 100644 --- a/templates/jobs.yaml +++ b/templates/jobs.yaml @@ -282,7 +282,6 @@ spec: secret: secretName: {{ .Release.Name }}-st2-kv restartPolicy: OnFailure - --- apiVersion: batch/v1 kind: Job @@ -324,8 +323,8 @@ spec: {{- if .Values.st2.packs.image.pullSecret }} - name: {{ .Values.st2.packs.image.pullSecret }} {{- end }} - {{- if .Values.st2.packs.image.repository }} initContainers: + {{- if .Values.st2.packs.image.repository }} # Merge packs and virtualenvs from st2actionrunner with those from the st2.packs image # Custom packs - name: st2-custom-packs @@ -342,6 +341,7 @@ spec: - | /bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared && /bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared + {{ end }} # System packs - name: st2-system-packs image: "{{ template "imageRepository" . }}/st2actionrunner{{ template "enterpriseSuffix" . }}:{{ .Chart.AppVersion }}" @@ -355,7 +355,6 @@ spec: - | /bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared && /bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared - {{ end }} containers: - name: st2-register-content image: "{{ template "imageRepository" . }}/st2actionrunner{{ template "enterpriseSuffix" . }}:{{ .Chart.AppVersion }}" @@ -378,9 +377,16 @@ spec: mountPath: /opt/stackstorm/configs/ {{- if .Values.st2.packs.image.repository }} - name: st2-packs-vol - mountPath: /opt/stackstorm/packs/ + mountPath: /opt/stackstorm/packs + readOnly: true - name: st2-virtualenvs-vol - mountPath: /opt/stackstorm/virtualenvs/ + mountPath: /opt/stackstorm/virtualenvs + readOnly: true + {{- else if .Values.st2.packs.nfs.server }} + - name: st2-packs-vol + mountPath: /opt/stackstorm/packs + - name: st2-virtualenvs-vol + mountPath: /opt/stackstorm/virtualenvs {{- end }} # TODO: Find out default resource limits for this specific service (#5) #resources: @@ -391,10 +397,5 @@ spec: - name: st2-pack-configs-vol configMap: name: {{ .Release.Name }}-st2-pack-configs - {{- if .Values.st2.packs.image.repository }} - - name: st2-packs-vol - emptyDir: {} - - name: st2-virtualenvs-vol - emptyDir: {} - {{- end }} + {{- include "packs-volumes" . | indent 8 }} restartPolicy: OnFailure diff --git a/values.yaml b/values.yaml index 44a32fe0..61ad816b 100644 --- a/values.yaml +++ b/values.yaml @@ -103,6 +103,18 @@ st2: # Optional name of the imagePullSecret if your custom packs image is hosted by a private Docker registry behind the auth #pullSecret: st2packs-auth + # Custom packs and virtualenvs can be mounted using NFS. + # Disclaimer: if you chose to use an NFS mount, be advised that the responsibility of installing and registering the + # packs is deferred to you. + # If both st2.packs.image.repository and st2.packs.nfs.server are set, the st2packs image will be used + nfs: + # Uncommend the following block to enabled NFS mounting of the packs + #server: 0.0.0.0 + # Required. This should point to a directory containing all the packs folders + virtualenvsPath: /var/nfsshare/virtualenvs + # Required. This should point to an empty directoy, or a directory containing pre built venvs for the packs + packsPath: /var/nfsshare/packs + # https://docs.stackstorm.com/reference/ha.html#st2sensorcontainer # It is possible to run st2sensorcontainer in HA mode by running one process on each compute instance. # Each sensor node needs to be provided with proper partition information to share work with other sensor