From 0b316ed68cdcf249ec11014bf581dbb79decf6b7 Mon Sep 17 00:00:00 2001 From: amanda Date: Fri, 11 Feb 2022 16:17:55 +0000 Subject: [PATCH 1/9] Initial documentation on overrides --- docs/source/packs.rst | 119 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) diff --git a/docs/source/packs.rst b/docs/source/packs.rst index 80c97ad4..6be0ea96 100644 --- a/docs/source/packs.rst +++ b/docs/source/packs.rst @@ -210,6 +210,125 @@ have to comply with Health and Safety. The generated file will be placed in For more nice tricks on pack configuration, see :doc:`/reference/pack_configs`. +Overriding Pack Defaults +------------------------ + +When installing a pack the state of the resources is taken from the metadata files in the pack. Occassionally, when installing community packs you might not want all resources enabled, e.g. you only want to use a subset of actions, or want to disable the sensors. + +Prior, to |st2| 3.7.0 then this state can be altered, by: + + * using the |st2| APIs to disable the resource. However, this will be forgotten upon a st2ctl reload or pack reinstall. + * changing the metadata file manually. This would be lost upon an upgrade, and cannot be easily tracked. + +In |st2| 3.7.0 we have introduced the override feature, so that the metadata of a packs resources can be overridden by configuration files. This will always be read upon a reload or pack install. The ST2 APIs will still allow you to override this, but as before any changes made by the |st2| APIs to enable/disable resources is forgotten upon reload or re-install. + +The override facility is currently restricted to allowing the enabled property to be overridden, and is controlled by the /opt/stackstorm/overrides directory. + +Upon a pack install or reload resource state is managed as follows: + +* State is read from the pack's resource metadata files that reside in /opt/stackstorm/packs/. These are downloaded from the relevant repository of the pack, e.g. GIT, StackStorm-Exchange. +* If /opt/stackstorm/overrides/global.yaml is present, then any global overrides are applied. The global.yaml allows you to specify the default state of a particular resource types, e.g. disable all sensors. + The format of the global.yaml is as follows (set to disable everything): + +.. code-block:: bash + + --- + sensors: + defaults: + enabled: false + actions: + defaults: + enabled: false + aliases: + defaults: + enabled: false + rules: + defaults: + enabled: false + +* If /opt/stackstorm/overrides/.yaml is present, then any any pack defaults, or resource specific overrides are applied. The .yaml allows you to alter the state of all the resources of a particular type, via the defaults property (e.g. all sensors within the pack), or by individual resource type, via the exceptions property (e.g. only alter state of individual actions). + The format of the .yaml is as follows (set to disable everything, except for individual resource of each type): + +.. code-block:: bash + + --- + sensors: + defaults: + enabled: false + exceptions: + sensorname1: + enabled: true + actions: + defaults: + enabled: false + exceptions: + actionname1: + enabled: true + aliases: + defaults: + enabled: false + exceptions: + aliasname1: + enabled: true + rules: + defaults: + enabled: false + exceptions: + rulename1: + enabled: true + +.. rubric:: Example: Disabling all sensors in one pack + +For example, to disable all sensors in an individual pack then we would create a /opt/stackstorm/overrides/.yaml that contained: + +.. code-block:: bash + + --- + sensors: + defaults: + enabled: false + +.. rubric:: Example: Disabling all sensors in all packs, except for one pack + +If, instead we wanted to override all sensors except for a single pack, then we would instead create a /opt/stackstorm/overrides/global.yaml to disable all ensors: + +.. code-block:: bash + + --- + sensors: + defaults: + enabled: false + +And an /opt/stackstorm/overrides/.yaml to enable the sensors for the one pack we needed: + +.. code-block:: bash + + --- + sensors: + defaults: + enabled: true + +.. rubric:: Example: Overriding state of individual resources + +In this next example, we disable all actions in pack1, except for the actions action1 and action2. We also change the state of just a single rule to disabled. To do this, we create a /opt/stackstorm/overrides/pack1.yaml with: + +.. code-block:: bash + + --- + actions: + defaults: + enabled: false + exceptions: + action1: + enabled: true + action2: + enabled: true + rules: + exceptions: + rule1: + enabled: false + + Developing a Pack ----------------- From 9540edaca1b5495f62ea37349a36f24935d22a75 Mon Sep 17 00:00:00 2001 From: amanda Date: Fri, 11 Feb 2022 16:30:15 +0000 Subject: [PATCH 2/9] Add examples of output to indicate overriding taking place --- docs/source/packs.rst | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/docs/source/packs.rst b/docs/source/packs.rst index 6be0ea96..0c4a0bc1 100644 --- a/docs/source/packs.rst +++ b/docs/source/packs.rst @@ -277,6 +277,44 @@ Upon a pack install or reload resource state is managed as follows: rulename1: enabled: true +If overrides are taking place, then the number of resources affected will be output on the st2ctl reload output, for example: + +.. code-block:: bash + + $ st2ctl reload --register-all + Registering content...[flags = --config-file /etc/st2/st2.conf --register-all] + 2022-02-07 12:56:43,694 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "None". + 2022-02-07 12:56:43,704 INFO [-] Successfully connected to database "st2" @ "127.0.0.1:27017" as user "None". + 2022-02-07 12:56:44,254 INFO [-] ========================================================= + 2022-02-07 12:56:44,254 INFO [-] ############## Registering triggers ##################### + 2022-02-07 12:56:44,254 INFO [-] ========================================================= + 2022-02-07 12:56:44,549 INFO [-] Registered 1 triggers. + 2022-02-07 12:56:44,549 INFO [-] ========================================================= + 2022-02-07 12:56:44,549 INFO [-] ############## Registering sensors ###################### + 2022-02-07 12:56:44,549 INFO [-] ========================================================= + 2022-02-07 12:56:44,832 INFO [-] Registered 9 sensors. + 2022-02-07 12:56:44,832 INFO [-] 7 sensors had their metadata overridden. + ... + +On a pack install, then the number of resources that have their metadata overridden will be reported in the output of the register_pack action by the "(overridden)" entries. If no resources of that type have their metadata overridden, then there will be no correspdoning overridden entry outputted: + + +.. code-block:: bash + + { + "runners": 14, + "rule_types": 2, + "policy_types": 3, + "triggers": 0, + "sensors": 2, + "sensors(overridden)": 2, + "actions": 1, + "rules": 1, + "aliases": 0, + "policies": 0, + "configs": 0 + } + .. rubric:: Example: Disabling all sensors in one pack For example, to disable all sensors in an individual pack then we would create a /opt/stackstorm/overrides/.yaml that contained: From 9af2b5572eb38ac8df5dd299676b064567fbfbc4 Mon Sep 17 00:00:00 2001 From: Amanda McGuinness Date: Thu, 17 Feb 2022 12:23:03 +0000 Subject: [PATCH 3/9] Apply suggestions from code review Co-authored-by: Eugen Cusmaunsa --- docs/source/packs.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/source/packs.rst b/docs/source/packs.rst index 0c4a0bc1..e12242bc 100644 --- a/docs/source/packs.rst +++ b/docs/source/packs.rst @@ -222,12 +222,12 @@ Prior, to |st2| 3.7.0 then this state can be altered, by: In |st2| 3.7.0 we have introduced the override feature, so that the metadata of a packs resources can be overridden by configuration files. This will always be read upon a reload or pack install. The ST2 APIs will still allow you to override this, but as before any changes made by the |st2| APIs to enable/disable resources is forgotten upon reload or re-install. -The override facility is currently restricted to allowing the enabled property to be overridden, and is controlled by the /opt/stackstorm/overrides directory. +The override facility is currently restricted to allowing the enabled property to be overridden, and is controlled by the ``/opt/stackstorm/overrides`` directory. Upon a pack install or reload resource state is managed as follows: -* State is read from the pack's resource metadata files that reside in /opt/stackstorm/packs/. These are downloaded from the relevant repository of the pack, e.g. GIT, StackStorm-Exchange. -* If /opt/stackstorm/overrides/global.yaml is present, then any global overrides are applied. The global.yaml allows you to specify the default state of a particular resource types, e.g. disable all sensors. +* State is read from the pack's resource metadata files that reside in ``/opt/stackstorm/packs/``. These are downloaded from the relevant repository of the pack, e.g. GIT, StackStorm-Exchange. +* If ``/opt/stackstorm/overrides/global.yaml`` is present, then any global overrides are applied. The ``global.yaml`` allows you to specify the default state of a particular resource types, e.g. disable all sensors. The format of the global.yaml is as follows (set to disable everything): .. code-block:: bash @@ -246,8 +246,8 @@ Upon a pack install or reload resource state is managed as follows: defaults: enabled: false -* If /opt/stackstorm/overrides/.yaml is present, then any any pack defaults, or resource specific overrides are applied. The .yaml allows you to alter the state of all the resources of a particular type, via the defaults property (e.g. all sensors within the pack), or by individual resource type, via the exceptions property (e.g. only alter state of individual actions). - The format of the .yaml is as follows (set to disable everything, except for individual resource of each type): +* If ``/opt/stackstorm/overrides/.yaml`` is present, then any pack defaults, or resource specific overrides are applied. The .yaml allows you to alter the state of all the resources of a particular type, via the defaults property (e.g. all sensors within the pack), or by individual resource type, via the exceptions property (e.g. only alter state of individual actions). + The format of the ``.yaml`` is as follows (set to disable everything, except for individual resource of each type): .. code-block:: bash @@ -277,7 +277,7 @@ Upon a pack install or reload resource state is managed as follows: rulename1: enabled: true -If overrides are taking place, then the number of resources affected will be output on the st2ctl reload output, for example: +If overrides are taking place, then the number of resources affected will be output on the ``st2ctl reload`` output, for example: .. code-block:: bash @@ -317,7 +317,7 @@ On a pack install, then the number of resources that have their metadata overrid .. rubric:: Example: Disabling all sensors in one pack -For example, to disable all sensors in an individual pack then we would create a /opt/stackstorm/overrides/.yaml that contained: +For example, to disable all sensors in an individual pack then we would create a ``/opt/stackstorm/overrides/.yaml`` that contained: .. code-block:: bash @@ -328,7 +328,7 @@ For example, to disable all sensors in an individual pack then we would create a .. rubric:: Example: Disabling all sensors in all packs, except for one pack -If, instead we wanted to override all sensors except for a single pack, then we would instead create a /opt/stackstorm/overrides/global.yaml to disable all ensors: +If, instead we wanted to override all sensors except for a single pack, then we would instead create a ``/opt/stackstorm/overrides/global.yaml`` to disable all sensors: .. code-block:: bash @@ -337,7 +337,7 @@ If, instead we wanted to override all sensors except for a single pack, then we defaults: enabled: false -And an /opt/stackstorm/overrides/.yaml to enable the sensors for the one pack we needed: +And an ``/opt/stackstorm/overrides/.yaml`` to enable the sensors for the one pack we needed: .. code-block:: bash @@ -348,7 +348,7 @@ And an /opt/stackstorm/overrides/.yaml to enable the sensors for the o .. rubric:: Example: Overriding state of individual resources -In this next example, we disable all actions in pack1, except for the actions action1 and action2. We also change the state of just a single rule to disabled. To do this, we create a /opt/stackstorm/overrides/pack1.yaml with: +In this next example, we disable all actions in ``pack1``, except for the actions ``action1`` and ``action2``. We also change the state of just a single rule to disabled. To do this, we create a ``/opt/stackstorm/overrides/pack1.yaml`` with: .. code-block:: bash From 5f9c9c7c465512f86f9835593b002da6ae846d6b Mon Sep 17 00:00:00 2001 From: Amanda McGuinness Date: Thu, 17 Feb 2022 12:25:28 +0000 Subject: [PATCH 4/9] Update packs.rst --- docs/source/packs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/packs.rst b/docs/source/packs.rst index e12242bc..8e0b32bd 100644 --- a/docs/source/packs.rst +++ b/docs/source/packs.rst @@ -217,7 +217,7 @@ When installing a pack the state of the resources is taken from the metadata fil Prior, to |st2| 3.7.0 then this state can be altered, by: - * using the |st2| APIs to disable the resource. However, this will be forgotten upon a st2ctl reload or pack reinstall. + * using the |st2| APIs to disable the resource. However, this will be forgotten upon a ``st2ctl reload`` or pack reinstall. * changing the metadata file manually. This would be lost upon an upgrade, and cannot be easily tracked. In |st2| 3.7.0 we have introduced the override feature, so that the metadata of a packs resources can be overridden by configuration files. This will always be read upon a reload or pack install. The ST2 APIs will still allow you to override this, but as before any changes made by the |st2| APIs to enable/disable resources is forgotten upon reload or re-install. From 24e7a263db705d05507b01f8e2951b09fa012917 Mon Sep 17 00:00:00 2001 From: amanda Date: Fri, 18 Feb 2022 11:52:16 +0000 Subject: [PATCH 5/9] Mention resource types restricted to --- docs/source/packs.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/packs.rst b/docs/source/packs.rst index 8e0b32bd..cdd6376a 100644 --- a/docs/source/packs.rst +++ b/docs/source/packs.rst @@ -222,7 +222,7 @@ Prior, to |st2| 3.7.0 then this state can be altered, by: In |st2| 3.7.0 we have introduced the override feature, so that the metadata of a packs resources can be overridden by configuration files. This will always be read upon a reload or pack install. The ST2 APIs will still allow you to override this, but as before any changes made by the |st2| APIs to enable/disable resources is forgotten upon reload or re-install. -The override facility is currently restricted to allowing the enabled property to be overridden, and is controlled by the ``/opt/stackstorm/overrides`` directory. +The override facility is currently restricted to allowing the enabled property to be overridden only for resources of type: action, alias, rule and sensor. It is controlled by the ``/opt/stackstorm/overrides`` directory. Upon a pack install or reload resource state is managed as follows: From 70e6b54c8548a786610328d83a2452ad8d2203aa Mon Sep 17 00:00:00 2001 From: amanda Date: Fri, 4 Mar 2022 14:55:45 +0000 Subject: [PATCH 6/9] Add note about global pack name reservation --- docs/source/upgrade_notes.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index dc44a501..f0fe44f9 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -3,6 +3,15 @@ Upgrade Notes ============= +.. _ref-upgrade-notes-v3-7: + +|st2| v3.7 +---------- + +* As part of introducing the override pack metadata functionality, the name ``global`` is + reserved, and cannot be used for pack names or pack references, to avoid conflict between + the global override file and individual pack override files. + .. _ref-upgrade-notes-v3-6: |st2| v3.6 From 27b781fdf6303f7c40d7e3d55bc63a998999fd90 Mon Sep 17 00:00:00 2001 From: amanda Date: Fri, 4 Mar 2022 16:07:32 +0000 Subject: [PATCH 7/9] Add global reserved to develop pack section --- docs/source/reference/packs.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/source/reference/packs.rst b/docs/source/reference/packs.rst index 275cf473..cad6e292 100644 --- a/docs/source/reference/packs.rst +++ b/docs/source/reference/packs.rst @@ -169,6 +169,8 @@ available at :github_st2:`st2/contrib/hello_st2 `. ``name`` value in ``pack.yaml`` must only contain lowercase letters, digits, and underscores, unless you set the ``ref`` value explicitly in ``pack.yaml``. Finally the email attribute in ``pack.yaml`` must contain a properly formatted email address. + + From |st2| 3.7.0, the name ``global`` is reserved, and cannot be used as a pack name. The ``python_versions`` field declares which Python versions this pack supports. As |st2| only supports Python **3** from version 3.4.0 onwards, packs submitted to the StackStorm From 922f6141b141cba104e907e5678965e7616058b9 Mon Sep 17 00:00:00 2001 From: amanda Date: Mon, 7 Mar 2022 13:33:50 +0000 Subject: [PATCH 8/9] Rename global overrides to _global --- docs/source/packs.rst | 6 +++--- docs/source/reference/packs.rst | 2 +- docs/source/upgrade_notes.rst | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/packs.rst b/docs/source/packs.rst index cdd6376a..b9c413ed 100644 --- a/docs/source/packs.rst +++ b/docs/source/packs.rst @@ -227,8 +227,8 @@ The override facility is currently restricted to allowing the enabled property t Upon a pack install or reload resource state is managed as follows: * State is read from the pack's resource metadata files that reside in ``/opt/stackstorm/packs/``. These are downloaded from the relevant repository of the pack, e.g. GIT, StackStorm-Exchange. -* If ``/opt/stackstorm/overrides/global.yaml`` is present, then any global overrides are applied. The ``global.yaml`` allows you to specify the default state of a particular resource types, e.g. disable all sensors. - The format of the global.yaml is as follows (set to disable everything): +* If ``/opt/stackstorm/overrides/_global.yaml`` is present, then any global overrides are applied. The ``_global.yaml`` allows you to specify the default state of a particular resource types, e.g. disable all sensors. + The format of the _global.yaml is as follows (set to disable everything): .. code-block:: bash @@ -328,7 +328,7 @@ For example, to disable all sensors in an individual pack then we would create a .. rubric:: Example: Disabling all sensors in all packs, except for one pack -If, instead we wanted to override all sensors except for a single pack, then we would instead create a ``/opt/stackstorm/overrides/global.yaml`` to disable all sensors: +If, instead we wanted to override all sensors except for a single pack, then we would instead create a ``/opt/stackstorm/overrides/_global.yaml`` to disable all sensors: .. code-block:: bash diff --git a/docs/source/reference/packs.rst b/docs/source/reference/packs.rst index cad6e292..98bef1f4 100644 --- a/docs/source/reference/packs.rst +++ b/docs/source/reference/packs.rst @@ -170,7 +170,7 @@ available at :github_st2:`st2/contrib/hello_st2 `. unless you set the ``ref`` value explicitly in ``pack.yaml``. Finally the email attribute in ``pack.yaml`` must contain a properly formatted email address. - From |st2| 3.7.0, the name ``global`` is reserved, and cannot be used as a pack name. + From |st2| 3.7.0, the name ``_global`` is reserved, and cannot be used as a pack name. The ``python_versions`` field declares which Python versions this pack supports. As |st2| only supports Python **3** from version 3.4.0 onwards, packs submitted to the StackStorm diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index f0fe44f9..485cacfc 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -8,7 +8,7 @@ Upgrade Notes |st2| v3.7 ---------- -* As part of introducing the override pack metadata functionality, the name ``global`` is +* As part of introducing the override pack metadata functionality, the name ``_global`` is reserved, and cannot be used for pack names or pack references, to avoid conflict between the global override file and individual pack override files. From 0487d36ba36784d073430de1d034868087b63272 Mon Sep 17 00:00:00 2001 From: amanda Date: Tue, 8 Mar 2022 15:40:31 +0000 Subject: [PATCH 9/9] Add review comment --- docs/source/packs.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/packs.rst b/docs/source/packs.rst index b9c413ed..3e825757 100644 --- a/docs/source/packs.rst +++ b/docs/source/packs.rst @@ -246,7 +246,7 @@ Upon a pack install or reload resource state is managed as follows: defaults: enabled: false -* If ``/opt/stackstorm/overrides/.yaml`` is present, then any pack defaults, or resource specific overrides are applied. The .yaml allows you to alter the state of all the resources of a particular type, via the defaults property (e.g. all sensors within the pack), or by individual resource type, via the exceptions property (e.g. only alter state of individual actions). +* If ``/opt/stackstorm/overrides/.yaml`` is present, then any pack defaults, or resource specific overrides are applied. The .yaml allows you to alter the state of all the resources of a particular type, via the defaults property (e.g. all sensors within the pack), or by individual resource type, via the exceptions property (e.g. only alter state of individual actions). As the pack name is determined by the name of the overrides file, the name of the resource in the file should omit the pack name, e.g. ``action1`` in the ``pack1.yaml`` refers to the resource ``pack1.action1``. The format of the ``.yaml`` is as follows (set to disable everything, except for individual resource of each type): .. code-block:: bash @@ -348,7 +348,7 @@ And an ``/opt/stackstorm/overrides/.yaml`` to enable the sensors for t .. rubric:: Example: Overriding state of individual resources -In this next example, we disable all actions in ``pack1``, except for the actions ``action1`` and ``action2``. We also change the state of just a single rule to disabled. To do this, we create a ``/opt/stackstorm/overrides/pack1.yaml`` with: +In this next example, we disable all actions in ``pack1``, except for the actions ``pack1.action1`` and ``pack1.action2``. We also change the state of just a single rule to disabled. To do this, we create a ``/opt/stackstorm/overrides/pack1.yaml`` with: .. code-block:: bash