diff --git a/README.md b/README.md index fc4891d3a..804a5c9a8 100644 --- a/README.md +++ b/README.md @@ -145,14 +145,17 @@ sphinx-autobuild -H 127.0.0.1 -b html ./docs/source/ ./docs/build/html :github_st2:`st2/st2common/st2common/operators.py ` -* Link to Github st2contrib repo: +* Link to Github StackStorm-Exchange org: - :github_contrib:`Link to docker README on st2contrib` + :github_exchange:`Link to a sensu pack repo inside Exchange` -* Link to st2contrib and st2incubator repos on Github (using a global we set up in source/conf.py) +* Link to StackStorm Exchange website with a filter query: - `st2contrib`_ - `st2incubator`_ + :web_exchange:`Sensu` + +* Link to the Exchange website on Github (using a global we set up in source/conf.py) + + `exchange`_ * The pattern to include an example from `/st2/contrib/examples`: make example file name a reference on github. May say that it is deployed to `/usr/share/doc/st2/examples/`, and auto-include the file: diff --git a/docs/source/_static/images/packs-chatops-get.png b/docs/source/_static/images/packs-chatops-get.png new file mode 100644 index 000000000..cccd96efc Binary files /dev/null and b/docs/source/_static/images/packs-chatops-get.png differ diff --git a/docs/source/_static/images/packs-chatops-install.png b/docs/source/_static/images/packs-chatops-install.png new file mode 100644 index 000000000..3473b1926 Binary files /dev/null and b/docs/source/_static/images/packs-chatops-install.png differ diff --git a/docs/source/_static/images/packs-chatops-search.png b/docs/source/_static/images/packs-chatops-search.png new file mode 100644 index 000000000..8eb2c4c8e Binary files /dev/null and b/docs/source/_static/images/packs-chatops-search.png differ diff --git a/docs/source/_static/images/packs-chatops-show.png b/docs/source/_static/images/packs-chatops-show.png new file mode 100644 index 000000000..e463c8c6a Binary files /dev/null and b/docs/source/_static/images/packs-chatops-show.png differ diff --git a/docs/source/actions.rst b/docs/source/actions.rst index f9f3d0801..d95e60fb0 100644 --- a/docs/source/actions.rst +++ b/docs/source/actions.rst @@ -552,7 +552,7 @@ passed in, action will be considered as failed (first flag in the result tuple indicating action status is ``False``). For a more complex example, please refer to the `actions in the Libcloud pack in -the contrib repository `_. +StackStorm Exchange `_. Configuration File ~~~~~~~~~~~~~~~~~~ @@ -663,7 +663,7 @@ To see all predefined actions: .. rubric:: What's Next? -* Explore packs and actions contributed by |st2| developers and community in the `StackStorm st2contrib repo on Github `_. +* Explore packs and actions contributed by |st2| developers and community in the `StackStorm Exchange `_. * Check out `tutorials on stackstorm.com `__ on creating actions, and other practical examples of automating with |st2|. .. _JSON Schema: http://json-schema.org/documentation.html diff --git a/docs/source/chatops/notifications.rst b/docs/source/chatops/notifications.rst index 19685eec9..6ba525e17 100644 --- a/docs/source/chatops/notifications.rst +++ b/docs/source/chatops/notifications.rst @@ -4,7 +4,7 @@ Notifications If you read through the :ref:`ref-chatops` section, you are familiar with notifications. Even without ChatOps, notifications can be used to post messages to external systems like Chat clients, send emails etc. Notifications require an action that is registered with -|st2| (e.g., the `post_message `_ action in the the ``slack`` pack) +|st2| (e.g., the `post_message `_ action in the the ``slack`` pack) and a notification rule to go with it. Notifications are implemented as triggers, rules and actions. A special ``core.st2.notifytrigger`` is emitted by the system on completion of every action. A rule to match the trigger to a notify action results in notifications being sent out. diff --git a/docs/source/chatops/pack_deploy.rst b/docs/source/chatops/pack_deploy.rst index 274d5e322..fa986a712 100644 --- a/docs/source/chatops/pack_deploy.rst +++ b/docs/source/chatops/pack_deploy.rst @@ -1,89 +1,78 @@ Packs ===== -Installing packs from st2incubator or st2contrib via ChatOps ------------------------------------------------------------- +Installing packs with ChatOps +----------------------------- -When you have the bot listening for ChatOps commands in a channel, installing extra packs -can be done by running a single command: +When you have the bot listening for ChatOps commands in a channel, installing extra packs +from StackStorm Exchange can be done by running a single command: .. code-block:: bash - ! pack deploy st2contrib elasticsearch,travis_ci - bot: Deploying the requested pack(s) from *st2contrib* for you.... - @my_user: Successful deployment of *elasticsearch* *travis_ci* ! - > from https://github.com/StackStorm/st2contrib.git (branch: _master_). + !pack install github,slack,trello + bot: Installing the requested pack(s) for you. + @my_user: + > Successful deployment of *github*, *slack*, *trello* packs! -If you're adventurous and wish to install and help develop a pack that's -currently in st2incubator you can install one with the following command: +You can install a pack from any github repository just as easily: supply the full URL +to the bot. Pack name is unnecessary, as it will be read from ``pack.yaml`` later: .. code-block:: bash - ! pack deploy st2incubator vsphere,debian - Deploying the requested pack(s) from *st2incubator* for you - @my_user: Successful deployment of *vsphere* *debian* ! - > from https://github.com/StackStorm/st2incubator.git (branch: _master_). + !pack install https://github.com/stackstorm/openstack + bot: Installing the requested pack(s) for you. -The command takes the following very simple format: +You can even mix the two formats in one string: .. code-block:: bash - - ! pack deploy {{repo_name}} {{packs}} {{branch=master}} - Download StackStorm packs via ChatOps -Deploying Custom packs via ChatOps ----------------------------------- + !pack install github,slack,trello,https://github.com/stackstorm/openstack -The same commands can be used to install your own packs just by adding an entry to `/opt/stackstorm/packs/packs/config.yaml` in the following format: +.. figure :: /_static/images/packs-chatops-install.png + :align: center -.. code-block:: yaml - --- - repositories: - NameOfRep: - repo: "https://github.com//my-st2.git" - subtree: true +Getting information about an installed pack +------------------------------------------- -This will allow you to install a pack via: +The ``!pack get `` command shows you information about an installed +pack. There is two sections in the output: the first is pack metadata from +``pack.yaml``, and the optional second is git information, if your pack has +been installed from a git source like StackStorm Exchange or a single repo. -.. code-block:: bash - - ! pack deploy NameOfRep MyAwesomePack - -If you don't have multiple packs within the same repository under a `packs` directory, just set `subtree` to `false` and issue the following command: - -.. code-block:: bash +Git status will tell you if there's a difference between your local pack version +and the latest version in the origin repository, and also show the remotes +for your pack. - ! pack deploy NameOfRep NameOfRep +If a pack is not installed, but available in StackStorm Exchange, the bot will +gallantly offer to install it: -Automating Custom Pack Deployment ---------------------------------- +.. figure :: /_static/images/packs-chatops-get.png + :align: center -Building on the above it's possible to enable auto deployment for a single branch of a -repository which has has `subtree` set to `false` by adding an `auto_deployment` section as shown below: +Getting information about an available pack +------------------------------------------- -.. code-block:: yaml +The remote counterpart to ``pack get`` is ``pack show``: it will show an +entry from the StackStorm Exchange, our pack directory, if a pack with the +given name is available. - --- - repositories: - my-st2: - repo: "https://github.com//my-st2.git" - subtree: false - auto_deployment: - branch: "master" - notify_channel: "my-chatops-channel" +.. figure :: /_static/images/packs-chatops-show.png + :align: center -Then you need a rule (or a sensor) that will trigger the `packs.deploy` action with the right -parameters. The following is based on an post-commit hook from BitBucket Server: +Searching for a pack +-------------------- -.. code-block:: yaml +To search for a pack in StackStorm Exchange, use ``!pack search ``. Note that +your query will match results in all pack parameters: for example, you can search for +an author or a keyword, not just name and description. The results are ordered by +relevance: if you search for "cloud", first you will get packs with "cloud" in their +name—because they are the most likely to be what you were looking for—and then packs +with "cloud" in keywords or description. And then packs authored by "Mr. Cloud", if +any. - action: - ref: "packs.deploy" - parameters: - auto_deploy: true - repo_name: "{{trigger.body.repository.name}}" - branch: "{{trigger.body.refChanges[0].refId}}" - packs: [ "{{trigger.body.repository.name}}" ] - message: "{{trigger.body.changesets.get('values')[0].toCommit.message}}" - author: "{{trigger.body.changesets.get('values')[0].toCommit.author.name}}" +**Pro-tip:** if you configure an additional pack index (see "Working with pack indexes" in :doc:`/packs`), +it will be queried alongside StackStorm Exchange by commands like ``show`` or ``search``. +.. figure :: /_static/images/packs-chatops-search.png + :align: center diff --git a/docs/source/conf.py b/docs/source/conf.py index d70ea052b..a6a9b0855 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -106,10 +106,14 @@ def previous_version(ver): 'github_mistral': ('https://github.com/StackStorm/mistral/tree/master/%s', None), 'github_contrib': ('https://github.com/StackStorm/st2contrib/tree/master/%s', None), + 'github_exchange': + ('https://github.com/StackStorm-Exchange/%s', None), 'github_devenv': ('https://github.com/StackStorm/devenv/tree/master/%s', None), 'github_st2web': ('https://github.com/StackStorm/st2web/tree/master/%s', None), 'ops_latest': - ('https://downloads.stackstorm.net/releases/st2/' + release + '/%s/', None) + ('https://downloads.stackstorm.net/releases/st2/' + release + '/%s/', None), + 'web_exchange': + ('https://exchange.stackstorm.org/#%s', None), } # Inserted at the bottom of all rst files. @@ -124,8 +128,7 @@ def previous_version(ver): rst_epilog = """ %s -.. _st2contrib: http://www.github.com/stackstorm/st2contrib -.. _st2incubator: http://www.github.com/stackstorm/st2incubator +.. _exchange: https://exchange.stackstorm.org/ .. |bwc| replace:: Brocade Workflow Composer .. |ipf| replace:: IP Fabric Automation Suite """ % product_replace diff --git a/docs/source/install/config/config.rst b/docs/source/install/config/config.rst index 35c2609e2..454592cf9 100644 --- a/docs/source/install/config/config.rst +++ b/docs/source/install/config/config.rst @@ -245,10 +245,10 @@ By default, the logs can be found in ``/var/log/st2``. args=("logs/mysensor.audit.log",) * To configure logging with syslog, grab the configuration and follow - instructions at :github_contrib:`st2contrib/extra/syslog ` + instructions at :github_exchange:`exchange-misc/syslog ` * Check out LogStash configuration and Kibana dashboard for pretty logging and - audit at :github_contrib:`st2contrib/extra/logstash ` + audit at :github_exchange:`exchange-misc/logstash ` Configure Mistral diff --git a/docs/source/install/deb.rst b/docs/source/install/deb.rst index c9d3f1a47..d38006eb2 100644 --- a/docs/source/install/deb.rst +++ b/docs/source/install/deb.rst @@ -1,7 +1,7 @@ Ubuntu / Debian =============== -If you're just looking for a "one-liner" installation, check the :doc:`top-level install guide `. Otherwise, you +If you're just looking for a "one-liner" installation, check the :doc:`top-level install guide `. Otherwise, you can use this guide for step-by step instructions for installing |st2| on a single Ubuntu/Debian 64 bit system as per the :doc:`Reference deployment `. @@ -111,7 +111,7 @@ Verify ----------------- At this point you have a minimal working installation, and can happily play with |st2|: -follow :doc:`/start` tutorial, :ref:`deploy examples `, explore and install packs from `st2contrib`_. +follow :doc:`/start` tutorial, :ref:`deploy examples `, explore and install packs from `StackStorm Exchange `__. But there is no joy without WebUI, no security without SSL termination, no fun without ChatOps, and no money without Brocade Workflow Composer. Read on, move on! diff --git a/docs/source/install/rhel6.rst b/docs/source/install/rhel6.rst index 44322f4b2..66ac5c69a 100644 --- a/docs/source/install/rhel6.rst +++ b/docs/source/install/rhel6.rst @@ -1,7 +1,7 @@ RHEL 6 / CentOS 6 ================= -If you're just looking for a "one-liner" installation, check the :doc:`top-level install guide `. Otherwise, you +If you're just looking for a "one-liner" installation, check the :doc:`top-level install guide `. Otherwise, you can use this guide for step-by step instructions for installing |st2| on a single RHEL 6/CentOS 6 64 bit system per the :doc:`Reference deployment `. @@ -181,7 +181,7 @@ Verify At this point you have a minimal working installation, and can happily play with |st2|: follow :doc:`/start` tutorial, :ref:`deploy examples `, explore and install packs -from `st2contrib`_. +from `StackStorm Exchange `__. But there is no joy without WebUI, no security without SSL termination, no fun without ChatOps, and no money without Brocade Workflow Composer. Read on, move on! diff --git a/docs/source/install/rhel7.rst b/docs/source/install/rhel7.rst index 0d7b5ab8a..0377f0eab 100644 --- a/docs/source/install/rhel7.rst +++ b/docs/source/install/rhel7.rst @@ -1,7 +1,7 @@ RHEL 7 / CentOS 7 ================= -If you're just looking for a "one-liner" installation, check the :doc:`top-level install guide `. Otherwise, you +If you're just looking for a "one-liner" installation, check the :doc:`top-level install guide `. Otherwise, you can use this guide for step-by step instructions for installing |st2| on a single RHEL 7/CentOS 7 64 bit system per the :doc:`Reference deployment `. @@ -153,7 +153,7 @@ Verify ----------------- At this point you have a minimal working installation, and can happily play with |st2|: -follow :doc:`/start` tutorial, :ref:`deploy examples `, explore and install packs from `st2contrib`_. +follow :doc:`/start` tutorial, :ref:`deploy examples `, explore and install packs from `StackStorm Exchange `__. But there is no joy without WebUI, no security without SSL termination, no fun without ChatOps, and no money without Brocade Workflow Composer. Read on, move on! diff --git a/docs/source/packs.rst b/docs/source/packs.rst index 8bcc767da..3d20dfaf4 100644 --- a/docs/source/packs.rst +++ b/docs/source/packs.rst @@ -6,114 +6,184 @@ What is a Pack? Pack is the unit of deployment for integrations and automations that extend |st2|. Typically a pack is organized along service or product boundaries e.g. AWS, Docker, Sensu etc. A pack can contain :doc:`Actions `, :doc:`Workflows `, :doc:`Rules `, :doc:`Sensors `, :doc:`Aliases `. -It is best to view a pack as the means to extend |st2| and allow it to integrate with an external systems. See `next section` to learn more about pack management. +It is best to view a pack as the means to extend |st2| and allow it to integrate with external systems. Everything you create will also be part of a pack. -Packs Location and Discovery ----------------------------- +Managing Packs +-------------- + +.. note:: + + Everything about packs got better and easier in |st2| 2.1! There are new API endpoints, CLI commands, repository + structure for hosting packs, and a pack collection called StackStorm Exchange. Some of the older things are now deprecated. If you're using a previous version, it's time to upgrade or at least read the :doc:`upgrade notes ` and :doc:`/reference/packmgmt`. + +|st2| packs are managed through ``st2 pack ...`` commands: ``st2 pack -h`` will give you a useful overview if you just need a quick start. + +A few packs (such as the ``core`` pack for basic StackStorm actions) come pre-installed with StackStorm. ``list`` and ``get`` are the primary commands to work with local packs: + +.. code-block:: bash + + # List all installed packs + st2 pack list + + # Get detailed information about an installed pack + st2 pack get core -When using |st2| and pack management actions, all the packs are by default installed into the +When using |st2| and pack management actions, all the packs are installed into the system packs directory which defaults to ``/opt/stackstorm/packs``. -When |st2| searches for available packs it looks into the system packs directory and -into any additional directories which are specified in the ``packs_base_paths`` setting. +Discovering Packs +----------------- -To look for packs in additional directories, set the value of ``packs_base_paths`` in ``st2.conf`` -(typically in :github_st2:`/etc/st2/st2.conf `, as described in -:doc:`Configuration `). The value must be a colon delimited string of directory paths. +There's over a hundred StackStorm packs already available to you! `StackStorm Exchange `__ is a collection of ready-made packs submitted by StackStorm users and engineers, and there are packs for most of the popular cloud providers and DevOps tools. -For example: +In addition to the pack listing at `exchange.stackstorm.org `__, you can search the pack index in CLI with ``st2 pack search`` and ``st2 pack show``: -:: +.. code-block:: bash - [content] - packs_base_paths=/home/myuser1/packs:/home/myuser2/packs + # Search query is applied across all pack parameters. -Note: Directories are always searched from left to right in the order they are -specified, with the system packs directory always searched first. + # It will search through pack names: + st2 pack search sensu + # And keywords: + st2 pack search monitoring + # And description (use quotes for multi-word search): + st2 pack search "Amazon Web Services" + # And even pack author: + st2 pack search "Jon Middleton" -Getting a Pack --------------- + # Show an index entry for the pack + st2 pack show sensu -Pack management is done by |st2| actions from `packs` pack, pun intended. Run ``st2 action list --pack packs`` for a list of pack management actions. +Installing a Pack +----------------- -Some packs can be installed and run "as is" without any configurations. +Installing a pack is simple: .. code-block:: bash - st2 run packs.install packs=docker,sensu repo_url=https://github.com/StackStorm/st2contrib.git + st2 pack install sensu -This downloads the Sensu and Docker packs from the `StackStorm/st2contrib community repo on GitHub `__, places them as local content under ``/opt/stackstorm/packs``, registers with |st2| and loads the content. + # You can also install multiple packs: + st2 pack install datadog github -By default packs are installed from the |st2| community repo. Use ``repo_url`` parameter to install a pack from a fork of `st2contrib `__, or from a custom repo. The following example installs all the packs from `StackStorm/st2incubator `__ - the repo where you find our experiments and work-in-progress: +This command will download packs from the `StackStorm Exchange organization on GitHub `__, place them under ``/opt/stackstorm/packs``, and register them with |st2|. + +Essentially, ``st2 pack install`` works with git repositories: there is one for every pack in the Exchange, and you can install your own packs from git just as easily. .. code-block:: bash - st2 run packs.install packs=* register=all repo_url=https://github.com/StackStorm/st2incubator.git + st2 pack install https://github.com/emedvedev/chatops-training -If you are using a private repo to host your pack(s) on github, use: -``repo_url=https://username:password@github.com/username/repository.git``. -You can optionally pass ``subtree=True`` if the repo contains multiple packs within ``packs`` directory -of repo root just like `StackStorm/st2contrib community repo on GitHub `_. -If the repo contains only one pack at the repo root, then use ``subtree=False``. +By default, the latest version of a pack will be installed, but you can specify a particular version, branch, tag, or even a commit hash. -To uninstall packs: ``st2 run packs.uninstall packs=docker,sensu``. This unloads and unregisters the content and deletes the packs from the disk. +.. code-block:: bash -The integration packs often require configurations to adjust to the environment. e.g. you will need to specify SMTP server for email, a puppet master URL for Puppet, or a Keystone endpoint and tenant credentials for OpenStack. The installation process is: + st2 pack install cloudflare=77ee04e + st2 pack install cloudflare=0.1.0 + st2 pack install https://github.com/emedvedev/chatops-training=testing -1. Download the pack with ``packs.download`` -2. Check out the `README.md`. Adjust configurations per your environment. For - more information on pack configuration and how to configure the pack, please - refer to :doc:`/reference/pack_configs`. -3. Run pack setup via ``packs.setup_virtualenv``. It sets up a virtual environment and installs the dependencies listed in requirements.txt. -4. Load the pack into |st2| with ``pack.load register=all|actions|rules|sensors``. +Note that running ``st2 pack install`` on an installed pack will start an upgrade: your pack will be replaced with the version you're asking |st2| to install. -Let's install the Docker pack: +To uninstall a pack, use ``remove``: .. code-block:: bash - # Download Docker pack from http://github.com/stackstorm/st2contrib - st2 run packs.download packs=docker + st2 pack remove sensu - # Set up a virtual environment for this pack and install all the pack dependencies - # listed in requirements.txt (if any). - # Virtual environment provides isolated Python environment for sensors and Python runner - # actions. - st2 run packs.setup_virtualenv packs=docker +Configuring a Pack +------------------ - # Check out README.md and if necessary, adjust configuration for your environment - less /opt/stackstorm/packs/docker/README.md +Integration packs often require configuration to adjust to the environment. e.g. you will need to specify SMTP server for email, a puppet master URL for Puppet, or a Keystone endpoint and tenant credentials for OpenStack. - # Load ALL the content: actions, sensors, rules - # If you don't want to load sample rules by default, do - # st2 run packs.load register=sensors && st2 run packs.load register=actions - st2 run packs.load register=all +Most packs that require configuration can be configured interactively: - # To pick up sensors, we need to bounce the st2sensorcontainer process. - # Note: live update coming soon and this won't be needed. - st2 run packs.restart_component servicename=st2sensorcontainer +.. code-block:: bash - # Verify that the Docker pack was installed - st2 action list --pack=docker - st2 sensor list --pack=docker - st2 trigger list --pack=docker + st2 pack config cloudflare -The Docker pack is now installed and ready to use. +You will be prompted for configuration parameters in a nice interactive tool with descriptions, suggestions, and defaults. You will also be asked to verify your final config file in a text editor before saving it: it's optional, and most packs don't require more than two or three fields, but we have to comply with Health and Safety. -Packs may contain automations - rules and workflows. Rules are not loaded by default - you may want to review and adjust them before loading. Pass ``register=all`` option to ``packs.install`` and ``packs.load`` actions to get the rules loaded. Use ``st2ctl reload`` for fine control - ``packs.load`` is an st2 action wrapper around it. +Some packs will require old-school manual configuration: in that case, you will have to edit a ``config.yaml`` file inside the ``/opt/stackstorm/packs/`` directory and reload the pack with ``st2 pack register ``. Your config will not be overwritten on pack upgrades, and to be extra safe, you can save it as ``/opt/stackstorm/configs/.yaml`` to keep it away from the pack source, which is something we recommend. -.. note:: Pack management is implemented as a pack of st2 actions. Check out :github_st2:`/opt/stackstorm/packs ` for examples of defining actions and workflows. +If you edit files in a pack manually, make sure to reload its content for the file changes to be reflected in StackStorm: -Creating a Pack ---------------- +.. code-block:: bash + + st2 pack register sensu -See :doc:`/reference/packs` for details on how to package your integrations and automations in a pack, how to publish it, and how to contribute it to the |st2| community. +Developing a Pack +----------------- + +See :doc:`/reference/packs` for details on how to package your integrations and automations in a pack, how to fork a pack for development or create your own, how to publish it, and how to contribute it to the |st2| community. If you're planning to develop any |st2| content, we would strongly suggest getting yourself familiar with that page: every piece of content in StackStorm has to belong to a pack, and a good understanding of pack workflow will make your development process much easier! .. rubric:: What's Next? -* Explore existing packs for many common products and tools from `StackStorm community `__ - `st2contrib `__. +* Explore existing packs for many common products and tools: `StackStorm Exchange `__. * Learn how to write a pack and contribute to the community - :doc:`/reference/packs`. * Learn how to write :ref:`custom sensors ` and :ref:`custom actions `. * Check out `tutorials on stackstorm.com `__ - a growing set of practical examples of automating with |st2|. * For information on pack testing, please see the :doc:`Pack Testing ` page. +Advanced Topics +--------------- + +Installing packs from private repositories +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you're installing a pack from a private repository on GitHub, you can use https auth with a `personal access token `__, or create a `deploy key `__ to use SSH. + +Access tokens are used with HTTPS auth: ``st2 pack install https://:@github.com/username/repo.git``. + +Deploy keys are used with ``git@`` urls, and require the system user running the command (stanley or root, depending on your configuration) to have a private key, but they are more secure and can be configured on the per-repo basis. + +Other git hosting services should also support either SSH or HTTPS auth, and would be configured in a similar fashion. + +Pack location +~~~~~~~~~~~~~ + +When |st2| searches for available packs, it looks into the system packs directory and +into any additional directories which are specified in the ``packs_base_paths`` setting. + +To look for packs in additional directories, set the value of ``packs_base_paths`` in ``st2.conf`` +(typically in :github_st2:`/etc/st2/st2.conf `, as described in +:doc:`Configuration `). The value must be a colon delimited string of directory paths. + +For example: + +:: + + [content] + packs_base_paths=/home/myuser1/packs:/home/myuser2/packs + +Note: Directories are always searched from left to right in the order they are +specified, with the system packs directory always searched first. + +Working with pack indexes +~~~~~~~~~~~~~~~~~~~~~~~~~ + +When you run pack management commands like ``st2 pack install sensu`` or ``st2 pack search git``, |st2| uses a pack index file to resolve short names and perform searches. A pack index is, essentially, a JSON object: it contains metadata and URLs for all available packs. + +`The StackStorm Exchange index file `__ is a default file used by all |st2| instances, and a good example of the index format. The file is hosted on GitHub (`StackStorm-Exchange/index `__) and proxied through CloudFlare CDN. + +The index path is specified in ``st2.conf`` as ``content.index_url``. You can replace the default index, or even use more than one with a comma-separated list: + +:: + + [content] + index_url=https://my-super-index.org/index.json,https://exchange.stackstorm.org/v1/index.json + +Contents from all specified indexes will merge, priority arranged left to right. In the example above, ``sensu`` pack in your own index would override ``sensu`` pack from the Exchange. + +There are multiple reasons to consider hosting your own index, especially with HA deployments or multi-server setups: + +* mirroring: in case the main index is not available, your mirror will be used; +* forking: if you fork Exchange packs, you can create an index that is going to use your forks; +* enterprise restrictions: if you need pack names to resolve, but can't install from github, you can specify your own index as the only source; +* a centralized resolver: in a multi-server deployment, you can host an index to keep repo URLs in a centralized location; +* bragging rights: get your own packs resolvable by short names because the cool kids are doing it. + +In most cases there are many other ways to solve the same problem, but sometimes a pack index is a viable alternative. Create your index file and make it accessible over HTTPS, change the config—and you're good! + +When you have to monitor index health, the ``/packs/index/health`` API endpoint will show you the state of all indexes used by your |st2| instance. + .. include:: /__engage_community.rst diff --git a/docs/source/reference/history.rst b/docs/source/reference/history.rst index 81aaacf87..7e61388c2 100644 --- a/docs/source/reference/history.rst +++ b/docs/source/reference/history.rst @@ -82,7 +82,7 @@ Logstash The audit logs are a lot more comprehensive in the information they contain. It is useful to view these in a tool like LogStash, which provides excellent search, sort and aggregation features. -Check out the LogStash configuration and Kibana dashboard for pretty logging and audit at :github_contrib:`st2contrib/extra/logstash ` +Check out the LogStash configuration and Kibana dashboard for pretty logging and audit at :github_exchange:`exchange-misc/logstash ` Coming soon diff --git a/docs/source/reference/packmgmt.rst b/docs/source/reference/packmgmt.rst new file mode 100644 index 000000000..18600a74c --- /dev/null +++ b/docs/source/reference/packmgmt.rst @@ -0,0 +1,44 @@ +:orphan: + +Pack Management Transition +========================== + +Integration packs have received significant overhaul in StackStorm 2.1: with new dedicated tools, working with packs becomes very close to the "usual" package management you know from working with programming languages and operating systems. Installing, updating, and managing StackStorm packs has become a smoother, more streamlined experience. + +This page serves as a migration guide, as well as the list of features that are being changed or deprecated. If you are upgrading from 2.0 and earlier versions, read on. + + +StackStorm Exchange +------------------- + +`StackStorm Exchange `__ is the new pack directory maintained by the StackStorm team. Any pack from StackStorm Exchange can be installed with ``st2 pack install `` in CLI. + +All Exchange packs are hosted in the StackStorm-Exchange organization on GitHub: `StackStorm-Exchange `__. + +To submit your own pack to Exchange, follow the instructions in `exchange-incubator `__. + + +Earlier StackStorm versions +--------------------------- + +The old `st2contrib `__ repository is frozen: it is still functional, but it will not receive updates anymore, and new submissions are not accepted. This can be an inconvenience for people using the old StackStorm versions, but we want to make sure StackStorm packs is maintained in the best possible way; unfortunately, this also means we cannot support two separate pack directories. We strongly encourage everyone to update to 2.1 and try out the new pack experience. + +In case a StackStorm upgrade is not an option, the packs from the `StackStorm-Exchange `__ organization can still be installed with the ``packs.install`` action even on pre-2.1. + + +Migration notes +--------------- + +* The ``packs`` pack is deprecated: it is present in 2.1, but everyone is strongly encouraged to use the new CLI commands ``st2 pack <...>`` or API endpoints. + +* The ``packs.install`` parameters have been streamlined: ``subtree`` is not supported anymore, and the ``packs`` parameter now supports both pack names (resolved through Exchange) and pack URLs, as well as installing particular versions with the `=` separator. + +* Subtree repositories (repositories containing multiple packs inside the ``packs/`` subdir) are not supported anymore. Subtrees will have to be split into multiple single-pack repositories in order for StackStorm to be able to install the packs. + +* Pack versions in ``pack.yaml`` are finally required to conform to semver: ``0.1.0`` is acceptable, ``0.1`` is not. + +* Dashes in pack refs should be replaced with underscores. Note that you can now separate the ``name`` and ``ref`` attributes: if you don't like the short reference name, your pack can have a beautiful display name visible in Web UI and Flow. + +------------- + +.. include:: ../__engage_community.rst diff --git a/docs/source/reference/packs.rst b/docs/source/reference/packs.rst index bde9e2219..6fdda4142 100644 --- a/docs/source/reference/packs.rst +++ b/docs/source/reference/packs.rst @@ -86,7 +86,7 @@ The ``policies`` folder contains Policies. See :doc:`Policies `. +If you would like to create a pack yourself then follow these *simple* steps. In the example below, we will create a simple pack named **hello_st2**. The full example is also available at :github_st2:`st2/contrib/hello_st2 `. 1. First, let's create the pack folder structure and related files. Let's keep the metadata files such as pack.yaml, config.schema.yaml, and requirements.txt empty for now: @@ -149,18 +149,21 @@ Copy the following content to policies/policy1.yaml .. literalinclude:: /../../st2/contrib/hello_st2/policies/policy1.yaml -7. Deploy this pack manually: +7. Install this pack manually. Your pack should be versioned in git for this step to work, or you can move the files to ``/opt/stackstorm/packs``. It's up to you to figure out an optimal workflow for editing and versioning your packs, but we recommend using git and installing from your repositories. .. code-block:: bash - # Assuming that hello_st2 is on the same server - cp -R ./hello_st2 /opt/stackstorm/packs + # 1. If hello_st2 is on the same server. + st2 pack install ./hello_st2 + + # 2. If hello_st2 is hosted on github. + st2 pack install https://github.com/ - # Create virtual environment for hello_st2. - st2 run packs.setup_virtualenv packs=hello_st2 + # 3. If you want to move the pack without git. + cp -R ./hello_st2 /opt/stackstorm/packs - # Reloads the content - st2 run packs.load register=all + # Reload the content. + st2 pack register hello_st2 Once you follow steps 1-7 you will have created your first pack. Commands like ``st2 action list``, ``st2 rule list`` and ``st2 trigger list`` will show you the loaded content. To check if the sensor triggering action is working, run ``st2 execution list``, there should be an entry for executing ``hello_st2.greet`` every minute. @@ -170,45 +173,45 @@ Next steps would be to create an integration pack for you favorite tool or servi Pushing a Pack to the Community ------------------------------- -So, now you forged this uber-awesome pack in |st2|, what's next? Do you want to share your awesome pack and knowledge with the community? For this purpose we have created the `StackStorm community repo `__ where you can share and pull other content packs. Submit a pull request! Here are the steps: +So, now you forged this uber-awesome pack in |st2|, what's next? Do you want to share your awesome pack and knowledge with the community? For this purpose we have created the `StackStorm Exchange `__ where you can share and pull other content packs. Submit a pull request! Here are the steps: -1. Fork the |st2| community repository (st2contrib) on Github +1. Fork the `exchange-incubator `__ repository on Github. - * Go to https://github.com/StackStorm/st2contrib and click "Fork" button on + * Go to https://github.com/StackStorm-Exchange/exchange-incubator and click "Fork" button on the right 2. Clone your fork .. code-block:: bash - git clone https://github.com//st2contrib.git + git clone https://github.com//exchange-incubator.git 3. Create a branch for your changes .. code-block:: bash - cd st2contrib + cd exchange-incubator git checkout -b my_uber_new_pack 4. Put your pack in the repo .. code-block:: bash - cp -R ~/uber_new_pack ./packs/ + cp -R ~/uber_new_pack . 5. Create a local commit and push to remote repo .. code-block:: bash - git add packs/uber_new_pack + git add uber_new_pack git commit -m "Awesomeness!!!" git push origin my_uber_new_pack 6. Create pull request - * Go to `StackStorm repo `__. You will see a yellow banner with a button ``Compare & Pull request``. Click the button. + * Go to `exchange-incubator `__. You will see a yellow banner with a button ``Compare & Pull request``. Click the button. * Fill in details describing the pack. Click the ``Create pull request`` button. - * Github will notify us of a new pull request (PR) and we shall review the code, make sure everything looks pristine and merge it in to make your pack publicly available via st2contrib. + * Github will notify us of a new pull request (PR) and we shall review the code, make sure everything looks pristine and merge it in to make your pack publicly available via StackStorm Exchange. .. hint:: If you are new to git/GitHub, `here `__ is an excellent interactive learning resource. diff --git a/docs/source/rules.rst b/docs/source/rules.rst index 3f87ec80a..0be0b0858 100644 --- a/docs/source/rules.rst +++ b/docs/source/rules.rst @@ -718,7 +718,7 @@ Run action every full hour every day of the week .. rubric:: What's Next? -* Explore automations on the `st2contrib`_ community repo. +* Explore automations in the `StackStorm Exchange `_. * Learn more about :doc:`sensors`. * Check out `tutorials on stackstorm.com `__ - a growing set of practical examples of automating with |st2|. diff --git a/docs/source/sensors.rst b/docs/source/sensors.rst index fbb4af62b..237ede8aa 100644 --- a/docs/source/sensors.rst +++ b/docs/source/sensors.rst @@ -138,7 +138,7 @@ datastore. This way if the sensor is restarted or if it crashes, the sensor can resume from where it left off without injecting duplicate triggers into the system. -For the implementation, see :github_contrib:`twitter_search_sensor.py on st2contrib` +For the implementation, see :github_exchange:`twitter_search_sensor.py in StackStorm Exchange` 1. list_values(local=True, prefix=None) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -240,5 +240,5 @@ For example: Examples -------- -For more examples, please reference packs in the `st2contrib repo -`__. +For more examples, please reference packs in `StackStorm Exchange +`__. diff --git a/docs/source/start.rst b/docs/source/start.rst index bc9038e72..6ebe1ef96 100644 --- a/docs/source/start.rst +++ b/docs/source/start.rst @@ -236,7 +236,7 @@ to /opt/stackstorm/packs/, setup, and reload the content: # Reload stackstorm context st2ctl reload --register-all -For more content - actions, sensors, rules - checkout `st2contrib`_ community repo on GitHub. +For more content — actions, sensors, rules — checkout `StackStorm Exchange `__. Datastore --------- @@ -262,7 +262,7 @@ For more information on datastore, check :doc:`datastore` * Get more actions, triggers, rules: - * Install integration packs from `st2contrib`_ - follow guide on :doc:`/packs`. + * Install integration packs from `StackStorm Exchange `__ - follow guide on :doc:`/packs`. * :ref:`Convert your scripts into StackStorm actions. ` * Learn how to :ref:`write custom actions `. diff --git a/docs/source/upgrade_notes.rst b/docs/source/upgrade_notes.rst index 923ea482c..c6c81d71c 100644 --- a/docs/source/upgrade_notes.rst +++ b/docs/source/upgrade_notes.rst @@ -6,12 +6,26 @@ Upgrade Notes |st2| in development -------------------- +* StackStorm Exchange is introduced, and st2contrib is now deprecated. For more information see + :doc:`/reference/packmgmt`. + * Pack names and action parameter names can now only contain valid word characters (``a-z``, ``0-9`` and ``_``). If you have an existing pack or action which uses parameter name which doesn't fall into this criteria it needs to be updated otherwise pack / action registration will fail with an error. +* The ``packs`` pack is deprecated starting from 2.1, in future versions it will be completely + replaced with the ``st2 pack <...>`` commands and API endpoints. See :doc:`/reference/packmgmt` + for the transition reference. + +* The ``packs.install`` action has been reworked, and subtree (multi-pack) repositories are no + longer supported due to the StackStorm Exchange transfer. + +* Support for ``.gitinfo`` file has been removed and as such ``packs.info`` action has also been + removed. All the local pack directories are now direct git checkouts of the corresponding pack + repositories so this file is not needed anymore. + * Datastore scopes are now ``st2kv.system`` and ``st2kv.user`` as opposed to ``system`` and ``user``. So if you are accessing datastore items in your content, you should now use jinja expressions ``{{st2kv.system.foo}}`` and ``{{st2kv.user.foo}}``. The older jinja expressions @@ -117,6 +131,10 @@ Upgrade Notes type: "integer" required: true +* ``version`` field in the pack metadata file must now contain a version string which is a + valid semver identifier - ``..``. For example ``0.1.0``, ``1.0.0,``2.0.1``, + etc. + |st2| v2.0.0 ------------