diff --git a/docs/source/_includes/internal_trigger_types.rst b/docs/source/_includes/internal_trigger_types.rst index 6493045db..e1e944972 100644 --- a/docs/source/_includes/internal_trigger_types.rst +++ b/docs/source/_includes/internal_trigger_types.rst @@ -1,42 +1,44 @@ -.. NOTE: This file has been generated automatically, don't manually edit it +.. NOTE: This file has been generated automatically, don't manually edit it. + Edit st2common/st2common/constants/triggers.py and rebuild the + documentation. Action ~~~~~~ -+---------------------------+-----------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ -| Reference | Description | Properties | -+===========================+=================================================================+=============================================================================================================+ -| st2.generic.actiontrigger | Trigger encapsulating the completion of an action execution. | execution_id, status, start_timestamp, action_name, action_ref, runner_ref, parameters, result | -+---------------------------+-----------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ -| st2.generic.notifytrigger | Notification trigger. | execution_id, status, start_timestamp, end_timestamp, action_ref, runner_ref, channel, route, message, data | -+---------------------------+-----------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ -| st2.action.file_writen | Trigger encapsulating action file being written on disk. | ref, file_path, host_info | -+---------------------------+-----------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ -| st2.generic.inquiry | Trigger indicating a new "inquiry" has entered "pending" status | id, route | -+---------------------------+-----------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ ++--------------------------------+-----------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ +| Reference | Description | Properties | ++================================+=================================================================+=============================================================================================================+ +| core.st2.generic.actiontrigger | Trigger encapsulating the completion of an action execution. | execution_id, status, start_timestamp, action_name, action_ref, runner_ref, parameters, result | ++--------------------------------+-----------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ +| core.st2.generic.notifytrigger | Notification trigger. | execution_id, status, start_timestamp, end_timestamp, action_ref, runner_ref, channel, route, message, data | ++--------------------------------+-----------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ +| core.st2.action.file_written | Trigger encapsulating action file being written on disk. | ref, file_path, host_info | ++--------------------------------+-----------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ +| core.st2.generic.inquiry | Trigger indicating a new "inquiry" has entered "pending" status | id, route | ++--------------------------------+-----------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+ Sensor ~~~~~~ -+--------------------------+--------------------------------------------------+------------+ -| Reference | Description | Properties | -+==========================+==================================================+============+ -| st2.sensor.process_spawn | Trigger indicating sensor process is started up. | object | -+--------------------------+--------------------------------------------------+------------+ -| st2.sensor.process_exit | Trigger indicating sensor process is stopped. | object | -+--------------------------+--------------------------------------------------+------------+ ++-------------------------------+--------------------------------------------------+------------+ +| Reference | Description | Properties | ++===============================+==================================================+============+ +| core.st2.sensor.process_spawn | Trigger indicating sensor process is started up. | object | ++-------------------------------+--------------------------------------------------+------------+ +| core.st2.sensor.process_exit | Trigger indicating sensor process is stopped. | object | ++-------------------------------+--------------------------------------------------+------------+ Key Value Pair ~~~~~~~~~~~~~~ -+---------------------------------+---------------------------------------------------------+------------------------+ -| Reference | Description | Properties | -+=================================+=========================================================+========================+ -| st2.key_value_pair.create | Trigger encapsulating datastore item creation. | object | -+---------------------------------+---------------------------------------------------------+------------------------+ -| st2.key_value_pair.update | Trigger encapsulating datastore set action. | object | -+---------------------------------+---------------------------------------------------------+------------------------+ -| st2.key_value_pair.value_change | Trigger encapsulating a change of datastore item value. | old_object, new_object | -+---------------------------------+---------------------------------------------------------+------------------------+ -| st2.key_value_pair.delete | Trigger encapsulating datastore item deletion. | object | -+---------------------------------+---------------------------------------------------------+------------------------+ ++--------------------------------------+---------------------------------------------------------+------------------------+ +| Reference | Description | Properties | ++======================================+=========================================================+========================+ +| core.st2.key_value_pair.create | Trigger encapsulating datastore item creation. | object | ++--------------------------------------+---------------------------------------------------------+------------------------+ +| core.st2.key_value_pair.update | Trigger encapsulating datastore set action. | object | ++--------------------------------------+---------------------------------------------------------+------------------------+ +| core.st2.key_value_pair.value_change | Trigger encapsulating a change of datastore item value. | old_object, new_object | ++--------------------------------------+---------------------------------------------------------+------------------------+ +| core.st2.key_value_pair.delete | Trigger encapsulating datastore item deletion. | object | ++--------------------------------------+---------------------------------------------------------+------------------------+ diff --git a/docs/source/actions.rst b/docs/source/actions.rst index e7ac71794..9950316c5 100644 --- a/docs/source/actions.rst +++ b/docs/source/actions.rst @@ -103,17 +103,15 @@ the following runners: 7. ``action-chain`` - This runner supports executing simple linear work-flows. For more information, please refer to the :doc:`Workflows ` and :doc:`ActionChain ` documentation. -8. ``mistral-v2`` - This runner is built on top of the Mistral OpenStack project and supports - executing complex work-flows. For more information, please refer to the - :doc:`Workflows ` and :doc:`Mistral ` documentation. -9. ``inquirer`` - This runner provides the core logic of the :doc:`Inquiries ` + +8. ``inquirer`` - This runner provides the core logic of the :doc:`Inquiries ` feature. Note: This runner is an implementation detail for the ``core.ask`` action, and in most cases should not be referenced in other actions. -10. ``winrm-cmd`` - The WinRM command runner allows you to run the command-line interpreter (``cmd``) commands on Windows hosts using the WinRM protocol. -11. ``winrm-ps-cmd`` - The WinRM PowerShell command runner allows you to run the PowerShell commands on Windows hosts using the WinRM protocol. -12. ``winrm-ps-script`` - WinRM PowerShell script runner allows you to run PowerShell scripts on Windows hosts. +9. ``winrm-cmd`` - The WinRM command runner allows you to run the command-line interpreter (``cmd``) commands on Windows hosts using the WinRM protocol. +10. ``winrm-ps-cmd`` - The WinRM PowerShell command runner allows you to run the PowerShell commands on Windows hosts using the WinRM protocol. +11. ``winrm-ps-script`` - WinRM PowerShell script runner allows you to run PowerShell scripts on Windows hosts. Runners come with their own set of input parameters. When an action is executed, it inherits the diff --git a/docs/source/chatops/notifications.rst b/docs/source/chatops/notifications.rst index b527de517..65511983b 100644 --- a/docs/source/chatops/notifications.rst +++ b/docs/source/chatops/notifications.rst @@ -178,54 +178,65 @@ task notify is shown below: timeout: 180 on-success: "make_tests" -How do I Setup Notifications for Mistral? ------------------------------------------ +How do I Setup Notifications for Orquesta? +------------------------------------------ -The method for global notifications for the workflow is the same as ActionChain. You have a notify -section in the action meta when registering. See an -`example `_. -Unfortunately, notifications per task are not supported in Mistral as a first class citizen yet. -This will be added in later releases. +The Orquesta metadata supports an optional notify input parameter, this specifies the tasks in the workflow that notifications will be generated for. For example: -How do I Skip Notifications for Tasks in a Workflow? +.. code-block:: yaml + + --- + name: notify + description: A basic sequential workflow with notify enabled. + pack: orquesta_tests + runner_type: orquesta + entry_point: workflows/sequential.yaml + enabled: true + parameters: + who: + required: true + type: string + default: Stanley + notify: + type: array + default: + - task1 + - task2 + - task3 + + +How do I Skip Notifications for Tasks in a chain ? ----------------------------------------------------------- -This is implemented as a runner parameter ``skip_notify``. If your chain or workflow contains +This is implemented as a runner parameter ``skip_notify``. If your chain contains multiple tasks and you want some tasks to be "muted", you can do so by specifying skip_notify and call out tasks to mute. For example: .. code-block:: yaml --- - name: mistral-basic-two-tasks-with-notifications - pack: examples - description: Run mistral workflow with two tasks. - runner_type: mistral-v2 - entry_point: workflows/mistral-basic-two-tasks-with-notifications.yaml + # Action definition metadata + name: "echochain" + description: "Simple Action Chain workflow" + + # `runner_type` has value `action-chain` to identify that action is an ActionChain. + runner_type: "action-chain" + + # `entry_point` path to the ActionChain definition file, relative to the pack's action directory. + entry_point: "chains/echochain.yaml" + enabled: true parameters: skip_notify: default: - - "task2" - context: - default: {} - immutable: true - type: object - task: - default: null - immutable: true - type: string - workflow: - default: null - immutable: true - type: string + - c2 notify: on-complete: message: "\"@channel: Action succeeded.\"" routes: - "slack" -In the above example, notifications for "task2" will not be sent out. This feature is particularly +In the above example, notifications for "c2" will not be sent out. This feature is particularly useful in combination with ChatOps where you don't want noisy tasks to pollute the Chat client. Note that it is not currently possible to have a default ``skip_notify`` policy. @@ -237,4 +248,4 @@ If you enabled ChatOps, you get all the things wired for you. You don't have to metadata etc. You can still use ``skip_notify`` to skip notifications for certain tasks in a chain or workflow. If you specified a notify section in metadata or in tasks, those notification routes will override ChatOps. Therefore, you might not see notifications in the chat client. -See `this issue `_ for an example. \ No newline at end of file +See `this issue `_ for an example. diff --git a/docs/source/datastore.rst b/docs/source/datastore.rst index f9614206a..8f422b3ae 100644 --- a/docs/source/datastore.rst +++ b/docs/source/datastore.rst @@ -459,7 +459,7 @@ A simple action example: --- description: Remediates a host. enabled: true - runner_type: mistral-v2 + runner_type: orquesta entry_point: workflows/remediate.yaml name: remediate pack: default @@ -490,7 +490,7 @@ the parameter definition: --- description: Provisions a VM enabled: true - runner_type: mistral-v2 + runner_type: orquesta entry_point: workflows/vm_provision.yaml name: vm_provision pack: default diff --git a/docs/source/mistral.rst b/docs/source/mistral.rst deleted file mode 100644 index 1da5b00ed..000000000 --- a/docs/source/mistral.rst +++ /dev/null @@ -1,260 +0,0 @@ -Mistral -======= - -.. note:: - - Our new workflow engine :doc:`Orquesta ` is now GA. We recommend writing all new - workflows in Orquesta. Mistral is still supported, but will be removed in a future release. - -`Mistral `_ is an OpenStack project that -manages and executes workflows as a service. Mistral is automatically installed as a separate -service named "mistral" along with |st2|. A Mistral workflow can be defined as a |st2| action in a -Mistral workbook using the `v2 Workflow Language `_. - -Expression languages such as YAQL are used for formatting variables and condition evaluations. -Starting with |st2| v2.2, Jinja2 is also supported where YAQL expressions are accepted. Both -workbook and workflow definitions are supported. - -On action execution, |st2| writes the definition to Mistral and executes the workflow. A workflow -can invoke other |st2| actions natively as subtasks. |st2| handles the translations and calls -transparently in Mistral and actively polls Mistral for execution results. |st2| actions in the -workflow can be traced back to the original parent action that invoked the workflow. - -**Essential Mistral Links:** - -.. note:: - - The following tools and resources are not owned by |st2|. Please use at your own risk. - -* Mistral workflow definition language, aka `v2 WorkFlow Language - `_ -* `YAQL documentation `_ and `YAQL online evaluator - `_ -* `Jinja2 template engine documentation `_ and `Jinja2 online evaluator - `_ - - -.. note:: - - Workflow examples in this documentation use YAQL expressions unless otherwise stated. - -Basic Workflow --------------- - -Similarly to ActionChains, Mistral workflows have an action metadata file in -``/opt/stackstorm/packs//actions``, and the workflow definition itself in -``/opt/stackstorm/packs//actions/workflows``. - -Let's start with a very basic workflow that calls a |st2| action and notifies |st2| when the -workflow is done. The files used in this example are also located under -:github_st2:`/usr/share/doc/st2/examples ` if |st2| is already installed (see -also :ref:`deploy examples `). - -The first task is named ``run-cmd``. It executes a shell command on the server where |st2| is -installed. A task can reference any registered |st2| action directly. In this example, the -``run-cmd`` task calls ``core.local`` and passing the cmd as input. ``core.local`` is an action -that comes installed with |st2|. When the workflow is invoked, |st2| will translate the workflow -definition appropriately before sending it to Mistral. Let's save this as -``/opt/stackstorm/packs/examples/actions/workflows/mistral-basic.yaml`` on our |st2| server. - -.. literalinclude:: /../../st2/contrib/examples/actions/workflows/mistral-basic.yaml - :language: yaml - -This is the corresponding |st2| action metadata for the example above. The |st2| pack for this -workflow action is named "examples". Note that the workflow is named fully qualified as -``.`` in the definition above. The |st2| action runner is ``mistral-v2``. The entry -point for the |st2| action refers to the YAML file of the workflow definition. Let's save this -metadata as ``/opt/stackstorm/packs/examples/actions/mistral-basic.yaml``: - -.. literalinclude:: /../../st2/contrib/examples/actions/mistral-basic.yaml - :language: yaml - -The following table list optional parameters that can be defined in the workflow action. In the -example, these optional parameters are set to immutable. It is good practice to set them to -immutable even if they are empty since these are Mistral-specific parameters for the workflow -author. - -+------------+--------------------------------------------------------+ -| options | description | -+============+========================================================+ -| workflow | If definition is a workbook containing many workflows, | -| | this specifies the main workflow to execute. | -+------------+--------------------------------------------------------+ -| task | If the type of workflow is "reverse", this specifies | -| | the task to invoke. | -+------------+--------------------------------------------------------+ -| context | A dictionary containing additional workflow start up | -| | parameters. | -+------------+--------------------------------------------------------+ - -Next, run ``st2 action create /opt/stackstorm/packs/examples/actions/mistral-basic.yaml`` to -create this workflow action. This will register the workflow as ``examples.mistral-basic`` in -|st2|. To execute the workflow, run ``st2 run examples.mistral-basic cmd=date -a`` where ``-a`` -tells the command to return and not wait for the workflow to complete. - -If the workflow completed successfully, both the workflow ``examples.mistral-basic`` and the -action ``core.local`` should have a ``succeeded`` status in the |st2| action execution list. By -default, ``st2 execution list`` only returns top level executions. This means subtasks are not -displayed. - -.. code-block:: shell - - +--------------------------+--------------+--------------+-----------+-----------------+---------------+ - | id | action.ref | context.user | status | start_timestamp | end_timestamp | - +--------------------------+--------------+--------------+-----------+-----------------+---------------+ - | 54ee54c61e2e24152b769a47 | examples | stanley | succeeded | Wed, 25 Feb | Wed, 25 Feb | - | | .mistral- | | | 2015 23:03:34 | 2015 23:03:34 | - | | basic | | | UTC | UTC | - +--------------------------+--------------+--------------+-----------+-----------------+---------------+ - -To display subtasks, run ``st2 execution get --show-tasks``: - -.. code-block:: shell - - +--------------------------+------------+--------------+-----------+------------------------------+------------------------------+ - | id | action.ref | context.user | status | start_timestamp | end_timestamp | - +--------------------------+------------+--------------+-----------+------------------------------+------------------------------+ - | 54ee54c91e2e24152b769a49 | core.local | stanley | succeeded | Wed, 25 Feb 2015 23:03:37 | Wed, 25 Feb 2015 23:03:37 | - | | | | | UTC | UTC | - +--------------------------+------------+--------------+-----------+------------------------------+------------------------------+ - -The following is a simple extension of the previous workflow definition. In this example, we have -a second task named "task2". It might be natural to think that "task2" will be executed after -"task1", i.e, in sequential order. However, when no tasks attributes like ``on-complete``, -``on-success`` and ``on-error`` are defined, tasks are run in parallel. This is possible with -Mistral because it provides a join flow control which allows us to synchronize multiple parallel -workflow branches and aggregate their data. - -.. literalinclude:: /../../st2/contrib/examples/actions/workflows/mistral-basic-two-tasks-with-notifications.yaml - :language: yaml - -Publishing Variables --------------------- - -A Mistral task can publish results from a task as variables that can be consumed in other tasks: - -.. sourcecode:: yaml - - tasks: - get_hostname: - action: core.local - input: - cmd: "hostname" - publish: - hostname: <% task(get_hostname).result.stdout %> - -In the above example, ``get_hostname`` is a ``core.local`` action which runs the command -``hostname``. The ``core.local`` action produces output consisting of the fields ``stdout``, -``stderr``, ``exit_code`` etc. - -We just want to publish the variable ``stdout`` from it, for the rest of tasks to consume. To -reference the result of the task, use the ``task`` function, which returns a dictionary containing -attributes for the task such as id, state, result, and additional info. - -Another example is shown below: - -.. sourcecode:: yaml - - tasks: - create_new_node: - action: rackspace.create_vm - input: - name: <% $.hostname %> - flavor_id: <% $.vm_size_id %> - image_id: <% $.vm_image_id %> - key_material: <% $.ssh_pub_key %> - metadata: - asg: <% $.asg %> - publish: - ipv4_address: '<% task(create_new_node).result.result.public_ips[1] %>' - ipv6_address: '<% task(create_new_node).result.result.public_ips[0] %>' - -In the above example, the action ``rackspace.create_vm`` is a Python action that produces a result -object. We just want to publish the IP addresses from the ``public_ips`` list field from the -result object. - -Please note that ``result.result`` is not a typo. The Python action posts output to a key named -``result`` for the st2 action execution and the Mistral task function puts the result of the Python -action in ``result`` of its output dictionary. - -Such published variables are accessible as input parameters to other tasks in the workflow. An -example of using ``ipv4_address`` from the above example in another task is shown below: - -.. sourcecode:: yaml - - tasks: - # ... - - setup_ipv4_dns: - action: rackspace.create_dns_record - wait-before: 1 # delay, in seconds - input: - name: '<% $.hostname %>.<% $.asg %>.<% $.domain %>' - zone_id: <% $.dns_zone_id %> - type: 'A' - data: <% $.ipv4_address %> - - # .... - -Stitching Together a More Complex Workflow ------------------------------------------- - -The following is a mock up of a more complex workflow. In this mock up running simple ``printf`` -and ``sleep`` commands, the workflow demonstrates nested workflows, fork, and join: - -.. literalinclude:: /../../st2/contrib/examples/actions/workflows/mistral-workbook-complex.yaml - :language: yaml - -Since there are multiple workflows defined in this workbook, the workflow author has to specify which workflow to execute in the metadata: - -.. literalinclude:: /../../st2/contrib/examples/actions/mistral-workbook-complex.yaml - :language: yaml - -To test out this workflow, save the metadata file to ``/opt/stackstorm/packs/examples/actions/`` -and the workflow file to ``/opt/stackstorm/packs/examples/actions/workflows``. Run -``st2 action create /opt/stackstorm/packs/examples/actions/mistral-workbook-complex.yaml`` to -create the action and run ``st2 run examples.mistral-workbook-complex vm_name="vmtest1" -a`` to -test. - -Validation ----------- - -The Mistral CLI includes tools for performing high-level sanity checks of Mistral workflow YAML -files: - -.. code-block:: bash - - # Validate a workflow - mistral workflow-validate /path/to/workflow.yaml - - # Validate a workbook - mistral workbook-validate /path/to/workbook.yaml - - -.. note:: - - These sanity checks simply provide a test against the Mistral DSL schema. They do NOT test YAQL - or Jinja2 expressions. - - -More Examples -------------- - -There are more workflow examples under :github_st2:`/usr/share/doc/st2/examples `. These include error handling, repeat, and retries. - -Check out this step-by-step tutorial on building a workflow in |st2| https://stackstorm.com/2015/07/08/automating-with-mistral-workflow/ - -More details about Mistral can be found at https://docs.openstack.org/mistral/latest/. - -More Topics ------------ - -The following sections go into more details on specific topics. - -.. toctree:: - :maxdepth: 1 - - YAQL Expressions - Jinja Expressions - Completion and Latency - Workflow Operations diff --git a/docs/source/mistral_jinja.rst b/docs/source/mistral_jinja.rst deleted file mode 100644 index 2dd0e99cd..000000000 --- a/docs/source/mistral_jinja.rst +++ /dev/null @@ -1,256 +0,0 @@ -Mistral + Jinja -=============== - -Starting with |st2| v2.2, Jinja expressions are also supported where YAQL expressions are accepted. -Jinja expressions can be used for simple conditional evaluation and data transformation in Mistral -workflows. There will be many cases where you did not author the actions but there's a need to -decide from the result of the action whether to continue or there's a need to transform the -result to another value or structure for the next action in the workflow. - -Here are use cases where Jinja can be applied in Mistral workflows: - -* Define input values that are passed to tasks. -* Define output values published from tasks and workflows. -* Define conditions that determine transitions between tasks. - -Knowing where Jinja can be applied in Mistral workflows, the following are some cool things that you can do with Jinja: - -* Access values from lists and dictionary. -* Simple arithmetic. -* Evaluation of boolean logic. -* Use conditional logic and builtin filters to evaluate and transform data. - -.. note:: - - Please refer to the official documentation for Mistral and Jinja. The documentation here - is meant to help |st2| users get started quickly, but is not a complete reference. - -Basics ------- - -The following are statements in the workflow and task definition that accepts Jinja: - -* task action input -* task concurrency -* task on-complete -* task on-error -* task on-success -* task pause-before -* task publish -* task retry break-on -* task retry continue-on -* task retry count -* task retry delay -* task timeout -* task wait-before -* task wait-after -* task with-items -* workflow output - -Each of the statements can take a string with one or more Jinja expressions. Each expression in the -string should be encapsulated with ``{{ }}``. Code block using ``{% %}`` is also supported. Please -note that the symbols ``{`` and ``}`` will conflict with JSON and Jinja expressions, and must -always be encapsulated with quotes or double quotes in the workflow definition. - -.. note:: - - Mixing of both YAQL and Jinja expressions in a single statement is not supported. - -When evaluating a Jinja expression, Mistral also passes a JSON dictionary (aka context) to the -Jinja templating engine. The context contains all the workflow inputs, published variables, and -result of completed tasks up to this point of workflow execution, including the current task. The -Jinja expression can refer to one or more variables in the context. The reserved symbol ``_`` is -used to reference the context. For example, given the context -``{"ip": "127.0.0.1", "port": 8080}``, the string ``https://{{ _.ip }}:{{ _.port }}/api`` returns -``https://127.0.0.1:8080/api``. - -The following is the same example used in a workflow: - -.. code-block:: yaml - - version: '2.0' - - examples.yaql-basic: - type: direct - input: - - ip - - port - tasks: - task1: - action: examples.call-api - input: - endpoint: https://{{ _.ip }}:{{ _.port }}/api - -.. note:: - - The reserved symbol to reference the workflow context for Jinja is different than for YAQL - expressions. This is due to the differences between the two engines. As mentioned above, the - symbol ``_`` is used to access context in Jinja whereas the symbol ``$`` is used in YAQL. - -The following is a more complex workbook example with a few more Jinja expressions. There are -variables passed to input parameters and being published after task completion. Please take note -of the ``install_apps`` task in the ``configure_vm`` workflow. The input parameter ``cmd`` is -given the value formatted by the Jinja for loop. Unlike YAQL, a string in a Jinja expression must -be explicitly encapsulated in quotes (i.e. ``{{ 'this is a string.' }}``). - -.. literalinclude:: /../../st2/contrib/examples/actions/workflows/mistral-jinja-workbook-complex.yaml - :language: yaml - -Certain statements in Mistral such as on-success and on-error can evaluate boolean logic. The -``on-condition`` related statements are used for transition from one task to another. If a -boolean logic is defined with these statements, it can be used to evaluate whether the transition -should continue or not. Complex boolean logic using a combination of ``not``, ``and``, ``or``, and -parentheses is possible. - -Take the following workflow as an example: Execution of a certain branch in the workflow depends -on the value of ``_.path``. If ``_.path == a``, then task ``a`` is executed. If ``_.path == b``, -then task ``b``. Finally task ``c`` is executed if neither matches. - -.. literalinclude:: /../../st2/contrib/examples/actions/workflows/mistral-jinja-branching.yaml - :language: yaml - -The statement ``with-items`` in Mistral is used to execute an action over iteration of one or more -lists of items. The following is a sample Mistral workflow that iterates over the list of given -names to invoke the action to create an individual VM: - -.. code-block:: yaml - - version: '2.0' - - examples.create-vms: - type: direct - input: - - names - tasks: - task1: - with-items: "name in {{ _.names }}" - action: examples.create-vm - input: - name: "{{ _.name }}" - -``with-items`` can take more than one list as the following example illustrates. In this case, a -list of VMs and IP addresses are passed as inputs and then iterated through step by step together: - -.. code-block:: yaml - - version: '2.0' - - examples.create-vms: - type: direct - input: - - names - - ips - tasks: - task1: - with-items: - - "name in {{ _.names }}" - - "ip in {{ _.ips }}" - action: examples.create-vm - input: - name: "{{ _.name }}" - ip: "{{ _.ip }}" - -.. note:: - - The Jinja expression(s) passed to with-items is slightly different than YAQL expression(s). - If using Jinja, the entire statement (i.e. ``"name in {{ _.names }}"``) require quotation - because the symbols ``{`` and ``}`` for the delimiters conflict with JSON. Whereas if using - YAQL, the statement does not necessarily require quotation. - -Jinja Filters -------------- - -Jinja has a list of built-in filters to work with strings, dictionaries, lists, etc. Please -refer to the Jinja `documentation -`_ -for the list of available filters. - -A number of Mistral and |st2| specific custom functions (aka filters in Jinja) such as ``st2kv``, -``task``, ``env``, and ``execution`` that are available in YAQL are also made available in Jinja. - -Mistral -^^^^^^^ - -* ``env()`` returns the environment variables passed to the workflow execution on invocation such - as the |st2| Action Execution ID ``st2_execution_id``. - - For example, the expression - ``{{ env().st2_action_api_url }}/actionexecutions/{{ env().st2_execution_id }}`` returns the API - endpoint for the current workflow execution in |st2| as something like - ``https://127.0.0.1:9101/v1/actionexecutions/874d3d5b3f024c1aa93225ef0bcfcf3a``. -* To access information about the parent action, the following expressions can be used - ``{{ env()['__actions']['st2.action']['st2_context']['parent']['api_user'] }}``, - ``{{ env()['__actions']['st2.action']['st2_context']['parent']['source_channel'] }}`` - or ``{{ env()['__actions']['st2.action']['st2_context']['parent']['user'] }}``. - - Note that this similar to the ActionChain expressions - ``{{action_context.parent.source_channel}}``, ``{{action_context.parent.user}}`` or - ``{{action_context.parent.api_user}}``. -* ``task('task_name')`` returns the state, state_info, and the result of the given task_name. - -StackStorm -^^^^^^^^^^ - -``st2kv('st2_key_id')`` queries |st2|'s datastore and returns the value for the given key. For -example, the expression ``{{ st2kv('system.shared_key_x') }}`` returns the value for a system -scoped key named ``shared_key_x`` while the expression ``{{ st2kv('my_key_y') }}`` returns the -value for the user scoped key named ``my_key_y``. - -Please note that the key name should be in quotes otherwise Jinja treats a key name with a dot -like ``system.shared_key_x`` as a dict access. - -.. note:: - - If the retrieved value was stored encrypted, ``st2kv`` no longer attempts decryption by default - (as of version 2.4). To decrypt the retrieved value, you must explicitly enable it through the - ``decrypt`` parameter: ``st2kv('st2_key_id', decrypt=true)``. - -Testing Expressions -------------------- - -Sometimes Jinja expressions can become complex and the need to verify the expression outside of -|st2| is necessary. To accomplish this there are a few options: - -- `Jinja2 online evaluator `_ -- Write a small test script: - - .. code-block:: python - - #!/usr/bin/env python - import jinja2 - import os - - - class JinjaUtils: - - @staticmethod - def render_file(filename, context): - path, filename = os.path.split(filename) - env = jinja2.Environment(loader=jinja2.FileSystemLoader(path or './')) - tmpl = env.get_template(filename) - return tmpl.render(context) - - @staticmethod - def render_str(jinja_template_str, context): - env = jinja2.Environment() - tmpl = env.from_string(jinja_template_str) - return tmpl.render(context) - - - if __name__ == "__main__": - context = {'results': {'name': 'Stanley'}} - template = "Hello {{ results.name }}" - print JinjaUtils.render_str(template, context) - - .. note:: - - The test script does NOT include the custom |st2| Jinja filters or functions - such as ``st2kv``. - - -More Examples -------------- - -More workflow examples using Jinja expressions can be found at -:github_st2:`/usr/share/doc/st2/examples `. The examples are -prefixed with ``mistral-jinja``. diff --git a/docs/source/mistral_operations.rst b/docs/source/mistral_operations.rst deleted file mode 100644 index 8349e01d7..000000000 --- a/docs/source/mistral_operations.rst +++ /dev/null @@ -1,107 +0,0 @@ -Workflow Operations -=================== - -Pausing and Resuming Workflow Execution ---------------------------------------- - -An execution of a Mistral workflow can be paused by running ``st2 execution pause ``. -An execution must be in a running state in order for pause to be successful. The execution will -initially go into a ``pausing`` state, and will go into a ``paused`` state when no more tasks are -in an active state such as ``running``, ``pausing``, or ``canceling``. When a workflow execution -is paused, it can be resumed by running ``st2 execution resume ``. - -The ``pause`` and ``resume`` operation will cascade down to subworkflows, whether it's another -workflow defined in a workbook or it's another |st2| action that is a Mistral workflow or Action -Chain. If the ``pause`` operation is performed from a subworkflow or subchain, then the ``pause`` -will cascade up to the parent workflow or parent chain. However, if the ``resume`` operation is -performed from a subworkflow or subchain, the ``resume`` will not cascade up to the parent workflow -or parent chain. This allows users to resume and troubleshoot branches individually. - -Canceling Workflow Execution ----------------------------- - -An execution of a Mistral workflow can be cancelled by running -``st2 execution cancel ``. Workflow tasks that are still running will not be -canceled and will run to completion. No new tasks for the workflow will be scheduled. - -Re-running Workflow Execution ------------------------------ - -An execution of a Mistral workflow can be re-run on error. The execution either can be re-run from -the beginning or from the task(s) that failed. The latter is useful for long running workflows with -temporary service or network outages. Re-running the workflow execution from the beginning is -exactly like re-running any |st2| execution with the command -``st2 execution re-run ``. - -The re-run is a completely separate execution with a new execution ID in both |st2| and Mistral. -Re-running the workflow from where it errored is slightly different. To retain context, the -original workflow execution is reused in Mistral but a new |st2| execution will be created to stay -consistent in |st2|. The re-run command has a new ``--tasks`` option that takes a list of task -names to re-run. - -For example, given a workflow that fails at task3 and task4 on separate parallel branches, the -command ``st2 execution re-run --tasks task3 task4`` will resume the Mistral -workflow execution and re-run both task3 and task4 using original inputs. Both the workflow and -task execution in Mistral have to be in an ``errored`` state for re-run. - -If using a Mistral workbook, tasks of subworkflows can also be re-run. For example, if the main -workflow has a task1 that calls subflow1, then to re-run subtask1 of subflow1, the syntax for the -``st2 execution re-run`` command would be -``st2 execution re-run --tasks task1.subtask1``. - -If the task to re-run is a "with-items" task, there is an option to re-run only failed iterations. -For example, task1 is a with-items task with 5 items. Let's say 2 of the items failed. By -specifying the ``st2 execution re-run --tasks task1 task2 --no-reset task1`` option, task1 will -only re-run the 2 items that failed. If the ``--no-reset`` option is not provided, then all 5 -items will be re-run. - -.. note:: - - Re-running workflow execution from the task(s) that failed is currently an experimental - feature and subject to bug(s) and change(s). Please also note that re-running a subtask nested - in another |st2| action is not currently supported. - -Task Timeout vs Action Timeout ------------------------------- - -Mistral supports a task ``timeout:`` parameter. This sets the maximum amount of time Mistral will -wait before marking a task as failed. **However**, |st2| actions implement their own timeouts. -The default value for each action timeout depends upon the action runner used. Typically this is -60s for SSH-based actions, and 600s for Python actions. The default can be changed on a per-action -basis, and can be over-ridden for each execution. - -This can cause confusion when you need to extend the timeout for some tasks. Setting a longer Mistral -timeout does **not** extend the underlying action timeout. For example, if you have a long-running -command, this will **not** achieve the desired result: - -.. code-block:: yaml - - version: '2.0' - - examples.task-timeout: - type: direct - input: - - command - tasks: - task1: - timeout: 120 - action: core.local - input: - cmd: <% $.command %> - -Instead, set the timeout on the underlying action. Note the indentation here: - -.. code-block:: yaml - - version: '2.0' - - examples.action-timeout: - type: direct - input: - - command - tasks: - task1: - action: core.local - input: - cmd: <% $.command %> - timeout: 120 diff --git a/docs/source/mistral_result.rst b/docs/source/mistral_result.rst deleted file mode 100644 index b6c73d023..000000000 --- a/docs/source/mistral_result.rst +++ /dev/null @@ -1,45 +0,0 @@ -.. _mistral-workflows-completion-latency-and-performance: - -Mistral Workflows Completion, Latency, and Performance -====================================================== - -During the workflow runtime, |st2| and Mistral handshake multiple times over HTTP. This applies when launching -the workflow execution, completing a task, or completing the workflow. - -Prior to v2.7, |st2| queries Mistral to check on workflow execution status and the status of individual tasks -via ``st2resultstracker``. This mechanism has a number of configuration settings. See :ref:`mistral-workflows-latency` -section about how to fine-tune the Mistral workflows completion time vs CPU consumption. - -Since v2.7, the results tracking mechanism is replaced with a callback mechanism from Mistral. Instead of |st2| -querying Mistral at regular interval, Mistral is configured to callback |st2| on task and workflow completion. -With the callback mechanism, it is possible to trace the events sent to |st2|. - -.. code-block:: bash - - # Identify the Mistral workflow execution ID which is different - st2 execution get -dj | grep workflow_execution_id - - # Grep the log entries from the Mistral log, typically at /var/log/mistral/mistral-server.log - sudo tail -n 1000 /var/log/mistral/mistral-server.log | grep stackstorm_notifier | grep - -The returned list of log entries will look similar to the following: - -:: - - 2018-03-28 22:40:05,811 140124959618704 INFO stackstorm_notifier [-] [839925d9-02c7-47be-ad8e-ce0943749a7b] The workflow event WORKFLOW_LAUNCHED for 839925d9-02c7-47be-ad8e-ce0943749a7b will be published to st2. - 2018-03-28 22:40:05,844 140124959618704 INFO stackstorm_notifier [-] [839925d9-02c7-47be-ad8e-ce0943749a7b] The workflow event WORKFLOW_LAUNCHED for 839925d9-02c7-47be-ad8e-ce0943749a7b is published to st2. - 2018-03-28 22:40:06,492 140124958584912 INFO stackstorm_notifier [-] [839925d9-02c7-47be-ad8e-ce0943749a7b] The task event TASK_SUCCEEDED for c8731e6a-2464-4a59-bf46-501a80215298 will be processed for st2. - 2018-03-28 22:40:06,492 140124958584912 INFO stackstorm_notifier [-] [839925d9-02c7-47be-ad8e-ce0943749a7b] The task event TASK_SUCCEEDED for c8731e6a-2464-4a59-bf46-501a80215298 is processed for st2. - 2018-03-28 22:40:07,195 140124956804432 INFO stackstorm_notifier [-] [839925d9-02c7-47be-ad8e-ce0943749a7b] The workflow event WORKFLOW_SUCCEEDED for 839925d9-02c7-47be-ad8e-ce0943749a7b will be published to st2. - 2018-03-28 22:40:07,371 140124956804432 INFO stackstorm_notifier [-] [839925d9-02c7-47be-ad8e-ce0943749a7b] The workflow event WORKFLOW_SUCCEEDED for 839925d9-02c7-47be-ad8e-ce0943749a7b is published to st2. - -The results tracking mechanism is still available for manual intervention and can be enabled on an individual workflow -basis in case |st2| or Mistral services are offline during a callback operation. - -.. code-block:: bash - - # Enable the results tracking for an individual workflow execution - st2-track-result --config-dir /etc/st2 - - # Disable the results tracking for an individual workflow execution - st2-track-result --config-dir /etc/st2 --delete diff --git a/docs/source/mistral_yaql.rst b/docs/source/mistral_yaql.rst deleted file mode 100644 index 75b4e0fde..000000000 --- a/docs/source/mistral_yaql.rst +++ /dev/null @@ -1,326 +0,0 @@ -Mistral + YAQL -============== - -YAQL is typically used for simple conditional evaluation and data transformation in Mistral -workflows. There will be many cases where you did not author the actions but there's a need to -decide from the result of the action whether to continue, or there's a need to transform the -result to another value or structure for the next action in the workflow. - -Here are use cases where YAQL can be applied in Mistral workflows: - -* Define input values that are passed to tasks. -* Define output values published from tasks and workflows. -* Define conditions that determine transitions between tasks. - -Knowing where YAQL can be applied in Mistral workflows, the following are some cool things that -you can do with YAQL: - -* Select key-value pairs from a list of dictionaries. -* Filter the list where one or more fields match condition(s). -* Transform a list to dictionary or vice versa. -* Simple arithmetic. -* Evaluation of boolean logic. -* Any combination of select, filter, transform, and evaluate. - -.. note:: - - Please refer to official OpenStack documentation for Mistral and YAQL. The documentation here - is meant to help |st2| users get started quickly, but does not cover everything. - `YAQL unit tests `_ are also a great - reference for how to use and what features are supported in YAQL. They help to cover some gaps - in OpenStack YAQL documentation. - -Basics ------- - -The following are statements in the workflow and task definition that accept YAQL: - -* task action input -* task concurrency -* task on-complete -* task on-error -* task on-success -* task pause-before -* task publish -* task retry break-on -* task retry continue-on -* task retry count -* task retry delay -* task timeout -* task wait-before -* task wait-after -* task with-items -* workflow output - -Each of the statements can take a string with one or more YAQL expressions. Each expression in the -string should be encapsulated with ``<% %>``. - -.. note:: - - Mixing of both YAQL and Jinja expressions in a single statement is not supported. - -When evaluating a YAQL expression, Mistral also passes a JSON dictionary (aka context) to the YAQL -engine. The context contains all the workflow inputs, published variables, and result of completed -tasks up to this point of workflow execution, including the current task. The YAQL expression can -refer to one or more variables in the context. The reserved symbol ``$`` is used to reference the -context. For example, given the context ``{"ip": "127.0.0.1", "port": 8080}``, the string -``https://<% $.ip %>:<% $.port %>/api`` returns ``https://127.0.0.1:8080/api``. The following is the -same example used in a workflow: - -.. code-block:: yaml - - version: '2.0' - - examples.yaql-basic: - type: direct - input: - - ip - - port - tasks: - task1: - action: examples.call-api - input: - endpoint: https://<% $.ip %>:<% $.port %>/api - -Certain statements in Mistral such as on-success and on-error can evaluate boolean logic. The -``on-condition`` related statements are used for transition from one task to another. If a -boolean logic is defined with these statements, it can be used to evaluate whether the transition -should continue or not. Complex boolean logic using a combination of ``not``, ``and``, ``or``, and -parentheses is possible. Take the following workflow as an example: Execution of certain branch -in the workflow depends on the value of ``$.path``. If ``$.path = a``, then task ``a`` is executed. -If ``$.path = b``, then task ``b``. Finally task ``c`` is executed if neither. - -.. literalinclude:: /../../st2/contrib/examples/actions/workflows/mistral-branching.yaml - :language: yaml - -The statement ``with-items`` in Mistral is used to execute an action over iteration of one or more -list of items. The following is a sample Mistral workflow that iterates over the list of given names -to invoke the action to create individual VM. - -.. code-block:: yaml - - version: '2.0' - - examples.create-vms: - type: direct - input: - - names - tasks: - task1: - with-items: name in <% $.names %> - action: examples.create-vm - input: - name: <% $.name %> - -``with-items`` can take more than one list as the following example illustrates. In this case, -a list of VMs and IP addresses are passed as inputs and then iterated through step by step together. - -.. code-block:: yaml - - version: '2.0' - - examples.create-vms: - type: direct - input: - - names - - ips - tasks: - task1: - with-items: - - name in <% $.names %> - - ip in <% $.ips %> - action: examples.create-vm - input: - name: <% $.name %> - ip: <% $.ip %> - -The sections below contain additional YAQL examples of how to work with lists and dictionaries, -that can be used in more advanced ``with-items`` use-cases. - -Dictionaries ------------- - -To create a dictionary, use the ``dict`` function. For example, ``<% dict(a=>123, b=>true) %>`` -returns ``{'a': 123, 'b': True}``. Let's say this dictionary is published to the context as -``dict1``. The keys function ``<% $.dict1.keys() %>`` returns ``['a', 'b']`` and -``<% $.dict1.values() %>`` returns the values ``[123, true]``. Concatenating dictionaries can be -done as ``<% dict(a=>123, b=>true) + dict(c=>xyz) %>`` which returns -``{'a': 123, 'b': True, 'c': 'xyz'}``. - -A specific key-value pair can be accessed by key name such as ``<% $.dict1.get(b) %>`` which -returns ``True``. Given the alternative ``<% $.dict1.get(b, false) %>``, if the key ``b`` does not -exist, then ``False`` will be returned by default. - -Lists ------ - -To create a list, use the ``list`` functions. For example, ``<% list(1, 2, 3) %>`` returns -``[1, 2, 3]`` and ``<% list(abc, def) %>`` returns ``['abc', 'def']``. List concatenation can be -done as ``<% list(abc, def) + list(ijk, xyz) %>`` which returns ``['abc', 'def', 'ijk', 'xyz']``. -If this list is published to the context as ``list1``, items can also be accessed via index such -as ``<% $.list1[0] %>``, which returns ``abc``. - -Queries -------- - -Let's take the following context as an example: - -.. code-block:: json - - { - "vms": [ - { - "name": "vmweb1", - "region": "us-east", - "role": "web" - }, - { - "name": "vmdb1", - "region": "us-east", - "role": "db" - }, - { - "name": "vmweb2", - "region": "us-west", - "role": "web" - }, - { - "name": "vmdb2", - "region": "us-west", - "role": "db" - } - ] - } - -The following YAQL expressions are some sample queries that YAQL is capable of: - -* ``<% $.vms.select($.name) %>`` returns the list of VM names - ``['vmweb1', 'vmdb1', 'vmweb2', 'vmdb2']``. -* ``<% $.vms.select([$.name, $.role]) %>`` returns a list of names and roles as - ``[['vmweb1', 'web'], ['vmdb1', 'db'], ['vmweb2', 'web'], ['vmdb2', 'db']]``. -* ``<% $.vms.select($.region).distinct() %>`` returns the distinct list of regions - ``['us-east', 'us-west']``. -* ``<% $.vms.where($.region = 'us-east').select($.name) %>`` selects only the VMs in us-east - ``['vmweb1', 'vmdb1']``. -* ``<% $.vms.where($.region = 'us-east' and $.role = 'web').select($.name) %>`` selects only the - web server in us-east ``['vmweb1']``. -* ``<% let(myregion => 'us-east', myrole => 'web') -> $.vms.where($.region = $myregion and $.role = $myrole).select($.name) %>`` - selects only the web server in us-east ``['vmweb1']``. - -List to Dictionary ------------------- - -There are cases where it is easier to work with dictionaries rather than lists (e.g. random access -of a value with the key). Let's take the same list of VM records from above and convert it to a -dictionary where VM name is the key and the value is the record. - -YAQL can convert a list of lists to a dictionary where each list contains the key and value. For -example, the expression ``<% dict(vms=>dict($.vms.select([$.name, $]))) %>`` returns the following -dictionary: - -.. code-block:: json - - { - "vms": { - "vmweb1": { - "name": "vmweb1", - "region": "us-east", - "role": "web" - }, - "vmdb1": { - "name": "vmdb1", - "region": "us-east", - "role": "db" - }, - "vmweb2": { - "name": "vmweb2", - "region": "us-west", - "role": "web" - }, - "vmdb2": { - "name": "vmdb2", - "region": "us-west", - "role": "db" - } - } - } - -In this expression, we took the original ``vms`` list, returned a list of ``[name, record]``, and then converted it to a dictionary. - -Other YAQL Functions --------------------- - -YAQL has a list of built-in functions to work with strings, dictionaries, lists, etc. Some of -these are passed through to Python built-in functions (i.e. int, float, pow, regex, round, etc.). - -Mistral adds additional workflow-related functions to the list. For example, the call to function -``<% len(foobar) %>`` to get the length of the string ``foobar`` returns the value ``6``. The -following is a curated list of commonly used functions. Please visit the YAQL documentation and -GitHub repo to explore more options. - -Built-in -^^^^^^^^ - -For the full list of built-in functions, see the `Standard Library section in YAQL docs -`_. Some notable examples: - -* ``float(value)`` converts value to float. -* ``int(value)`` converts value to integer. -* ``str(number)`` converts number to a string. -* ``len(list)`` and ``len(string)`` returns the length of the list and string respectively. -* ``max(a, b)`` returns the larger value between a and b. -* ``min(a, b)`` returns the smaller value between a and b. -* ``regex(expression).match(pattern)`` returns True if expression matches pattern. -* ``regex(expresssion).search(pattern)`` returns the first instance that matches the pattern. -* ``'some string'.toUpper()`` converts the string to all upper case. -* ``'some string'.toLower()`` converts the string to all lower case. -* ``['some', 'list'].contains(value)`` returns True if list contains value. -* ``"one, two, three, four".split(',').select(str($).trim())`` converts a comma separated string - to an array, trimming each element. - -Mistral -^^^^^^^ - -* ``env()`` returns the environment variables passed to the workflow execution on invocation such - as the |st2| Action Execution ID ``st2_execution_id``. - - For example, the expression - ``<% env().st2_action_api_url %>/actionexecutions/<% env().st2_execution_id %>`` returns the API - endpoint for the current workflow execution in |st2| as something like - ``https://127.0.0.1:9101/v1/actionexecutions/874d3d5b3f024c1aa93225ef0bcfcf3a``. - -* To access information about the parent action, the following expressions can be used - ``<% env().get('__actions').get('st2.action').st2_context.parent.api_user %>``, - ``<% env().get('__actions').get('st2.action').st2_context.parent.source_channel %>`` or - ``<% env().get('__actions').get('st2.action').st2_context.parent.user %>``. - - Note that this similar to the ActionChain expressions - ``{{action_context.parent.source_channel}}``, ``{{action_context.parent.user}}`` or - ``{{action_context.parent.api_user}}``. -* ``task(task_name)`` returns the state, state_info, and the result of the given task_name. - -|st2| -^^^^^ - -``st2kv('st2_key_id')`` queries |st2|'s datastore and returns the value for the given key. For -example, the expression ``<% st2kv('system.shared_key_x') %>`` returns the value for a system -scoped key named ``shared_key_x`` while the expression ``<% st2kv('my_key_y') %>`` returns the -value for the user scoped key named ``my_key_y``. - -Please note that the key name should be in quotes otherwise YAQL treats a key name with a dot like -``system.shared_key_x`` as a dict access. - -.. note:: - - If the retrieved value was stored encrypted, ``st2kv`` no longer attempts decryption by default - (as of version 2.4). To decrypt the retrieved value, you must explicitly enable it through the - ``decrypt`` parameter: ``st2kv('st2_key_id', decrypt => true)``. - -Testing YAQL Expressions ------------------------- - -The fastest way to test YAQL expressions with your data is to use the online YAQL evaluator at -http://yaqluator.com/. - -The website allows you to provide sample data and YAQL expressions which you can evaluate in real -time and see the result. This is especially handy when working with more complex expressions. diff --git a/docs/source/orquesta/index.rst b/docs/source/orquesta/index.rst index ebc86c803..effd3497e 100644 --- a/docs/source/orquesta/index.rst +++ b/docs/source/orquesta/index.rst @@ -1,12 +1,12 @@ Orquesta ======== -Orquesta is a workflow engine designed specifically to run natively in |st2|. The goal is to -replace ActionChain and Mistral in the future. Orquesta's workflow definition is described in -YAML and contains a number of changes and improvements over its predecessors. The workflow -definition supports simple sequential workflows to complex workflows with forks, joins, and -sophisticated data queries and transformation. Since Orquesta runs as a subcomponent of |st2|, -it does not require a separate authentication system and database like Mistral. +Orquesta is a workflow engine designed specifically to run natively in |st2|. It has replaced +the Mistral workflow engine, and the goal is to replace ActionChain in the future. Orquesta's +workflow definition is described in YAML and contains a number of changes and improvements over +its predecessors. The workflow definition supports simple sequential workflows to complex +workflows with forks, joins, and sophisticated data queries and transformation. Orquesta +does not require a separate authentication system and database like Mistral did. .. toctree:: :maxdepth: 2 diff --git a/docs/source/reference/runners.rst b/docs/source/reference/runners.rst index 141980b28..bd0b371b8 100644 --- a/docs/source/reference/runners.rst +++ b/docs/source/reference/runners.rst @@ -226,15 +226,3 @@ Runner Parameters ^^^^^^^^^^^^^^^^^ .. include:: /_includes/runner_parameters/action_chain.rst - -Mistral Runner (mistral-v2) ---------------------------- - -This runner is built on top of the Mistral OpenStack project and supports executing complex -workflows. For more information, please refer to the :doc:`Workflows ` and -:doc:`Mistral ` sections of the documentation. - -Runner Parameters -^^^^^^^^^^^^^^^^^ - -.. include:: /_includes/runner_parameters/mistral_v2.rst diff --git a/docs/source/troubleshooting/index.rst b/docs/source/troubleshooting/index.rst index d1562bc70..ed465ab1a 100644 --- a/docs/source/troubleshooting/index.rst +++ b/docs/source/troubleshooting/index.rst @@ -11,7 +11,6 @@ and how to debug and troubleshoot them. basic_chatops_troubleshooting rules sensors - mistral ssh database rest_api_access diff --git a/docs/source/troubleshooting/mistral.rst b/docs/source/troubleshooting/mistral.rst deleted file mode 100644 index 3627c7d3c..000000000 --- a/docs/source/troubleshooting/mistral.rst +++ /dev/null @@ -1,56 +0,0 @@ -Mistral Issues -============== - -This section contains information on how to troubleshoot Mistral-related issues. - -Troubleshooting Mistral Database Upgrade ----------------------------------------- - -The ``mistral`` and ``mistral-api`` services must not be running at time of upgrading the st2mistral -package and the Mistral database schema. If the st2mistral package has been upgraded and the -services are started before the ``mistral-db-manage upgrade head`` CLI command is executed, then the -``mistral-db-manage upgrade head`` command may fail. - -When the ``mistral`` and ``mistral-api`` services run, SQLAlchemy automatically creates tables, -relationships, and indices that do not exist. If there are new database objects in the upgraded -database schema, the ``mistral-db-manage upgrade head`` command will fail because the actual schema -in the database is now different than its specifications, and it no longer can create the new database -objects. - -When that happens, the new database tables, relationships, and indices must be deleted before the -``mistral-db-manage upgrade head`` command can be re-run. For more details, review the version-specific -notes in the :doc:`/install/upgrades` documentation, for the version of Mistral and |st2| you are upgrading -too. - -.. _mistral-workflows-latency: - -Troubleshooting Mistral Workflow Completion Latency ---------------------------------------------------- - -Since v2.7, the results tracking mechanism is replaced with a callback mechanism from Mistral. Instead of |st2| -querying Mistral at regular intervals, Mistral is configured to callback |st2| on task and workflow completion. -See :ref:`mistral-workflows-completion-latency-and-performance` - -Prior to v2.7, |st2| queries Mistral to check on workflow execution status and the status of individual tasks -via st2resultstracker. This ``st2resultstracker`` process saves the state of outstanding workflow executions -in the database, and once a workflow is complete, deletes the state from the database. The process uses -eventlets to simultaneously query multiple workflow results. This can consume significant CPU cycles. - -There are two configurable values for controlling this. These are ``thread_pool_size`` (number of eventlets) -and ``query_interval`` (interval to space out the subsequent queries to Mistral for a single execution). You -can configure these values by editing the ``results_tracker`` section in ``/etc/st2/st2.conf``: - -.. sourcecode:: ini - - [resultstracker] - query_interval = 5 # in seconds - thread_pool_size = 10 - -These values are subject to load conditions in your infrastructure and the number of workflows -you are running. The default value for ``query_interval`` is set to ``5`` (seconds) which is a balance -between the workflow speed and the CPU overhead. With |st2| 2.2 and earlier, this value was ``0.1``. -We have now set the default value to ``5`` seconds to be conservative. This also means the time to detect -a completed workflow in Mistral by |st2| could take as long as 5 seconds. - -If this is unacceptable for you, you can reduce the ``query_interval`` and also -simultaneously check CPU usage for the ``st2resultstracker`` process. diff --git a/docs/source/troubleshooting/self_verification.rst b/docs/source/troubleshooting/self_verification.rst index 5e10e9bd6..6e8c80dae 100644 --- a/docs/source/troubleshooting/self_verification.rst +++ b/docs/source/troubleshooting/self_verification.rst @@ -9,7 +9,7 @@ The script covers the following aspects of |st2|: * Examples pack installation * Commands described in :doc:`/start` * :doc:`/packs` actions -* :doc:`/actionchain` and :doc:`/mistral` Workflows +* :doc:`/actionchain` Workflows To run the |st2| self-verification script: diff --git a/docs/source/workflows.rst b/docs/source/workflows.rst index ef88c703d..e9b4de757 100644 --- a/docs/source/workflows.rst +++ b/docs/source/workflows.rst @@ -13,13 +13,12 @@ manually, or triggered by rules. Workflows can even be called from other workflo To create a workflow action, choose a workflow runner, connect the actions in a workflow definition, and provide the usual action meta data. -|st2| supports three types of workflows - :doc:`Orquesta `, :doc:`ActionChain ` -and :doc:`Mistral `. +|st2| supports two types of workflows - :doc:`Orquesta ` and :doc:`ActionChain `. * :doc:`Orquesta ` is a new workflow engine, designed specifically for |st2|, released in 2019. With Orquesta, you can define simple sequential workflows or complex workflows with forks, joins, - and sophisticated data transformation and queries. It will replace both ActionChains and Mistral. We - recommend you write all new workflows in Orquesta. + and sophisticated data transformation and queries. It has replaced the Mistral workflow engine, and will + also replace ActionChains. We recommend you write all new workflows in Orquesta. **Use Orquesta for all new workflows.** @@ -29,12 +28,8 @@ and :doc:`Mistral `. **Use ActionChain for simple legacy workflows.** -* :doc:`Mistral ` is a dedicated workflow service, originated in OpenStack, integrated - and bundled with |st2|. With Mistral, you can define complex workflow logic with nested - workflows, forks, joins, and policies for error handling, retries, and delays. This is maintained - for legacy reasons, but we recommend all new workflows are written in Orquesta. - - **Use Mistral for legacy complex workflows.** +Mistral workflows are no longer supported. See https://github.com/StackStorm/orquestaconvert for a tool to +convert Mistral workflows to Orquesta. Learn how to define and run workflows: @@ -43,6 +38,3 @@ Learn how to define and run workflows: Orquesta actionchain - mistral - mistral_yaql - mistral_jinja