Implement orquesta-rehearse command to run unit tests for workflow #209

guzzijones · 2020-06-11T20:30:07Z

Implement orquesta-rehearse command to allow users run unit tests for workflow definition. The command takes a base path (typically the StackStorm pack directory) and the relative path to the test spec which contains detail about which workflow definition and expectation to test.

The following is the output of displaying help for the command.

$ orquesta-rehearse -h
usage: A utility for testing orquesta workflows. [-h] -p BASE_PATH
                                                 (-f TEST_SPEC | -d TEST_SPEC_DIR)
                                                 [--debug]

optional arguments:
  -h, --help            show this help message and exit
  -p BASE_PATH, --base-path BASE_PATH
                        The base path where the workflow definition, tests,
                        and files are located.
  -f TEST_SPEC, --test-spec TEST_SPEC
                        The path to the test spec (relative from base path) to
                        run the test.
  -d TEST_SPEC_DIR, --test-spec-dir TEST_SPEC_DIR
                        The path to the test spec directory (relative from
                        base path) to run multiple tests.
  --debug               Set the log level to debug.

The following is the sample workflow used for the tests below.

version: 1.0

description: A basic sequential workflow.

input:
  - name

vars:
  - greeting: null

output:
  - greeting: <% ctx().greeting %>

tasks:
  task1:
    action: core.echo message=<% ctx().name %>
    next:
      - when: <% succeeded() %>
        publish: greeting=<% result() %>
        do: task2
  task2:
    action: core.echo
    input:
      message: "All your base are belong to us!"
    next:
      - when: <% succeeded() %>
        publish:
          - greeting: <% ctx("greeting") %>, <% result() %>
        do:
          - task3
  task3:
    action: core.echo message=<% ctx('greeting') %>
    next:
      - when: <% succeeded() %>
        publish: greeting=<% result() %>

The following is the sample test spec for the successful tests below.

---
workflow: workflows/sequential.yaml
inputs:
  name: Stanley
expected_task_sequence:
  - task1
  - task2
  - task3
  - continue
mock_action_executions:
  - task_id: task1
    result: Stanley
  - task_id: task2
    result: All your base are belong to us!
  - task_id: task3
    result: Stanley, All your base are belong to us!
expected_output:
  greeting: Stanley, All your base are belong to us!

The following is the output for executing a successful test.

$ orquesta-rehearse -p /path/to/stackstorm/pack s -f tests/success.yaml
2020-12-18 21:16:29,232 - INFO - The test spec "/path/to/stackstorm/pack/tests/success.yaml" is successfully loaded.
2020-12-18 21:16:29,232 - INFO - Start test for workflow "/path/to/stackstorm/pack/workflows/sequential.yaml".
2020-12-18 21:16:29,309 - INFO - Completed running test and the workflow execution matches the test spec.

The following is the output for executing a failed test.

$ orquesta-rehearse -p /path/to/stackstorm/pack s -f tests/failure.yaml
2020-12-18 21:23:47,677 - INFO - The test spec "/path/to/stackstorm/pack/tests/failure.yaml" is successfully loaded.
2020-12-18 21:23:47,678 - INFO - Start test for workflow "/path/to/stackstorm/pack/workflows/sequential.yaml".
2020-12-18 21:23:47,825 - ERROR - The actual task sequence ['task1', 'task2', 'task3', 'continue'] does not match expected ['task1', 'task3', 'task2', 'continue'].

The following is the output for executing a successful test with debug turned on.

$ orquesta-rehearse -p /path/to/stackstorm/pack s -f tests/success.yaml --debug
2020-12-18 21:16:32,303 - INFO - The test spec "/path/to/stackstorm/pack/tests/success.yaml" is successfully loaded.
2020-12-18 21:16:32,304 - INFO - Start test for workflow "/path/to/stackstorm/pack/workflows/sequential.yaml".
2020-12-18 21:16:32,334 - DEBUG - Task "task1__r0" is set to "running".
2020-12-18 21:16:32,334 - DEBUG - Applying action execution with "succeeded" status to task "task1__r0".
2020-12-18 21:16:32,337 - DEBUG - Task "task1__r0" is set to "succeeded".
2020-12-18 21:16:32,339 - DEBUG - Task "task2__r0" is identified to run next.
2020-12-18 21:16:32,346 - DEBUG - Task "task2__r0" is set to "running".
2020-12-18 21:16:32,346 - DEBUG - Applying action execution with "succeeded" status to task "task2__r0".
2020-12-18 21:16:32,350 - DEBUG - Task "task2__r0" is set to "succeeded".
2020-12-18 21:16:32,363 - DEBUG - Task "task3__r0" is identified to run next.
2020-12-18 21:16:32,371 - DEBUG - Task "task3__r0" is set to "running".
2020-12-18 21:16:32,371 - DEBUG - Applying action execution with "succeeded" status to task "task3__r0".
2020-12-18 21:16:32,392 - DEBUG - Task "task3__r0" is set to "succeeded".
2020-12-18 21:16:32,412 - DEBUG - The workflow completed with status "succeeded".
2020-12-18 21:16:32,413 - DEBUG - The workflow output with "{'greeting': 'Stanley, All your base are belong to us!'}".
2020-12-18 21:16:32,415 - DEBUG - The workflow completed with no errors.
2020-12-18 21:16:32,415 - DEBUG - Comparing with expected task execution sequence.
2020-12-18 21:16:32,416 - DEBUG - Comparing with expected workflow route(s).
2020-12-18 21:16:32,417 - DEBUG - Comparing with expected workflow status.
2020-12-18 21:16:32,419 - DEBUG - Comparing with expected workflow output.
2020-12-18 21:16:32,420 - INFO - Completed running test and the workflow execution matches the test spec.

The following is the output for executing multiple test specs in a directory. Please note that if using result_path in the mock action execution, then the results should be stored in separate files under a subdirectory.

$ orquesta-rehearse -p /path/to/stackstorm/pack -d tests
2020-12-18 22:05:06,306 - INFO - Using the test spec directory "/path/to/stackstorm/pack/tests".
2020-12-18 22:05:06,307 - INFO - Identified the following test specs in the directory: tests/failure.yaml, tests/success.yaml
2020-12-18 22:05:06,323 - INFO - The test spec "/path/to/stackstorm/pack/tests/failure.yaml" is successfully loaded.
2020-12-18 22:05:06,323 - INFO - Start test for workflow "/path/to/stackstorm/pack/workflows/sequential.yaml".
2020-12-18 22:05:06,363 - ERROR - The actual task sequence ['task1', 'task2', 'task3', 'continue'] does not match expected ['task1', 'task3', 'task2', 'continue'].
2020-12-18 22:05:06,376 - INFO - The test spec "/path/to/stackstorm/pack/tests/success.yaml" is successfully loaded.
2020-12-18 22:05:06,376 - INFO - Start test for workflow "/path/to/stackstorm/pack/workflows/sequential.yaml".
2020-12-18 22:05:06,422 - INFO - Completed running test and the workflow execution matches the test spec.
2020-12-18 22:05:06,422 - ERROR - There are errors processing test spec(s). Please review details above.

NOTE: Workflow definition that uses YAQL/Jinja functions that communicates with the StackStorm database is not supported in this PR.

guzzijones

fixes for fixture_mock

orquesta/fixture_mock.py

codecov-commenter · 2020-06-15T16:57:32Z

Codecov Report

Merging #209 into master will decrease coverage by 0.75%.
The diff coverage is 92.26%.

@@            Coverage Diff             @@
##           master     #209      +/-   ##
==========================================
- Coverage   94.04%   93.28%   -0.76%     
==========================================
  Files          41       45       +4     
  Lines        2735     3141     +406     
  Branches      545      644      +99     
==========================================
+ Hits         2572     2930     +358     
- Misses        100      128      +28     
- Partials       63       83      +20

Impacted Files	Coverage Δ
orquesta/composers/base.py	`91.66% <0.00%> (ø)`
orquesta/composers/mock.py	`57.14% <0.00%> (ø)`
orquesta/requests.py	`100.00% <ø> (ø)`
orquesta/specs/mistral/__init__.py	`100.00% <ø> (ø)`
orquesta/specs/mistral/v2/policies.py	`100.00% <ø> (ø)`
orquesta/specs/native/__init__.py	`100.00% <ø> (ø)`
orquesta/utils/context.py	`90.47% <ø> (ø)`
orquesta/utils/date.py	`89.65% <ø> (ø)`
orquesta/utils/dictionary.py	`90.90% <ø> (ø)`
orquesta/utils/expression.py	`100.00% <ø> (ø)`
... and 43 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8035754...f5638a9. Read the comment docs.

guzzijones

fixture errors are not workflow errors

orquesta/fixture_mock.py

… workflow conduct

guzzijones · 2020-06-22T16:31:41Z

This isn't so much a WIP anymore. I have done some internal testing and fixed some bugs. Still need to cleanup some python2 patching in the unit tests for builtins.

m4dcoder · 2020-06-22T23:06:44Z

@guzzijones

Please move test related changes under orquesta/tests.

Can you move the mock classes under orquesta/tests/mocks.py?
Can you move the exception classes under orquesta/tests/exceptions.py?

As for example fixture, is it possible to simplify to the following?

---
  workflow: file_mitigate.yaml

  routes: [[],["indicators_exist__t0"]]

  task_sequence:
    - task: file_exist
      # route default to 0 if not given
      route: 0
      status: succeeded
      result: true
    - task: extract_indicators
      status: succeeded
      result_file: /path/to/extract_indicators/result.json
    - task: indicators_exist
      status: succeeded
      result: None
    - task: fail
      route: 1
      result: None

  workflow_status: failed

  workflow_output: ...

For result too large to include in the test fixture, it is cleaner to save and point to a different file.

/path/to/extract_indicators/result.json

{
  "Hash": []
}

m4dcoder

Please address my comments. Thanks.

m4dcoder · 2020-06-22T23:08:05Z

@guzzijones Also, I don't think this address with-items task yet.

m4dcoder · 2020-06-22T23:11:17Z

@guzzijones I am also wondering we can add a separate command to list the routes for a workflow for reference. Therefore, users don't need to figure out expected routes for the workflow and users can use that reference to indicate which route number to use in the test fixtures.

guzzijones · 2020-06-23T14:23:53Z

@guzzijones

Please move test related changes under orquesta/tests.

Can you move the mock classes under orquesta/tests/mocks.py?

Yes will do

Can you move the exception classes under orquesta/tests/exceptions.py?

Yes will do

As for example fixture, is it possible to simplify to the following?

---
  workflow: file_mitigate.yaml

  routes: [[],["indicators_exist__t0"]]

  task_sequence:
    - task: file_exist
      # route default to 0 if not given

I need to understand what 0 means in this context?

  route: 0
  status: succeeded
  result: true
- task: extract_indicators
  status: succeeded
  result_file: /path/to/extract_indicators/result.json
- task: indicators_exist
  status: succeeded
  result: None
- task: fail
  route: 1
  result: None

workflow_status: failed

workflow_output: ...


For result too large to include in the test fixture, it is cleaner to save and point to a different file.

/path/to/extract_indicators/result.json

I will attempt to add result_file as a mutually exclusive property vs result

{
  "Hash": []
}

Add orquesta-rehearse command that takes -p base path and -t relative path for test spec as arguments to run the test for the workflow definition.

guzzijones · 2020-12-18T23:26:19Z

I will run through this on Monday with a fresh workflow and test the UI. I need to test a workflow with multiple routes. It helped me a lot in debug mode for the route to print out and then I could go back and fix my test fixture

guzzijones · 2020-12-18T23:35:04Z

So you can't interpret the expected_task_sequence from the mock_action_executions? @m4dcoder

m4dcoder · 2020-12-18T23:40:40Z

So you can't interpret the expected_task_sequence from the mock_action_executions?

@guzzijones Can you clarify your question just in case?

The expected_task_sequences and mock_action_executions are separated because there are use cases where a task can have multiple action executions. The relationship of task to action execution is one:many.

The most obvious example is with items. There can be multiple mock action executions for a with items task without triggering a new task sequence entry. The other example is task retry which doesn't write a new entry in the task sequence because that will inflate the size of the workflow state.

Also, there are cases where we want to test intermediate status changes (i.e. pending) and not necessary directly from running to succeeded, failed, or other completed status. This requires multiple mock action executions for the task.

I think I see what you mean. It is possible to have the mock action executions as an array under the task in the task sequence. I separate them for readability to avoid mock action executions interlace with the task sequence. It's just harder to read that way. Now I can look at a test spec and read the expected task sequence in its entirety.

guzzijones · 2020-12-18T23:53:02Z

So you can't interpret the expected_task_sequence from the mock_action_executions?

@guzzijones Can you clarify your question just in case?

The expected_task_sequences and mock_action_executions are separated because there are use cases where a task can have multiple action executions. The relationship of task to action execution is one:many.

The most obvious example is with items. There can be multiple mock action executions for a with items task without triggering a new task sequence entry.

Also, there are cases where we want to test intermediate status changes (i.e. pending) and not necessary directly from running to succeeded, failed, or other completed status. This requires multiple mock action executions for the task.

Ah yes, I don't think my way of using the task name as a key in the spec would have worked in the case of a cycle. I sorta cheated with with-items results and just used a list before.

m4dcoder · 2020-12-18T23:56:35Z

Btw, if the mock action execution is just default (succeeded and no result), then you don't need to provide a mock action execution entry for that task.

guzzijones · 2020-12-19T00:10:00Z

It appears this new test fixture spec diverges from this comment from before: #209 (comment).

I am curious why the change?

m4dcoder · 2020-12-19T02:15:52Z

@guzzijones From the comment you link to here, I see three differences 1) move class to namespace to orquesta/tests/mocks 2) move exceptions to namespace orquesta/tests/exceptions 3) different schema for task sequences.

1 + 2) I refactored and renamed the set of specs and classes that was implemented initially. On reconsideration where the classes should reside, the orquesta/tests namespace is specifically for unit tests and other modules related testing. Since WorkflowRehearsal and related classes are exposed as user features, this should be moved to the main orquesta namespace.
3) This is already covered in the comments above on task sequence and mock action executions.

guzzijones · 2020-12-19T02:42:12Z

Ah. Yes I missed this comment somehow.
"I think I see what you mean. It is possible to have the mock action executions as an array under the task in the task sequence. I separate them for readability to avoid mock action executions interlace with the task sequence. It's just harder to read that way. Now I can look at a test spec and read the expected task sequence in its entirety."

guzzijones · 2020-12-21T19:46:41Z

I have rewritten 2 workflow test specs in this new format and tested them against workflows. It works. 👍

If I put in an incorrect expected_routes I see is that it doesn't print out the routes array for me. So I can't paste it into the 'expected_routes' section. Instead I get this diff which is hard to decipher.
You notice some text is cut off has[49 chars]icket__t0 below on the task names which makes it hard to read.

 Comparing with expected workflow route(s).
2020-12-21 19:35:21,489 - ERROR - Lists differ: [[], [117 chars]__t0'], ['has_traffic__t0', 'bc_check__t0', 'h[75 chars]t0']] != [[], [117 chars]__t0', 'add_comment_mitigate_indicators_no_follow_up__t0']]

First differing element 3:
['has[49 chars]icket__t0']
['has[49 chars]icket__t0', 'add_comment_mitigate_indicators_no_follow_up__t0']

First list contains 1 additional elements.
First extra element 4:
['has_traffic__t0', 'bc_check__t0', 'has_traffic_pub_ticket__t0', 'add_comment_mitigate_indicators_no_follow_up__t0']

  [[],
   ['has_traffic__t0'],
   ['has_traffic__t0', 'bc_check__t0'],
-  ['has_traffic__t0', 'bc_check__t0', 'has_traffic_pub_ticket__t0'],
   ['has_traffic__t0',
    'bc_check__t0',
    'has_traffic_pub_ticket__t0',
    'add_comment_mitigate_indicators_no_follow_up__t0']] : The lists of workflow routes do not match.

The diff syntax is hard to read without color coding.
My solution was to just print out : X not equal to Y so I could then just copy paste the routes into my workflow test fixture.
I think you need to say found X vs expected Y instead of first differing element...
That UX needs work, imo.

The same UX goes for if I have missing tasks, etc.

After rewriting a large workflow in this new format I don't feel like separating the expected_task_sequence from the mock_action_results adds much value. I end up with tasks in 2 different parts of the spec and the routes are now formatted as __rX

m4dcoder · 2020-12-21T21:16:17Z

@guzzijones

Sure, I can look at improving the error messages.

The task sequences and mock action executions schema will stay as is. The list of action executions need to be treated as a stream of action execution updates into the workflow engine. I cannot do that if the list is merged with the task sequences.

Refactor error messages in workflow rehearsal to be more user friendly when printed out to console.

guzzijones · 2020-12-22T23:12:19Z

Works for me

(venv_orquesta_bin) [xxxx@xxxx]$ orquesta-rehearse -p /shared/dev/e390084/lmintel/actions/workflows -f tests/mitigate_success_no_traffic.yaml
2020-12-22 23:10:07,473 - INFO - The test spec "/shared/dev/e390084/lmintel/actions/workflows/tests/mitigate_success_no_traffic.yaml" is successfully loaded.
2020-12-22 23:10:07,473 - INFO - Start test for workflow "/shared/dev/e390084/lmintel/actions/workflows/mitigate.yaml".
2020-12-22 23:10:09,195 - ERROR - The actual routes [[], ['has_traffic__t0'], ['has_traffic__t0', 'bc_check__t0'], ['has_traffic__t0', 'bc_check__t0', 'has_traffic_pub_ticket__t0'], ['has_traffic__t0', 'bc_check__t0', 'has_traffic_pub_ticket__t0', 'add_comment_mitigate_indicators_no_follow_up__t0']] does not match expected [[], ['has_traffic__t0'], ['has_traffic__t0', 'bc_check__t0'], ['has_traffic__t0', 'bc_check__t0', 'has_traffic_pub_ticket__t0', 'add_comment_mitigate_indicators_no_follow_up__t0']].

guzzijones · 2021-01-13T13:48:16Z

Any plans to merge this soon?

m4dcoder · 2021-01-17T05:49:04Z

@guzzijones Yes, we will merge this before the next StackStorm release.

guzzijones added 5 commits June 11, 2020 19:09

WIP: initial working workflow testing user interface

4f5bf39

refactor orquesta-test console script

d95420d

fix python2 import mock

b89c02c

fix blank lines pep8

249504b

pep8 fixes

a865ce2

guzzijones commented Jun 15, 2020

View reviewed changes

orquesta/fixture_mock.py Outdated Show resolved Hide resolved

orquesta/fixture_mock.py Outdated Show resolved Hide resolved

black fixes

db2d68c

guzzijones commented Jun 15, 2020

View reviewed changes

orquesta/fixture_mock.py Outdated Show resolved Hide resolved

guzzijones added 10 commits June 15, 2020 17:21

log errors for fixture to fixture member var

3c31331

add success message

3f071ed

added path variable for workflow spec file, fixed exception missed on…

05359ac

… workflow conduct

black fix

9be6e96

unit test fix for fixture spec

485a567

test for spec error, do not throw on workflow assert error

a9cc96a

yaml does not handle tuples, force tuple to list

0d288cc

add tuple test for term tasks

1fd21fb

allow for lists for expected tasks instead of tuples

b88dcdf

fix property requirement for routes

22d4015

guzzijones changed the title ~~WIP - add console script to test orquesta workflows~~ add console script to test orquesta workflows Jun 22, 2020

guzzijones added 2 commits June 22, 2020 17:54

fix 2.7 mock builtin

2a35407

test an entire directory

e99f161

m4dcoder requested changes Jun 22, 2020

View reviewed changes

add process directory

4c3f170

guzzijones closed this Jun 23, 2020

m4dcoder added 5 commits December 16, 2020 23:20

Rename assert_conducting_sequences to assert_conducting_sequence

0d2eca2

Add result path to mock action execution to put result in separate files

dc6bd64

Support JSON and YAML files for the mock action execution result path

c72e9d9

Add orquesta-rehearse CLI to test workflow definition

313126a

Add orquesta-rehearse command that takes -p base path and -t relative path for test spec as arguments to run the test for the workflow definition.

Remove unused orquesta/tests/exceptions.py missed in rollback

821f780

m4dcoder changed the title ~~add console script to test orquesta workflows~~ Implement orquesta-rehearse command to allow users run unit tests for workflow definition Dec 18, 2020

m4dcoder changed the title ~~Implement orquesta-rehearse command to allow users run unit tests for workflow definition~~ Implement orquesta-rehearse command to run unit tests for workflow Dec 18, 2020

m4dcoder added 2 commits December 18, 2020 21:37

Refactor logging to allow orquesta-rehearse to specify log level

9193d46

Add option to pass a directory to run multiple test specs

957b617

m4dcoder removed the WIP label Dec 18, 2020

Refactor error messages in workflow rehearsal

a36a141

Refactor error messages in workflow rehearsal to be more user friendly when printed out to console.

m4dcoder added 3 commits January 17, 2021 06:07

Merge remote-tracking branch 'upstream/master' into test_module

7abc001

Add StackStorm authors copyright to affected files

af49f0d

Update chardet requirements as it is conflicting with requests

12f97c4

m4dcoder approved these changes Jan 17, 2021

View reviewed changes

m4dcoder merged commit 0cacb39 into StackStorm:master Jan 17, 2021

arm4b mentioned this pull request Aug 23, 2021

Add AJ (@guzzijones) to the TSC Maintainers StackStorm/st2#5340

Merged

Implement orquesta-rehearse command to run unit tests for workflow #209

Implement orquesta-rehearse command to run unit tests for workflow #209

Uh oh!

Conversation

guzzijones commented Jun 11, 2020 • edited by m4dcoder Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guzzijones left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Jun 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

guzzijones left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

guzzijones commented Jun 22, 2020

Uh oh!

m4dcoder commented Jun 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

m4dcoder left a comment

Choose a reason for hiding this comment

Uh oh!

m4dcoder commented Jun 22, 2020

Uh oh!

m4dcoder commented Jun 22, 2020

Uh oh!

guzzijones commented Jun 23, 2020

Uh oh!

guzzijones commented Dec 18, 2020

Uh oh!

guzzijones commented Dec 18, 2020

Uh oh!

m4dcoder commented Dec 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guzzijones commented Dec 18, 2020

Uh oh!

m4dcoder commented Dec 18, 2020

Uh oh!

guzzijones commented Dec 19, 2020

Uh oh!

m4dcoder commented Dec 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guzzijones commented Dec 19, 2020

Uh oh!

guzzijones commented Dec 21, 2020

Uh oh!

m4dcoder commented Dec 21, 2020

Uh oh!

guzzijones commented Dec 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guzzijones commented Jan 13, 2021

Uh oh!

m4dcoder commented Jan 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

guzzijones commented Jun 11, 2020 •

edited by m4dcoder

Loading

codecov-commenter commented Jun 15, 2020 •

edited

Loading

m4dcoder commented Jun 22, 2020 •

edited

Loading

m4dcoder commented Dec 18, 2020 •

edited

Loading

m4dcoder commented Dec 19, 2020 •

edited

Loading

guzzijones commented Dec 22, 2020 •

edited

Loading