Implement a WinRM runner

[nmaludy]

Description

This is a runner that utilizes WinRM to connect to a remote Windows host for command and script execution. I've chosen to implement it using the pywinrm library (https://github.com/diyan/pywinrm) which seems to be the Python standard, used by tools like Ansible and Salt.

The runner is fairly simplistic, only executing commands on a single host at a time (not multiple in parallel like SSH/remote_runner). Due to some limitations in the pywinrm library, some code needed to be copied over in order to support "standard" StackStorm features such as: environment variables, cwd, and timeouts. Ideally we will fork pywinrm and create a PR for these changes.

It supports both command and script executions. Currently parameters are not passed to the scripts when executing (in the TODO list)

Examples

Here are some example actions and commands to show it's working so others can play around:

/opt/stackstorm/packs/default/winrm_cmd.yaml

---
description: "Run an arbitrary WinRM command"
enabled: true
name: winrm_cmd
pack: default
parameters: {}
runner_type: "winrm-ps-cmd"

/opt/stackstorm/packs/default/winrm_script.yaml

---
description: "Run a WinRM script"
enabled: true
name: winrm_script
entry_point: ps_test.ps1
pack: default
parameters: {}
runner_type: "winrm-ps-script"

Testing winrm-ps-cmd

Simple test

$ st2 run default.winrm_cmd host=hostname.domain.tld username=user@domain.tld password=xxx verify_ssl_cert=false cmd='$PSVersionTable'
..
id: 5b1972fd595497f1ade242cb
status: succeeded
parameters: 
  cmd: $PSVersionTable
  host: hostname.domain.tld
  password: '********'
  username: user@domain.tld
  verify_ssl_cert: false
result: 
  failed: false
  return_code: 0
  stderr: ''
  stdout: "
Name                           Value                                           
----                           -----                                           
PSVersion                      4.0                                             
WSManStackVersion              3.0                                             
SerializationVersion           1.1.0.1                                         
CLRVersion                     4.0.30319.42000                                 
BuildVersion                   6.3.9600.18968                                  
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0}                            
PSRemotingProtocolVersion      2.2                                             


"

This proves that bot the env and cwd options work

$ st2 run default.winrm_cmd host=hostname.domain.tld username=user@domain.tld password=xxx  verify_ssl_cert=false cwd='C:\\Windows' env='{"NICK_TEST": "VALLLUE"}'  cmd='Get-Location; Get-Item Env:NICK_TEST'
..
id: 5b19772c595497f1ade242f8
status: succeeded
parameters: 
  cmd: Get-Location; Get-Item Env:NICK_TEST
  cwd: C:\Windows
  env:
    NICK_TEST: VALLLUE
  host: hostname.domain.tld
  password: '********'
  username: user@domain.tld
  verify_ssl_cert: false
result: 
  failed: false
  return_code: 0
  stderr: ''
  stdout: "
Path                                                                           
----                                                                           
C:\Windows                                                                     

PSPath        : Microsoft.PowerShell.Core\Environment::NICK_TEST
PSDrive       : Env
PSProvider    : Microsoft.PowerShell.Core\Environment
PSIsContainer : False
Key           : NICK_TEST
Value         : VALLLUE
Name          : NICK_TEST



"

This executes the following powershell script /opt/stackstorm/packs/default/actions/ps_test.ps1

$list_of_things = @("a", "b", "c", "d")

foreach ($thing in $list_of_things) {
        Write-Output $thing
}

$ st2 run default.winrm_script host=hostname.domain.tld username=user@domain.tld password=xxx verify_ssl_cert=false
..
id: 5b19800c595497f1ade24311
status: succeeded
parameters: 
  host: hostname@domain.tld
  password: '********'
  username: user@domain.tld
  verify_ssl_cert: false
result: 
  failed: false
  return_code: 0
  stderr: ''
  stdout: "a
b
c
d
"
  succeeded: true

TODO

• Pass parameters to script executions
• Changelog
• Unit tests
• core pack actions
• End-to-end tests
• Documentation
• Examples

[LindsayHill]

Cool. Resolves #1636

[nmaludy]

Support for parameters and string escaping

Just pushed support for pulling out named and positional parameters from the action metadata file and passing them into the PowerShell script.

Also added string quoting/escaping to help in that effort.

/opt/stackstorm/pack/default/actions/winrm_script.yaml

---
description: "Run an arbitrary WinRM script"
enabled: true
name: winrm_script
entry_point: ps_test.ps1
pack: default
parameters:
  p_bool:
    type: boolean
    default: true
  p_integer:
    type: integer
    default: 3
  p_number:
    type: number
    default: 12.34
  p_str:
    type: string
    default: "test"
  p_array:
    type: array
    default: ["a", "b", "c"]
  p_obj:
    type: object
    default:
      a: a_value
      b: b_value
  pos0:
    type: string
    position: 0
    default: "pos arg 0"
  pos1:
    type: string
    position: 1
    default: "pos arg 1"
runner_type: "winrm-ps-script"

/opt/stackstorm/pack/default/actions/ps_test.ps1

[CmdletBinding()]
Param(
  [bool]$p_bool,
  [int]$p_integer,
  [double]$p_number,
  [string]$p_str,
  [array]$p_array,
  [hashtable]$p_obj,
  [Parameter(Position=0)]
  [string]$p_pos0,
  [Parameter(Position=1)]
  [string]$p_pos1
)


Write-Output "p_bool = $p_bool"
Write-Output "p_integer = $p_integer"
Write-Output "p_number = $p_number"
Write-Output "p_str = $p_str"
Write-Output "p_array = $($p_array | ConvertTo-Json -Compress)"
Write-Output "p_obj = $($p_obj | ConvertTo-Json -Compress)"
Write-Output "p_pos0 = $p_pos0"
Write-Output "p_pos1 = $p_pos1"

Execution

$ st2 run default.winrm_script host=hostname.domain.tld username=user@domain.tld password=xxx verify_ssl_cert=false p_str='"test this `$string"'
..
id: 5b1acbcf595497f1ade243e9
status: succeeded
parameters: 
  host: hostname@domain.tld
  p_str: '"test this `$string"'
  password: '********'
  username: user@domain.tld
  verify_ssl_cert: false
result: 
  failed: false
  return_code: 0
  stderr: ''
  stdout: "p_bool = True
p_integer = 3
p_number = 12.34
p_str = "test this `$string"
p_array = ["a","b","c"]
p_obj = {"a":"a_value","b":"b_value"}
p_pos0 = pos arg 0
p_pos1 = pos arg 1
"
  succeeded: true

[LindsayHill]

Looks like this may need to be quoted, otherwise the runner fails to register with yaml.scanner.ScannerError: sequence entries are not allowed here

Seems to allow -- without quoting, but not -.

[nmaludy]

I realized through my testing that the requirements.txt in the runners directory wasn't actually being used by anything.

I just pushed a change that converts over all of the runners to using the same in-requirements.txt / requirements.txt pattern for "components".

Now, the Makefile will treat runners as other st2 components and generate a requirements.txt file from the in-requirements.txt file, consulting the fixed-requirements.txt file for version pins.

After this change, the virtualenv that is built during unit testing contains the requirements specified by the runners.

An additional change is necessary to st2-packages so that these requirements are included in the DEB and RPM packages: StackStorm/st2-packages#562

[nmaludy]

New update:

• Added lots of unit tests, should be almost full coverage now.

• Added a new runner winrm-cmd that executes commands via the Command Prompt instead of PowerShell. There are some commands in Windows that need to be executed on the Command Prompt, so i figured it would be a good option to provide.

• Added new actions to the core pack: core.winrm_cmd that executes remote Command Prompt commands, along with core.winrm_ps_cmd that executes remote PowerShell commands. Both of these are using the new runners.

[nmaludy]

For anyone interested in testing out the new runner. The steps for using it on an existing system are:

git clone https://github.com/nmaludy/st2.git
cd st2
git checkout feature/winrm-runner
cp -r contrib/runners/winrm_runner/
st2ctl reload --register-runners
/opt/stackstorm/st2/bin/pip install pywinrm

After this you can create actions that use the new runners winrm-cmd, winrm-ps-cmd and winrm-ps-script. There are some examples further up in this PR.

[Kami]

Nice work 👍

I will perform a more detailed review shortly.

As far as testing goes - we definitely need some kind of end to end tests which spin up Windows VM and test the functionality end to end, the same as we do for other runners (inside st2tests, etc.).

For that we will need to modify workflows in st2cd, st2ci repos and add new tests to st2tests repo.

[Kami]

Thanks for great working on those changes 👍

I'm looking into it / testing it today :)

[Kami]

Since it's based on an existing code, it would be good to also include under what license that code is (looks like MIT in this case).

[Kami]

I performed some basic testing locally (using Python 2 and 3) and everything looks good 👍

I tagged this PR with v2.9.0 milestone. This means we will need to wait a bit before we can merge it.

I believe @enykeev will start on the v2.8.0 release process soon. As soon as he unfreezes master for new changes I will merge this and work on the CI and all the related changes.

As discussed on Slack - it would be great if you can also work on the documentation change - how it works, how to use it and how to configure it :)

[Kami]

Ready to be merged once master is ready to accept changes for the v2.9.0 release.

[Kami]

Can you please look into the failing tests - I believe they just need to be updated to take into an account a new runner (orchestra).

[Kami]

I reverted ``from st2common import version` change because it needs more work (see #4211 (comment) for details).

This way I can continue with merging and testing those changes

[Kami]

Merged - thanks!

I will continue with merging and testing docs and st2tests PRs tomorrow. It took me way too long today to get this branch in sync and all the tests passing (mostly due to slow CI).