Decrypting K/V values in action parameters causing misleading errors if the keys don't exist and another K/V parameter needs to be converted to bool, array, or object

[jschoewe]

SUMMARY

In an action's parameters, decrypting an encrypted value in the K/V store results in a misleading error message if that key either doesn't exist or is not encrypted

STACKSTORM VERSION

3.5.0

OS, environment, install method

RHEL 8, Python 3.6.8

Steps to reproduce the problem

st2 key set test_bool "true"

test_action.yaml

---
name: 'test_action'
runner_type: 'remote-shell-cmd'
description: 'Test action inputs from the k/v store'
enabled: true
entry_point: ''
pack: 'test_pack'
parameters:
    username:
        type: 'string'
        description: 'Username to login to the remote system'
        required: true
    password:
        type: 'string'
        description: 'Password to login to the remote system'
        required: true
    hosts:
        type: 'string'
        description: 'Comma separated list of hosts to run the command on'
        required: true
    cmd:
        immutable: true
        default: 'echo "TEST: {{ test_bool }}"'
    connect_timeout:
        type: 'integer'
        description: 'SSH connect timeout in seconds'
        default: 30
    test_bool:
        type: boolean
        description: "Test boolean variable"
        default: "{{ st2kv.system.test_bool }}"
    test_encrypt:
      	type: string
        description: "Test encrypted variable"
        required: false
        secret: true
        default: "{{ st2kv.system.test_encrypt | decrypt_kv }}"

st2 run test_pack.test_action username="root" password="pass" hosts="localhost"

Expected Results

Prior to 3.5, if test_encrypt does not exist in the K/V store then a None value would be used for that parameter. If that can't be done then an error message about the test_encrypt parameter would be expected

Actual Results

Running the action above will result in the following misleading error if test_encrypt either is not in the k/v store or if it exists but isn't encrypted:

ERROR: 400 Client Error: Bad Request
MESSAGE: '{{ st2kv.system.test_bool }}' is not of type 'boolean' for url: http://127.0.0.1:9101/executions

[amanda11]

@jschoewe The error being reported is correct, as it is not about the value of test_encrypt that is the problem.

Test_bool is being set as a string value of "true", and therefore ST2 complains that is not of type boolean. Hence the error you see.

If I change the action so test_bool is a string, and dont' have test_encrypt specified then I get an error:

MESSAGE: Failed to render parameter "test_encrypt": Referenced datastore item "st2kv.system.test_encrypt" doesn't exist or it contains an empty string for url: http://127.0.0.1:9101/v1/executions

[jschoewe]

@amanda11 it is not about the value of test_encrypt that is the problem
Yes it is because when I add a correct test_encrypt value to the k/v store the action runs fine.
Test_bool is being set as a string value of "true", and therefore ST2 complains that is not of type boolean. Hence the error you see.
This is not right, stackstorm is smart enough to convert strings from the k/v store into booleans, arrays, objects, etc.
If I change the action so test_bool is a string, and dont' have test_encrypt specified then I get an error:
I don't want test_bool to be a string, and this doesn't change the fact that when it is a boolean it will cause hours of needless debugging due to a misleading error. This is still an issue that needs to be addressed.

[amanda11]

The error message in the example is about the test_bool parameter, not test_encrypt.

If I took test_bool out of the problem, then I couldn't reproduce the problem you are seeing. And got a clear error when test_encrypt was not in the store.

Could you perhaps provide an example that just has the encrypted parameter and provide the error you get?

As from my attempts to reproduce with the information given it is complaining that it has a string being provided for a Boolean, and not related to the test_encrypt parameter.
(It is possible that the behaviour of string to Boolean on key value store has changed - so trying to narrow down the problem).

[amanda11]

I am wondering as well if it's a combination of the two things, test_bool and test_encrypt.

I will try and confirm that if I keep the set on the test_bool as a Boolean and put the encrypted string in for test_encrypt if it works for me.

I am wondering if the action has just a parameter that is encrypted, and nothing else from the key value store - do you get the right error?

But if have both a variable that it has to convert type and a encrypted that isn't set then you get the wrong error.

As when I tried to simplify down to just the encrypted field I couldn't reproduce.

So it would be good if we can simplify the case to see if it's a combination of parameters that is causing your issue.

[jschoewe]

Sorry, I probably could have been more descriptive but I do think it's an issue with a combination of the 2. I am getting the correct error if I don't have to convert anything and I'm also getting bad type errors trying to pass arrays and objects from the k/v store too.

[amanda11]

I've verified it on my system, and agree it's a combination of the two that causes the error.

1. Action expects test_bool as string, and test_encrypt as encrypted.
   Send in no value for test_encrypt get error:

MESSAGE: Failed to render parameter "test_encrypt": Referenced datastore item "st2kv.system.test_encrypt" doesn't exist or it contains an empty string for url: http://127.0.0.1:9101/v1/executions

2. Action expects test_bool as boolean, and test_encrypt as encrypted.
   Send in no value for test_encrypt get error:

ERROR: 400 Client Error: Bad Request
MESSAGE: '{{ st2kv.system.test_bool }}' is not of type 'boolean' for url: http://127.0.0.1:9101/v1/executions

3. Action expects test_bool as boolean, and test_encrypt as encrypted
   Set test_encrypt to an encrypted string.
   Result. Input parameters expected

So I think we agree that it's the combination of both that causes the error. I'll change status to reproduced.

Would you mind amending the title so that we know it's a combination of the two that is the problem, that will make it clearer when trying to identify a fix.

[amanda11]

The problem occurs because when the ParamException is thrown for the invalid value for test_encrypt, it is caught at:

st2/st2api/st2api/controllers/v1/actionexecutions.py

Line 218 in ce1abf9

liveaction_db, actionexecution_db = action_service.create_request(

However in processing that failure, the call to create_request does a validate of parameters, and this doesn't seem to take into account the key/value conversion, and so throws a json error about test_bool.

Not sure, what has changed as couldn't find anything that broke it. Have applied a workaround fix locally so that it doesn't validate the parameter when it's just creating that request so that it can report it as failure. This does mean we then get the correct error returned, but need to do further investigation to see whether that is a viable fix or not.

[amanda11]

@jschoewe "Prior to 3.5, if test_encrypt does not exist in the K/V store then a None value would be used for that parameter. If that can't be done then an error message about the test_encrypt parameter would be expected".

I've just tried on 3.4.1, and I see the same problem (on a CentOS 8 machine), i.e. I'm getting the error about "test_bool" not being a boolean, when I don't set test_encrypt.

Is there a particular ST2 version/OS combination, where you know that you get the expected error?

Trying to investigate where things changed....

[amanda11]

I've gone through the code, and it looks like it's always (or for a very long time, at least since 2019) treated it as an error to decrypt a Key/Value that doesn't exist.

However if you had set the value to "" with --encrypt, then it would return success.

So I am proposing that the fix required is to make sure we get the correct error back, instead of the misleading one about the test_bool when the problem is with test_encrypt.