Connection to mongo db over SSL doesn't work

[alexku7]

SUMMARY

Unable to connect Stackstorm to Mongo DB over SSL.
Stackstorm can't function correctly when connection to MongoDB is encrypted by SSL,
There is exception of "aximum recursion depth exceeded while calling a Python object" in st2api.log and st2auth.log. In this conditions stackstorm is in non functional state at all.

STACKSTORM VERSION

st2 3.1.0, on Python 3.6.9
:

OS, environment, install method

OS: Both the stackstorm and moongodb on Ubuntu 18.04 LTS but installed in diffrent VMs in the same subnet.

All Stackstorm components are installed on one VM but Mongo DB, RAbitMQ on the second VM.

Steps to reproduce the problem

Without SSL all works fine, but after moving a connection to SSL, Stackstorm can't connect to Mongo.

1. Put ssl=true in st2.conf

2. Other parameters related to ssl ,don't really matter . Tested in all combinations of these parameters with and without client certificate authentication and so on.

3. mongo client can connect to monog over ssl without any problem. Simple python script using mongoengine module can connect to mongo over ssl. So it seems as not a problem of underlying infrastructure because all other tools and components can connect to mongo wih SSL but staskstorm can't/

Expected Results

Staskstoem should work with mongo over SSL. That a basic requirements .

Actual Results

Stackstorm starting up but in the st2api. log and st2auth.log we can see every few seconds error

2019-12-15 08:08:45,891 140651884441824 INFO (unknown file) [-] Connecting to database "somedatabase" @ "mongodb-host:27017" as user "someuser".
2019-12-15 08:08:45,892 140651884441824 WARNING (unknown file) [-] Retry on ConnectionError - Cannot connect to database default :
maximum recursion depth exceeded while calling a Python object

it seems like some bug in __init__py file in _db_connect function in line number 125 where the connection attempt goes to endless recursion or something likes this.

[Kami]

What version of MongoDB are you using?

Can you please also gist the whole log file (with sensitive information removed)? And I assume you installed StackStorm using Ubuntu Bionic packages?

There should likely be another error logged early in the logs before the "maximum recursion depth exceeded while calling a Python object".

[alexku7]

Hello
I used mongo 4 and i installed exactly by instructions here https://ewc-docs.extremenetworks.com/install/u18.html when all dependencies like mongo, rabitmq were installed on one VM and the staskstorm itself on another VM.

The full api.log in debug mode contains some additional warning but no clue for any endless recursion.

The very interesting thing that if i run st2ctl reload, i see in the console that st2 succeeds to connect to mongo and i even can see the created collections with some data in mongo. So, again, it's not a problem of infrastructure , because even st2 itself can connect over ssl and even create data in mongo while running st2reload command but in regular startup routine something goes wrong.

here is the log:

2019-12-16 10:27:58,567 140280891643712 DEBUG cfg [-] Expected default value of type(s) list but got '' of type str
2019-12-16 10:27:59,613 140280891643712 INFO (unknown file) [-] Creating st2api: StackStorm v3.1.0 as OpenAPI app.
2019-12-16 10:27:59,773 140280832733408 DEBUG cfg [-] Expected default value of type(s) list but got '' of type str
2019-12-16 10:27:59,872 140280832733408 INFO (unknown file) [-] Connecting to database "somedb" @ "mongo-host-vm:27017" as user "someuser".
2019-12-16 10:27:59,883 140280832733408 WARNING (unknown file) [-] Retry on ConnectionError - Cannot connect to database default :
maximum recursion depth exceeded while calling a Python object
2019-12-16 10:28:01,893 140280832733408 INFO (unknown file) [-] Connecting to database "somedb" @ "mongo-host-vm:27017" as user "someuser".
2019-12-16 10:28:01,896 140280832733408 WARNING (unknown file) [-] Retry on ConnectionError - Cannot connect to database default :
maximum recursion depth exceeded while calling a Python object
2019-12-16 10:28:05,899 140280832733408 INFO (unknown file) [-] Connecting to database "somedb" @ "mongo-host-vm:27017" as user "someuser".
2019-12-16 10:28:05,901 140280832733408 WARNING (unknown file) [-] Retry on ConnectionError - Cannot connect to database default :
maximum recursion depth exceeded while calling a Python object
2019-12-16 10:28:13,907 140280832733408 INFO (unknown file) [-] Connecting to database "somedb" @ "mongo-host-vm:27017" as user "someuser".
2019-12-16 10:28:13,909 140280832733408 WARNING (unknown file) [-] Retry on ConnectionError - Cannot connect to database default :
maximum recursion depth exceeded while calling a Python object

[Kami]

I was able to reproduce the issue.

It only affects Python3 (Python 3.x) installations and it's quite a nasty one.

The issue is related to eventlet monkey patch we do. This monkey patch happens after ssl module is already imported by some package so it breaks MongoDB ssl import.

Here is the original exception with more context:

Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/mongoengine/connection.py", line 286, in _create_connection
    return connection_class(**connection_settings)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 534, in __init__
    username, password, dbase, opts)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/client_options.py", line 157, in __init__
    self.__pool_options = _parse_pool_options(options)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/client_options.py", line 128, in _parse_pool_options
    ssl_context, ssl_match_hostname = _parse_ssl_options(options)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/client_options.py", line 103, in _parse_ssl_options
    match_hostname)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/ssl_support.py", line 127, in get_ssl_context
    ctx.options |= getattr(ssl, "OP_NO_SSLv2", 0)
  File "/usr/lib/python3.6/ssl.py", line 465, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/lib/python3.6/ssl.py", line 465, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/lib/python3.6/ssl.py", line 465, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  [Previous line repeated 321 more times]
RecursionError: maximum recursion depth exceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/cmd/api.py", line 80, in main
    _setup()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2api/cmd/api.py", line 54, in _setup
    service_registry=True, capabilities=capabilities)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/service_setup.py", line 160, in setup
    db_setup()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/database_setup.py", line 56, in db_setup
    connection = db_init.db_setup_with_retry(**db_cfg)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/persistence/db_init.py", line 81, in db_setup_with_retry
    ssl_match_hostname=ssl_match_hostname)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/persistence/db_init.py", line 63, in db_func_with_retry
    return db_func(*args, **kwargs)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 160, in db_setup
    ssl_match_hostname=ssl_match_hostname)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 129, in _db_connect
    **ssl_kwargs)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/mongoengine/connection.py", line 367, in connect
    return get_connection(alias)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/mongoengine/connection.py", line 275, in get_connection
    **conn_settings)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/mongoengine/connection.py", line 289, in _create_connection
    'Cannot connect to database %s :\n%s' % (alias, e))
mongoengine.connection.MongoEngineConnectionError: Cannot connect to database default :
maximum recursion depth exceeded

I'm looking into a fix.

[arm4b]

For reference cross-linking the original Forum thread: https://forum.stackstorm.com/t/cant-setup-stackstorm-with-mongodb-ssl-and-replicaset/962/11

[alexku7]

Hello
I tested the proposed fix and i strongly believe that the issue has been fixed partially.
While st2api actually is connecting to mongo over ssl, the other components still can't and I see exactly the same error.
I think you still missed to fix some files, For example, api.py file exists under st2stream component, and st2auth as well but this file wasn't fixed.
Also, other components, like st2actionrunner, sensorcontainer and so on - all of them can;t connect to mongo over ssl.
I found something about 10 files with monkey_patch that remained on the same place in the files and seems it has to be fixed as well ..

Can you check this issue again, please... it's very important for us. We really stuck without ability to connect over ssl as this one of major requirements for us