Containers constantly restarting - failing with `Failed to connect to database "st2" @ "mongo:27017" as user "None": No servers found yet`

[kbakk]

Describe the problem

I tried starting StackStorm with docker-compose using this repo (master). From what I can gather, the MongoDB container is running without issues - I can run docker-compose exec st2client bash, install telnet and do telnet mongo 27017.

In the logs I see stack traces such as this:

st2scheduler_1         | 2020-07-22 16:06:55,390 DEBUG [-] Using Python: 3.6.9 (/opt/stackstorm/st2/bin/python)
st2scheduler_1         | 2020-07-22 16:06:55,391 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
st2scheduler_1         | 2020-07-22 16:06:55,393 DEBUG [-] Using logging config: /etc/st2/logging.scheduler.conf
st2scheduler_1         | 2020-07-22 16:06:55,531 INFO [-] Connecting to database "st2" @ "mongo:27017" as user "None".
st2scheduler_1         | 2020-07-22 16:06:58,552 ERROR [-] Failed to connect to database "st2" @ "mongo:27017" as user "None": No servers found yet
st2scheduler_1         | 2020-07-22 16:06:58,554 ERROR [-] (PID=1) Scheduler quit due to exception.
st2scheduler_1         | Traceback (most recent call last):
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2actions/cmd/scheduler.py", line 124, in main
st2scheduler_1         |     _setup()
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2actions/cmd/scheduler.py", line 53, in _setup
st2scheduler_1         |     register_signal_handlers=True, service_registry=True, capabilities=capabilities)
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/service_setup.py", line 160, in setup
st2scheduler_1         |     db_setup()
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/database_setup.py", line 56, in db_setup
st2scheduler_1         |     connection = db_init.db_setup_with_retry(**db_cfg)
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/persistence/db_init.py", line 75, in db_setup_with_retry
st2scheduler_1         |     ssl_match_hostname=ssl_match_hostname)
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/persistence/db_init.py", line 58, in db_func_with_retry
st2scheduler_1         |     return retrying_obj.call(db_func, *args, **kwargs)
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 206, in call
st2scheduler_1         |     return attempt.get(self._wrap_exception)
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 247, in get
st2scheduler_1         |     six.reraise(self.value[0], self.value[1], self.value[2])
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/six.py", line 696, in reraise
st2scheduler_1         |     raise value
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 200, in call
st2scheduler_1         |     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 169, in db_setup
st2scheduler_1         |     ssl_match_hostname=ssl_match_hostname)
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 151, in _db_connect
st2scheduler_1         |     raise e
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 144, in _db_connect
st2scheduler_1         |     connection.admin.command('ismaster')
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/database.py", line 730, in command
st2scheduler_1         |     read_preference, session) as (sock_info, slave_ok):
st2scheduler_1         |   File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
st2scheduler_1         |     return next(self.gen)
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1298, in _socket_for_reads
st2scheduler_1         |     server = self._select_server(read_preference, session)
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1253, in _select_server
st2scheduler_1         |     server = topology.select_server(server_selector)
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 235, in select_server
st2scheduler_1         |     address))
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 193, in select_servers
st2scheduler_1         |     selector, server_timeout, address)
st2scheduler_1         |   File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 209, in _select_servers_loop
st2scheduler_1         |     self._error_message(selector))
st2scheduler_1         | pymongo.errors.ServerSelectionTimeoutError: No servers found yet

Full output of docker-compose logs: docker-compose-logs.txt

Versions

• Host OS: Debian GNU/Linux 9 (stretch)
• docker: 19.03.12
• docker-compose: 1.26.2
• stackstorm version: v3.3dev / 1f747fc (of docker-compose file)

To Reproduce

Doing a plain docker-compose up -d without any changes / setting environment variables. Was able to replicate both on MacOS 10.15.6 and Debian stretch.