504 Gateway-Timeout Error when trying to login to st2

[pkaramol]

Describe the problem
Communication with stackstorm when trying to authenticate fails with 504 Gateway Timeout nginx error.

We have set up stackstorm as per the official image but when trying to authenticate it fails with timeout.

Versions

• Host OS: Ubuntu 16.04
• docker: 18.05.0-ce
• docker-compose: 1.21.2
• stackstorm/stackstorm image: st2:latest

To Reproduce
Just setting up the docker version in a remote machine and try to log;

The weird thing is that when setting up the docker-compose locally it works.

Here is the effective nginx configuration from within the container;

root@f94068b4545e:/# nginx -T
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/st2.conf:42
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    application/font-woff                            woff;
    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/conf.d/default.conf:
server {
    listen       80;
    server_name  localhost;

    #charset koi8-r;
    #access_log  /var/log/nginx/host.access.log  main;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    #location ~ \.php$ {
    #    root           html;
    #    fastcgi_pass   127.0.0.1:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}


# configuration file /etc/nginx/conf.d/st2.conf:
#
# nginx configuration to expose st2 webui, redirect HTTP->HTTPS,
# provide SSL termination, and reverse-proxy st2api and st2auth API endpoint.
# To enable:
#    cp ${LOCATION}/st2.conf /etc/nginx/sites-available
#    ln -l /etc/nginx/sites-available/st2.conf /etc/nginx/sites-enabled/st2.conf
# see https://docs.stackstorm.com/install.html for details

# server {
#   listen *:80 default_server;
#
#   add_header Front-End-Https on;
#   add_header X-Content-Type-Options nosniff;
#
#   if ($ssl_protocol = "") {
#        return 301 https://$host$request_uri;
#   }
#
#   index  index.html;
#
#   access_log /var/log/nginx/st2webui.access.log combined;
#   error_log /var/log/nginx/st2webui.error.log;
# }

server {
  listen *:80 default_server;
  add_header Front-End-Https on;
  add_header X-Content-Type-Options nosniff;

  if ($ssl_protocol = "") {
       return 301 https://$host$request_uri;
  }

  index  index.html;

  access_log /var/log/nginx/st2webui.access.log combined;
  error_log /var/log/nginx/st2webui.error.log;
}

server {
  listen       *:443 ssl;
  ssl on;
  ssl_certificate           /etc/ssl/st2/st2.crt;
  ssl_certificate_key       /etc/ssl/st2/st2.key;
  ssl_session_cache         shared:SSL:10m;
  ssl_session_timeout       5m;
  ssl_protocols             TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers               <some-cipher>;
  ssl_prefer_server_ciphers on;
  index  index.html;
  access_log            /var/log/nginx/st2webui.access.log combined;
  error_log             /var/log/nginx/st2webui.error.log;

  add_header              X-Content-Type-Options nosniff;

  location @apiError {
    add_header Content-Type application/json always;
    return 503 '{ "faultstring": "Nginx is unable to reach st2api. Make sure service is running." }';
  }

  location /api/v2/atr_configuration{
    proxy_pass  http://127.0.0.1:8000;
  }

  location /api/v2/available_actions {
    proxy_pass  http://127.0.0.1:8000;
  }

  location /api/v2/sync_execution {
    proxy_pass  http://127.0.0.1:8000;
  }


  location /api/ {
    error_page 502 = @apiError;

    rewrite ^/api/(.*)  /$1 break;

    proxy_pass            http://127.0.0.1:9101/;
    proxy_read_timeout    90;
    proxy_connect_timeout 90;
    proxy_redirect        off;

    proxy_set_header      Host $host;
    proxy_set_header      X-Real-IP $remote_addr;
    proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_set_header Connection '';
    chunked_transfer_encoding off;
    proxy_buffering off;
    proxy_cache off;
    proxy_set_header Host $host;
  }

  location @streamError {
    add_header Content-Type text/event-stream;
    return 200 "retry: 1000\n\n";
  }

  # For backward compatibility reasons, rewrite requests from "/api/stream"
  # to "/stream/v1/stream" and "/api/v1/stream" to "/stream/v1/stream"
  location ~* (/stream/|/api(/v\d)?/stream/?) {
    error_page 502 = @streamError;

    rewrite ^/stream/(.*)  /$1 break;
    rewrite ^/api/stream/?$ /v1/stream break;
    rewrite ^/api(/v\d)?/stream/?$ $1/stream break;

    proxy_pass  http://127.0.0.1:9102;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    # Disable buffering and chunked encoding.
    # In the stream case we want to receive the whole payload at once, we don't
    # want multiple chunks.
    proxy_set_header Connection '';
    chunked_transfer_encoding off;
    proxy_buffering off;
    proxy_cache off;
  }

  location @authError {
    add_header Content-Type application/json always;
    return 503 '{ "faultstring": "Nginx is unable to reach st2auth. Make sure service is running." }';
  }

  location /auth/ {
    error_page 502 = @authError;

    rewrite ^/auth/(.*)  /$1 break;

    proxy_pass            http://127.0.0.1:9100/;
    proxy_read_timeout    90;
    proxy_connect_timeout 90;
    proxy_redirect        off;

    proxy_set_header      Host $host;
    proxy_set_header      X-Real-IP $remote_addr;
    proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass_header     Authorization;

    proxy_set_header Connection '';
    chunked_transfer_encoding off;
    proxy_buffering off;
    proxy_cache off;
    proxy_set_header Host $host;
  }

  location / {
    root      /opt/stackstorm/static/webui/;
    index     index.html;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
  }
}

[pkaramol]

This is a recurring error when performing tail -f /var/log/st2/*.log

Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 155, in db_ensure_indexes
    model_class.ensure_indexes()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 808, in ensure_indexes
    collection = cls._get_collection()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 206, in _get_collection
    cls.ensure_indexes()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 836, in ensure_indexes
    collection.create_index(fields, background=background, **opts)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/pymongo/collection.py", line 1754, in create_index
    self.__create_index(keys, kwargs, session, **cmd_options)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/pymongo/collection.py", line 1642, in __create_index
    with self._socket_for_writes() as sock_info:
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/pymongo/collection.py", line 194, in _socket_for_writes
    return self.__database.client._socket_for_writes()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 968, in _socket_for_writes
    server = self._get_topology().select_server(writable_server_selector)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/pymongo/topology.py", line 224, in select_server
    address))
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/pymongo/topology.py", line 183, in select_servers
    selector, server_timeout, address)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/pymongo/topology.py", line 199, in _select_servers_loop
    self._error_message(selector))
ServerSelectionTimeoutError: No servers found yet

[Kami]

It looks like MongoDB is not running / can't be reached.

I would check that MongoDB is running in that container.

[pkaramol]

It seems that I CAN perform a test connection to mongo from the st2 container

root@6d38dc3431e1:/# python
Python 2.7.6 (default, Nov 23 2017, 15:49:48) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymongo import MongoClient
>>> client = MongoClient('mongo', 27017)
>>> db = client.database_name
>>> collection = db.collection_name
>>> collection.find_one({"name":"name1"})
>>> print collection
Collection(Database(MongoClient(host=['mongo:27017'], document_class=dict, tz_aware=False, connect=True), u'database_name'), u'collection_name')
>>> print client
MongoClient(host=['mongo:27017'], document_class=dict, tz_aware=False, connect=True)

[pkaramol]

What is more, data seem to be created from st2 to mongo;

check this gist;

[pkaramol]

root@b30c2277e1bd:/# st2ctl status
st2actionrunner PID: 3129
st2actionrunner PID: 3131
st2actionrunner PID: 3154
st2actionrunner PID: 3164
st2api PID: 57
st2api PID: 3096
st2stream PID: 59
st2stream PID: 3086
st2auth PID: 48
st2auth PID: 3091
st2garbagecollector PID: 3074
st2notifier PID: 3111
st2resultstracker PID: 3062
st2rulesengine PID: 3176
st2sensorcontainer PID: 3105
st2chatops is not running.
mistral-server PID: 407
mistral.api PID: 399
mistral.api PID: 435
mistral.api PID: 436

[udaymadu]

@pkaramol ,

Have you resolved issue, am facing same issue in cluster environment.

and have started all st2 instances and ran /opt/stackstorm/st2/bin/pip install redis, postgres, rabbitmq, ..all of them showing requirement satisfied

[pkaramol]

Nope...it has been blocking me big time...

[udaymadu]

have you got any help form stackstorm community, can i go ahead and put t there.

[pkaramol]

by all means...! go ahead. any help / feedback on this would be highly valuable.