Increased x1.5-x2.0 memory usage after recent pip upgrade [staging-unstable]

[arm4b]

Critical issue after recent massive pip versions upgrade (#3538) is x1.5-x2.0 increased memory usage for entire st2 fleet.

This affects staging-unstable packages for now. I could spot it by a lot of Cannot allocate memory errors in 2GB Vagrant Xenial while installing recent staging-unstable st2 via curl|bash installer.

Further exploration:

st2 2.4dev-40 Xenial (OK)

st2 version:

root@ubuntu16:~# apt list --installed st2
Listing... Done
st2/xenial,now 2.4dev-40 amd64 [installed,upgradable to: 2.4dev-41]

st2 summ memory usage (exluding mistral and other services):

root@ubuntu16:~# ps -eo rss,pid,euser,args:100 --sort %mem | grep -v mistral | grep -i '/opt/stackstorm' | awk '{sum+=$1} END {print sum / 1024}'
1098.45

st2 memory usage by process:

root@ubuntu16:~# ps -eo rss,pid,euser,args:100 --sort %mem | grep -v mistral | grep -i '/opt/stackstorm' | awk '{printf $1/1024 "MB"; $1=""; print }'

25.8594MB 23853 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2auth.wsgi:application -k eventlet -b 127.0.0.1:9100 --workers 1 --threads 1 --graceful-timeout 10 --timeout 30
25.9023MB 23844 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2api.wsgi:application -k eventlet -b 127.0.0.1:9101 --workers 1 --threads 1 --graceful-timeout 10 --timeout 30
26.0625MB 23849 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2stream.wsgi:application -k eventlet -b 127.0.0.1:9102 --workers 1 --threads 10 --graceful-timeout 10 --timeout 30
40.7383MB 23857 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2garbagecollector --config-file /etc/st2/st2.conf
41.832MB 23873 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2sensorcontainer --config-file /etc/st2/st2.conf
45.8359MB 23865 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2resultstracker --config-file /etc/st2/st2.conf
46.1836MB 24110 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/local/lib/python2.7/site-packages/st2reactor/container/sensor_wrapper.py --pack=linux --file-path=/opt/stackstorm/packs/linux/sensor$
47.1016MB 23869 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2rulesengine --config-file /etc/st2/st2.conf
48.1523MB 23861 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2notifier --config-file /etc/st2/st2.conf
53.7891MB 23808 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
53.8281MB 23811 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
53.8398MB 23836 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
53.8867MB 23814 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
53.8867MB 23822 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
53.9102MB 23829 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
53.9492MB 23839 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
53.9531MB 23820 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
53.9961MB 23832 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
54MB 23817 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
69.0039MB 24049 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2stream.wsgi:application -k eventlet -b 127.0.0.1:9102 --workers 1 --threads 10 --graceful-timeout 10 --timeout 30
70.8867MB 24050 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2auth.wsgi:application -k eventlet -b 127.0.0.1:9100 --workers 1 --threads 1 --graceful-timeout 10 --timeout 30
71.1055MB 24055 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2api.wsgi:application -k eventlet -b 127.0.0.1:9101 --workers 1 --threads 1 --graceful-timeout 10 --timeout 30

st2 2.4dev-41 Xenial (x2 memory usage)

st2 version:

root@ubuntu16:~# apt list --installed st2
Listing... Done
st2/xenial,now 2.4dev-41 amd64 [installed]

st2 summ memory usage (excluding mistral and other services):

root@ubuntu16:~# ps -eo rss,pid,euser,args:100 --sort %mem | grep -v mistral | grep -i '/opt/stackstorm' | awk '{sum+=$1} END {print sum / 1024}'
1974.18

st2 memory usage by process:

root@ubuntu16:~# ps -eo rss,pid,euser,args:100 --sort %mem | grep -v mistral | grep -i '/opt/stackstorm' | awk '{printf $1/1024 "MB"; $1=""; print }'

26.332MB 25632 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2api.wsgi:application -k eventlet -b 127.0.0.1:9101 --workers 1 --threads 1 --graceful-timeout 10 --timeout 30
26.4141MB 25638 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2stream.wsgi:application -k eventlet -b 127.0.0.1:9102 --workers 1 --threads 10 --graceful-timeout 10 --timeout 30
26.4219MB 25641 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2auth.wsgi:application -k eventlet -b 127.0.0.1:9100 --workers 1 --threads 1 --graceful-timeout 10 --timeout 30
91.3164MB 25647 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2garbagecollector --config-file /etc/st2/st2.conf
92.5625MB 25667 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2sensorcontainer --config-file /etc/st2/st2.conf
95.6758MB 25658 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2resultstracker --config-file /etc/st2/st2.conf
96.1172MB 25969 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/local/lib/python2.7/site-packages/st2reactor/container/sensor_wrapper.py --pack=linux --file-path=/opt/stackstorm/packs/linux/sensor$
96.957MB 25652 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2notifier --config-file /etc/st2/st2.conf
97.1562MB 25662 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2rulesengine --config-file /etc/st2/st2.conf
101.133MB 25605 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
101.137MB 25599 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
101.152MB 25612 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
101.188MB 25602 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
101.195MB 25593 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
101.227MB 25609 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
101.242MB 25615 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
101.254MB 25621 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
101.277MB 25596 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
101.289MB 25618 root /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/st2actionrunner --config-file /etc/st2/st2.conf
101.684MB 25846 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2stream.wsgi:application -k eventlet -b 127.0.0.1:9102 --workers 1 --threads 10 --graceful-timeout 10 --timeout 30
105.188MB 25848 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2auth.wsgi:application -k eventlet -b 127.0.0.1:9100 --workers 1 --threads 1 --graceful-timeout 10 --timeout 30
105.312MB 25838 st2 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn st2api.wsgi:application -k eventlet -b 127.0.0.1:9101 --workers 1 --threads 1 --graceful-timeout 10 --timeout 30

---

This makes impossible to run st2 under some internal servers we have, also in 2GB st2vagrant VM.

Here is the Xenial revision I could isolate that points to recent #3538 pip change:
https://packagecloud.io/StackStorm/staging-unstable/packages/ubuntu/xenial/st2_2.4dev-41_amd64.deb (not OK)
vs
https://packagecloud.io/StackStorm/staging-unstable/packages/ubuntu/xenial/st2_2.4dev-40_amd64.deb (OK)

[Kami]

It looks like it's the gunicorn processes so it's probably gunicorn upgrade related, we can try reverting gunicorn upgrade and see if it changes anything.

[Kami]

I tried to replicate it locally by running st2api and st2auth and hitting them with many requests but memory didn't seem to increase at all so it looks like the leak will be harder to track down.

Did anything else also change?

[Kami]

We should also set up something like this https://github.com/tobami/codespeed, so we can continuously and more accurately track memory and other resource usage over commits and time.