Proposal: Support MongoDB 3.6

[LindsayHill]

Proposal: Support MongoDB 3.6 in the next version of ST2, and then make it the default in the next version after that.

Background: MongoDB 3.6 was released around 6 months ago, and we are now up to version 3.6.4, so stability should be good.

MongoDB 3.2 is supported until September 2018, while no end date has been set for MongoDB 3.4 or 3.6.

ST2 currently works with MongoDB 3.2 or 3.4. The recommendation (and default) is to use v3.4.

ST2 2.7.1 does not work consistently with MongoDB 3.6. See #4097.

Tasks:

• Figure out some workaround or upstream patch for mongoengine issue Increased x1.5-x2.0 memory usage after recent pip upgrade [staging-unstable] #3595
• Test ST2 master with MongoDB 3.6, and make any changes required to support both 3.4 & 3.6.
• Ship those changes in ST2 v2.8
• Update docs to reflect that initial v3.6 support has been added
• In the next version of ST2 after 2.8, update our CI to use 3.6
• Update install scripts to default to installing 3.6.
• Update docs to reflect that 3.6 is preferred version

[nmaludy]

+1

Also created tasks for ansible-st2 (StackStorm/ansible-st2#193) and puppet-st2 (StackStorm/puppet-st2#217)

[Kami]

I'm +1 for that.

In fact, I already wanted to do it in the past, but one of the pre-requisites for that is upgrading mongoengine and pymongo dependency.

Sadly, we are blocked on mongoengine upgrade due to performance regression in 0.13.0 - #3595, MongoEngine/mongoengine@466935e#commitcomment-23225459.

[Kami]

I had a look at the latest version of mongoengine and the same issue still exists.

Memory usage with mongoengine version 0.12.0 (regression was introduced in 0.13.0):

(virtualenv) vagrant@ubuntu-xenial:/data/stanley$ ps -eo rss,pid,euser,args:100 --sort %mem | grep -v mistral | grep -i 'st2' |grep -v grep | awk '{printf $1/1024 "MB"; $1=""; print }'
90.0547MB 4596 vagrant python st2api/bin/st2api

Memory usage with the latest version (0.15.3, post 0.13.0):

(virtualenv) vagrant@ubuntu-xenial:/data/stanley$ ps -eo rss,pid,euser,args:100 --sort %mem | grep -v mistral | grep -i 'st2' |grep -v grep | awk '{printf $1/1024 "MB"; $1=""; print }'
117.469MB 4695 vagrant python st2api/bin/st2api

This memory usage regression can be consistently reproduced.

Sadly at this point only option we are left with is forking mongoengine repo in StackStorm org and reverting the change which causes the regression.

@armab and others, what do you think?

I really wish we wouldn't need to fork it, but we have been stuck with an old version of mongoengine for more than a year now and I can't think of a better solution :/

[arm4b]

Before forking, I think we should at least open a valid Issue in mongoengine with a clear description and reproducible example.

That would be a better, comparing to commenting in a code commit/history. Formalizing it that way can hopefully lead to a better attention/awareness, which is not different to how we expect contributions to land in our repo, for example.

[nmaludy]

@Kami the other thing to consider is this bug / performance issue caused by mongoengine #4030

Not sure if there is maybe an alternative solution out there with better performance characteristics?

[Kami]

@armab Agreed.

Opened an issue so we can better track it - MongoEngine/mongoengine#1832. I also included a very simple script which reproduces the issue.

It only affects Python 2.7.x so it's likely some edge case / bug in CPython in 2.7.x, but that doesn't matter that much since a lot of people still use Python 2.7.

[Kami]

@nmaludy Sadly I didn't have time to look into that issue yet so I can't comment on it.

Having said that, I don't think totally moving away from mongoengine is something we can / should do at this point.

One think we can do though is to optimize queries and similar (and potentially skip mongoengine document abstraction) where it makes sense and where it impacts the performance.

[Kami]

This PR will allow us to support MongoDB 3.6 - #4291.

I tested it locally and things seem to be working fine.

I propose the following approach which we have already used in the past when we added support for 3.4 in addition to 3.2:

1. Update Travis so we also run Python 2.7 unit and integration tests under 3.6 in addition to 3.4. This will slow things down since it means one more builder, but there is really no way around that. Good news is that I recently improved build times a lot so hopefully overhead shouldn't be too big.
2. Upgrade st2cicd server to latest version of st2 2.9dev and MongoDB 3.6
3. After a while, if everything looks OK announce official support in the next release and make necessary docs changes.
4. Update documentation and installer scripts to install and use 3.6 by default (but still support 3.4 for a reasonable amount of time giving people enough time to upgrade)
5. Eventually remove 3.4 from Travis build and only run tests under 3.6 (this way we get rid of one builder and speed the whole build up)

@armab @LindsayHill @lakshmi-kannan and others - what do you think?

[LindsayHill]

Yes, that approach looks good to me. Maybe do something like this for timelines:
2.9 - announce MongoDB 3.6 support
3.0 - update installer scripts, install 3.6 by default, announce deprecation of 3.4 support (NB: announce only, not actually removing support yet)
3.2 - Formally remove 3.4 support - mainly just by removing 3.4 from Travis. Will probably still work, but we won't test it or claim support.

Also: I see MongoDB has released 4.0. Still too early for us though, don't want to test with that for at least a few patch cycles.