[WIP] Try adding back MongoDB changes to see if it helps with the tests performance #4263
Conversation
|
Even if it turns out the performance benefits are not large enough, we still want the That flag is very handy because it makes tests fail if some MongoDB query doesn't use an index and results in a full table scan. It also turns out that while this feature was disabled, we "regressed" / added some new features which don't correctly utilize indexes so tests are failing with that flag enabled. I will look into fixing the code once we get baseline performance numbers. |
database, don't ensure indexes twice.
Changes include: 1. No need to drop collections anymore before dropping the MongoDB database (only needed with older versions) 2. No need to synchronously ensure indexes for tests. Async index ensure is fine for tests and offers significat test speeds ups.
arm4b
left a comment
arm4b
left a comment
There was a problem hiding this comment.
I've looked at timings in this PR (before all jobs were restarted and canceled) and data tells that using custom mongo with ramdisk doesn't improve the CI timings.
However I agree that having control over settings like notablescan would be very helpful.
To avoid that amount of bash scripting and overcomplication, between the previously proposed options I'd chose modifying existing mongod config as a best way to inject custom settings and benefit from it, instead of hacking our own copy of mongo.
I'm also in favor of improvements like #4264 👍, compare the data preciously and pick up the optimizations that really worked.
| env: | ||
| global: | ||
| - CACHE_NAME=JOB1 | ||
| - MONGODB_VERSION=3.4.16 |
There was a problem hiding this comment.
Talking about version pinning, I believe what we have now /mongodb-org/3.4 in repo with major.minor would be overall a more preferable approach.
I think that in mongodb context pinning patch version instead of giving a better control would, in fact, work against us.
|
|
||
| # Let's enable rabbitmqadmin | ||
| # See https://github.com/messagebus/lapine/wiki/Testing-on-Travis. | ||
| before_script: |
There was a problem hiding this comment.
For reference of editing existing mongo conf, this looks like a good hint:
http://georgeshank.com/how-to-enable-a-mongodb-replica-set-on-travis/
before_script:
- echo "foo = bar" | sudo tee -a /etc/mongodb.conf
- sudo service mongodb restart
which overall is more clean.
There was a problem hiding this comment.
I would be fine with that, although we might also need to use or sed or similar if config option already exists, so it could get a bit messy.
|
Closing. Will open a new PR which sets those options in the config file as discussed above with @armab. |
Adding back MongoDB performance optimization changes which were (accidentally?) reverted quite a long time ago to see if it's they help with the tests performance and if it's still worth having those changes.