Various test speed ups #4264
Various test speed ups #4264
Conversation
Changes include: 1. No need to drop collections anymore before dropping the MongoDB database (only needed with older versions) 2. No need to synchronously ensure indexes for tests. Async index ensure is fine for tests and offers significat test speeds ups.
|
It looks like the slowest build now takes around ~18 minutes and it used to take around ~22 minutes. I'm also looking into some other improvements (coverage seems to have a huge overhead, so perhaps we could only run it on master branches and not for all the PRs). Before: https://travis-ci.org/StackStorm/st2/builds/407625613 |
|
It looks like the coverage changes in #4147 broke codecov coverage reporting - https://travis-ci.org/StackStorm/st2/jobs/407657640#L9507, https://codecov.io/github/StackStorm/st2?branch=master (/cc @blag). Having said that, I'm looking if it's possible to only run coverage on master and version branches and not for pull request since coverage has huge negative performance overhead on the tests run time. |
|
It turns out that code coverage related changes in #4147 have a massive negative performance overhead - build time went from ~10 minutes to ~20+ minutes after that PR has been merged. Because of that, I'm working with @blag to only instrument, generate and submit coverage report to coveralls on master branch changes. |
|
Also, to back my claim that the coverage PR introduced a massive 40 or so percent test slow down with some actual data: Travis build page before the change was merged: https://travis-ci.org/StackStorm/st2/builds/385220238 So ~22 vs ~14 minutes. And now with those changes tests take less than 10 minutes. Big chunk of that is due to the coverage change and other 3-5 minutes speed up is due to my changes in 2dd5b78. |
| coverage combine .coverage.integration.* | ||
| @if [ -n "$(NOSE_COVERAGE_FLAGS)" ]; then \ | ||
| . $(VIRTUALENV_DIR)/bin/activate; COVERAGE_FILE=.coverage.integration \ | ||
| coverage combine .coverage.integration.*; \ |
There was a problem hiding this comment.
BTW, combine itself looks suspicious, because codecov already combines coverage natively, sent from different jobs within one build: https://docs.codecov.io/docs/merging-reports
There was a problem hiding this comment.
That's true, but I also wanted to combine the tests locally. I can remove that if we want to.
|
@blag Thanks - LGTM, but it would be great if you can also make the environment variable change we discussed on Slack so it's not tied / coupled to Travis (using new In addition to that it would also be great if you can have a look in coverage combining and submitting coverage to coveralls which I mentioned on Slack and which @armab mentioned above. |
dependency won't be updated locally when running "make requirements" and either virtualenv needs to be re-created for new version to be installed or pip uninstall orchestra ran before make requirements.
|
Sorry hit the wrong button there. I'll implement notifying coveralls in a separate PR, since that's not really related to test speed ups at all. |
|
While at it, I also pushed a small change (3991bc1) for more user-friendly named builds - https://travis-ci.org/StackStorm/st2/builds/407952313 |
This pull request cherry picks two big test performance improvements from #4263:
All those changes are independent from other MongoDB start up and download related changes in #4263.
Based on the local testing, this should speed Travis tests, by 20% at the minimum. More accurate numbers are coming up soon once the Travis builds finishes.