Pattern for running particular Travis tasks as part of a nightly build #4760
Conversation
Make sure we only generate coverage for runners and orquesta integration tests when ENABLE_COVERAGE environment variable is set to "yes". Coverage adds a lot of overhead so this should speed up PR builds.
invocation basis. Update tests which have a race / rely on timing to use longer wait time to avoid failure.
nightly build. This should substantially speed up PR builds.
…ts_to_nighly_build
part of the nightly builds now.
There was a problem hiding this comment.
Good stuff 👍
As we're moving some of the tasks to nightly builds, let's please make sure that Slack notifications are configured with https://docs.travis-ci.com/user/notifications/ to #stackstorm.
Quick example: https://github.com/StackStorm/ansible-st2/blob/master/.travis.yml#L34-L39
Otherwise there is a high chance that Travis Nightly build will be ignored.
are specified, but not all of them have corresponding nightly tasks. For example: TASK="foo1 foo2 foo3" and only "foo2-nightly" task exists. In this case, only "foo2-nightly" would run and other would be ignored.
|
On a related note - I really dislike our current Makefile. It's way too convoluted and hard to reason about and understand (especially the coverage targets). Eventually, I propose moving all the test targets (unit, integration tests, pack tests) to tox, similar as we do for Python 3 tasks. You can check It does require a bit more work aka specifying a list of components manually, but I think it's still much better than current convoluted and confusing Makefile. |
|
@armab I agree, I will enable it. It's also worth noting that you can't only enable notifications for cron / nightly builds, so it will be enabled for failures for all non-PR builds. |
Also enable it so we can test it's working.
Kami
changed the title
|
@armab Added Slack notification for non-PR builds (master, nightly crob builds) and verified it's working. @m4dcoder Per slack - now that we have optimized Orquesta integration tests and made them substantially faster, do you think we still need to move them to a nightly build? Perhaps, just the concurrent load one which seems to take ~2 minutes. |
| # We want to be notified when a master or nightly build fails | ||
| notifications: | ||
| # Post build failures to '#stackstorm' channel in 'stackstorm' Slack | ||
| slack: |
@Kami There are few more that can be moved into nightly build. The tests need some reorganizing though. We will certain do this but unless the recent improvements aren't sufficient, I probably won't work on it until I have a little more space on my plate. |
|
@m4dcoder Yeah, I didn't really mean you need to work on it. I planned to do some of that re-organization in the near future. For now, I will merge this PR as is and we can tackle moving more "slow" tests into a nightly build as part of a separate PR and larger reorganization of tests. |
This pull request updates travis config and introduces a new pattern for running particular make targets / checks as part of a nightly build.
For each of the existing tasks which runs as part of a Travis build such as
ci-integration,ci-checks, etc. we can now have additional Make target called<task>-nightly(e.g.ci-checks-nightly) which will only run as part of a nightly build.This will allow us to offload some slow tests (e.g. Mistral unit tests, etc.) to a nightly build and reduce some of the overhead and speed things up for the PR (and master merge) builds.
As part of this PR, I will also move mistral runner tests to the nightly build (Mistral will eventually be deprecated) and update existing st2client checks which run as part of a nightly build to utilize this new consistent pattern for nightly builds.