The Journey to a Faster Release Process

[punkrokk]

During the July 2021 TSC, the notion of moving towards quicker releases came up. This issue is to track a conversation around some of the ideas and clarify a path forward.

Things that could speed up releases:

• More frequent releases e.g. can move towards a release strategy that can quickly release updates/patches/bugfixes/defects that are low risk to release to master?
• Maintain the existing release process for higher risk updates (OS changes, core software updates (Mongo/Rabbit, etc)) while enabling smaller fixes to get out quickly?
• What fluff can we remove from the release process to enable faster releases that are low risk?
• How might the versioning strategy change? (e.g. minor (e.g. v2.6) is bumped at significant releases, and allow small features into patches? Or alternatively, bump minor version more frequently?
• What else is blocking faster releases?

[amanda11]

At the moment I would say that a large amount of time on the release process is spent on the manual testing states. Of which a key part is the UI where we don't have automated tests. In the process we perform the manual testing process on the staging-unstable, staging-stable and stable releases and that takes some time.

[amanda11]

In regard to having low-risk and higher risk, then the only issue I can see is how we manage that in terms of branches. It would seem that we would have to keep the major ones off the master branch until we were near a release point, so that the master can be released more rapidly.
This will add overhead on merges between branches, and probably increase the number of CI runs we need to do - as we'd need to have the runs on both the "master" and "next-release" branches.

[amanda11]

If we start having bug-fix releases, then it would also increase the number of upgrade paths, and if we have to increase the upgrade runs then that would maybe increase the release process. So if we have 3.5.0, 3.5.1, 3.5.2, 3.5.3 etc as the quick releases, then we release 3.6.0 - which upgrade paths do we test. We would presumably set a strategy here e.g. first and last maybe?
We may also want to have upgrade runs for the maintaince patches as well.

We do have upgrade workflows but we only run them at release time, I don't see why they couldn't be run more often to automate that, however that would increase our costs as we'd need the servers running for longer. Running them prior to release would be good anyway, as its a bit late in the day to find the upgrade doesn't work. As it stands at moment the upgrade is tested in the week of pre-release testing, so running the automated upgrade as part of the CI process would also cut down the amount of time spent in our pre-release testing. So perhaps an automated upgrade once a week for instance woudln't be too costly, and would help the release process.

[Kami]

I'm +1 for anything which allows us to release more often - we've been on that journey for a long time and trying to improve that process, but it appears that releases still need tons of hand holding and manual work.

I do think we should enable all the end to end tests and checks (including upgrade ones) to run as often as possible (e.g. daily).

Ideally after each merge to master we should have packages in staging stable which we should be able to release at any time.

[blag]

One thing that would really help the release process is some Selenium or whatever tests for the frontend.

Our testing is pretty comprehensive, especially our end-to-end tests. But our end-user testing, especially just basic frontend functionality (logging in, logging out, logging in again, executing an action, viewing the results in the executions tab, switching to the rules tab, switching to the actions tab, creating a workflow in the editor, saving it, and executing it, etc.) is still done manually during the release cycle. If we automated that, it would drastically reduce the amount of pre-release testing down to just testing out code that has changed during the release. And as long as we add tests for changed code, that should also be covered.

We don't need to run the full tests for the workflow editor, just the incredibly basic ones.

So, what I would do, is setup frontend testing as part of the end-to-end tests, and expand the CLI tests that we do to cover what is currently manually done in the pre-release testing.

That will get us a lot closer to "push button" releases. And if we really wanted to stick to time based releases, we could just schedule that on st2cicd and let the process take care of itself.

Most of what slows down the release process is running tests on new OSes. When we don't introduce support for new OSes during releases, testing is tons easier to do.