CI/CD speed up headrooms

[Totktonada]

I'll collect ideas about speed up of CI/CD in tarantool/tarantool here.

1. Packaging jobs: skip repository metadata update and dependencies installation.

   • Docker image caching on GitHub Actions cache infrastructure.
     • Suitable for self-hosted runners?
     • Requires some integration of packpack with GitHub Actions.
     • We should periodically expire/renew the cache, otherwise we may stuck at some old state forever. And it should be done automatically.
   • https://github.com/tarantool/infra/issues/127
     • The main downside: it'll require extra attention and maintenance (periodical manual work).

2. Testing jobs: cache dependencies.

   • As above, whether it is suitable for self-hosted runners?
   • As above, periodical automatic cache renew/refresh is required.

3. ARM64 jobs: use ccache (complation here is long).

   • I'm afraid of unintended effects due to ccache: as false negative (build failure) as well as false positive (missing of expected build failure). So I want ccache files to be expired each week. Also it worth to never use ccache files on tag jobs.
   • GitHub Actions integration (suitable for self-hosted runners?).

4. Run small set of tests on most of targets on developer branches.

5. Skip previous run, when a developer branch is force-pushed.

   • See https://github.com/fkirc/skip-duplicate-actions#cancel-outdated-workflow-runs (search for alternatives as well).
   • Important: don't do that on master and release branches: prior to 2.10 it'll break live deployments, after 2.10 it'll just miss some testing statuses in commits history.
   • Issue: github-ci: prevent outdated workflows runs on developers branches #100.

6. Don't run ARM64 jobs on developer branches (if the branch is not named as somethinf-full-ci).

   Implemented in ci: reduce number of arm64 job runs tarantool#6569

7. In Jepsen testing we can build a package once (on a fast machine) and install it on MCS virtual machine (similarly how we do it in integration.yml). See timestamps:

   2021-10-29T16:38:55.4755076Z INFO [2021-10-29 16:38:55,472] jepsen node 87.239.105.42 - tarantool.db Building Tarantool nil
   2021-10-29T16:45:06.1806054Z INFO [2021-10-29 16:45:06,177] jepsen node 87.239.105.42 - tarantool.db Configure instance on 87.239.105.42
   

   Six minutes!

8. We can run Jepsen testing directly on GitHub hosted runners on Ubuntu, without docker container with Debian Stretch inside (I guess we implemented this for our own CentOS machines, when they work as GitLab CI runners, and it has no sense anymore). We spend ~1 minute to fetch the image and ~40 seconds for apt-get update inside each time.

9. We can use GitHub's caching infrastructure to don't install clojure, jepsen, terraform and its dependencies each time. See the idea here.

10. Jepsen testing has a client machine (from where we send requests to tarantool instances) and MCS virtual machines with tarantool server instances. The client machine is where the GitHub runner resides and now we use GitHub hosted runners in this type of testing. Moving the client machine to MCS may decrease timings between the client and the tarantool server(s). The tests are time bound, so it should not affect testing time much. But it'll allow to make more requests per second and so will make the testing more powerful within the same job run time.

11. Move FreeBSD CI to cloud (MCS) #143

Feel free to add more!

[ligurio]

I would split the task for three parts: speedup in infrastructure, speedup in building Tarantool and speedup in testing.

Please consider my ideas:

• (infra) ignore branches, paths and files on CI, see https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#excluding-branches-and-tags (we don't need to run CI when README, gitignore or other non-code files were changed)
• (infra) disable man-db update (short description: update of man-db happen on every package installation and may require significant time (it's a step Processing triggers for man-db). We don't need manual pages on CI, so it is worth to disable triggers or remove man-db at all sudo apt remove man-db --purge)
• (testing) tune timeouts in tests and never use "stupid" timeouts (aka sleep())
• (testing) profile test framework and make sure there are no bottlenecks there (Fix slowdown on Python 3 test-run#290)
• (infra) make sure that hardware on CI is not a bottleneck (especially storage)
• (infra) disable fsync() in cases when you don't need consistency guarantees
• (testing) add parallelism to luatest
• split package building and testing and make packages only after successful static analysis and testing.
• Reuse repository metadata cache by CI workers https://github.com/tarantool/infra/issues/113
• add Ninja support to Tarantool build system, see cmake: support build using Ninja tarantool#7178
• (infra) use git -depth, however it may non-applicable to tarantool repository due to using submodules

[tsafin]

* (infra) use `git -depth`, however it may non-applicable to tarantool repository due to using submodules

Submodules is not a problem, tagging is a problem. You supposed to guess such depth length, that will bring commit which has been tagged properly, so Tarantool version could be successfully calculated. From simple experiments, looks like 1024 is good enough, but was not reliable, and at the end of a day I had to back to infinite history https://github.com/tarantool/tpch/blob/master/.github/workflows/quick-check.yml#L35 (it's not that bad given our relative small repository sizes)

[tsafin]

Let me ask simple question though - did we collect and analyzed timings information for our build/tests logs? i.e. parse all targets log timestamps, and aggregate them to the stages, to see the bottleneck? We have a lot of assumptions here what should be improved here and there, but it's not yet "data-driven decision"

[Totktonada]

We recently started to collect logs (see https://github.com/tarantool/multivac), @kyukhin is working to make it be like a service. We can add some analysis for timings of different CI steps and run it on top of the collected logs database.

[Totktonada]

(Added point 6 re ARM64 and 7, 8, 9, 10 re Jepsen.)

[Totktonada]

(Added point 11 re FreeBSD.)

[ligurio]

We can use GitHub's caching infrastructure to don't install clojure, jepsen, terraform and its dependencies each time. See the idea here.

Now it is possible to use pre-compiled Jepsen tests, see https://github.com/tarantool/jepsen.tarantool/releases

Let me ask simple question though - did we collect and analyzed timings information for our build/tests logs?

@tsafin I know that there is a github-actions-profiler for that.