On the future of CI post Travis

[IAlibay]

Just putting this up as there's been a lot of discussion internally, off-github, and on the Twitterverse :)

Current status

As of today Travis CI has switched over to using a credit based system for OSS CI. Unfortunately the amount of credits given are far too low to keep MDAnalysis going. As of writing this, we have already used up all our allocated credits (that's ~ 5 commits and 2 merges...).

Thankfully, mostly due to the work of @tylerjereddy, half our CI infrastructure is still working (mostly Windows testing, but we have all the bits in place to add cover for other OSes).

Solution

The current proposed solution (based on some off-github discusions) is that we should both move part of our CI matrix to existing still-free solutions (i.e. appveyor and azure), and also offload some of that to Github Actions.

Timeframe

I'm currently in the process of testing out github actions on IAlibay/mdanalysis and am hoping to open a PR at some point tomorrow (how early will probably depend on how much more coffee I decide to drink tonight 🙃).

If someone wants to take the lead on reviewing the current matrix and see how things should be spread out to actions/azure/appveyor (maybe some minimal dependency pip based installs?) that would be awesome :)

[tylerjereddy]

@IAlibay Thanks! I'm pretty swamped dealing with the exact same problem for SciPy, and I was hoping to cut the initial SciPy 1.6.0 release branch on Tuesday.. what a mess!

[tylerjereddy]

To keep it concise, this is a summary of what I've been able to find out: numpy/numpy#17824 (comment)

[tylerjereddy]

Travis CI support has added 400,000 credits to our account this morning (see below); I haven't been able to follow our migration progress lately, but if most of the x86 jobs have been moved off we might be able to run a decent number of jobs on the more "exotic" architectures until another solution is found for those. If our Travis matrix is still large I suppose temporarily turning off the credit consumption might be an option--I'll leave that up to the folks actively working on the migration right now.

[IAlibay]

Thanks for working this out @tylerjereddy !

I think there's a PR from master running at the moment, but after that I'll turn off the consumption of paid credits for now to give us time to clean up the travis config.

Probably the best thing here would be to make it exclusive to the arm build. My suggestion would be to set it up either as a daily cron job or to run on merge to develop? That way we can keep some kind of arm testing going without burning through all our units in one go.

[IAlibay]

Ok "Consume paid credits for OSS" is now off.