split CI into "core" and "analysis"

[orbeckst]

Given that travis-ci/coverall works so well, we can think about how to get more out of it and the flexibility it provides us. I propose to split the CI and especially the coverage reports into two batches whose status would also be displayed separately in the README.rst:

• analysis: MDAnalysis.analysis and MDAnalysis.visualization
• core: everything else (core, topology, coordinate, lib, util, ...)

advantages

• more accurate reflection of the coverage of the most crucial code parts (the core) because at the moment, analysis has much fewer unit tests than core (we need to change that but it won't happen overnight)
• (small) speed up of the CI runs

disadvantages

• no global view anymore
• mildly confusing when we have multiple badges on the front page (although we could simply put core library and analysis modules in front of the badges)

Opinions?

EDIT: added visualization to the analysis batch. 2015-05-26

[richardjgowers]

So I like the idea of this split. So +1 to the proposal.

There's a technical issue in that coveralls (and coverage in general) is usually done per repository, so we cant have 2 badges from a single build.

Possible solutions to this:

1. multiple repos / branches for core/analysis (same code hosted twice)
2. split out analysis into it's own package (messy and might not be found)
3. don't include analysis submodule in calculating total coverage (but hopefully will still show in detailed breakdown)

I think option 3 is the best

[orbeckst]

I don't like multiple repos for the same code, that's just messy; we could have a second branch that is identical to develop, especially if we can install a hook that automatically merges. But this still won't work well for feature branches. And I don't think we would do our users a favor by splitting out analysis yet, maybe down the road if the individual parts have grown more or are growing at different speeds.

I think we can try 3 but we still need to have a way to establish coverage of the analysis part. And of course we have to advertise properly what coverage means.

The build pass would still be for all unit tests, right?

[richardjgowers]

Yeah the tests are independent to coverage, so any written tests would run regardless of coverage metric.

I'll have a go at this soonish and see what we can achieve.

[seb-buch]

Hi,

+1, it also makes sense for the end users that, for instance, build
their own script based on the core features but don't use the analysis
modules that much... Even though I have no idea what percentage of users
it represents.

On 05/24/2015 07:57 PM, Richard Gowers wrote:

Possible solutions to this:

• multiple repos / branches for core/analysis (same code hosted twice)
• split out analysis into it's own package (messy and might not be found)
• don't include analysis submodule in calculating total coverage (but hopefully will still show in detailed breakdown)

I am also in favor of soltuion 3.

My two cents,

Séb

[orbeckst]

Sounds like a reasonable quorum and a smashing "all in favour" ;-). Let's do it.

Eventually, we should strive to have full coverage near 100% but for right now, core appears the most important one (especially if we move more stuff into lib #287).

Will there still be an automated way to obtain coverage reports for analysis?

[orbeckst]

@richardjgowers you seem to be making progress #289 – Note that MDAnalysis.visualizations should be grouped with analysis, at least for the time being. I'm not even sure how to write tests for that, maybe @tylerjereddy knows more.

[richardjgowers]

Sorry yeah, this is done now. Analysis isn't included in coverage measurement (but tests are ran). We got about a 13% bump in measured coverage.

There's currently no way to see if analysis is covered however.... Which I'm not a huge fan of.