keep up to date with flaky tags

[gdams]

we need to keep up to date with flaky tags as I fear that we are often marking modules flaky without doing much research and often then leaving them marked as flaky despite the fact that they may be fixed now. I propose we add a tag to citgm-all such as citgm-all --check-flaky which reports all the modules at the end that are passing but are marked as flaky. Then we can hopefully prevent this from being a problem.

[MylesBorins]

I've done periodic checking of flakyness. We are about due for another.

[gdams]

@MylesBorins how have you currently been doing this? would it not be easier to have a tidy report at the end of the output?

[MylesBorins]

Our output currently reports flaky modules. I've just been running CI with forcing things to fail for each release line and checking

[gibfahn]

I like the idea of always letting you know somehow (possibly at the end) what the results (pass/fail) were for the flaky tests. I don't think it needs to be a flag though, if it's only a couple of lines somewhere I can't see a reason you wouldn't want that info.

[gdams]

so you're more for making this a permanent fixture?

[MylesBorins]

@gibfahn are you suggesting that we report passing "flaky" module differently than passing "non-flaky" modules?

[gdams]

No the way I view it is that at the end of the citgm run there will be an output summarising any modules that are marked as flaky but are actually passing

[MylesBorins]

I think we are saying the same thing.... ☺️

[gdams]

Yes I think we are actually, is there any harm to implementing this? It means that we catch fixed modules quicker

[gibfahn]

@gdams If you do an implementation we can discuss.