Mark flaky test as xfail on Python 3.5.0

[nicoddemus]

I've tried to reproduce the issue, but it seems to only happen sporadically on Linux.

Mark it is flaky now to avoid people wasting time on PRs when they encounter this problem.

Related to #5795

[blueyed]

Have you seen it without xdist?
It is also related to 3.5.0 likely, not 3.5.x?

[nicoddemus]

Have you seen it without xdist?

That's what I've tried, yes.

It is also related to 3.5.0 likely, not 3.5.x?

Good point, have not tried that.

Either way, I think it is fine to merge this; if/when someone champions this, we just remove that line of code. It is worst to have this failing randomly on PRs and generating friction, IMHO.

[blueyed]

Sure. I just got the impression that it started to happen after explicitly testing 3.5.0.

[nicoddemus]

Sure. I just got the impression that it started to happen after explicitly testing 3.5.0.

You got a point, I will mention it in the issue. If you are OK with the PR, could you approve please? Thanks

[blueyed]

You got a point, I will mention it in the issue.

Why not just amend it here instead? :)

[blueyed]

Should be 3.5.0, but ok with me.

[blueyed]

Have you seen it without xdist?

That's what I've tried, yes.

So you've not seen it yourself? "it seems to only happen sporadically on Linux." refers to the CI job then?

[nicoddemus]

So you've not seen it yourself? "it seems to only happen sporadically on Linux." refers to the CI job then?

Yep, and on Linux only, which we use 3.5.0 (on Windows we use 3.5.4 I think). So you are probably right that is 3.5.0 related.

I have changed the check to use 3.5.0 then, thanks!

[nicoddemus]

Wait, another one failed now: test_continue_on_collection_errors

[blueyed]

__________________________ testing/test_collection.py __________________________
[gw1] linux -- Python 3.5.0 /home/travis/build/pytest-dev/pytest/.tox/py35-xdist/bin/python
worker 'gw1' crashed while running 'testing/test_collection.py::test_continue_on_collection_errors'

It is a bit unfortunate that xdist provides not more information there, isn't it?

[blueyed]

Maybe the job should be changed to not use xdist?

[nicoddemus]

Hmmm that would probably help, let me try it.

[nicoddemus]

I will remove the xfail marker, let's see how it fares without xdist for awhile then.

[nicoddemus]

🤞

[blueyed]

Cool, for reference it might still break when using coverage with it (#5772).

[nicoddemus]

😓