pytest-xdist is not reporting the same nodeid as pytest does

[elyezer]

Running pytest and pytest -n <executors> does not produce the same nodeid for the tests and therefore this is preventing lastfailed from running properly. Since the nodeids in the cache don't match any test collected, it ends up running tests that are not meant to run.

I have the following workflow:

1. Run the tests that can run in parallel using xdist
2. Run the tests that need to run serially without using xdist's -n option
3. Run pytest with the --last-failed option to try rerunning the failures to check if they are flaky. This is again without using xdist's -n option

Here is the reproducer project structure:

$ ls -1R
.:
pytest.cfg
tests

./tests:
api

./tests/api:
__pycache__
test_bar.py
test_foo.py

Here are the file contents:

$ cat tests/api/test_bar.py 
def test_bar_1():
    assert True


def test_bar_2():
    assert True


def test_bar_3():
    assert False

$ cat tests/api/test_foo.py 
import os


def test_foo_fail():
    assert os.environ.get('FOO_PASS', 'false') == 'true'


def test_foo_ok():
    assert True

$ cat pytest.cfg 
[tool:pytest]
python_files = *.py
testpaths = tests/api

Considering the project described above, if you run pytest -n <executors> it shows the following (mind the tests' nodeids):

$ pytest -v --tb=no -c pytest.cfg -n 2
================================ test session starts ================================
platform linux -- Python 3.7.3, pytest-4.6.3, py-1.8.0, pluggy-0.12.0 -- /home/elyezer/.local/venvs/pytest/bin/python3
cachedir: .pytest_cache
rootdir: /home/elyezer/code/xdist-reproducer, inifile: pytest.cfg, testpaths: tests/api
plugins: xdist-1.29.0, forked-1.0.2
[gw0] linux Python 3.7.3 cwd: /home/elyezer/code/xdist-reproducer
[gw1] linux Python 3.7.3 cwd: /home/elyezer/code/xdist-reproducer
[gw0] Python 3.7.3 (default, May 11 2019, 00:38:04)  -- [GCC 9.1.1 20190503 (Red Hat 9.1.1-1)]
[gw1] Python 3.7.3 (default, May 11 2019, 00:38:04)  -- [GCC 9.1.1 20190503 (Red Hat 9.1.1-1)]
gw0 [5] / gw1 [5]
scheduling tests via LoadScheduling

test_bar.py::test_bar_1 
test_bar.py::test_bar_2 
[gw1] [ 20%] PASSED test_bar.py::test_bar_2 
[gw0] [ 40%] PASSED test_bar.py::test_bar_1 
test_bar.py::test_bar_3 
test_foo.py::test_foo_fail 
[gw0] [ 60%] FAILED test_bar.py::test_bar_3 
test_foo.py::test_foo_ok 
[gw0] [ 80%] PASSED test_foo.py::test_foo_ok 
[gw1] [100%] FAILED test_foo.py::test_foo_fail 

======================== 2 failed, 3 passed in 0.32 seconds =========================

But if we run without the -n option here is the output:

$ pytest -v --tb=no -c pytest.cfg     
================================ test session starts ================================
platform linux -- Python 3.7.3, pytest-4.6.3, py-1.8.0, pluggy-0.12.0 -- /home/elyezer/.local/venvs/pytest/bin/python3
cachedir: .pytest_cache
rootdir: /home/elyezer/code/xdist-reproducer, inifile: pytest.cfg, testpaths: tests/api
plugins: xdist-1.29.0, forked-1.0.2
collected 5 items                                                                   

tests/api/test_bar.py::test_bar_1 PASSED                                      [ 20%]
tests/api/test_bar.py::test_bar_2 PASSED                                      [ 40%]
tests/api/test_bar.py::test_bar_3 FAILED                                      [ 60%]
tests/api/test_foo.py::test_foo_fail FAILED                                   [ 80%]
tests/api/test_foo.py::test_foo_ok PASSED                                     [100%]

======================== 2 failed, 3 passed in 0.04 seconds =========================

As you can see both calls has the same rootdir but produce different nodeids for the tests. And if we check the cache for lastfailed:

$ pytest -c pytest.cfg --cache-show=cache/lastfailed
================================ test session starts ================================
platform linux -- Python 3.7.3, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
rootdir: /home/elyezer/code/xdist-reproducer, inifile: pytest.cfg, testpaths: tests/api
plugins: xdist-1.29.0, forked-1.0.2
cachedir: /home/elyezer/code/xdist-reproducer/.pytest_cache
------------------------ cache values for 'cache/lastfailed' ------------------------
cache/lastfailed contains:
  {'test_bar.py::test_bar_3': True,
   'test_foo.py::test_foo_fail': True,
   'tests/api/test_bar.py::test_bar_3': True,
   'tests/api/test_foo.py::test_foo_fail': True}

=========================== no tests ran in 0.00 seconds ============================

pytest-xdist test nodeid generation is not consistent with plain pytest nodeid generation and this inconsistency is breaking plugins such as the lastfailed that relies on the nodeids to do its job.

[nicoddemus]

Hi @elyezer,

Really sorry for the delay in answering this!

Thanks a lot for the detailed report, we appreciate it.

The problem is related to the testpaths option; commenting that out produces the correct node ids:

 λ pytest -v --tb=no -c pytest.cfg -n 2
======================== test session starts ========================
platform win32 -- Python 3.6.3, pytest-4.6.4.dev15+g09dee292c.d20190630, py-1.5.2, pluggy-0.12.0 -- x:\pytest-xdist\.env36\scripts\python.exe
cachedir: .pytest_cache
rootdir: x:\pytest-xdist\.tmp\node-bug, inifile: pytest.cfg
plugins: forked-0.2, xdist-1.28.1.dev9+g407d6e1
[gw0] win32 Python 3.6.3 cwd: x:\pytest-xdist\.tmp\node-bug
[gw1] win32 Python 3.6.3 cwd: x:\pytest-xdist\.tmp\node-bug
[gw0] Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)]
[gw1] Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)]
gw0 [5] / gw1 [5]
scheduling tests via LoadScheduling

tests/api/test_bar.py::test_bar_2
tests/api/test_bar.py::test_bar_1
[gw0] [ 20%] PASSED tests/api/test_bar.py::test_bar_1
[gw1] [ 40%] PASSED tests/api/test_bar.py::test_bar_2
tests/api/test_bar.py::test_bar_3
tests/api/test_foo.py::test_foo_fail
[gw0] [ 60%] FAILED tests/api/test_bar.py::test_bar_3
tests/api/test_foo.py::test_foo_ok
[gw0] [ 80%] PASSED tests/api/test_foo.py::test_foo_ok
[gw1] [100%] FAILED tests/api/test_foo.py::test_foo_fail

================ 2 failed, 3 passed in 1.29 seconds =================

Definitely a bug, thanks for the report.

[nicoddemus]

Actually the underlying cause is the -c flag. Unfortunately we have a technical debt regarding config initialization (tracked in https://github.com/pytest-dev/pytest/projects/2).

For now a workaround is to rename your pytest.cfg to setup.cfg (the standard name), and then the node ids are generated as expected:

λ pytest -v --tb=no -n2
======================== test session starts ========================
platform win32 -- Python 3.6.3, pytest-4.6.4.dev15+g09dee292c.d20190630, py-1.5.2, pluggy-0.12.0 -- x:\pytest-xdist\.env36\scripts\python.exe
cachedir: .pytest_cache
rootdir: x:\pytest-xdist\.tmp\node-bug, inifile: setup.cfg, testpaths: tests/api
plugins: forked-0.2, xdist-1.28.1.dev9+g407d6e1
[gw0] win32 Python 3.6.3 cwd: x:\pytest-xdist\.tmp\node-bug
[gw1] win32 Python 3.6.3 cwd: x:\pytest-xdist\.tmp\node-bug
[gw0] Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)]
[gw1] Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)]
gw0 [5] / gw1 [5]
scheduling tests via LoadScheduling

tests/api/test_bar.py::test_bar_2
[gw1] [ 20%] PASSED tests/api/test_bar.py::test_bar_2
tests/api/test_bar.py::test_bar_1
[gw0] [ 40%] PASSED tests/api/test_bar.py::test_bar_1
tests/api/test_foo.py::test_foo_fail
tests/api/test_bar.py::test_bar_3
[gw0] [ 60%] FAILED tests/api/test_bar.py::test_bar_3
[gw1] [ 80%] FAILED tests/api/test_foo.py::test_foo_fail
tests/api/test_foo.py::test_foo_ok
[gw1] [100%] PASSED tests/api/test_foo.py::test_foo_ok

================ 2 failed, 3 passed in 1.05 seconds =================

[nicoddemus]

Related: #6

[elyezer]

@nicoddemus thanks for digging into that. I was thinking if it would make sense to have some sort of helper on pytest that would provide the nodeid since I assume the logic to deal with the config file is there already and therefore we could avoid having to somewhat repeat the same logic here.

I am willing to contribute, so if you could give me some pointers I could try getting this done. This is going to be the first time that I will be contribute and I still learning more about the source code of both pytest and pytest-xdist.

Thank you

[nicoddemus]

Hi @elyezer,

Great to hear that you are considering contributing, we definitely appreciate it. 👍

But this topic in particular is not really recommended for newcomers to the project, as it involves a lot of complex iteration during the config initialization between pytest and pytest-xdist.

Good news is that while I was debugging your report I decided to take the approach shown in #448 to fix the config initialization problem, and it looks promising. Hopefully I will find the time required to move this forward in the next few weeks.

I suggest to take a look at other issues in the tracker that might be easier to get started. 👍

[elyezer]

@nicoddemus I will keep a eye on it. Thank you for working on it.

I will check other issues and hopefully no can start contributing sooner rather than later.

Thank you very much