Regression: in 3.10 tests are no longer de-dupped resulting in mutations propagating through parameterized test runs #4322
Description
What happens
In python 2 with pytest 3.10, when you pass a file or directory argument to pytest multiple times, it executes tests multiple times instead of once. This can result in parameterized tests that mutate their parameters running again with the mutations visible, causing spurious test failures.
This is particularly problematic when using mocks, because the mock value gets re-used and checks such as "mock was only called once" start to fail.
This does not happen in python 3 and did not happen in previous versions of pytest.
Repro steps
-
Create a fresh python 2 virtual environment and install pytest 3.10
-
Create a file
test.pycontaining the following code:
@pytest.mark.parametrize(
'x,y',
[
[
[], 'passed_in_correctly'
]
]
)
def test_repro(x, y):
assert y == 'passed_in_correctly'
x.append(1)
assert x == [1]-
Run
pytest . . -
The test runs twice in the same session and fails the second time
================ test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-3.10.0, py-1.7.0, pluggy-0.8.0
rootdir: [...]
collected 2 items
test.py .F
===================== FAILURES ==============================
_________ test_repro[x0-test] _________________________
x = [1, 1], y = 'test'
@pytest.mark.parametrize(
'x,y',
[
[
[], 'test'
]
]
)
def test_repro(x, y):
assert y == 'test'
x.append(1)
> assert x == [1]
E assert [1, 1] == [1]
E Left contains more items, first extra item: 1
E Use -v to get the full diff
test_repro.py:15: AssertionError
===================1 failed, 1 passed in 0.07 seconds ===================