Flaky reruns autouse session-scoped fixtures when a tests fails one of its runs

[JimArnow]

When a test fails and is rerun, Flaky reruns session-scoped autouse fixtures. For example, running this test:

import pytest

@pytest.fixture(scope="session", autouse=True)
def my_session_scoped_autouse_fixture():
    print("\nIn session scope autouse fixture")

@pytest.mark.flaky(max_runs=3)
def test_flaky_reruns(local_list = []):
    local_list.append(0)
    assert len(local_list)==3

calls the fixture 3 times:

============================= test session starts ==============================
collecting ... collected 1 item

test_it.py::test_flaky_reruns 
In session scope autouse fixture

In session scope autouse fixture

In session scope autouse fixture
PASSED

============================== 1 passed in 0.01s ===============================

Process finished with exit code 0

This can cause unexpected behaviors. In my case, I'm using the pytest-playwright package which has a session-scoped autouse fixture that deletes the output directory where traces are stored for failing tests. If a test fails and is rerun, any previously created trace files are deleted.