pytest.mark.filterwarnings doesn't work if warning already shown on python2.7

[duncanmmacleod]

It looks like pytest.mark.filterwarnings doesn't work properly if a warning has already been shown only on python2.7. See the following example:

import pytest
import warnings


def func():
    warnings.warn('test warning')


def test_1():
    func()


@pytest.mark.filterwarnings('always:test warning')
def test_2():
    with pytest.warns(UserWarning):
        func()

This gives the following result:

============================= test session starts ==============================
platform darwin -- Python 2.7.15, pytest-3.8.0, py-1.5.3, pluggy-0.7.1
rootdir: /Users/duncan/tmp, inifile:
collected 2 items

test-pytest.py .F                                                        [100%]

=================================== FAILURES ===================================
____________________________________ test_2 ____________________________________

    @pytest.mark.filterwarnings('always:test warning')
    def test_2():
        with pytest.warns(UserWarning):
>           func()
E           Failed: DID NOT WARN. No warnings of type (<type 'exceptions.UserWarning'>,) was emitted. The list of emitted warnings is: [].

test-pytest.py:16: Failed
=============================== warnings summary ===============================
/Users/duncan/tmp/test-pytest.py:6: UserWarning: test warning
  warnings.warn('test warning')

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================ 1 failed, 1 passed, 1 warnings in 0.05 seconds ================

On python 3.x this works fine. If I reverse the order of the tests, it works fine. Is there a workaround for this?

[asottile]

The docs say:

One thing to be aware of is that if a warning has already been raised because of a once/default rule, then no matter what filters are set the warning will not be seen again unless the warnings registry related to the warning has been cleared.

Perhaps pytest should clear the warnings registry?

[Zac-HD]

So long as we clear and then restore the warnings registry, that sounds good to me.

This will be best-effort only though - code under test might use a different registry, or the Python implementation might change - so we should also document that we attempt to clear and restore the registry for pytest.warns.

[Zac-HD]

#2586 might also be related.

[asottile]

via #4104 / #4192