Captured warnings missing file and test that triggered them in Pytest >=3.8.0

[CAM-Gerlach]

In pytest >= 3.8.0 (3.6.2-3..7.4 confirmed to not have this problem, while 3.8.0-3.9.1 all do), captured warnings are missing a very crucial piece of information: the file and test that originally triggered them. In a large codebase like that of the Spyder IDE (of which I'm a developer) that relies on pytest for its testing framework (as well as integrating support within the IDE), with hundreds of files and thousands of tests its crippling to only see the file and line in whatever third party dependency actually issued the warning, rather than also including the actual test and file. Most likely, this has something to do with the changes in #3251 . However, I don't see how this isn't a serious regression since it makes DeprecationWarnings and the like all but useless in most circumstances, save for manually hunting through the code for what might have triggered it.

This occurs across the Spyder codebase when tested on e.g. a Windows 8.1 x64 machine running under Anaconda (nominally 5.2.0), and on both Python 3.6 and Python 3.7 at least; the only difference in environments that trigger it is simply an upgrade/downgrade of Pytest to 3.8.0/3.7.4 respectively. I tested it on a minimal example project as well, Here's the module, eggs, to be tested:

import warnings

def foo(spam):
    message = "This is foo: " + str(spam)
    warnings.warn("This is a test warning")
    print(message)
    return message

and here's the test file:

import pytest
import eggs

def test_foo():
    expected_message = "This is foo: bar"
    actual_message = eggs.foo("bar")
    assert actual_message == expected_message
    
if __name__ == "__main__":
    pytest.main()

Here's the pytest output for the 3.7.4 case:

(base) C:\Users\C. A. M. Gerlach\Documents\dev\TestPackage>pytest ham/tests/test_eggs.py -vv -rw
============================= test session starts =============================
platform win32 -- Python 3.6.6, pytest-3.7.4, py-1.5.3, pluggy-0.8.0 -- C:\Anaconda3\python.exe
cachedir: .pytest_cache
PyQt5 5.9.2 -- Qt runtime 5.9.6 -- Qt compiled 5.9.4
rootdir: C:\Users\C. A. M. Gerlach\Documents\dev\TestPackage, inifile:
plugins: timeout-1.2.1, remotedata-0.2.1, qt-2.1.0, openfiles-0.3.0, mock-1.6.3, doctestplus-0.1.3, cov-2.4.0, arraydiff-0.2, hypothesis-3.38.5, flaky-3.3.0
collected 1 item

ham/tests/test_eggs.py::test_foo PASSED                                  [100%]
===Flaky Test Report===


===End Flaky Test Report===

============================== warnings summary ===============================
ham/tests/test_eggs.py::test_foo
  C:\Users\C. A. M. Gerlach\Documents\dev\TestPackage\ham\eggs.py:7: UserWarning: This is a test warning
    warnings.warn("This is a test warning")

-- Docs: https://docs.pytest.org/en/latest/warnings.html
==================== 1 passed, 1 warnings in 0.06 seconds =====================

and for 3.8.0:

(base) C:\Users\C. A. M. Gerlach\Documents\dev\TestPackage>pytest ham/tests/test_eggs.py -vv -rw
============================= test session starts =============================
platform win32 -- Python 3.6.6, pytest-3.8.0, py-1.5.3, pluggy-0.8.0 -- C:\Anaconda3\python.exe
cachedir: .pytest_cache
PyQt5 5.9.2 -- Qt runtime 5.9.6 -- Qt compiled 5.9.4
rootdir: C:\Users\C. A. M. Gerlach\Documents\dev\TestPackage, inifile:
plugins: timeout-1.2.1, remotedata-0.2.1, qt-2.1.0, openfiles-0.3.0, mock-1.6.3, doctestplus-0.1.3, cov-2.4.0, arraydiff-0.2, hypothesis-3.38.5, flaky-3.3.0
collected 1 item

ham/tests/test_eggs.py::test_foo PASSED                                  [100%]
===Flaky Test Report===


===End Flaky Test Report===

============================== warnings summary ===============================
C:\Users\C. A. M. Gerlach\Documents\dev\TestPackage\ham\eggs.py:7: UserWarning: This is a test warning
  warnings.warn("This is a test warning")

-- Docs: https://docs.pytest.org/en/latest/warnings.html
==================== 1 passed, 1 warnings in 0.03 seconds =====================

Thanks.

[asottile]

This is likely do the major refactor/rewrite of the warnings system that occurred in 3.8.0 -- I wonder if this bit slipped under the radar CC @nicoddemus

[nicoddemus]

Seems like it, thanks @CAM-Gerlach. I'll look into it. 👍

[CAM-Gerlach]

Thanks @asottile and @nicoddemus ! I figured it had to be something related to that big overhaul.

[nicoddemus]

PR open: #4229. 👍