pytest.warns and pytest.raises ordering issue #9036
Description
- a detailed description of the bug or problem you are having
- output of
pip listfrom the virtual environment you are using - pytest and operating system versions
- minimal example if possible
Description
It appears that when testing that both a warning is emitted and an exception is raised, the order in which pytest.raises and pytest.warns are used matters and can lead to potentially confusing behavior.
Minimal example
For example, the following test passes:
import warnings
import pytest
def my_func():
# This both emits a `FutureWarning` and raises a `ValueError`
warnings.warn("my warning", category=FutureWarning)
raise ValueError("my error")
def test():
with pytest.raises(ValueError, match="my error"):
with pytest.warns(SyntaxWarning, match="my warning"): # <-- I expected this to fail because `my_func` doesn't emit a `SyntaxWarning`
my_func()Naively I expected this test to fail as my_func is emitting a FutureWarning, but the test checking that a SyntaxWarning is emitted.
However, if I switch the order of pytest.raises and pytest.warns, then the test fails as expected. That is
import warnings
import pytest
def my_func():
# This both emits a `FutureWarning` and raises a `ValueError`
warnings.warn("my warning", category=FutureWarning)
raise ValueError("my error")
def test():
with pytest.warns(SyntaxWarning, match="my warning"):
with pytest.raises(ValueError, match="my error"):
my_func()outputs
__________________________________________________________________________ test __________________________________________________________________________
def test():
with pytest.warns(SyntaxWarning, match="my warning"):
with pytest.raises(ValueError, match="my error"):
> my_func()
E Failed: DID NOT WARN. No warnings of type (<class 'SyntaxWarning'>,) was emitted. The list of emitted warnings is: [FutureWarning('my warning')].
test.py:13: Failed
Extra info
pip list:
Package Version Location
----------------------------- -------------------------- --------------------------------------
aiobotocore 1.3.3
aiohttp 3.7.4.post0
aioitertools 0.7.1
alabaster 0.7.12
anyio 3.3.0
appdirs 1.4.4
appnope 0.1.2
argon2-cffi 20.1.0
async-generator 1.10
async-timeout 3.0.1
asyncssh 2.7.0
attrs 21.2.0
awscli 1.19.106
Babel 2.9.1
backcall 0.2.0
backports.functools-lru-cache 1.6.4
bcrypt 3.2.0
bleach 3.3.1
blosc 1.10.2
bokeh 2.3.3
botocore 1.20.106
brotlipy 0.7.0
cached-property 1.5.2
certifi 2021.5.30
cffi 1.14.6
cfgv 3.3.0
cftime 1.5.0
chardet 4.0.0
charset-normalizer 2.0.0
click 8.0.1
cloudpickle 1.6.0
cmarkgfm 0.6.0
colorama 0.4.3
cryptography 3.4.7
cytoolz 0.11.0
dask 2021.8.1+3.ga6cc44fd.dirty /Users/james/projects/dask/dask
dask-labextension 5.1.0
dask-sphinx-theme 1.3.6
debugpy 1.4.1
decorator 5.0.9
defusedxml 0.7.1
distlib 0.3.2
distributed 2021.8.1 /Users/james/projects/dask/distributed
docutils 0.16
editdistance-s 1.0.0
entrypoints 0.3
execnet 1.9.0
filelock 3.0.12
fsspec 2021.7.0+8.gee22435
future 0.18.2
gssapi 1.6.14
h5py 3.3.0
HeapDict 1.0.1
identify 2.2.11
idna 3.1
imagesize 1.2.0
importlib-metadata 4.6.1
iniconfig 1.1.1
ipykernel 6.0.3
ipython 7.25.0
ipython-genutils 0.2.0
ipywidgets 7.6.3
jedi 0.18.0
Jinja2 3.0.1
jmespath 0.10.0
joblib 1.1.0.dev0
json5 0.9.5
jsonschema 3.2.0
jupyter-client 6.1.12
jupyter-core 4.7.1
jupyter-server 1.10.1
jupyter-server-proxy 3.1.0
jupyterlab 3.1.1
jupyterlab-pygments 0.1.2
jupyterlab-server 2.6.1
jupyterlab-widgets 1.0.0
Keras 2.4.3
keyring 23.0.1
locket 0.2.0
lz4 3.1.3
MarkupSafe 2.0.1
matplotlib-inline 0.1.2
mistune 0.8.4
more-itertools 8.8.0
msgpack 1.0.2
multidict 5.1.0
nbclassic 0.3.1
nbclient 0.5.3
nbconvert 6.1.0
nbformat 5.1.3
nest-asyncio 1.5.1
netCDF4 1.5.7
nodeenv 1.6.0
notebook 6.4.0
numpy 1.21.1
numpydoc 1.1.0
olefile 0.46
packaging 21.0
pandas 1.3.1
pandocfilters 1.4.2
paramiko 2.7.2
parso 0.8.2
partd 1.2.0
pexpect 4.8.0
pickleshare 0.7.5
Pillow 8.2.0
pip 21.2.1
pkginfo 1.7.1
pluggy 0.13.1
pre-commit 2.13.0
prometheus-client 0.11.0
prompt-toolkit 3.0.19
psutil 5.8.0
ptyprocess 0.7.0
py 1.10.0
pyasn1 0.4.8
pycparser 2.20
Pygments 2.9.0
PyNaCl 1.4.0
pynvml 11.0.0
pyOpenSSL 20.0.1
pyparsing 2.4.7
pyrsistent 0.17.3
PySocks 1.7.1
pytest 6.2.4
pytest-asyncio 0.12.0
pytest-faulthandler 2.0.1
pytest-forked 1.3.0
pytest-repeat 0.8.0
pytest-rerunfailures 10.1
pytest-timeout 1.4.2
pytest-xdist 2.3.0
python-dateutil 2.8.2
python-snappy 0.6.0
pytz 2021.1
PyYAML 5.4.1
pyzmq 22.1.0
readme-renderer 27.0
requests 2.26.0
requests-toolbelt 0.9.1
requests-unixsocket 0.2.0
rfc3986 1.5.0
rsa 4.7.2
s3fs 2021.7.0+5.g0df4f28
s3transfer 0.4.2
scikit-learn 0.24.2
scipy 1.7.0
Send2Trash 1.7.1
setuptools 49.6.0.post20210108
simpervisor 0.4
six 1.16.0
sniffio 1.2.0
snowballstemmer 2.1.0
sortedcollections 2.1.0
sortedcontainers 2.4.0
Sphinx 4.1.2
sphinx-click 3.0.1
sphinx-rtd-theme 0.5.2
sphinxcontrib-applehelp 1.0.2
sphinxcontrib-devhelp 1.0.2
sphinxcontrib-htmlhelp 2.0.0
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.3
sphinxcontrib-serializinghtml 1.1.5
tblib 1.7.0
terminado 0.10.1
testpath 0.5.0
threadpoolctl 2.2.0
toml 0.10.2
toolz 0.11.1
torch 1.9.0
torchvision 0.9.0a0
tornado 6.1
tqdm 4.61.2
traitlets 5.0.5
twine 0.0.0
typing-extensions 3.10.0.0
urllib3 1.26.6
virtualenv 20.4.7
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 0.57.0
wheel 0.36.2
widgetsnbextension 3.5.1
wrapt 1.12.1
yarl 1.6.3
zict 2.0.0
zipp 3.5.0
zstandard 0.15.2
I'm using pytest==6.2.4 on a Mac running Big Sur.