pytest-dev · RonnyPfannschmidt · Nov 4, 2018 · Oct 27, 2018 · Oct 27, 2018 · Oct 28, 2018
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -24,12 +24,12 @@ repos:
     -   id: flake8
         language_version: python3
 -   repo: https://github.com/asottile/reorder_python_imports
-    rev: v1.3.2
+    rev: v1.3.3
     hooks:
     -   id: reorder-python-imports
         args: ['--application-directories=.:src']
 -   repo: https://github.com/asottile/pyupgrade
-    rev: v1.8.0
+    rev: v1.10.1
     hooks:
     -   id: pyupgrade
         args: [--keep-percent-format]

diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -18,6 +18,72 @@ with advance notice in the **Deprecations** section of releases.
 
 .. towncrier release notes start
 
+pytest 3.10.0 (2018-11-03)
+==========================
+
+Features
+--------
+
+- `#2619 <https://github.com/pytest-dev/pytest/issues/2619>`_: Resume capturing output after ``continue`` with ``__import__("pdb").set_trace()``.
+
+  This also adds a new ``pytest_leave_pdb`` hook, and passes in ``pdb`` to the
+  existing ``pytest_enter_pdb`` hook.
+
+
+- `#4147 <https://github.com/pytest-dev/pytest/issues/4147>`_: Add ``-sw``, ``--stepwise`` as an alternative to ``--lf -x`` for stopping at the first failure, but starting the next test invocation from that test.  See `the documentation <https://docs.pytest.org/en/latest/cache.html#stepwise>`__ for more info.
+
+
+- `#4188 <https://github.com/pytest-dev/pytest/issues/4188>`_: Make ``--color`` emit colorful dots when not running in verbose mode. Earlier, it would only colorize the test-by-test output if ``--verbose`` was also passed.
+
+
+- `#4225 <https://github.com/pytest-dev/pytest/issues/4225>`_: Improve performance with collection reporting in non-quiet mode with terminals.
+
+  The "collecting …" message is only printed/updated every 0.5s.
+
+
+
+Bug Fixes
+---------
+
+- `#2701 <https://github.com/pytest-dev/pytest/issues/2701>`_: Fix false ``RemovedInPytest4Warning: usage of Session... is deprecated, please use pytest`` warnings.
+
+
+- `#4046 <https://github.com/pytest-dev/pytest/issues/4046>`_: Fix problems with running tests in package ``__init__.py`` files.
+
+
+- `#4260 <https://github.com/pytest-dev/pytest/issues/4260>`_: Swallow warnings during anonymous compilation of source.
+
+
+- `#4262 <https://github.com/pytest-dev/pytest/issues/4262>`_: Fix access denied error when deleting stale directories created by ``tmpdir`` / ``tmp_path``.
+
+
+- `#611 <https://github.com/pytest-dev/pytest/issues/611>`_: Naming a fixture ``request`` will now raise a warning: the ``request`` fixture is internal and
+  should not be overwritten as it will lead to internal errors.
+
+
+
+Improved Documentation
+----------------------
+
+- `#4255 <https://github.com/pytest-dev/pytest/issues/4255>`_: Added missing documentation about the fact that module names passed to filter warnings are not regex-escaped.
+
+
+
+Trivial/Internal Changes
+------------------------
+
+- `#4272 <https://github.com/pytest-dev/pytest/issues/4272>`_: Display cachedir also in non-verbose mode if non-default.
+
+
+- `#4277 <https://github.com/pytest-dev/pytest/issues/4277>`_: pdb: improve message about output capturing with ``set_trace``.
+
+  Do not display "IO-capturing turned off/on" when ``-s`` is used to avoid
+  confusion.
+
+
+- `#4279 <https://github.com/pytest-dev/pytest/issues/4279>`_: Improve message and stack level of warnings issued by ``monkeypatch.setenv`` when the value of the environment variable is not a ``str``.
+
+
 pytest 3.9.3 (2018-10-27)
 =========================
 
@@ -366,7 +432,7 @@ Features
   the standard warnings filters to manage those warnings. This introduces ``PytestWarning``,
   ``PytestDeprecationWarning`` and ``RemovedInPytest4Warning`` warning types as part of the public API.
 
-  Consult `the documentation <https://docs.pytest.org/en/latest/warnings.html#internal-pytest-warnings>`_ for more info.
+  Consult `the documentation <https://docs.pytest.org/en/latest/warnings.html#internal-pytest-warnings>`__ for more info.
 
 
 - `#2908 <https://github.com/pytest-dev/pytest/issues/2908>`_: ``DeprecationWarning`` and ``PendingDeprecationWarning`` are now shown by default if no other warning filter is
@@ -541,7 +607,7 @@ Bug Fixes
 - `#3473 <https://github.com/pytest-dev/pytest/issues/3473>`_: Raise immediately if ``approx()`` is given an expected value of a type it doesn't understand (e.g. strings, nested dicts, etc.).
 
 
-- `#3712 <https://github.com/pytest-dev/pytest/issues/3712>`_: Correctly represent the dimensions of an numpy array when calling ``repr()`` on ``approx()``.
+- `#3712 <https://github.com/pytest-dev/pytest/issues/3712>`_: Correctly represent the dimensions of a numpy array when calling ``repr()`` on ``approx()``.
 
 - `#3742 <https://github.com/pytest-dev/pytest/issues/3742>`_: Fix incompatibility with third party plugins during collection, which produced the error ``object has no attribute '_collectfile'``.
 

diff --git a/HOWTORELEASE.rst b/HOWTORELEASE.rst
@@ -46,5 +46,3 @@ taking a lot of time to make a new one.
    * testing-in-python@lists.idyll.org (only major/minor releases)
 
    And announce it on `Twitter <https://twitter.com/>`_ with the ``#pytest`` hashtag.
-
-#. After a minor/major release, merge ``release-X.Y.Z`` into ``master`` and push (or open a PR).
diff --git a/changelog/2619.feature.rst b/changelog/2619.feature.rst
diff --git a/changelog/4147.feature.rst b/changelog/4147.feature.rst
diff --git a/changelog/4188.feature.rst b/changelog/4188.feature.rst
diff --git a/changelog/4225.feature.rst b/changelog/4225.feature.rst
diff --git a/changelog/4272.trivial.rst b/changelog/4272.trivial.rst
diff --git a/changelog/4277.trivial.rst b/changelog/4277.trivial.rst
diff --git a/changelog/4287.bugfix.rst b/changelog/4287.bugfix.rst
@@ -0,0 +1 @@
+Fix nested usage of debugging plugin (pdb), e.g. with pytester's ``testdir.runpytest``.
diff --git a/changelog/611.bugfix.rst b/changelog/611.bugfix.rst
diff --git a/doc/4266.bugfix.rst b/doc/4266.bugfix.rst
@@ -0,0 +1 @@
+Handle (ignore) exceptions raised during collection, e.g. with Django's LazySettings proxy class.
diff --git a/doc/en/announce/index.rst b/doc/en/announce/index.rst
@@ -6,6 +6,7 @@ Release announcements
    :maxdepth: 2
 
 
+   release-3.10.0
    release-3.9.3
    release-3.9.2
    release-3.9.1

diff --git a/doc/en/announce/release-3.10.0.rst b/doc/en/announce/release-3.10.0.rst
@@ -0,0 +1,43 @@
+pytest-3.10.0
+=======================================
+
+The pytest team is proud to announce the 3.10.0 release!
+
+pytest is a mature Python testing tool with more than a 2000 tests
+against itself, passing on many different interpreters and platforms.
+
+This release contains a number of bugs fixes and improvements, so users are encouraged
+to take a look at the CHANGELOG:
+
+    https://docs.pytest.org/en/latest/changelog.html
+
+For complete documentation, please visit:
+
+    https://docs.pytest.org/en/latest/
+
+As usual, you can upgrade from pypi via:
+
+    pip install -U pytest
+
+Thanks to all who contributed to this release, among them:
+
+* Anders Hovmöller
+* Andreu Vallbona Plazas
+* Ankit Goel
+* Anthony Sottile
+* Bernardo Gomes
+* Brianna Laugher
+* Bruno Oliveira
+* Daniel Hahler
+* David Szotten
+* Mick Koch
+* Niclas Olofsson
+* Palash Chatterjee
+* Ronny Pfannschmidt
+* Sven-Hendrik Haase
+* Ville Skyttä
+* William Jamir Silva
+
+
+Happy testing,
+The Pytest Development Team
diff --git a/doc/en/cache.rst b/doc/en/cache.rst
@@ -244,6 +244,8 @@ You can always peek at the content of the cache using the
       {'test_caching.py::test_function': True}
     cache/nodeids contains:
       ['test_caching.py::test_function']
+    cache/stepwise contains:
+      []
     example/value contains:
       42
 

diff --git a/doc/en/contents.rst b/doc/en/contents.rst
@@ -33,6 +33,7 @@ Full pytest documentation
    reference
 
    goodpractices
+   flaky
    pythonpath
    customize
    example/index

diff --git a/doc/en/example/multipython.py b/doc/en/example/multipython.py
@@ -33,7 +33,7 @@ def dumps(self, obj):
         dumpfile = self.picklefile.dirpath("dump.py")
         dumpfile.write(
             textwrap.dedent(
-                """\
+                r"""
                 import pickle
                 f = open({!r}, 'wb')
                 s = pickle.dump({!r}, f, protocol=2)
@@ -49,7 +49,7 @@ def load_and_is_true(self, expression):
         loadfile = self.picklefile.dirpath("load.py")
         loadfile.write(
             textwrap.dedent(
-                """\
+                r"""
                 import pickle
                 f = open({!r}, 'rb')
                 obj = pickle.load(f)

diff --git a/doc/en/example/nonpython.rst b/doc/en/example/nonpython.rst
@@ -85,8 +85,9 @@ interesting to just look at the collection tree::
     rootdir: $REGENDOC_TMPDIR/nonpython, inifile:
     collected 2 items
     <Package '$REGENDOC_TMPDIR/nonpython'>
-      <YamlFile 'test_simple.yml'>
-        <YamlItem 'hello'>
-        <YamlItem 'ok'>
+      <Package '$REGENDOC_TMPDIR/nonpython'>
+        <YamlFile 'test_simple.yml'>
+          <YamlItem 'hello'>
+          <YamlItem 'ok'>
 
     ======================= no tests ran in 0.12 seconds =======================
diff --git a/doc/en/fixture.rst b/doc/en/fixture.rst
@@ -153,7 +153,7 @@ This makes use of the automatic caching mechanisms of pytest.
 
 Another good approach is by adding the data files in the ``tests`` folder.
 There are also community plugins available to help managing this aspect of
-testing, e.g. `pytest-datadir <https://github.com/gabrielcnr/pytest-datadir>`__
+testing, e.g. `pytest-datadir <https://pypi.org/project/pytest-datadir/>`__
 and `pytest-datafiles <https://pypi.org/project/pytest-datafiles/>`__.
 
 .. _smtpshared:

diff --git a/doc/en/flaky.rst b/doc/en/flaky.rst
@@ -0,0 +1,125 @@
+
+Flaky tests
+-----------
+
+A "flaky" test is one that exhibits intermittent or sporadic failure, that seems to have non-deterministic behaviour. Sometimes it passes, sometimes it fails, and it's not clear why. This page discusses pytest features that can help and other general strategies for identifying, fixing or mitigating them.
+
+Why flaky tests are a problem
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Flaky tests are particularly troublesome when a continuous integration (CI) server is being used, so that all tests must pass before a new code change can be merged. If the test result is not a reliable signal -- that a test failure means the code change broke the test -- developers can become mistrustful of the test results, which can lead to overlooking genuine failures. It is also a source of wasted time as developers must re-run test suites and investigate spurious failures.
+
+
+Potential root causes
+^^^^^^^^^^^^^^^^^^^^^
+
+System state
+~~~~~~~~~~~~
+
+Broadly speaking, a flaky test indicates that the test relies on some system state that is not being appropriately controlled - the test environment is not sufficiently isolated. Higher level tests are more likely to be flaky as they rely on more state.
+
+Flaky tests sometimes appear when a test suite is run in parallel (such as use of pytest-xdist). This can indicate a test is reliant on test ordering.
+
+-  Perhaps a different test is failing to clean up after itself and leaving behind data which causes the flaky test to fail.
+- The flaky test is reliant on data from a previous test that doesn't clean up after itself, and in parallel runs that previous test is not always present
+- Tests that modify global state typically cannot be run in parallel.
+
+
+Overly strict assertion
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Overly strict assertions can cause problems with floating point comparison as well as timing issues. `pytest.approx <https://docs.pytest.org/en/latest/reference.html#pytest-approx>`_ is useful here.
+
+
+Pytest features
+^^^^^^^^^^^^^^^
+
+Xfail strict
+~~~~~~~~~~~~
+
+:ref:`pytest.mark.xfail ref` with ``strict=False`` can be used to mark a test so that its failure does not cause the whole build to break. This could be considered like a manual quarantine, and is rather dangerous to use permanently.
+
+
+PYTEST_CURRENT_TEST
+~~~~~~~~~~~~~~~~~~~
+
+:ref:`pytest current test env` may be useful for figuring out "which test got stuck".
+
+
+Plugins
+~~~~~~~
+
+Rerunning any failed tests can mitigate the negative effects of flaky tests by giving them additional chances to pass, so that the overall build does not fail. Several pytest plugins support this:
+
+* `flaky <https://github.com/box/flaky>`_
+* `pytest-flakefinder <https://github.com/dropbox/pytest-flakefinder>`_ - `blog post <https://blogs.dropbox.com/tech/2016/03/open-sourcing-pytest-tools/>`_
+* `pytest-rerunfailures <https://github.com/pytest-dev/pytest-rerunfailures>`_
+* `pytest-replay <https://github.com/ESSS/pytest-replay>`_: This plugin helps to reproduce locally crashes or flaky tests observed during CI runs.
+
+Plugins to deliberately randomize tests can help expose tests with state problems:
+
+* `pytest-random-order <https://github.com/jbasko/pytest-random-order>`_
+* `pytest-randomly <https://github.com/pytest-dev/pytest-randomly>`_
+
+
+Other general strategies
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Split up test suites
+~~~~~~~~~~~~~~~~~~~~
+
+It can be common to split a single test suite into two, such as unit vs integration, and only use the unit test suite as a CI gate. This also helps keep build times manageable as high level tests tend to be slower. However, it means it does become possible for code that breaks the build to be merged, so extra vigilance is needed for monitoring the integration test results.
+
+
+Video/screenshot on failure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For UI tests these are important for understanding what the state of the UI was when the test failed. pytest-splinter can be used with plugins like pytest-bdd and can `save a screenshot on test failure <https://pytest-splinter.readthedocs.io/en/latest/#automatic-screenshots-on-test-failure>`_, which can help to isolate the cause.
+
+
+Delete or rewrite the test
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If the functionality is covered by other tests, perhaps the test can be removed. If not, perhaps it can be rewritten at a lower level which will remove the flakiness or make its source more apparent.
+
+
+Quarantine
+~~~~~~~~~~
+
+Mark Lapierre discusses the `Pros and Cons of Quarantined Tests <https://dev.to/mlapierre/pros-and-cons-of-quarantined-tests-2emj>`_ in a post from 2018.
+
+
+
+CI tools that rerun on failure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Azure Pipelines (the Azure cloud CI/CD tool, formerly Visual Studio Team Services or VSTS) has a feature to `identify flaky tests <https://docs.microsoft.com/en-us/azure/devops/release-notes/2017/dec-11-vsts#identify-flaky-tests>`_ and rerun failed tests.
+
+
+
+Research
+^^^^^^^^
+
+This is a limited list, please submit an issue or pull request to expand it!
+
+* Gao, Zebao, Yalan Liang, Myra B. Cohen, Atif M. Memon, and Zhen Wang. "Making system user interactive tests repeatable: When and what should we control?." In *Software Engineering (ICSE), 2015 IEEE/ACM 37th IEEE International Conference on*, vol. 1, pp. 55-65. IEEE, 2015.  `PDF <http://www.cs.umd.edu/~atif/pubs/gao-icse15.pdf>`__
+* Palomba, Fabio, and Andy Zaidman. "Does refactoring of test smells induce fixing flaky tests?." In *Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on*, pp. 1-12. IEEE, 2017. `PDF in Google Drive <https://drive.google.com/file/d/10HdcCQiuQVgW3yYUJD-TSTq1NbYEprl0/view>`__
+*  Bell, Jonathan, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tifany Yung, and Darko Marinov. "DeFlaker: Automatically detecting flaky tests." In *Proceedings of the 2018 International Conference on Software Engineering*. 2018. `PDF <https://www.jonbell.net/icse18-deflaker.pdf>`__
+
+
+Resources
+^^^^^^^^^
+
+* `Eradicating Non-Determinism in Tests <https://martinfowler.com/articles/nonDeterminism.html>`_ by Martin Fowler, 2011
+* `No more flaky tests on the Go team <https://www.thoughtworks.com/insights/blog/no-more-flaky-tests-go-team>`_ by Pavan Sudarshan, 2012
+* `The Build That Cried Broken: Building Trust in your Continuous Integration Tests <https://www.youtube.com/embed/VotJqV4n8ig>`_ talk (video) by `Angie Jones <http://angiejones.tech/>`_ at SeleniumConf Austin 2017
+* `Test and Code Podcast: Flaky Tests and How to Deal with Them <https://testandcode.com/50>`_ by Brian Okken and Anthony Shaw, 2018
+* Microsoft:
+
+  * `How we approach testing VSTS to enable continuous delivery <https://blogs.msdn.microsoft.com/bharry/2017/06/28/testing-in-a-cloud-delivery-cadence/>`_ by Brian Harry MS, 2017
+  * `Eliminating Flaky Tests <https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/eliminating-flaky-tests>`_ blog and talk (video) by Munil Shah, 2017
+
+* Google:
+
+  * `Flaky Tests at Google and How We Mitigate Them <https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html>`_ by John Micco, 2016
+  * `Where do Google's flaky tests come from? <https://docs.google.com/document/d/1mZ0-Kc97DI_F3tf_GBW_NB_aqka-P1jVOsFfufxqUUM/edit#heading=h.ec0r4fypsleh>`_  by Jeff Listfield, 2017
diff --git a/doc/en/proposals/parametrize_with_fixtures.rst b/doc/en/proposals/parametrize_with_fixtures.rst
@@ -75,7 +75,7 @@ Issues
 ------
 
 * By using ``request.getfuncargvalue()`` we rely on actual fixture function
-  execution to know what fixtures are involved, due to it's dynamic nature
+  execution to know what fixtures are involved, due to its dynamic nature
 * More importantly, ``request.getfuncargvalue()`` cannot be combined with
   parametrized fixtures, such as ``extra_context``
 * This is very inconvenient if you wish to extend an existing test suite by

diff --git a/doc/en/reference.rst b/doc/en/reference.rst
@@ -117,6 +117,7 @@ Add warning filters to marked test items.
         A *warning specification string*, which is composed of contents of the tuple ``(action, message, category, module, lineno)``
         as specified in `The Warnings filter <https://docs.python.org/3/library/warnings.html#warning-filter>`_ section of
         the Python documentation, separated by ``":"``. Optional fields can be omitted.
+        Module names passed for filtering are not regex-escaped.
 
         For example:
 

diff --git a/doc/en/talks.rst b/doc/en/talks.rst
@@ -23,6 +23,8 @@ Books
 Talks and blog postings
 ---------------------------------------------
 
+- pytest: recommendations, basic packages for testing in Python and Django, Andreu Vallbona, PyconES 2017 (`slides in english <http://talks.apsl.io/testing-pycones-2017/>`_, `video in spanish <https://www.youtube.com/watch?v=K20GeR-lXDk>`_)
+
 - `Pythonic testing, Igor Starikov (Russian, PyNsk, November 2016)
   <https://www.youtube.com/watch?v=_92nfdd5nK8>`_.
Original file line number	Diff line number	Diff line change
Expand Up		@@ -46,5 +46,3 @@ taking a lot of time to make a new one.
		* testing-in-python@lists.idyll.org (only major/minor releases)

		And announce it on `Twitter <https://twitter.com/>`_ with the ``#pytest`` hashtag.

		#. After a minor/major release, merge ``release-X.Y.Z`` into ``master`` and push (or open a PR).
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Fix nested usage of debugging plugin (pdb), e.g. with pytester's ``testdir.runpytest``.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Handle (ignore) exceptions raised during collection, e.g. with Django's LazySettings proxy class.