Add docs page discussing flaky tests

[pfctdayelise]

Adding a docs page giving guidance on treatment of flaky tests. Discovered there are no less than 5 plugins that will rerun failed tests!

I tried to build the docs locally but got this error which I am not sure how to resolve:

Warning, treated as error:
autodoc: failed to import function 'hookspec.pytest_assertrepr_compare' from module '_pytest'; the following exception was raised:
No module named 'pluggy'
ERROR: InvocationError for command '/home/brianna/workspace/pytest/.tox/docs/bin/sphinx-build -W -b html . _build' (exited with code 2)

The markup looks ok in https://livesphinx.herokuapp.com/ , aside from the :ref: references.

[nicoddemus]

Hi @pfctdayelise, thanks!

To build the docs locally, the simplest way is to run tox -e docs. 😁

[nicoddemus]

Awesome work @pfctdayelise! 🎉

[codecov]

Codecov Report

Merging #4276 into master will decrease coverage by 1.88%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4276      +/-   ##
==========================================
- Coverage   95.87%   93.98%   -1.89%     
==========================================
  Files         109      109              
  Lines       24626    24621       -5     
  Branches     2395     2392       -3     
==========================================
- Hits        23609    23140     -469     
- Misses        721     1160     +439     
- Partials      296      321      +25

Flag	Coverage Δ
#docs	?
#doctesting	?
#linting	?
#linux	93.85% <ø> (-1.8%)	⬇️
#nobyte	?
#numpy	?
#pexpect	?
#py27	91.91% <ø> (-2.13%)	⬇️
#py34	92.22% <ø> (-0.1%)	⬇️
#py35	?
#py36	92.21% <ø> (-1.79%)	⬇️
#py37	?
#trial	?
#windows	91.16% <ø> (-2.99%)	⬇️
#xdist	?

Impacted Files	Coverage Δ
testing/test_pdb.py	43.94% <0%> (-51.06%)	⬇️
src/_pytest/debugging.py	63.02% <0%> (-31.1%)	⬇️
testing/python/approx.py	79.84% <0%> (-19.77%)	⬇️
src/_pytest/unittest.py	75.14% <0%> (-19.21%)	⬇️
testing/test_unittest.py	85.32% <0%> (-13.77%)	⬇️
src/_pytest/reports.py	86.9% <0%> (-10.72%)	⬇️
src/_pytest/python_api.py	87.12% <0%> (-10.31%)	⬇️
src/_pytest/assertion/util.py	92.99% <0%> (-5.15%)	⬇️
testing/test_terminal.py	96.68% <0%> (-3.32%)	⬇️
src/_pytest/doctest.py	94.46% <0%> (-2.38%)	⬇️
... and 15 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ff04a1f...cc25256. Read the comment docs.

[pfctdayelise]

Tox -e docs is when I got that error!

I suspect some install instruction is missing from the contributing doc.

[nicoddemus]

Oh! Hmm any chance you were using an old environment? If so please try tox -re docs so tox recreates the environment.