WIP Use a custom cache directory for test_stepwise tests

[nicoddemus]

Otherwise the tests will use tox's env cache which makes them flaky

Example failure:

https://travis-ci.org/pytest-dev/pytest/jobs/454100204

[RonnyPfannschmidt]

wrong fix - pytester should fix this in general for everything

[nicoddemus]

wrong fix - pytester should fix this in general for everything

Hmm indeed. Any suggestion on how to do this? I'm thinking of changing the cache_dir attribute directly, as writing a ini file for every test won't work for tests which write their own pytest.ini.

[RonnyPfannschmidt]

@nicoddemus i believe it would do us good to clean up the env variables

[nicoddemus]

i believe it would do us good to clean up the env variables

Sorry, I'm not following, what do environment variables have to do with the cache dir? 🤔

[RonnyPfannschmidt]

i was thinking of #4271

[blueyed]

#4271 appears to not have fixed it by itself: it was merged already with the above failure, no?

[nicoddemus]

In #4271, handle_env is defined in test_cacheprovider.py, the test this PR addresses is in test_stepwise.py.

[blueyed]

Let's rename it to clean_env or something and move it to conftest then maybe?

[blueyed]

But maybe pytester by itself could do it already.

[nicoddemus]

But maybe pytester by itself could do it already.

Yeah I think that was @RonnyPfannschmidt's suggestion.

[codecov]

Codecov Report

Merging #4378 into master will increase coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #4378      +/-   ##
==========================================
+ Coverage   95.82%   95.84%   +0.02%     
==========================================
  Files         111      111              
  Lines       24928    24929       +1     
  Branches     2431     2431              
==========================================
+ Hits        23887    23893       +6     
+ Misses        740      738       -2     
+ Partials      301      298       -3

Flag	Coverage Δ
#docs	29.66% <0%> (+0.07%)	⬆️
#doctesting	29.66% <0%> (+0.07%)	⬆️
#linting	29.66% <0%> (+0.07%)	⬆️
#linux	95.67% <100%> (+0.02%)	⬆️
#nobyte	92.33% <100%> (ø)	⬆️
#numpy	94.92% <100%> (+1.98%)	⬆️
#pexpect	41.59% <0%> (+0.02%)	⬆️
#py27	94.03% <100%> (+0.02%)	⬆️
#py34	92.15% <100%> (+0.06%)	⬆️
#py35	92.16% <100%> (+0.06%)	⬆️
#py36	93.9% <100%> (+0.02%)	⬆️
#py37	92.38% <100%> (+0.06%)	⬆️
#trial	94.92% <100%> (+1.98%)	⬆️
#windows	94.09% <100%> (ø)	⬆️
#xdist	93.86% <100%> (+0.12%)	⬆️

Impacted Files	Coverage Δ
testing/test_stepwise.py	100% <100%> (ø)	⬆️
src/_pytest/terminal.py	93.02% <0%> (+0.85%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 93a9836...70a072a. Read the comment docs.

[blueyed]

Follow-up in #4382.