test_junitxml.py::TestPython::test_failure_function broke on CI

[The-Compiler]

See e.g. https://github.com/pytest-dev/pytest/runs/5086149463?check_suite_focus=true - happens in multiple unrelated PRs. No idea why off-hand...

[asottile]

looks like test pollution -- let me see if I can find out why

[asottile]

these are the polluting tests:

$ cat t8
testing/test_doctest.py::TestDoctests::test_valid_setup_py
testing/test_junitxml.py::TestPython::test_failure_function[xunit1-all]

[asottile]

looks like a regression in setuptools -- between 60.5.0 and 60.6.0

[asottile]

cc @jaraco

[jaraco]

Thanks @asottile. I see you have a workaround for pytest. If that isn't a permanent fix, can you at least summarize or characterize the missed expectation from setuptools? I have a clue, but you seem to have a deeper grasp of it at the moment.

[asottile]

I'm not really sure, something about the changed version there causes the root logger to get misconfigured as a side effect of setup()

[jaraco]

Between those versions, the only change that seems at all relevant is pypa/setuptools#3042. If someone wishes to investigate deeper, I welcome it.

[Lewiscowles1986]

An even weirder part of this is that Anthony has a YouTube video which starts with this commit, and shows a totally different test polution failure...
https://www.youtube.com/watch?v=FRteianaPMo

I think, the version of setuptools used at the time [futureproofing statement as this may be different when someone reads, or when the code was added]:

When running setup(), configures Python’s logging module. Specifically, it adds a NullHandler to the 'distutils' logger and, in some cases, modifies the global logging configuration, which can affect log levels.

https://github.com/pypa/setuptools/blob/1fe0c5d2f46e141d6097910d2905a3fce5e30c52/setuptools/__init__.py#L111-L117

@jaraco I think you are mentioned on a related PR pypa/setuptools#2974, Issue pypa/setuptools#2973 and commit pypa/setuptools@382037c