pytest 5.2.3 breaks tests on Python 3.5 – 3.8

[asottile]

In GitLab by @ericvw on Nov 14, 2019, 22:59

pytest 5.2.3 is causing test failures for Python 3.5 up to 3.8, inclusive. I believe re-kicking off CI should also result in the observed behavior.

Downgrading pytest to 5.2.2 results in all tests passing as expected.

I've bisected pytest 5.2.2...5.2.3 and have identified github:pytest-dev/pytest@9275012ef789048998d1b87b9536bd7c2c063596 as the commit that changes the behavior of the flake8 test suite. The PR is github:pytest-dev/pytest#5831.

I'm working on determining and understanding the root cause better and will update with my findings unless someone else beats me to it. I wanted to report what I have been able to find out so far.

To reproduce: In a clean virtual environment (I used Python 3.8)…

$ pip install -e .
$ pip install -U pytest mock
$ pytest

# Alternatively...
$ pip install -U tox
$ tox

[…]

========================================= FAILURES =========================================
_____________________________________ test_legacy_api ______________________________________

tmpdir = local('/private/var/folders/t8/80zbh8_x0tb4yz8nctl91kcw0000gs/T/pytest-of-xxxx/pytest-28/test_legacy_api0')

    def test_legacy_api(tmpdir):
        """A basic end-to-end test for the legacy api reporting errors."""
        with tmpdir.as_cwd():
            t_py = tmpdir.join('t.py')
            t_py.write('import os  # unused import\n')

>           style_guide = legacy.get_style_guide()

tests/integration/test_api_legacy.py:11:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/flake8/api/legacy.py:37: in get_style_guide
    application.register_plugin_options()
src/flake8/main/application.py:190: in register_plugin_options
    self.check_plugins.register_options(self.option_manager)
src/flake8/plugins/manager.py:490: in register_options
    list(self.manager.map(register_and_enable))
src/flake8/plugins/manager.py:298: in map
    yield func(self.plugins[name], *args, **kwargs)
src/flake8/plugins/manager.py:486: in register_and_enable
    call_register_options(plugin)
src/flake8/plugins/manager.py:398: in generated_function
    return method(optmanager, *args, **kwargs)
src/flake8/plugins/manager.py:217: in register_options
    add_options(optmanager)
src/flake8/plugins/pyflakes.py:100: in add_options
    parser.add_option(
src/flake8/options/manager.py:405: in add_option
    self._current_group.add_argument(*option_args, **option_kwargs)
/Users/[omitted]/.pyenv/versions/3.8.0/lib/python3.8/argparse.py:1386: in add_argument
    return self._add_action(action)
/Users/[omitted]/.pyenv/versions/3.8.0/lib/python3.8/argparse.py:1590: in _add_action
    action = super(_ArgumentGroup, self)._add_action(action)
/Users/[omitted]/.pyenv/versions/3.8.0/lib/python3.8/argparse.py:1400: in _add_action
    self._check_conflict(action)
/Users/[omitted]/.pyenv/versions/3.8.0/lib/python3.8/argparse.py:1539: in _check_conflict
    conflict_handler(action, confl_optionals)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <argparse._ArgumentGroup object at 0x10b470dc0>
action = _StoreAction(option_strings=['--builtins'], dest='builtins', nargs=None, const=None, default=None, type=functools.part...separated_list=True, normalize_paths=False), choices=No
ne, help='define more built-ins, comma separated', metavar=None)
conflicting_actions = [('--builtins', _StoreAction(option_strings=['--builtins'], dest='builtins', nargs=None, const=None, default=None, typ...parated_list=True, normalize_paths=False),
 choices=None, help='define more built-ins, comma separated', metavar=None))]

    def _handle_conflict_error(self, action, conflicting_actions):
        message = ngettext('conflicting option string: %s',
                           'conflicting option strings: %s',
                           len(conflicting_actions))
        conflict_string = ', '.join([option_string
                                     for option_string, action
                                     in conflicting_actions])
>       raise ArgumentError(action, message % conflict_string)
E       argparse.ArgumentError: argument --builtins: conflicting option string: --builtins

/Users/[omitted]/.pyenv/versions/3.8.0/lib/python3.8/argparse.py:1548: ArgumentError

[asottile]

In GitLab by @ericvw on Nov 14, 2019, 23:05

changed the description

[asottile]

In GitLab by @ericvw on Nov 14, 2019, 23:26

The pytest change is causing FlakesChecker.add_options() to be called twice 😕, at least for test_legacy_api.

[asottile]

In GitLab by @ericvw on Nov 14, 2019, 23:30

Just as a sanity check, the tests fail when going back to flake8==3.7.8 (before the argparse changes).

[asottile]

In GitLab by @ericvw on Nov 15, 2019, 24:13

Commenting out https://github.com/pytest-dev/pytest/pull/5831/files#diff-88cb3736bed2f144d42f07ccbeb97eb0R642 in pytest causes the test suite to pass 🤷.

[asottile]

In GitLab by @ericvw on Nov 15, 2019, 01:16

https://gitlab.com/pycqa/flake8/issues/594#note_245056475 is observed depending on how pytest was invoked.

This gets really strange:

$ pytest --show-capture=no -k test_legacy_api

tests/integration/test_api_legacy.py F                                                                                                                                            [  7%]
tests/unit/test_legacy_api.py ............                                                                                                                                        [100%]

$ pytest --show-capture=no -k tests/integration/test_api_legacy.py

tests/integration/test_api_legacy.py F                                                                                                                                            [100%]

$ pytest --show-capture=no tests/integration/test_api_legacy.py

tests/integration/test_api_legacy.py .                                                                                                                                            [100%]

It seems to be isolated to pytests's dynamic mechanism of test discovery.

[asottile]

In GitLab by @ericvw on Nov 15, 2019, 02:49

It appears that the "F" plugin is created twice in the case of running pytest [-k]. However, the def test_api_legacy is invoked only once.

The first time the stack trace looks like:

[... omitted pytest frames ...]
/Users/eric/.pyenv/versions/3.8.0/envs/flake8-dev/lib/python3.8/site-packages/_pytest/python.py(170)pytest_pyfunc_call()
-> result = testfunction(**testargs)
  /Volumes/development/repos/flake8.git/tests/integration/test_api_legacy.py(11)test_legacy_api()
-> style_guide = legacy.get_style_guide()
  /Volumes/development/repos/flake8.git/src/flake8/api/legacy.py(36)get_style_guide()
-> application.find_plugins(prelim_opts.config, prelim_opts.isolated)
  /Volumes/development/repos/flake8.git/src/flake8/main/application.py(176)find_plugins()
-> self.check_plugins = plugin_manager.Checkers(
  /Volumes/development/repos/flake8.git/src/flake8/plugins/manager.py(354)__init__()
-> self.manager = PluginManager(
  /Volumes/development/repos/flake8.git/src/flake8/plugins/manager.py(240)__init__()
-> self._load_entrypoint_plugins()
  /Volumes/development/repos/flake8.git/src/flake8/plugins/manager.py(263)_load_entrypoint_plugins()
-> self._load_plugin_from_entrypoint(entry_point)
  /Volumes/development/repos/flake8.git/src/flake8/plugins/manager.py(274)_load_plugin_from_entrypoint()
-> self.plugins[name] = Plugin(name, entry_point, local=local)
> /Volumes/development/repos/flake8.git/src/flake8/plugins/manager.py(35)__init__()

The second time the stack trace looks like:

/Users/eric/.pyenv/versions/3.8.0/envs/flake8-dev/lib/python3.8/site-packages/_pytest/python.py(170)pytest_pyfunc_call()
-> result = testfunction(**testargs)
  /Volumes/development/repos/flake8.git/tests/integration/test_api_legacy.py(11)test_legacy_api()
-> style_guide = legacy.get_style_guide()
  /Volumes/development/repos/flake8.git/src/flake8/api/legacy.py(36)get_style_guide()
-> application.find_plugins(prelim_opts.config, prelim_opts.isolated)
  /Volumes/development/repos/flake8.git/src/flake8/main/application.py(176)find_plugins()
-> self.check_plugins = plugin_manager.Checkers(
  /Volumes/development/repos/flake8.git/src/flake8/plugins/manager.py(354)__init__()
-> self.manager = PluginManager(
  /Volumes/development/repos/flake8.git/src/flake8/plugins/manager.py(240)__init__()
-> self._load_entrypoint_plugins()
  /Volumes/development/repos/flake8.git/src/flake8/plugins/manager.py(263)_load_entrypoint_plugins()
-> self._load_plugin_from_entrypoint(entry_point)
  /Volumes/development/repos/flake8.git/src/flake8/plugins/manager.py(274)_load_plugin_from_entrypoint()
-> self.plugins[name] = Plugin(name, entry_point, local=local)
> /Volumes/development/repos/flake8.git/src/flake8/plugins/manager.py(35)__init__()

It appears to be that entrypoints.get_group_all('flake8.extensions') is duplicating the entrypoints being returned with pytest 5.2.3, but not with 5.2.2

[asottile]

In GitLab by @ericvw on Nov 15, 2019, 03:37

When running with pytest==5.2.3, the ./flake8/src the directory appears to be in sys.path twice whereas it is in there once with pytest==5.2.2.

I quickly put together:

# test_entrypoints.py
from os import path
import sys
import entrypoints

def print_entrypoint(name):
    for e in entrypoints.get_group_all('flake8.extension'):
        if e.name == name:
            print(e)

print('### Path inserted ###')
sys.path.insert(0, path.abspath('src'))
print_entrypoint('F')

print('### Path appended ###')
sys.path.append(path.abspath('src'))
print_entrypoint('F')

$ python test_entrypoints.py
### Path inserted ###
EntryPoint('F', 'flake8.plugins.pyflakes', 'FlakesChecker', None)
### Path appended ###
EntryPoint('F', 'flake8.plugins.pyflakes', 'FlakesChecker', None)
EntryPoint('F', 'flake8.plugins.pyflakes', 'FlakesChecker', None)

I'll let pytest know about this so they are aware and also follow-up with entrypoints about its expected behavior.

[asottile]

In GitLab by @ericvw on Nov 15, 2019, 03:40

Looks like pytest is getting other reports of the commit I was able to bisect (see pytest-dev/pytest#6194).

[asottile]

In GitLab by @ericvw on Nov 15, 2019, 03:54

mentioned in merge request !383

[asottile]

In GitLab by @asottile on Nov 15, 2019, 06:57

closed via commit c920950

[asottile]

In GitLab by @asottile on Nov 15, 2019, 06:59

awesome, thanks for the thorough investigation! note also that I plan to switch to importlib_metadata (#569) which might (?) not have this same oddity