8.1.0: testing of pytest fails with installed and disabled plugins

[mtelka]

• a detailed description of the bug or problem you are having

When I run tests for pytest 8.1.0 with some additional pytest plugins installed and disabled, then more than 10 tests fails with errors like:

============================= test session starts ==============================
platform sunos5 -- Python 3.9.18, pytest-8.1.0, pluggy-1.4.0 -- $(BUILD_DIR)/.tox/py39/bin/python
cachedir: .tox/py39/.pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('$(BUILD_DIR)/.hypothesis/examples'))
rootdir: $(BUILD_DIR)
configfile: pyproject.toml
testpaths: testing
plugins: hypothesis-6.98.15, xdist-3.5.0
collecting ... collected 3626 items

... snip ...

__________________ TestConfigFromdictargs.test_basic_behavior __________________

self = <test_config.TestConfigFromdictargs object at 0x7fffaba39610>
_sys_snapshot = None

    def test_basic_behavior(self, _sys_snapshot) -> None:
        option_dict = {"verbose": 444, "foo": "bar", "capture": "no"}
        args = ["a", "b"]

>       config = Config.fromdictargs(option_dict, args)

testing/test_config.py:991:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../prototype/i386/usr/lib/python3.9/vendor-packages/_pytest/config/__init__.py:1192: in fromdictargs
    config.parse(args, addopts=False)
../prototype/i386/usr/lib/python3.9/vendor-packages/_pytest/config/__init__.py:1490: in parse
    self._preparse(args, addopts=addopts)
../prototype/i386/usr/lib/python3.9/vendor-packages/_pytest/config/__init__.py:1377: in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
/usr/lib/python3.9/vendor-packages/pluggy/_manager.py:415: in load_setuptools_entrypoints
    self.register(plugin, name=ep.name)
../prototype/i386/usr/lib/python3.9/vendor-packages/_pytest/config/__init__.py:497: in register
    plugin_name = super().register(plugin, name)
/usr/lib/python3.9/vendor-packages/pluggy/_manager.py:167: in register
    self._verify_hook(hook, hookimpl)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_pytest.config.PytestPluginManager object at 0x7fffa95f9580>
hook = <HookCaller 'pytest_collect_file'>
hookimpl = <HookImpl plugin_name='black', plugin=<module 'pytest_black' from '/usr/lib/python3.9/vendor-packages/pytest_black.py'>>

    def _verify_hook(self, hook: HookCaller, hookimpl: HookImpl) -> None:
        if hook.is_historic() and (hookimpl.hookwrapper or hookimpl.wrapper):
            raise PluginValidationError(
                hookimpl.plugin,
                "Plugin %r\nhook %r\nhistoric incompatible with yield/wrapper/hookwrapper"
                % (hookimpl.plugin_name, hook.name),
            )

        assert hook.spec is not None
        if hook.spec.warn_on_impl:
            _warn_for_function(hook.spec.warn_on_impl, hookimpl.function)

        # positional arg checking
        notinspec = set(hookimpl.argnames) - set(hook.spec.argnames)
        if notinspec:
>           raise PluginValidationError(
                hookimpl.plugin,
                "Plugin %r for hook %r\nhookimpl definition: %s\n"
                "Argument(s) %s are declared in the hookimpl but "
                "can not be found in the hookspec"
                % ( 
                    hookimpl.plugin_name,
                    hook.name,
                    _formatdef(hookimpl.function),
                    notinspec,
                ),
            )
E           pluggy._manager.PluginValidationError: Plugin 'black' for hook 'pytest_collect_file'
E           hookimpl definition: pytest_collect_file(file_path, path, parent)
E           Argument(s) {'path'} are declared in the hookimpl but can not be found in the hookspec

/usr/lib/python3.9/vendor-packages/pluggy/_manager.py:342: PluginValidationError

In this case the pytest-black plugin is installed and disabled using PYTEST_ADDOPTS=-p 'no:black'.

This is regression since pytest 8.0.2.

• output of pip list from the virtual environment you are using

The pytest testing is not run in virtual environment, but in real environment.

• pytest and operating system versions

pytest 8.1.0
OpenIndiana (rolling release, up-to-date)

• minimal example if possible

[bluetech]

The plugins need to update, but we didn't issue proper deprecation warnings for it, so we've yanked the 8.1.0 release for now.

[mtelka]

I'm sorry, but I believe that plugins (and their update) have nothing to do with this. Or, are you trying to suggest that plugins should no longer use the pytest11 entry point?

I think this might be related: pytest-dev/pluggy#457.

[bluetech]

Ah, I thought you were complaining about the failure itself, but you were complaining that disabling the plugin didn't fix the failure.

Yeah, it seems like the pluggy issue.

[mtelka]

Failures reported above are something new. I do not see these failures with pytest 8.0.2, while everything else in the testing environment is same. So this is clear regression caused by some change in pytest. I know the 8.1.0 is yanked, but there is high possibility that this will appear again in some new future version.

I believe this new issue is not the same as pytest-dev/pluggy#457. Maybe they are related, but definitely not the same (and pluggy claims the bug is in pytest :-)).

[bluetech]

There are two issues:

• pytest 8.1.0 removed some stuff without proper deprecation which caused some plugins to fail. This causes the pluggy._manager.PluginValidationError: Plugin 'black' for hook 'pytest_collect_file' error. This version has been yanked and proper deprecation will be introduced.

• Even when a setuptools plugin is disabled with -p no:foo the error still happens. This is pluggy loads disabled pytest plugins pluggy#457.

I know the 8.1.0 is yanked, but there is high possibility that this will appear again in some new future version.

Hopefully after we introduce the deprecations the plugin will be fixed and when we remove the deprecated stuff again it will be OK.

[mtelka]

Okay, hopefully you are right.

But still, are you sure that pytest-dev/pluggy#457 is really a bug at the pluggy side? How should pluggy know that it should not load some plugins from the following?

../prototype/i386/usr/lib/python3.9/vendor-packages/_pytest/config/__init__.py:1377: in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")

How pytest pass the information about disabled plugins to pluggy?

[bluetech]

But still, are you sure that pytest-dev/pluggy#457 is really a bug at the pluggy side?

I am not.

How pytest pass the information about disabled plugins to pluggy?

IIRC it uses https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluginManager.set_blocked

[mtelka]

How pytest pass the information about disabled plugins to pluggy?

IIRC it uses https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluginManager.set_blocked

Great info! Thank you for that. Quick search in pytest sources shows the set_blocked is never called.

[nicoddemus]

Actually it is called here:

pytest/src/_pytest/config/__init__.py

Lines 785 to 806 in 03e5471

	def consider_pluginarg(self, arg: str) -> None:
	""":meta private:"""
	if arg.startswith("no:"):
	name = arg[3:]
	if name in essential_plugins:
	raise UsageError("plugin %s cannot be disabled" % name)
	# PR #4304: remove stepwise if cacheprovider is blocked.
	if name == "cacheprovider":
	self.set_blocked("stepwise")
	self.set_blocked("pytest_stepwise")
	self.set_blocked(name)
	if not name.startswith("pytest_"):
	self.set_blocked("pytest_" + name)
	else:
	name = arg
	# Unblock the plugin.
	self.unblock(name)
	if not name.startswith("pytest_"):
	self.unblock("pytest_" + name)
	self.import_plugin(arg, consider_entry_points=True)

[mtelka]

Actually it is called here:

Indeed. I do not know why I failed to find it. I assume the information about disabled plugins is not passed down to the runpytest call.