required_plugins false-positive with prereleases of plugins

[The-Compiler]

With a pytest.ini like

[pytest]
required_plugins = pytest-qt pytest-bdd

when those plugins are installed from VCS, pytest doesn't seem to detect pytest-qt (and some others), while pytest-bdd (and some others) are fine.

To reproduce:

$ python3 -m venv .venv
$ .venv/bin/pip install -qU pip setuptools wheel
$ .venv/bin/pip install pytest git+https://github.com/pytest-dev/pytest-qt.git git+https://github.com/pytest-dev/pytest-bdd.git
[...]
Successfully installed [...] packaging-20.9 parse-1.19.0 [...] pluggy-0.13.1 py-1.10.0 [...] pytest-6.2.2 pytest-bdd-4.0.2 pytest-qt-3.3.1.dev71+g1fbdd94 [...]

$ cat > pytest.ini
[pytest]
required_plugins = pytest-qt pytest-bdd

$ ./.venv/bin/pytest   
ERROR: Missing required plugins: pytest-qt

$ .venv/bin/pip freeze                  
[...]
pytest-bdd @ git+https://github.com/pytest-dev/pytest-bdd.git@5d58fe3bbbf38d2408701f4876281a3440890522
pytest-qt @ git+https://github.com/pytest-dev/pytest-qt.git@1fbdd944d33c49662894661b4071fb641a681b21
[...]

Looking at it with a debugger, apparently this condition:

pytest/src/_pytest/config/__init__.py

Line 1280 in dd37097

elif Version(plugin_dist_info[spec.name]) not in spec.specifier:

works fine for pytest-bdd:

> .../_pytest/config/__init__.py(1273)_validate_plugins()
-> try:
(Pdb) required_plugin
'pytest-bdd'

(Pdb) n
> .../_pytest/config/__init__.py(1281)_validate_plugins()
-> elif Version(plugin_dist_info[spec.name]) not in spec.specifier:

(Pdb) plugin_dist_info[spec.name]
'4.0.2'

(Pdb) spec.specifier
<SpecifierSet('')>

(Pdb) plugin_dist_info[spec.name] in spec.specifier
True

but not pytest-qt:

(Pdb) required_plugin
'pytest-qt'

(Pdb) n
> .../_pytest/config/__init__.py(1281)_validate_plugins()
-> elif Version(plugin_dist_info[spec.name]) not in spec.specifier:

(Pdb) plugin_dist_info[spec.name]
'3.3.1.dev71+g1fbdd94'

(Pdb) spec.specifier
<SpecifierSet('')>

(Pdb) plugin_dist_info[spec.name] in spec.specifier
False

(Pdb) n
> /home/florian/proj/pytest/src/_pytest/config/__init__.py(1282)_validate_plugins()
-> missing_plugins.append(required_plugin)

I'm not sure why that is - I suspect setuptools-scm comes into play somehow?

[RonnyPfannschmidt]

It's a bug in the version class I suspect, will debug when I get back to the computer

[The-Compiler]

Looking at packaging, here's what in does on a SpecifierSet:

    def contains(
        self, item: UnparsedVersion, prereleases: Optional[bool] = None
    ) -> bool:

        # Ensure that our item is a Version or LegacyVersion instance.
        if not isinstance(item, (LegacyVersion, Version)):
            item = parse(item)

        # Determine if we're forcing a prerelease or not, if we're not forcing
        # one for this particular filter call, then we'll use whatever the
        # SpecifierSet thinks for whether or not we should support prereleases.
        if prereleases is None:
            prereleases = self.prereleases

        # We can determine if we're going to allow pre-releases by looking to
        # see if any of the underlying items supports them. If none of them do
        # and this item is a pre-release then we do not allow it and we can
        # short circuit that here.
        # Note: This means that 1.0.dev1 would not be contained in something
        #       like >=1.0.devabc however it would be in >=1.0.debabc,>0.0.dev0
        if not prereleases and item.is_prerelease:
            return False

        # We simply dispatch to the underlying specs here to make sure that the
        # given version is contained within all of them.
        # Note: This use of all() here means that an empty set of specifiers
        #       will always return True, this is an explicit design decision.
        return all(s.contains(item, prereleases=prereleases) for s in self._specs)

note the:

Note: This use of all() here means that an empty set of specifiers will always return True, this is an explicit design decision.

but the early short-circuit here:

        if not prereleases and item.is_prerelease:
            return False

In the case of pytest-qt, Version(plugin_dist_info[spec.name]).is_prerelease is True, while it is False for pytest-bdd.

pytest does spec = Requirement(required_plugin) and then uses spec.specifier to get a SpecifierSet from that. It looks like those by default don't accept prereleases at all.

What do you think about:

diff --git i/src/_pytest/config/__init__.py w/src/_pytest/config/__init__.py
index 144f1c9d1..687b16a37 100644
--- i/src/_pytest/config/__init__.py
+++ w/src/_pytest/config/__init__.py
@@ -1270,14 +1270,17 @@ def _validate_plugins(self) -> None:
         missing_plugins = []
         for required_plugin in required_plugins:
             try:
-                spec = Requirement(required_plugin)
+                req = Requirement(required_plugin)
             except InvalidRequirement:
                 missing_plugins.append(required_plugin)
                 continue
 
-            if spec.name not in plugin_dist_info:
+            specifier = req.specifier
+            specifier.prereleases = True
+
+            if req.name not in plugin_dist_info:
                 missing_plugins.append(required_plugin)
-            elif Version(plugin_dist_info[spec.name]) not in spec.specifier:
+            elif Version(plugin_dist_info[req.name]) not in specifier:
                 missing_plugins.append(required_plugin)
 
         if missing_plugins:

If that sounds reasonable, I'll see if I can add a test for this and open a PR.

[RonnyPfannschmidt]

it looks reasonable at first glance, thanks for checking

[asottile]

instead of mutating .prereleases = true one can do specifier.contains(version, prereleases=True) (I at least would prefer that)