Solver fails when dependency has multiple (constrained) dependency definitions for same package

[NicolasT]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Fedora 35
• Poetry version: 1.1.8 (on host, installed from Fedora repo/RPM) and 1.1.13 (in container to test with latest version, installed via pip)
• Link of a Gist with the contents of your pyproject.toml file:

Just including the file (which is the output of a simple `poetry init` accepting all defaults):

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "^3.10"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Issue

When trying to poetry add psychopy, which currently (rightfully) selects version 2022.1.2, the solver bails out:

Using version ^2022.1.2 for PsychoPy                                                                                                                                                                                                          
                                                                                                                                                                                                                                              
Updating dependencies                                                                                                                                                                                                                         
Resolving dependencies... (2.4s)                                                                                                                                                                                                              
                                                                                                                                                                                                                                              
  SolverProblemError                                                                                                                                                                                                                          
                                                                                                                                                                                                                                              
  Because psychopy (2022.1.2) depends on both python-vlc (>=3.0.12118) and python-vlc (3.0.11115), psychopy is forbidden.                                                                                                                     
  So, because no versions of psychopy match >2022.1.2,<2023.0.0                                                                                                                                                                               
   and test depends on PsychoPy (^2022.1.2), version solving failed.                                                                                                                                                                          
                                                                                                                                                                                                                                              
  at ~/.local/lib/python3.10/site-packages/poetry/puzzle/solver.py:241 in _solve                                                                                                                                                              
      237│             packages = result.packages                                                                                                                                                                                             
      238│         except OverrideNeeded as e:                                                                                                                                                                                                
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)                                                                                                                                            
      240│         except SolveFailure as e:                                                                                                                                                                                                  
    → 241│             raise SolverProblemError(e)                                                                                                                                                                                            
      242│                                                                                                                                                                                                                                    
      243│         results = dict(                                                                                                                                                                                                            
      244│             depth_first_search(                                                                                                                                                                                                    
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

(verbose backtrace below)

This is (likely) caused by a bit of a weird dependency listing for python-vlc in the package (from https://github.com/psychopy/psychopy/blob/2022.1.2/setup.cfg):

...
python-vlc <= 3.0.11115; platform_system == "Windows"
python-vlc >= 3.0.12118; platform_system != "Windows"
...
python-vlc >= 3.0.12118; platform_system != "Windows"
python-vlc == 3.0.11115; platform_system == "Windows"
...

Of course this reduces to

python-vlc == 3.0.11115; platform_system == "Windows"
python-vlc >= 3.0.12118; platform_system != "Windows"

and indeed, when making this change in a local fork and poetry add'ing this directory, poetry is able to resolve all dependencies and install the package.

Hence, it appears odd is going on when the same package is declared as a dependency with platform constraints multiple times.

Full backtrace

PyPI: No release information found for psychopy-0.93, skipping
PyPI: No release information found for psychopy-0.93.7, skipping
PyPI: No release information found for psychopy-0.95.3, skipping
PyPI: No release information found for psychopy-0.95.4, skipping
PyPI: No release information found for psychopy-0.95.7, skipping
PyPI: No release information found for psychopy-0.96.02, skipping
PyPI: No release information found for psychopy-0.97.00, skipping
PyPI: No release information found for psychopy-0.97.01, skipping
PyPI: No release information found for psychopy-1.00.00, skipping
PyPI: No release information found for psychopy-1.00.02, skipping
PyPI: No release information found for psychopy-1.00.03, skipping
PyPI: No release information found for psychopy-1.50.00, skipping
PyPI: No release information found for psychopy-1.50.01, skipping
PyPI: No release information found for psychopy-1.50.02, skipping
PyPI: No release information found for psychopy-1.50.04, skipping
PyPI: No release information found for psychopy-1.51.00, skipping
PyPI: No release information found for psychopy-1.51.01, skipping
PyPI: No release information found for psychopy-1.60.00, skipping
PyPI: No release information found for psychopy-1.60.01, skipping
PyPI: No release information found for psychopy-1.60.03, skipping
PyPI: No release information found for psychopy-1.61.01, skipping
PyPI: No release information found for psychopy-1.61.02, skipping
PyPI: No release information found for psychopy-1.61.03, skipping
PyPI: No release information found for psychopy-1.62.02, skipping
PyPI: No release information found for psychopy-1.63.03, skipping
PyPI: No release information found for psychopy-1.65.00, skipping
PyPI: No release information found for psychopy-1.70.01, skipping
PyPI: No release information found for psychopy-1.71.00, skipping
PyPI: No release information found for psychopy-1.74.03, skipping
PyPI: No release information found for psychopy-1.75.01, skipping
PyPI: No release information found for psychopy-1.80.02, skipping
PyPI: 56 packages found for psychopy *
Using version ^2022.1.2 for PsychoPy

Updating dependencies
Resolving dependencies...
   1: fact: test is 0.1.0
   1: derived: test
   1: fact: test depends on PsychoPy (^2022.1.2)
   1: selecting test (0.1.0)
   1: derived: PsychoPy (>=2022.1.2,<2023.0.0)
PyPI: No release information found for psychopy-0.93, skipping
PyPI: No release information found for psychopy-0.93.7, skipping
PyPI: No release information found for psychopy-0.95.3, skipping
PyPI: No release information found for psychopy-0.95.4, skipping
PyPI: No release information found for psychopy-0.95.7, skipping
PyPI: No release information found for psychopy-0.96.02, skipping
PyPI: No release information found for psychopy-0.97.00, skipping
PyPI: No release information found for psychopy-0.97.01, skipping
PyPI: No release information found for psychopy-1.00.00, skipping
PyPI: No release information found for psychopy-1.00.02, skipping
PyPI: No release information found for psychopy-1.00.03, skipping
PyPI: No release information found for psychopy-1.50.00, skipping
PyPI: No release information found for psychopy-1.50.01, skipping
PyPI: No release information found for psychopy-1.50.02, skipping
PyPI: No release information found for psychopy-1.50.04, skipping
PyPI: No release information found for psychopy-1.51.00, skipping
PyPI: No release information found for psychopy-1.51.01, skipping
PyPI: No release information found for psychopy-1.60.00, skipping
PyPI: No release information found for psychopy-1.60.01, skipping
PyPI: No release information found for psychopy-1.60.03, skipping
PyPI: No release information found for psychopy-1.61.01, skipping
PyPI: No release information found for psychopy-1.61.02, skipping
PyPI: No release information found for psychopy-1.61.03, skipping
PyPI: No release information found for psychopy-1.62.02, skipping
PyPI: No release information found for psychopy-1.63.03, skipping
PyPI: No release information found for psychopy-1.65.00, skipping
PyPI: No release information found for psychopy-1.70.01, skipping
PyPI: No release information found for psychopy-1.71.00, skipping
PyPI: No release information found for psychopy-1.74.03, skipping
PyPI: No release information found for psychopy-1.75.01, skipping
PyPI: No release information found for psychopy-1.80.02, skipping
PyPI: 1 packages found for psychopy >=2022.1.2,<2023.0.0
   0: Duplicate dependencies for pyglet
   0: Different requirements found for pyglet (<1.5) with markers platform_system != "Darwin" and pyglet (>=1.5) with markers platform_system == "Darwin".
   1: Version solving took 0.010 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('psychopy', '2022.1.2'): {'pyglet': <Dependency pyglet (<1.5)>}}).
   1: fact: test is 0.1.0
   1: derived: test
   1: fact: test depends on PsychoPy (^2022.1.2)
   1: selecting test (0.1.0)
   1: derived: PsychoPy (>=2022.1.2,<2023.0.0)
   0: Duplicate dependencies for python-vlc
   1: fact: psychopy (2022.1.2) depends on requests (*)
   1: fact: psychopy (2022.1.2) depends on cryptography (*)
   1: fact: psychopy (2022.1.2) depends on numpy (*)
   1: fact: psychopy (2022.1.2) depends on scipy (*)
   1: fact: psychopy (2022.1.2) depends on matplotlib (*)
   1: fact: psychopy (2022.1.2) depends on pandas (*)
   1: fact: psychopy (2022.1.2) depends on pillow (*)
   1: fact: psychopy (2022.1.2) depends on glfw (*)
   1: fact: psychopy (2022.1.2) depends on pygame (*)
   1: fact: psychopy (2022.1.2) depends on pyo (*)
   1: fact: psychopy (2022.1.2) depends on soundfile (*)
   1: fact: psychopy (2022.1.2) depends on sounddevice (*)
   1: fact: psychopy (2022.1.2) depends on python-bidi (*)
   1: fact: psychopy (2022.1.2) depends on arabic_reshaper (*)
   1: fact: psychopy (2022.1.2) depends on json_tricks (*)
   1: fact: psychopy (2022.1.2) depends on xlrd (*)
   1: fact: psychopy (2022.1.2) depends on openpyxl (*)
   1: fact: psychopy (2022.1.2) depends on pyserial (*)
   1: fact: psychopy (2022.1.2) depends on pyyaml (*)
   1: fact: psychopy (2022.1.2) depends on gevent (*)
   1: fact: psychopy (2022.1.2) depends on msgpack (*)
   1: fact: psychopy (2022.1.2) depends on msgpack-numpy (*)
   1: fact: psychopy (2022.1.2) depends on psutil (*)
   1: fact: psychopy (2022.1.2) depends on tables (*)
   1: fact: psychopy (2022.1.2) depends on pyzmq (*)
   1: fact: psychopy (2022.1.2) depends on ujson (*)
   1: fact: psychopy (2022.1.2) depends on moviepy (*)
   1: fact: psychopy (2022.1.2) depends on opencv-python (*)
   1: fact: psychopy (2022.1.2) depends on python-gitlab (*)
   1: fact: psychopy (2022.1.2) depends on gitpython (*)
   1: fact: psychopy (2022.1.2) depends on astunparse (*)
   1: fact: psychopy (2022.1.2) depends on esprima (*)
   1: fact: psychopy (2022.1.2) depends on freetype-py (*)
   1: fact: psychopy (2022.1.2) depends on jedi (>=0.16)
   1: fact: psychopy (2022.1.2) depends on psychtoolbox (*)
   1: fact: psychopy (2022.1.2) depends on pocketsphinx (*)
   1: fact: psychopy (2022.1.2) depends on SpeechRecognition (*)
   1: fact: psychopy (2022.1.2) depends on google-api-core (*)
   1: fact: psychopy (2022.1.2) depends on google-auth (*)
   1: fact: psychopy (2022.1.2) depends on google-cloud (*)
   1: fact: psychopy (2022.1.2) depends on google-cloud-speech (*)
   1: fact: psychopy (2022.1.2) depends on googleapis-common-protos (*)
   1: fact: psychopy (2022.1.2) depends on websocket_client (*)
   1: fact: psychopy (2022.1.2) depends on markdown-it-py (*)
   1: fact: psychopy (2022.1.2) depends on pyglet (<1.5)
   1: fact: psychopy (2022.1.2) depends on wxPython (<4.0.2 || >4.0.2,<4.0.3 || >4.0.3)
   1: fact: psychopy (2022.1.2) depends on python-vlc (>=3.0.12118)
   1: fact: psychopy (2022.1.2) depends on python-vlc (<=3.0.11115)
   1: fact: psychopy (2022.1.2) depends on python-vlc (3.0.11115)
   1: fact: psychopy (2022.1.2) depends on pyobjc-core (<8.0)
   1: fact: psychopy (2022.1.2) depends on pyobjc-framework-Quartz (<8.0)
   1: fact: psychopy (2022.1.2) depends on pyobjc (<8.0)
   1: fact: psychopy (2022.1.2) depends on pyparallel (*)
   1: fact: psychopy (2022.1.2) depends on python-xlib (*)
   1: fact: psychopy (2022.1.2) depends on distro (*)
   1: fact: psychopy (2022.1.2) depends on pyWinhook (*)
   1: fact: psychopy (2022.1.2) depends on pyqmix (>=2018.12.13)
   1: fact: psychopy (2022.1.2) depends on pypiwin32 (*)
   1: fact: psychopy (2022.1.2) depends on imageio (>=2.5)
   1: fact: psychopy (2022.1.2) depends on imageio-ffmpeg (*)
   1: fact: psychopy (2022.1.2) depends on pyqt5 (*)
   1: fact: psychopy (2022.1.2) depends on javascripthon (*)
   1: fact: psychopy (2022.1.2) depends on questplus (>=2019.3)
   1: selecting psychopy (2022.1.2)
   1: derived: questplus (>=2019.3)
   1: derived: javascripthon
   1: derived: pyqt5
   1: derived: imageio-ffmpeg
   1: derived: imageio (>=2.5)
   1: derived: pypiwin32
   1: derived: pyqmix (>=2018.12.13)
   1: derived: pyWinhook
   1: derived: distro
   1: derived: python-xlib
   1: derived: pyparallel
   1: derived: pyobjc (<8.0)
   1: derived: pyobjc-framework-Quartz (<8.0)
   1: derived: pyobjc-core (<8.0)
   1: derived: python-vlc (==3.0.11115)
   1: conflict: psychopy (2022.1.2) depends on python-vlc (>=3.0.12118)
   1: ! not  python-vlc (>=3.0.12118) is satisfied by  python-vlc (3.0.11115)
   1: ! which is caused by "psychopy (2022.1.2) depends on python-vlc (3.0.11115)"
   1: ! thus: psychopy is forbidden
   1: fact: psychopy is forbidden
   1: derived: not psychopy (==2022.1.2)
   1: fact: no versions of psychopy match >2022.1.2,<2023.0.0
   1: conflict: no versions of psychopy match >2022.1.2,<2023.0.0
   1: !  PsychoPy (>2022.1.2,<2023.0.0) is partially satisfied by not  psychopy (2022.1.2)
   1: ! which is caused by "psychopy is forbidden"
   1: ! thus: psychopy is forbidden
   1: !  psychopy (>=2022.1.2,<2023.0.0) is satisfied by  PsychoPy (^2022.1.2)
   1: ! which is caused by "test depends on PsychoPy (^2022.1.2)"
   1: ! thus: version solving failed
   1: Version solving took 0.020 seconds.
   1: Tried 1 solutions.

  Stack trace:

  10  ~/.local/lib/python3.10/site-packages/clikit/console_application.py:131 in run
       129│             parsed_args = resolved_command.args
       130│
     → 131│             status_code = command.handle(parsed_args, io)
       132│         except KeyboardInterrupt:
       133│             status_code = 1

   9  ~/.local/lib/python3.10/site-packages/clikit/api/command/command.py:120 in handle
       118│     def handle(self, args, io):  # type: (Args, IO) -> int
       119│         try:
     → 120│             status_code = self._do_handle(args, io)
       121│         except KeyboardInterrupt:
       122│             if io.is_debug():

   8  ~/.local/lib/python3.10/site-packages/clikit/api/command/command.py:171 in _do_handle
       169│         handler_method = self._config.handler_method
       170│
     → 171│         return getattr(handler, handler_method)(args, io, self)
       172│
       173│     def __repr__(self):  # type: () -> str

   7  ~/.local/lib/python3.10/site-packages/cleo/commands/command.py:92 in wrap_handle
        90│         self._command = command
        91│
     →  92│         return self.handle()
        93│
        94│     def handle(self):  # type: () -> Optional[int]

   6  ~/.local/lib/python3.10/site-packages/poetry/console/commands/add.py:173 in handle
       171│             self._installer.whitelist([r["name"] for r in requirements])
       172│
     → 173│             status = self._installer.run()
       174│         except BaseException:
       175│             # Using BaseException here as some exceptions, eg: KeyboardInterrupt, do not inherit from Exception

   5  ~/.local/lib/python3.10/site-packages/poetry/installation/installer.py:103 in run
       101│         local_repo = Repository()
       102│
     → 103│         return self._do_install(local_repo)
       104│
       105│     def dry_run(self, dry_run=True):  # type: (bool) -> Installer

   4  ~/.local/lib/python3.10/site-packages/poetry/installation/installer.py:235 in _do_install
       233│             )
       234│
     → 235│             ops = solver.solve(use_latest=self._whitelist)
       236│         else:
       237│             self._io.write_line("Installing dependencies from lock file")

   3  ~/.local/lib/python3.10/site-packages/poetry/puzzle/solver.py:65 in solve
        63│         with self._provider.progress():
        64│             start = time.time()
     →  65│             packages, depths = self._solve(use_latest=use_latest)
        66│             end = time.time()
        67│

   2  ~/.local/lib/python3.10/site-packages/poetry/puzzle/solver.py:239 in _solve
       237│             packages = result.packages
       238│         except OverrideNeeded as e:
     → 239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
       240│         except SolveFailure as e:
       241│             raise SolverProblemError(e)

   1  ~/.local/lib/python3.10/site-packages/poetry/puzzle/solver.py:207 in solve_in_compatibility_mode
       205│             )
       206│             self._provider.set_overrides(override)
     → 207│             _packages, _depths = self._solve(use_latest=use_latest)
       208│             for index, package in enumerate(_packages):
       209│                 if package not in packages:

  SolverProblemError

  Because psychopy (2022.1.2) depends on both python-vlc (>=3.0.12118) and python-vlc (3.0.11115), psychopy is forbidden.
  So, because no versions of psychopy match >2022.1.2,<2023.0.0
   and test depends on PsychoPy (^2022.1.2), version solving failed.

  at ~/.local/lib/python3.10/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

[kmsquire]

I'm having a similar (but less easily fixable) issue with installing different versions of numpy depending on the python version (^1.22.0 on >= 3.8, <1.22.0 on 3.7).

The issue can be seen here, which is part of this commit to this pull request to pyserde.

The current version of this test (in the links above) is

numpy = [
    { version = "<1.22.0", markers = "python_version ~= '3.7' and (extra == 'numpy' or extra == 'all')", optional = true },
    { version = "^1.22.0", markers = "(python_version ~= '3.8' or python_version ~= '3.9' or python_version ~= '3.10') and (extra == 'numpy' or extra == 'all')", optional = true },
]

But I've also tried splitting up the python versions to separate lines, as well as using, e.g.,

numpy = [
    { version = "<1.22.0", python = "3.7", markers = "extra == 'numpy' or extra == 'all'", optional = true },
    { version = "^1.22.0", python = "3.8", markers = "extra == 'numpy' or extra == 'all'", optional = true },
    { version = "^1.22.0", python = "3.9", markers = "extra == 'numpy' or extra == 'all'", optional = true },
    { version = "^1.22.0", python = "3.10", markers = "extra == 'numpy' or extra == 'all'", optional = true },
]

When running update, poetry attempts to install two versions of numpy (1.21.6 and 1.22.3) simultaneously, at least with the first example. With other variations, the behavior varies, but is never what I want (1.21.x on 3.7, 1.22.x on >3.8)

If you feel this is a different issue, please let me know and I'll open a new issue.

[radoering]

@kmsquire

Your first example is not solvable. Unfortunately, poetry solves it anyway. From PEP 440:

~=3.1: version 3.1 or later, but not version 4.0 or later.

Thus, numpy < 1.22 is required for python 3.7, 3.8 and so on. However, numpy >= 1.22 is required for python 3.8, 3.9, ... That's a conflict! At the moment poetry is not good at handling multiple constraints with overlapping markers. There is already an issue about that: #5066.

Your second example looks better but is running into this issue: #3444

[kmsquire]

Your first example is not solvable. Unfortunately, poetry solves it anyway. From PEP 440:

Thanks--I had forgotten the details of ~. Updating it to

numpy = [
    { version = "<1.22.0", markers = "python_version ~= '3.7.0' and (extra == 'numpy' or extra == 'all')", optional = true },
    { version = "^1.22.0", markers = "python_version ~= '3.8' and (extra == 'numpy' or extra == 'all')", optional = true },
]

seems like it should work, but it still tries to to upgrade to 1.22 on python 3.7, despite the poetry.lock file containing

[[package]]
name = "numpy"
version = "1.21.1"
description = "NumPy is the fundamental package for array computing with Python."
category = "main"
optional = false
python-versions = ">=3.7"

[[package]]
name = "numpy"
version = "1.21.6"
description = "NumPy is the fundamental package for array computing with Python."
category = "main"
optional = false
python-versions = ">=3.7,<3.11"

[[package]]
name = "numpy"
version = "1.22.3"
description = "NumPy is the fundamental package for array computing with Python."
category = "main"
optional = false
python-versions = ">=3.8"

I'll also note that optional should be true...

[radoering]

Considering your updated example without any other dependencies everything seems fine. That's what I get when running poetry lock:

[[package]]
name = "numpy"
version = "1.21.6"
description = "NumPy is the fundamental package for array computing with Python."
category = "main"
optional = true
python-versions = ">=3.7,<3.11"

[[package]]
name = "numpy"
version = "1.22.3"
description = "NumPy is the fundamental package for array computing with Python."
category = "main"
optional = true
python-versions = ">=3.8"

I tried poetry install --extras numpy with Python 3.7 and 3.9 and it works as expected (1.21.6 for 3.7 and 1.22.3 for 3.9).

The optional = false and the third numpy version comes from the dev dependency numpy = "*". However, even with this third entry and optional = false I can only observe numpy 1.21.6 (and no other version) being installed for Python 3.7.

[kmsquire]

Thanks. I'll try to explore further why I'm getting different behavior.

Just to check, what version of poetry are you using? I'm using 1.1.13.

[radoering]

I used 1.1.13, too.